Real-time self-adaptive human musculoskeletal modeling and muscle force simulation system

Abstract

The invention discloses a real-time self-adaptive human musculoskeletal modeling and muscle force simulation system, and belongs to the field of computer vision. The system comprises a multi-sensor data acquisition module, a space-time calibration module, a three-dimensional force prediction module, a real-time modeling and parameter adjustment module, a muscle force simulation module and a data visualization module, and realizes space-time calibration fusion by acquiring and summarizing positions and pressure distribution data of human body joint points; a joint diagram structure and a body segment diagram structure are established, spatiotemporal features are obtained through feature extraction and fusion, three-dimensional foot hip bottom pressure is predicted, a human body musculoskeletal model containing a muscle geometric path is constructed, model parameters are optimized in real time, a joint objective function is established to calculate the muscle activation degree, and muscle force simulation is completed. According to the system provided by the invention, the connection between the muscle and the skeleton can be directly established, the individual adaptability is improved, the physiological rationality of muscle activation distribution is improved, and high-precision real-time muscle force simulation analysis is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of biomedical engineering, and particularly relates to a real-time adaptive human musculoskeletal modeling and muscle force simulation system. BACKGROUND

[0002] In recent years, with the in-depth development of human biomechanics research, musculoskeletal modeling and muscle force analysis technology has been widely used in the fields of clinical rehabilitation, sports science, and human engineering. Measuring and analyzing muscle force data is of great value and necessity for users in the fields of rehabilitation medicine, scientific sports, and human-computer interaction.

[0003] Traditional musculoskeletal modeling methods mainly rely on expensive optical motion capture systems (such as Vicon) combined with force platforms for kinematic and dynamic data acquisition, and muscle force simulation analysis through software platforms such as OpenSim. This method mainly relies on multi-body dynamics simulation and optimization algorithms, and its process usually includes collecting human kinematic data based on optical motion capture systems in a laboratory environment, combining with the dynamic data collected synchronously during the movement, establishing a model of the human musculoskeletal system through multi-body dynamics equations, and solving the muscle force of each muscle block using static or dynamic optimization algorithms. This method uses optimization solving to solve the problem of muscle redundancy in human physiology, and has good rationality and accuracy based on physiological anatomy, but the data acquisition relies on high-precision laboratory equipment, which is not only cumbersome to operate and limited by the site, but also has high usage cost.

[0004] Data-driven methods represented by deep learning can directly establish a mapping from kinematic data to muscle force, thereby bypassing the complex simulation process. However, such methods are mostly for scientific research purposes, and their accuracy is heavily dependent on the amount of data, which requires the collection of human kinematic and dynamic data for each case, consuming a large amount of manpower and resources, and limiting the data size in model training.

[0005] In muscle force simulation calculations, traditional static optimization methods usually use an objective function that minimizes the sum of squares of muscle activation, which often leads to a few muscles bearing the main load, while other muscle activations are close to zero. This phenomenon does not match the actual muscle coordination pattern of the human body. In addition, the tendon parameters in existing models are usually fixed and cannot be adaptively adjusted according to individual differences and actual measurement data, which affects the accuracy and physiological rationality of muscle force calculation and hinders the in-depth application of this technology in personalized rehabilitation assessment and sports guidance.

[0006] The technical solution based on a depth camera combined with a pressure sensor array has the advantages of low cost, small space occupation, and easy deployment, and provides new possibilities for musculoskeletal modeling. The depth camera directly obtains three-dimensional coordinate information of human joint points through infrared light or structured light technology, and the pressure sensor array can obtain contact force information, and the combination of the two has the potential to realize real-time muscle force simulation. However, the existing research on musculoskeletal modeling based on depth cameras is extremely limited, mainly due to the lack of a musculoskeletal model specifically constructed for depth camera data, and the technical difficulties of multi-sensor data fusion. SUMMARY

[0007] To solve the above technical problems, the present application provides a real-time adaptive human musculoskeletal modeling and muscle force simulation system, which directly establishes a musculoskeletal model based on depth camera data, and further improves the individual adaptability of the model by adjusting the adaptive parameters. By constructing a heterogeneous multi-modal model with a graph structure conversion layer, the three-dimensional force on the contact surface is predicted, and when muscle force simulation is performed, a joint objective function is established to improve the physiological reasonableness of muscle activation distribution, realizing low-cost, high-precision real-time muscle force simulation analysis.

[0008] To achieve the above-mentioned purposes, the embodiment provides a real-time adaptive human musculoskeletal modeling and muscle force simulation system, which comprises: A multi-sensor data acquisition module, which includes a depth camera, a plantar pressure sensor array, a hip pressure sensor array, and a data acquisition end, is used to synchronously acquire three-dimensional coordinates and pressure distribution data of human joints; A space-time calibration module adopts a space calibration algorithm to establish a coordinate system mapping relationship by clicking on the feature points of the pressure sensor array in the field of view of the depth camera, realizing the space calibration fusion of multiple coordinate systems; A three-dimensional force prediction module extracts pressure distribution features from the pressure distribution data, constructs joint graph structure and body segment graph structure based on the pressure distribution features and the position of human joints, and extracts spatial features and time sequence features through graph attention network and time convolution network respectively, and performs feature fusion to predict three-dimensional foot and hip pressure; A real-time modeling and parameter adjustment module is used to construct a human musculoskeletal model containing muscle geometric paths based on three-dimensional coordinates of human joints, to real-time optimize human musculoskeletal model parameters according to the actual measured muscle length range, and to output reasonable muscle length and muscle speed; A muscle force simulation module is used to calculate human joint moments through three-dimensional coordinates of human joints and three-dimensional foot and hip pressure, to establish a joint objective function based on the calculated human joint moments and combined with reasonable muscle length and speed, to calculate muscle activation and obtain corresponding muscle force, and to complete muscle force simulation.

[0009] In one embodiment, the real-time adaptive human musculoskeletal modeling and muscle force simulation system further comprises a data visualization module for providing real-time animation of muscle activation, muscle force profiles and human musculoskeletal model, and representing muscle activation intensity by color depth.

[0010] In one embodiment, in the multi-sensor data acquisition module, the human joint points captured by the depth camera include head, neck, pelvis, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, right hand, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot.

[0011] In one embodiment, in the multi-sensor data acquisition module, the plantar pressure sensor array and the hip pressure sensor array are both matrix arrays composed of multiple rows and multiple columns of flexible pressure sensor units, for collecting vertical pressure distribution.

[0012] In one embodiment, in the space-time calibration module, the mapping relationship between the coordinate system is established by clicking the feature points of the pressure sensor array in the depth camera field of view, including: clicking the sensor units of the plantar sensor array and the hip sensor array in the depth camera field of view as feature points, obtaining the coordinate system position of the feature points in the depth camera field of view, calculating the rotation matrix of the pressure array coordinate system to the depth camera coordinate system, and establishing the mapping relationship.

[0013] In one embodiment, in the three-dimensional force prediction module, the joint graph structure is constructed based on the pressure distribution features and the human joint point positions, and the body segment graph structure is obtained by converting the joint graph structure, including: The pressure distribution features containing plantar pressure distribution and hip pressure distribution extracted through convolution layer and self-attention mechanism are modeled into a joint graph structure with human joint point positions wherein is a set of human joint points, containing human joint position information and pressure distribution features, is a set of human joint connection edges; The joint graph structure is converted into a body segment graph structure through an edge-node conversion layer wherein is a set of body segment nodes, is a set of body segment connection edges.

[0014] In one embodiment, when the graph structure conversion is performed through the edge-node conversion layer, the following conversion rules are followed: Each edge in the joint graph structure becomes a node in the body segment graph structure, and the node feature is a weighted combination of the features of the two end nodes of the original edge: , wherein, The first Node features, is the weight parameter, The first Node features, The first Node features; In the joint graph structure, if two edges share a joint node, then there is an undirected edge connection between the corresponding two nodes in the segment graph structure.

[0015] The present invention implements a learnable feature fusion operation by performing graph structure conversion through an edge-node conversion layer. It effectively converts the structural information implied by an edge in a joint graph (defined by two joints) into the feature information of a node in a segment graph, thereby providing a high-quality data foundation for subsequent segment-based mechanical analysis.

[0016] In one embodiment, in a 3D force prediction module, spatial features and temporal features are extracted and fused based on the joint graph structure and the body segment graph structure using a graph attention network and a temporal convolutional network, respectively, to predict 3D foot and hip pressure, including: The joint graph structure and the body segment graph structure are respectively subjected to the graph attention network and the temporal convolutional network for spatial feature and temporal feature extraction to capture the importance relationship and dynamic changes of the nodes in the joint graph structure and the body segment graph structure; wherein the attention coefficient of the graph attention network is: , in, is the weight matrix, is the attention vector, is the node feature, || represents the splicing operation; The spatiotemporal features extracted from the joint graph structure and the body segment graph structure are input into the feature fusion layer for feature fusion, and then pass through the temporal convolutional network, the global pooling layer and the fully connected layer in sequence. The output is the three-dimensional force vector containing the soles of both feet and the bottom of the buttocks, which is the three-dimensional foot and buttocks pressure.

[0017] By using the temporal convolutional network twice to process the features extracted from the joint graph and the body segment graph respectively, the first time using the local temporal convolutional network ensures that the independent motion pattern of each joint and each body segment is captured. The second time using the convolutional network, after fusing the features of all joints and body segments, it can capture the overall global motion trend, greatly enhancing the ability to understand dynamic movements, and ultimately achieving high-precision prediction of three-dimensional forces.

[0018] In one embodiment, in the real-time modeling and parameter adjustment module, the three-dimensional coordinates of the human joint points acquired by the depth camera are used to construct a human musculoskeletal model including muscle geometric paths, including: Based on the human joint points collected by the depth camera and combined with the muscle system, the geometric path of each muscle is defined, each muscle is defined by a plurality of path points, and the direct connection between the muscle position and the human joint point is established; wherein the muscle system includes: hamstrings, biceps femoris, gluteus maximus, iliopsoas, rectus femoris, quadriceps femoris, gastrocnemius, soleus, tibialis anterior, erector spinae, internal oblique, external oblique, and the corresponding left and right sides.

[0019] In one embodiment, in the real-time modeling and parameter adjustment module, the real-time optimization of the human musculoskeletal model parameters includes preliminary parameter adjustment and parameter fine adjustment. The preliminary parameter adjustment includes: preliminary adjustment of the human musculoskeletal model parameters by a linear scaling preprocessing algorithm, and the calculation formula of linear scaling is: , , , wherein, is the measured muscle length of the first muscle, is the predicted muscle length of the first muscle; The parameter fine adjustment includes: establishing a multi-objective optimal function including a maximum length normalized deviation term, a minimum length normalized deviation term, a uniform distribution term and a regularization term to fine adjust the muscle parameters, so as to make the normalized muscle length and speed within a reasonable physiological range, and maintain the biological rationality of muscle mechanical characteristics, and the calculation is as follows: , wherein, is the optimal fiber length of the first muscle, is the tendon slack length of the first muscle, is the initial value of the optimal fiber length of the first muscle in the original human musculoskeletal model, is the initial value of the tendon slack length of the first muscle in the original human musculoskeletal model, is the total number of optimized muscles, , , and are the corresponding weight coefficients of each term, is the variance.

[0020] In one embodiment, the joint objective function in the muscle force simulation module includes: an overall activation term, a maximum activation penalty term, an activation range term and a time smoothing term; the overall activation term is used to minimize the square of all muscle activations to reduce the overall muscle energy consumption; the maximum activation penalty term is used to inhibit the excessive activation of a single muscle; the activation range term is used to promote the uniform distribution of muscle activations; and the time smoothing term is used to inhibit the sharp fluctuations of muscle activations over time to improve the physiological rationality, and is calculated as follows: , wherein, is the activation of the i-th muscle, is the maximum value of all muscle activations, is the difference between the maximum and minimum values of muscle activations, is the change of the same muscle activation at adjacent time points, , , , , are weight coefficients.

[0021] In one embodiment, the corresponding constraint conditions are set when the joint objective function is used to calculate muscle activations, and are calculated as follows: , wherein, R is the force arm matrix, is the maximum isometric contraction force, , , are the tension-length, force-velocity and passive tension coefficients respectively, is the muscle fiber length, is the muscle fiber velocity, is the fiber length at the maximum muscle force, is the maximum fiber velocity, is the joint torque.

[0022] In one embodiment, the muscle force simulation module, based on the calculated muscle activations, the corresponding muscle forces are obtained by substituting into the Hill-type muscle model, and the muscle force calculation formula is as follows: , wherein, is the muscle force.

[0023] In one embodiment, after obtaining the muscle force, a signal noise adaptive sensing filtering technology is further used to realize the real-time smoothing processing of the muscle force signal.

[0024] Compared with the prior art, the present application has at least the following beneficial effects: (1) The general muscle model parameters are used for the traditional method, which is easy to cause the extreme value of the muscle force characteristic curve, and the optimization result is poor. The space-time calibration module and the real-time modeling and parameter adjustment module are introduced, the multi-sensor data fusion problem is solved through the space calibration algorithm, and the human muscle and bone model based on the depth camera data is further constructed, so that the muscle position is directly proportional to the joint position of the human body instead of the general model, without manual scaling and calibration operation of the traditional method, real-time processing can be realized, and individual adaptability is significantly improved. (2) The heterogeneous multi-modal three-dimensional force prediction module is introduced, the joint graph is converted into a body segment graph which is more in line with the mechanical transmission rule through graph structure conversion, the space-time feature modeling is carried out combined with the graph attention network and the time convolution network, and finally the high-precision three-dimensional force prediction is realized, which provides a high-quality data basis for subsequent body segment-based mechanical analysis.

[0025] (3) The joint objective function is further established to calculate the muscle activation and obtain the corresponding muscle force, so that the muscle activation distribution is more uniform and more consistent with the actual situation, and the physiological rationality of the muscle activation distribution is improved.

[0026] (4) The real-time adaptive human muscle and bone modeling and muscle force simulation system provided by the present application not only improves the performance, but also enhances the structural rationality and interpretability of the model, and is suitable for human motion analysis, rehabilitation engineering, robot control and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced as follows.

[0028] Figure 1 is a structural schematic diagram of the real-time adaptive human muscle and bone modeling and muscle force simulation system provided by the present application.

[0029] Figure 2 is a scene of the multi-sensor data acquisition module.

[0030] Figure 3 is a structural schematic diagram of the space-time calibration module.

[0031] Figure 4 is a structural schematic diagram of the three-dimensional force prediction module.

[0032] Figure 5 is a human muscle and bone model schematic diagram.

[0033] Figure 6 is a flowchart of the human muscle and bone model parameter adjustment.

[0034] Figure 7 It is a structural diagram of the muscle force simulation module.

[0035] Figure 8 It is a structural diagram of the data visualization module.

[0036] Figure 9 This is a comparison chart of muscle strength for different standing-up strategies.

[0037] Figure 10 This is a comparison chart of muscle activation patterns of different optimization methods. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] In order to achieve low-cost, high-precision real-time muscle force simulation analysis and adapt to individual differences, the embodiment provides a real-time adaptive human musculoskeletal modeling and muscle force simulation system, as shown in the attached Figure 1 The figure shows the overall design scheme structure diagram of the system, which includes: multi-sensor data acquisition module, spatiotemporal calibration module, three-dimensional force prediction module, real-time modeling and parameter adjustment module, muscle force simulation module and data visualization module.

[0040] In an embodiment, a multi-sensor data acquisition module, which includes a depth camera, a foot pressure sensor array, a buttocks pressure sensor array and a data acquisition terminal, is used to synchronously acquire position and pressure distribution data of human joints.

[0041] Specifically, Figure 2 As shown, the depth camera, a Microsoft Kinect Azure, is positioned directly in front of the subject, within a range of 1.5-3 meters, with a sampling frequency of 30 Hz. Both the foot and buttock pressure sensor arrays consist of a matrix array of 32×32 flexible pressure sensor units, with an effective sensing area of ​​400 mm×400 mm and a unit resolution of 11.5 mm. These units are used to collect vertical pressure distribution. The subject's feet and buttocks are positioned above their respective pressure sensor arrays, and test movements such as standing still and standing up are performed. Joint kinematic data and pressure distribution data are simultaneously recorded at the data acquisition end.

[0042] In an embodiment, a spatiotemporal calibration module adopts a spatial calibration algorithm to establish a coordinate system mapping relationship by clicking on feature points of a pressure sensor array in the field of view of a depth camera, thereby achieving spatial calibration fusion of multiple coordinate systems.

[0043] Specifically,Figure 3 As shown, signals from multiple monitoring devices are time-aligned and captured within a unified time range to achieve time domain calibration. Spatial calibration involves clicking on four corner sensor units in the foot and buttocks sensor arrays in the depth camera field of view software, a total of eight points, as feature points. The positions of these feature points in the depth camera coordinate system are then acquired. Based on these feature points, the rotation matrix from the pressure array coordinate system to the depth camera coordinate system is calculated, establishing a mapping relationship. Finally, the data is unified in the depth camera coordinate system to achieve spatial calibration.

[0044] In an embodiment, a three-dimensional force prediction module extracts pressure distribution features from pressure distribution data, constructs a joint graph structure and a body segment graph structure based on the pressure distribution features and the positions of human joints, and extracts spatial features and temporal features from the joint graph structure and the body segment graph structure through a graph attention network and a temporal convolution network respectively, and performs feature fusion to predict the three-dimensional foot and hip sole pressure.

[0045] Specifically, Figure 4 As shown in the figure, the 3D force prediction module builds a novel heterogeneous multimodal fusion architecture. First, a convolutional network combined with a self-attention mechanism is used to extract pressure distribution features from the 64×32 pixel pressure distribution data of each frame.

[0046] The extracted pressure distribution features and 19 joint position features are then constructed into a joint graph structure ,in is a set of human joint points, including the position information and pressure distribution characteristics of human joint points. The set of edges connecting the human joints; Through the edge-node conversion layer, the joint graph structure is converted into a body segment graph structure ,in is the set of body segment nodes, The set of edges connecting body segments.

[0047] The edge-node conversion layer converts the graph structure Convert to a new graph structure ,picture Edge In the new picture Corresponding node , The definition of a node in is as follows: 1. Node characteristics: ,in, and for middle Corresponding edges The characteristics of the two endpoints, are learnable weights.

[0048] 2. Connection relationship of nodes: for two nodes and , if they share an end point of two corresponding edges and in , there is an undirected edge connection between them.

[0049] By adopting the above edge-node conversion layer for edge-node conversion, the following purposes can be achieved: (1) Conversion of perspective from joint center to body segment center: the joint graph structure takes the human body joint node as the node, and the edge represents the bone; while the body segment graph structure takes the bone (body segment) as the node, and the edge represents the joint. This conversion is more in line with the force transmission path in biomechanics, and the force is transmitted through the bone (body segment), not the joint.

[0050] (2) Enhanced feature representation capability: through the learnable weight fuse the features of the two end joints, so that each body segment node can adaptively absorb the information of the two end joints, which is more conducive to force modeling.

[0051] (3) Structural induction bias: the body segment graph more directly reflects the human biomechanical structure, which is conducive to learning the generation and transmission mechanism of human muscle force.

[0052] Then, the joint graph structure and the body segment graph structure are respectively subjected to graph attention coding and time convolution feature extraction through a graph attention network and a time convolution network, for capturing the importance relationship between nodes in the joint graph structure and the body segment graph structure and the dynamic changes of actions; wherein the attention coefficient of the graph attention network is: , wherein is a weight matrix, is an attention vector, is a node feature, and || represents a splicing operation; The spatial features obtained by the graph attention coding and the time sequence features extracted are input into a feature fusion layer for feature fusion, and then sequentially pass through a time convolution network, a global pooling layer and a full connection layer, to output a three-dimensional force vector containing a total of 9 components of the double foot bottom and the hip bottom. The network training data set contains 464 standing up action samples, and the prediction accuracy is above 95% in the vertical component and above 85% in the horizontal component.

[0053] Through such heterogeneous multi-modal fusion and graph structure conversion, the joint graph is converted into a body segment graph that is more in line with the mechanical transmission law, and the graph attention network and the time convolution network are combined for spatio-temporal feature modeling, ultimately realizing high-precision three-dimensional foot and hip bottom pressure prediction.

[0054] In the embodiment, the real-time modeling and parameter adjustment module is configured to construct a human musculoskeletal model containing muscle geometric paths based on the positions of the human body joint nodes, to optimize the parameters of the human musculoskeletal model in real time according to the actually measured muscle length range, and to output reasonable muscle length and speed.

[0055] Specifically, as shown in Figure 5 The constructed human musculoskeletal model containing muscle geometric paths is shown in the figure, and the 19 joint node positions obtained based on the depth camera include head, neck, pelvis, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hand, right hand, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, and right foot.

[0056] The model contains 18 main muscles: hamstrings, biceps femoris, gluteus maximus, iliopsoas, rectus femoris, quadriceps femoris, gastrocnemius, soleus, tibialis anterior, erector spinae, internal oblique, external oblique, and left and right sides, respectively.

[0057] Each muscle is defined by 2-3 path points to define the geometric path, and the muscle position directly corresponds to the joint node position instead of the general model, without the need for scaling operation in OpenSim, and real-time processing can be achieved.

[0058] Further, as shown in Figure 6 A double-layer optimization strategy is constructed to cope with individual differences in muscle parameters, which is used to optimize the parameters of the human musculoskeletal model in real time, including preliminary parameter adjustment and parameter fine-tuning. First, the human musculoskeletal model parameters are preliminarily adjusted by a linear scaling preprocessing algorithm, and the calculation formula of linear scaling is: , , , Wherein is the measured muscle length of the first muscle, is the predicted muscle length of the first muscle; Subsequently, a multi-objective optimization algorithm is used to fine-tune the muscle parameters, and the objective function of the multi-objective optimization algorithm is established as: ,

[0059] Wherein, is the optimal fiber length of the first muscle, is the tendon slack length of the first muscle, is the tendon slack length of the first Optimal fiber length initial value of block muscle in original human musculoskeletal model, For the first Tendon slack length initial value of block muscle in original human musculoskeletal model, Total number of optimized muscles, , , And Respectively, the corresponding weight coefficient of each item, Variance.

[0060] The composition of the objective function: the first item is the maximum length normalized deviation term, which makes the measured maximum muscle length normalized value and the model maximum muscle length normalized value as close as possible; the second item is the minimum length normalized deviation term, which makes the measured minimum muscle length normalized value and the model minimum muscle length normalized value as close as possible; the third item is the uniform distribution term, which makes the optimal fiber length distribution of all muscles more uniform (smaller variance); the fourth item is the regularization term, which prevents the optimization result from deviating too much from the original model parameters and ensures physiological reasonableness.

[0061] And set the constraint condition , By constraining the length parameter of the muscle to be a positive number, the optimal solution is found under the premise of meeting this condition. By optimizing the human musculoskeletal model parameters in real time, reasonable muscle length and speed are output.

[0062] Through the adjustment, the normalized muscle length and speed are within the reasonable physiological range, and the biological reasonableness of the muscle mechanical characteristics is maintained.

[0063] In the embodiment, the muscle force simulation module is used to calculate the human joint moment through the position of the human joint and the three-dimensional foot-hip bottom pressure, establish a joint objective function based on the calculated human joint moment and combined with reasonable muscle length and speed, calculate muscle activation and obtain corresponding muscle force, and complete muscle force simulation.

[0064] Specifically, as Figure 7 Indicated, in order to solve the problem of unreasonable muscle activation allocation, an optimization framework is constructed to improve the muscle force simulation effect.

[0065] Firstly, based on the human musculoskeletal model, input the human joint data collected by the depth camera and the three-dimensional foot bottom pressure and three-dimensional hip bottom pressure data predicted by the three-dimensional force prediction module, calculate the human joint moment according to the Newton-Euler equation, and associate the joint with the corresponding muscle controlling the flexion and extension function of the joint.

[0066] Based on the calculated human joint moment and combined with reasonable muscle length and speed, a joint objective function is established to calculate muscle activation, and the objective function is:

[0067] in, For the The activation of the muscles is the maximum value of all muscle activations, is the difference between the maximum and minimum muscle activation values, For adjacent moments Variations in activation of the same muscle.

[0068] The first term (overall activation term) of the joint objective function is used to minimize the square of all muscle activations and reduce overall muscle energy consumption; the second term (maximum activation penalty term) is used to suppress excessive activation of a single muscle; the third term (extreme activation term) is used to encourage a uniform distribution of activations; and the fourth term (temporal smoothing term) is used to suppress drastic fluctuations in activations over time and improve physiological rationality. 、 、 are weight coefficients for each item, which are 1, 5, and 100 respectively in this embodiment.

[0069] When establishing the joint objective function to calculate muscle activation, set the corresponding constraint conditions. The constraint conditions are the joint torque balance equations, which are calculated as follows:

[0070] in, R is the force arm matrix, is the maximum isometric force, 、 、 are tension-length, force-velocity and passive tension coefficients, is the muscle fiber length, is the muscle fiber velocity, is the fiber length at which muscle force is maximum, is the maximum fiber speed, is the joint torque.

[0071] Next, the optimized muscle activation , substitute into the Hill muscle model to get the muscle force , the muscle force calculation formula is: .

[0072] In the muscle force simulation module of this embodiment, the Hill-type muscle model includes a variety of optional mechanical characteristic curve models, such as: tension-length relationship curve, force-velocity relationship curve and passive tension curve. Users can select a specific curve model for each curve according to different application requirements.

[0073] Further, the adaptive filtering technology is used to filter the simulated muscle force signal, and the filtering parameters are dynamically adjusted according to the noise level and frequency characteristics of the signal. The filter is a 4th order Butterworth low-pass filter, and the basic cutoff frequency is 8 Hz, which is adaptively adjusted in the range of 4-12 Hz according to the signal characteristics. The noise threshold is set to 0.15, and when the signal noise level exceeds the threshold, the cutoff frequency is automatically reduced to enhance the filtering effect.

[0074] The data visualization module is used to provide real-time animation of muscle activation, muscle force characteristic curve and human musculoskeletal model, and the muscle activation intensity is represented by color depth. Specifically, as shown in Figure 8 In order to better show the real-time adaptive human musculoskeletal modeling and muscle force, the real-time animation of muscle activation, muscle force characteristic curve and human musculoskeletal model is visualized.

[0075] In order to verify the effectiveness of the real-time adaptive human musculoskeletal modeling and muscle force simulation system proposed in the application, a healthy subject is tested.

[0076] In the experiment, the healthy subject adopts three different standing strategies: (1) Momentum Transfer (MT) standing strategy, (2) Exaggerated Trunk Flexion (ETF) standing strategy, (3) Dominant Vertical Rise (DVR) standing strategy. At the same time, the high-precision Vicon motion capture system and three-dimensional force platform are used to record data, which are used as actual measurement values for comparison.

[0077] As shown in Figure 9 The typical muscle (tibialis anterior muscle) force average trend comparison chart of multiple tests of different standing strategies is shown. The overall trend of the muscle force calculated by the method of the application is consistent with that obtained by the traditional Vicon+OpenSim method, and compared with the traditional Vicon+OpenSim method, the muscle force curve of the method of the application shows more detailed changes, which proves the accuracy and reliability of the application. Through the real-time adaptive human musculoskeletal modeling and muscle force simulation system of the application, the significant differences in the change patterns of the tibialis anterior muscle force of the three strategies can be clearly identified, and the ETF standing strategy shows more dramatic muscle force changes, which is consistent with the actual situation of the ETF strategy, i.e. the excessive dorsiflexion of the ankle joint caused by the trunk past forward leaning. This result verifies the effectiveness of the application in distinguishing different movement strategies.

[0078] As shown in Figure 10As shown, it is a muscle activation pattern comparison chart of different optimization methods. Compared with the traditional method of minimizing the square sum of muscle activation, the muscle activation distribution obtained by the real-time modeling and parameter adjustment module and the establishment of the joint objective function optimization is more even, especially the muscle activation of key muscles such as the hamstrings and the short head of the biceps femoris changes from close to zero to significant, which is more consistent with the actual situation.

[0079] Unlike the prior art, the present application constructs a muscle and bone model specifically for depth camera data, with muscle positions directly corresponding to joint positions rather than a general model, without the need for scaling operations in OpenSim, enabling real-time processing. Unlike traditional methods that use general muscle model parameters, the problem of poor optimization results caused by the presence of extreme values in the muscle force characteristic curve is solved by introducing a space-time calibration module and a real-time modeling and parameter adjustment module. The spatial calibration algorithm solves the problem of multi-sensor data fusion, and further constructs a human muscle and bone model based on depth camera data, so that muscle positions directly correspond to human joint positions in a proportional relationship rather than a general model, without the need for manual scaling and calibration operations, enabling real-time processing and significantly improving individual adaptability. Further, a three-dimensional force prediction module is introduced, which converts the joint graph into a body segment graph that better conforms to the laws of mechanical transmission through heterogeneous multi-modal fusion and graph structure conversion, and combines graph attention networks and temporal convolution networks for spatio-temporal feature modeling, ultimately achieving high-precision three-dimensional force prediction, providing a high-quality data basis for subsequent body segment-based mechanical analysis. A joint objective function is established to calculate muscle activation and obtain corresponding muscle force, making muscle activation distribution more even and more consistent with the actual situation, improving the physiological reasonableness of muscle activation distribution, enhancing the flexibility of the model, and enabling a combination of low-cost depth cameras and pressure sensor arrays to achieve high-precision real-time muscle force simulation analysis, which can be widely used in human motion analysis, rehabilitation engineering, robot control and other fields.

[0080] The specific embodiments described above detail the technical solutions and beneficial effects of the present application. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the scope of the principles of the present application should be included within the scope of protection of the present application.

Claims

1. A real-time adaptive human musculoskeletal modeling and muscle force simulation system, characterized in that: include: The multi-sensor data acquisition module includes a depth camera, a foot pressure sensor array, a buttocks pressure sensor array, and a data acquisition terminal, which is used to synchronously collect the position and pressure distribution data of human joints; The spatiotemporal calibration module uses a spatial calibration algorithm to establish a coordinate system mapping relationship by clicking on the feature points of the pressure sensor array in the depth camera's field of view, thereby achieving spatial calibration fusion of multiple coordinate systems; The 3D force prediction module extracts pressure distribution features from the pressure distribution data and constructs joint graph structures and body segment graph structures based on the pressure distribution features and the positions of human joints. The joint graph structure and body segment graph structure respectively extract spatial features and temporal features through a graph attention network and a temporal convolutional network, and perform feature fusion to predict the 3D foot and hip pressure. The real-time modeling and parameter adjustment module is used to build a human musculoskeletal model including muscle geometric paths based on the positions of human joints. It optimizes the parameters of the human musculoskeletal model in real time according to the actual measured muscle length range and outputs reasonable muscle length and speed. The muscle force simulation module is used to calculate the human joint torque through the position of the human joint points and the three-dimensional foot and hip pressure. Based on the calculated human joint torque and combined with reasonable muscle length and speed, a joint objective function is established to calculate the muscle activation degree and obtain the corresponding muscle force to complete the muscle force simulation.

2. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: The real-time adaptive human musculoskeletal modeling and muscle force simulation system also includes: a data visualization module for providing real-time animation of muscle activation degree, muscle force characteristic curve and human musculoskeletal model, and indicating muscle activation intensity by color depth.

3. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: In the multi-sensor data acquisition module, the plantar pressure sensor array and the buttocks pressure sensor array are both matrix arrays composed of multiple rows and columns of flexible pressure sensor units, which are used to collect vertical pressure distribution.

4. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: In the spatiotemporal calibration module, the coordinate system mapping relationship is established by clicking the feature points of the pressure sensor array in the depth camera field of view, including: clicking the sensor units of the sole sensor array and the buttocks sensor array in the depth camera field of view as feature points, obtaining the coordinate system position of the feature points in the depth camera, calculating the rotation matrix from the pressure array coordinate system to the depth camera coordinate system, and establishing a mapping relationship.

5. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: In the three-dimensional force prediction module, the joint graph structure is constructed based on the pressure distribution characteristics and the positions of the human joint points, and the body segment graph structure is obtained by converting the joint graph structure, including: The pressure distribution features including the plantar pressure distribution and buttocks pressure distribution extracted by the convolution layer and self-attention mechanism are modeled into a joint graph structure together with the position of the human joints. ,in is a set of human joint points, including the position information and pressure distribution characteristics of human joint points. The set of edges connecting the human joints; Through the edge-node conversion layer, the joint graph structure is converted into a body segment graph structure ,in is the set of body segment nodes, The set of edges connecting body segments.

6. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 5, characterized in that: When performing graph structure conversion through the edge-node conversion layer, the following conversion rules are followed: Each edge in the joint graph structure becomes a node in the segment graph structure, and the node feature is a weighted combination of the node features at both ends of the original edge: ， in, The first Node features, is the weight parameter, The first Node features, The first Node features; In the joint graph structure, if two edges share a joint node, then there is an undirected edge connection between the corresponding two nodes in the segment graph structure.

7. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 5, characterized in that: In the 3D force prediction module, spatial features and temporal features are extracted and fused based on the joint graph structure and body segment graph structure through a graph attention network and a temporal convolutional network, respectively, to predict the 3D foot and hip pressure. This includes: The joint graph structure and the body segment graph structure are respectively subjected to the graph attention network and the temporal convolutional network for spatial feature and temporal feature extraction to capture the importance relationship and dynamic changes of the nodes in the joint graph structure and the body segment graph structure; wherein the attention coefficient of the graph attention network is: ， in, is the weight matrix, is the attention vector, is the node feature, || represents the splicing operation; The spatiotemporal features extracted from the joint graph structure and the body segment graph structure are input into the feature fusion layer for feature fusion, and then pass through the temporal convolutional network, the global pooling layer and the fully connected layer in sequence. The output is the three-dimensional force vector containing the soles of both feet and the bottom of the buttocks, which is the three-dimensional foot and buttocks pressure.

8. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: In the real-time modeling and parameter adjustment module, the three-dimensional coordinates of the human joint points obtained by the depth camera are used to construct a human musculoskeletal model including muscle geometric paths, including: Based on the human joint points captured by the depth camera and combined with the muscle system, the geometric path of each muscle is defined. Each muscle defines a geometric path through multiple path points, and a direct connection is established between the muscle position and the human joint points; wherein, the muscle system includes: hamstrings, biceps femoris, gluteus maximus, iliopsoas, rectus femoris, quadriceps femoris, gastrocnemius, soleus, tibialis anterior, erector spinae, internal oblique muscles, external oblique muscles, and their corresponding left and right sides.

9. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: In the real-time modeling and parameter adjustment module, the real-time optimization of the human musculoskeletal model parameters includes preliminary parameter adjustment and parameter fine-tuning; The preliminary parameter adjustment includes: performing preliminary adjustment on the parameters of the human musculoskeletal model by using a linear scaling preprocessing algorithm. The linear scaling calculation formula is: ， ， ， Among them, For the The measured muscle length of each muscle, For the predicted muscle length of the muscle; The parameter fine-tuning includes establishing a multi-objective optimal function including a maximum length normalized deviation term, a minimum length normalized deviation term, a uniform distribution term, and a regularization term to fine-tune the muscle parameters, so as to keep the normalized muscle length and velocity within a reasonable physiological range and maintain the biological rationality of the muscle mechanical characteristics. The calculation is as follows: ， in, For the The optimal fiber length for a muscle For the The relaxed length of the tendon of the muscle For the The optimal initial value of fiber length of each muscle in the original human musculoskeletal model, For the The initial value of the tendon relaxation length of the muscle in the original human musculoskeletal model, To optimize total muscle mass, 、 、 and are the weight coefficients corresponding to each item, is the variance.

10. The real-time adaptive human musculoskeletal modeling and muscle force simulation system according to claim 1, characterized in that: In the muscle force simulation module, the joint objective function includes: an overall activation term, a maximum activation penalty term, an activation range term, and a time smoothing term; the overall activation term is used to minimize the square of all muscle activations and reduce overall muscle energy consumption; the maximum activation penalty term is used to suppress excessive activation of a single muscle; the activation range term is used to promote uniform distribution of muscle activation; and the time smoothing term is used to suppress drastic fluctuations in muscle activation over time and improve physiological rationality. It is calculated as follows: ， in, For the The activation of the muscles is the maximum value of all muscle activations, is the difference between the maximum and minimum muscle activation values, For adjacent moments Changes in activation of the same muscle, 、 、 is the weight coefficient of each item.

Images