Physiotherapy path planning method based on human body 3D reconstruction
By using a method based on 3D reconstruction of the human body, TSDF voxel network and SMPL model to generate real-time physiotherapy paths, the problem of inaccurate physiotherapy paths in existing technologies is solved, personalized and real-time adjustment is achieved, and the physiotherapy effect is improved.
Patent Information
- Application Number
- CN202510873497.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-10-14
AI Technical Summary
The existing physiotherapy path generation process cannot fully integrate the actual parameter information and dynamic movement of the human body, resulting in the generated physiotherapy path being inaccurate and unable to be adjusted in real time, which weakens the physiotherapy effect.
A method based on human body 3D reconstruction is adopted. The scene is reconstructed through the TSDF voxel network, and the SMPL model is combined to perform dynamic 3D reconstruction of the human body. Bone segment information and acupoint coordinates are extracted to generate a real-time physical therapy path. Temporal consistency optimization is achieved through a sparsity optimization model and a graph convolution LSTM structure.
It improves the accuracy and personalization of therapy path planning, can respond to human body parameters and movement changes in real time, and improves the therapy effect.
Smart Images

Figure CN120783944A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of physiotherapy path planning, in particular to a physiotherapy path planning method based on human body 3D reconstruction. BACKGROUND
[0002] The current physiotherapy path generation process has a significant limitation, that is, it cannot fully integrate and utilize the actual parameter information of the target human body and the dynamic movement of the human body during physiotherapy to make real-time adjustments and optimizations. This defect leads to the generated physiotherapy path often being not accurate enough to perfectly fit the individual's unique needs and real-time state changes. In actual application, the slight movement of the human body during physiotherapy can have an important impact on the physiotherapy effect. However, due to the lack of real-time response capability to such dynamic changes in the prior art, the physiotherapy path often appears rigid and not personalized enough, which may weaken the overall effect of physiotherapy. Therefore, developing an intelligent physiotherapy path generation system that can capture and respond to changes in human body parameters and movements in real time has become the key to improving the accuracy and effect of physiotherapy, which is of great significance for promoting the progress and popularization of physiotherapy technology. Therefore, how to overcome the above-mentioned technical problems and defects becomes a problem that needs to be focused on. SUMMARY
[0003] In order to overcome the above-mentioned problems in the prior art, the present application provides a physiotherapy path planning method based on human body 3D reconstruction, which adopts the following technical scheme:
[0004] In a first aspect, the present application provides a physiotherapy path planning method based on human body 3D reconstruction, comprising:
[0005] Obtaining N pictures based on a preset trajectory, and preprocessing the N pictures;
[0006] Converting the coordinates of the preprocessed depth map and projecting it into an RGB image to obtain a fused image;
[0007] Taking the first camera pose as the origin of the world coordinate system, reconstructing the overall scene based on the fused image through a TSDF voxel network;
[0008] Separating the human body region from the overall scene, dynamically reconstructing the human body through SMPL to generate a human body surface triangular mesh;
[0009] Extracting features from the human body surface triangular mesh to obtain bone segment information of the human body, and performing bone degree mapping based on the bone segment information; performing UV mapping after registering the human body surface triangular mesh through a standard acupoint model, and generating acupoint coordinates of the human body based on the UV mapping and the bone degree mapping;
[0010] Based on the acupuncture point coordinates and meridian theory of the human body, the initial therapy path is generated. Based on the constitution path weight adjustment and the avoidance of dangerous anatomical structures, the initial therapy path is fitted with a surface path to generate the final real-time therapy path.
[0011] Furthermore, N pictures are obtained based on the preset trajectory, including:
[0012] The deep filtering module extracts spatial features from RGB-D sequences. The deep filtering module uses a convolutional LSTM structure. The convolutional layer at the front end extracts the spatial features of a single-frame depth map. The LSTM layer aggregates the spatial features in the temporal dimension, memorizes the depth features of previous depth maps through the hidden state of the LSTM, and optimizes the temporal consistency of the depth map of the current frame.
[0013] Furthermore, the first camera pose is used as the origin of the world coordinate system, and the entire scene is reconstructed based on the fused image through the TSDF voxel network, including:
[0014] A coarse-grained voxel grid is constructed for the scene based on the first camera pose and the fused image, with each voxel associated with an LSTM unit. As new depth maps are input, the state of each voxel is updated based on the current observation.
[0015] The LSTM unit balances current observations with historically accumulated information. For stable surfaces, the LSTM increases the weight of the corresponding voxel to suppress noise. For short-lived dynamic objects, the LSTM automatically forgets their contributions through an internal gating mechanism. The state of each voxel includes the SDF value and weight.
[0016] Furthermore, a coarse-grained voxel grid is constructed for the scene based on the first camera pose and the fused image. This involves using an asymmetric encoder-decoder structure from a sparsity optimization model, with sparse 3D convolution as the core, to compress and reconstruct the voxel grid. Specifically,
[0017] A dense voxel grid is obtained, and the encoder is used to reduce the dimension and extract sparse features of the dense voxel grid. The sparse features are compactly represented through the bottleneck layer, and the features of the compact representation are sparsely reconstructed based on the decoder to obtain a compressed sparse voxel map.
[0018] The overall loss function expression used when training the sparsity optimization model is: Total Loss = Loss_BCE + 0.5 × Loss_Grad + 0.1 × Loss_TV;
[0019] Where Total Loss represents the overall loss function, Loss_BCE, Loss_Grad, and Loss_TV represent the binary cross entropy loss function, 3D gradient difference loss, and total variation regularization loss, respectively.
[0020] Furthermore, the human body region is separated from the overall scene, including:
[0021] Based on Mask R-CNN, single-frame 2D detection is performed from the entire scene to generate an initial human mask. Based on the initial human mask, the human point cloud is reconstructed by using 3D projection. The human point cloud is input into the temporal segmentation module to obtain the human body area. The temporal segmentation module adopts a graph convolution LSTM structure to extract the spatial features of the human point cloud through graph convolution. Based on the LSTM fusion of temporal information, the probability of each point belonging to the human body is predicted.
[0022] Furthermore, SMPL is used to perform dynamic 3D reconstruction of the human body, including:
[0023] Based on the compressed sparse voxel mapping, a feature pyramid network is constructed for the human body area. The global feature vector is obtained according to the feature pyramid network. The SMPL parameter decoder receives the output posture parameters and shape parameters, and generates the corresponding human body mesh through SMPL to obtain a parameterized human body mesh.
[0024] Furthermore, a corresponding human body mesh is generated through SMPL to obtain a parameterized human body mesh, including: based on a preset differentiable optimization framework, using SMPL as a deformable template, LSTM modeling parameter temporal dynamics, minimizing the geometric difference between the SMPL mesh and the TSDF field, and when the geometric difference between the SMPL mesh and the TSDF field is minimized, using the corresponding human body mesh generated by SMPL as the parameterized human body mesh;
[0025] Minimizing the geometric difference between the SMPL grid and the TSDF field includes: transforming the essence of human posture estimation into an optimization problem under spatiotemporal constraints based on a preset formula; the preset formula is:
[0026]
[0027] in is a geometric constraint, is a timing constraint, is the anatomical constraint, Θ = {θ1, θ2...θ T} is the timing attitude parameter, is the mapping function of the SMPL model, is the TSDF data field at time t. Based on the preset formula, it is easy to obtain that there is a shrinkage factor that makes the Riemann gradient descent method converge exponentially.
[0028] Furthermore, dynamic three-dimensional reconstruction of the human body through SMPL also includes: based on posture parameters and shape parameters, segmenting the sequence slices through the sliding window technology, extracting the temporal features of the sequence slices and vectorizing them to obtain a temporal feature vector, using the temporal feature vector as the input of a bidirectional LSTM network to obtain future situation prediction values θ', β', and obtaining a continuous action sequence based on the future situation prediction values θ', β' to complete the temporal three-dimensional reconstruction of the human body.
[0029] Furthermore, feature extraction is performed on the triangular mesh of the human body surface, wherein the feature extraction includes multimodal feature extraction, wherein the multimodal features include multimodal features obtained by detecting anatomical landmarks based on curvature, detecting anatomical landmarks based on edges, detecting visible landmarks on the body surface, and searching for geometric constraint rules.
[0030] Furthermore, based on the acupuncture point coordinates and meridian theory of the human body, an initial treatment path is generated, including:
[0031] Based on the acupoint coordinates, the meridian association information of each acupoint is determined, and the meridian association information is converted into an ordered graph structure of acupoint nodes. The nodes of the graph structure are acupoints, and the edges are meridian connections. Based on the parameterized human body grid, the node coordinates are mapped to three-dimensional space. Based on the preset physical therapy needs, the starting and target acupoints are selected, and the acupoint sequential path is generated through a graph search algorithm. The path coordinates are smoothly interpolated to achieve the continuity of the physical therapy path trajectory and obtain the initial physical therapy path.
[0032] In a second aspect, the present application also provides a physical therapy path planning system based on 3D reconstruction of the human body, comprising:
[0033] An image acquisition module is used to acquire N images based on a preset trajectory and pre-process the N images, where the images include depth images and RGB images;
[0034] The fused image acquisition module is used to perform coordinate transformation on the pre-processed depth map and project it into the RGB map to obtain the fused image;
[0035] The voxel network reconstruction module is used to reconstruct the entire scene through the TSDF voxel network based on the fused image, taking the first camera pose as the origin of the world coordinate system;
[0036] The human body surface triangular mesh generation module is used to separate the human body area from the overall scene, perform dynamic 3D reconstruction of the human body through SMPL, and generate the human body surface triangular mesh;
[0037] The human acupoint coordinate generation module is used to extract features from the human body surface triangular mesh, obtain human bone segment information, and perform bone depth mapping based on the bone segment information; after aligning the human body surface triangular mesh with the standard acupoint model, UV mapping is performed, and based on the UV mapping and bone depth mapping, the human acupoint coordinates are generated;
[0038] The real-time therapy path generation module is used to generate the initial therapy path based on the acupuncture point coordinates and meridian theory of the human body. Based on the constitution path weight adjustment and the avoidance of dangerous anatomical structures, the initial therapy path is fitted with a surface path to generate the final real-time therapy path.
[0039] In a third aspect, the present application provides an electronic device, comprising:
[0040] One or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the method as described in the first aspect.
[0041] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first aspect.
[0042] In a fifth aspect, the present application provides a computer program, which, when executed by a computer, is used to execute the method described in the first aspect.
[0043] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor.
[0044] This application has the following beneficial effects:
[0045] This application obtains N pictures based on a preset trajectory, preprocesses the N pictures, performs coordinate conversion on the preprocessed depth map, and projects it into an RGB map to obtain a fused image; uses the first camera pose as the origin of the world coordinate system, and reconstructs the overall scene based on the fused image through the TSDF voxel network; separates the human body area from the overall scene, and dynamically reconstructs the human body in three dimensions through SMPL to generate a triangular mesh of the human body surface; aligns the triangular mesh of the human body surface with the standard acupoint model and then performs UV mapping, and generates the coordinates of the acupoints of the human body based on UV mapping and bone mapping; generates an initial physiotherapy path based on the acupoint coordinates and meridian theory of the human body, and performs surface path fitting on the initial physiotherapy path based on the constitution path weight adjustment and the avoidance of dangerous anatomical structures to generate a final real-time physiotherapy path, thereby improving the accuracy of human physiotherapy path planning. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram illustrating a logic execution of a method for physical therapy path planning based on 3D reconstruction of the human body according to an embodiment of the present application;
[0047] Figure 2 This is a flow chart of a method for physical therapy path planning based on 3D reconstruction of the human body according to an embodiment of the present application;
[0048] Figure 3 This is a schematic diagram of the process of obtaining a compressed sparse voxel map according to an embodiment of the present application;
[0049] Figure 4 Schematic diagram of the spatiotemporal consistency constraints of multi-frame point clouds and TSDF data consistency verification in an embodiment of the present application;
[0050] Figure 5 Schematic diagram of a differentiable optimization framework according to an embodiment of the present application;
[0051] Figure 6 This is a system flow chart of an embodiment of the present application. DETAILED DESCRIPTION
[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0053] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0054] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0055] Glossary of terms in this application: SMPL is a parameterized human body model, TSDF is a truncated signed distance function, and LSTM is a long short-term memory network.
[0056] Continue to refer Figure 1 and Figure 2 , the figure shows a logic execution diagram and a flow chart of a physical therapy path planning method based on 3D reconstruction of the human body of the present application, the method comprising the following steps:
[0057] Step 201: obtain N pictures based on a preset trajectory and preprocess the N pictures, wherein the pictures include depth pictures and RGB pictures, and the N pictures are obtained by an RGB-D camera.
[0058] It should be noted that before acquiring N pictures based on the preset trajectory, the internal and external parameters of the camera are calibrated and the camera distortion coefficients are corrected. The goal of the internal parameter calibration of this application is to determine the focal length, principal point, and distortion coefficient. This application adopts the Zhang Zhengyou calibration method, which includes the derivation of the homography matrix and the solution of the internal parameter and the calibration of the distortion parameters. The purpose of the external parameter calibration of this application is to find the position and orientation of the camera in the 3D world (i.e., the rotation matrix R and the translation vector t). In Zhang Zhengyou's joint calibration, the initial external parameters r1, r2, t (r3 = r1 × r2) are decomposed from the single adaptation matrix H.
[0059] In an embodiment of the present application, N pictures are obtained based on a preset trajectory, including: spatial feature extraction of the RGB-D sequence based on a depth filtering module, wherein the depth filtering module adopts a convolutional LSTM structure, extracts the spatial features of a single-frame depth map through the convolutional layer of the front end, aggregates spatial features in the time dimension based on the LSTM layer, remembers the depth features of the previous multiple-frame depth maps through the hidden state of the LSTM, and optimizes the temporal consistency of the depth map of the current frame. The RGB-D sequence is a time series data containing continuous frame RGB images and depth maps. Spatial feature extraction of the RGB-D sequence by the depth filtering module can achieve edge and surface feature extraction, providing a data basis for subsequent overall scene reconstruction (static environment).
[0060] In an embodiment of the present application, N pictures are preprocessed, including: applying bilateral filtering to the N acquired depth maps and invalidating them by linear interpolation based on neighborhood valid points, and performing color and lighting correction on the N captured RGB images; wherein the bilateral filtering is implemented by weighted averaging of spatial distance and grayscale similarity.
[0061] Step 202: perform coordinate transformation on the pre-processed depth map and project it into the RGB map to obtain a fused image.
[0062] In the embodiment of the present application, the coordinate transformation of the pre-processed depth map is performed, including converting the pixel coordinates and depth values into 3D coordinates to obtain a 3D point cloud, which is specifically performed as follows: for the pixel (u d ,v d ), then obtain the three-dimensional coordinate point P corresponding to the depth mapd =[x d ,y d ,z d ] T The calculation process is: d =d(u d ,v d ), in, is the focal length of the depth camera, is the principal point of the depth camera.
[0063] In the embodiment of the present application, the depth map after coordinate conversion is projected into the RGB map to obtain a fused image, including: using the external parameters of the depth camera to the RGB camera to convert the 3D point cloud to the coordinate system of the RGB camera, where the camera external parameters include the rotation matrix R and the translation vector T, which can be specifically expressed as follows: using the camera external parameters: the rotation matrix R 3×3 and the translation vector t 3×1 , point P d Convert to the color camera coordinate system: Among them, P c is the coordinate after transformation. c =[x c ,y c ,z c ] T . The three-dimensional point cloud P in the color camera coordinate system c Projected to the color image pixel coordinates (u c ,v c ): in is the focal length of the color camera, This application fuses the depth map with the color map to give the point cloud accurate color information.
[0064] In step 203, the first camera pose is used as the origin of the world coordinate system. Based on the fused image, the entire scene is reconstructed through the TSDF voxel network, including: establishing a coarse-grained voxel grid for the scene based on the first camera pose and the fused image, with each voxel associated with an LSTM unit; as the new depth map is input, the state of each voxel is updated according to the current observation. The LSTM unit balances the current observation and historical accumulated information. For stable surfaces, the LSTM increases the weight of the corresponding voxel to suppress noise; for short-lived dynamic objects, the LSTM automatically forgets their contribution through an internal gating mechanism. The state of each voxel includes the SDF value and weight. The fused image in this application is a multi-frame point cloud. In this embodiment of the application, the SDF value is the distance from the point to the nearest surface, and a negative value indicates that the point is inside the object.
[0065] In an embodiment of the present application, a coarse-grained voxel grid is established for the scene based on the first camera pose and the fused image, including using an asymmetric encoder-decoder structure in a sparsity optimization model, with sparse 3D convolution as the core, to compress and reconstruct the voxel grid.
[0066] In an embodiment of the present application, an asymmetric encoder-decoder structure in a sparsity optimization model is adopted, with sparse 3D convolution as the core, to compress and reconstruct the voxel grid, including: obtaining a dense voxel grid, performing dimensionality reduction and sparse feature extraction on the dense voxel grid through an encoder, compactly representing the sparse features through a bottleneck layer, sparsely reconstructing the features of the compact representation based on the decoder, and obtaining a compressed sparse voxel map.
[0067] In the examples of this application, please refer to Figure 3 The key layer structure of the encoder uses a sparse 3D convolution layer, in which the convolution kernel uses a small 3x3x3 kernel to reduce the amount of computation, and the stride is 1 to maintain the resolution. The output channel gradually expands the feature dimension by increasing layer by layer. The activation function uses LeakyReLu (α=0.01) to balance the gradient flow of sparse features. The pooling kernel of the sparse maximum pooling layer is 2x2x2, with a stride of 2. It only operates on non-empty voxel areas and discards 75% of redundant voxels. The residual connection adds a skip connection every two convolution layers. The input size of the bottleneck layer is (16, 16, 16, 64), and the compression operation (64→8) is performed through a 1×1×1 convolution compression channel to generate low-dimensional features (16, 16, 16, 8). To enhance sparsity, L1 regularization is applied to the output features (threshold set to 0.01×mean), forcing more than 90% of the channel values to approach zero.
[0068] The key layer structure of the decoder is the sparse transposed convolution layer, with a kernel size of 2x2x2 and a stride of 2, which gradually restores the resolution. The number of channels decreases with each layer, matching the inverse structure of the encoder. Skip connections concatenate feature maps of the same resolution layer from the encoder to the corresponding layer in the decoder. Dynamic sparse masking is implemented by generating a binary mask based on the voxel locations activated by the encoder, and deconvolution is performed only on the masked area.
[0069] The overall loss function expression used when training the sparsity optimization model is: Total Loss = Loss_BCE + 0.5 × Loss_Grad + 0.1 × Loss_TV; where Total Loss represents the overall loss function, Loss_BCE, Loss_Grad, and Loss_TV represent the binary cross entropy loss function, 3D gradient difference loss, and total variation regularization loss, respectively. The main loss is the sparse-aware reconstruction loss, which compares the difference between the original voxel value and the reconstructed voxel value through binary cross entropy. Where y_true is the original voxel value, y_pred is the reconstructed voxel value, Loss_BCE = -∑[y_true·log(y_pred)+(1-y_true)·log(1-y_pred)]; the 3D gradient difference loss is used to constrain the neighborhood gradient of the reconstructed voxel to be consistent with the original data, protecting the edge of the object. Based on total variation regularization: Overall loss function: Total Loss = Loss_BCE + 0.5 × Loss_Grad + 0.1 × Loss_TV. It should be noted that this application adopts an asymmetric encoder-decoder structure with sparse 3D convolution as the core to optimize the efficient compression and reconstruction of voxel data.
[0070] In an embodiment of the present application, a coarse-grained voxel grid is established for the scene based on the first camera pose and the fused image, including: locating the spatial coordinate system through the first camera pose, determining the scene range through the fused image, and generating a coarse-grained voxel grid. For each voxel, the SDF (signed distance) distance from the center to the scene surface is calculated, and the approximate SDF value from the sampling point to the scene surface is calculated through the depth data of the fused image. Based on the fused image after multi-frame registration, the TSDF representation of the scene is incrementally updated through the LSTM gating mechanism, and the TSDF-LSTM state update formula is used: Among them C t : The state tensor of the 3D voxel grid at time t, F t : Forget gate tensor, I t : input gate tensor, τ: candidate state transformation tanh(W c *[H t-1 ,X t ]),∈: numerical stability constant, H t : Implicit state tensor at time t (H is the feature dimension).
[0071] This application provides an initial spatial coordinate system through the first camera pose, provides color and depth information through image fusion, and guides the initialization of the voxel grid in combination with the LSTM timing network.
[0072] In the embodiment of the present application, for each voxel, the SDF (signed distance) distance from the center to the scene surface is calculated, and after the approximate SDF value from the sampling point to the scene surface is calculated by fusing the depth data of the image, the following is further included:
[0073] From the historical TSDF voxel grid, the historical surface triangle mesh is extracted by the Marching Cubes algorithm to obtain the historical surface point set P old , extract the current surface point set P from the current frame TSDF voxel grid new , for each point set p in the historical surface point set i , find the nearest point q in the current surface point set through KD tree search i , forming the initial corresponding point pair (p_i,q_i), and designing the energy function based on the initial corresponding point pair Where T is the non-rigid deformation field, and the data item is used to measure the deformation point T(p i ) and the target point q i The Euclidean distance error, w i is the weight of the point pair, where the weight of the point pair can be dynamically adjusted based on the SDF gradient amplitude. The smoothing term is obtained by penalizing the first-order derivative of the deformation field T. Force the deformation of adjacent points to remain smooth and avoid local mutations, λ s is the smoothing coefficient. The regularization term ensures that the deformation is reasonable by introducing additional constraints, where λ r represents the regularization term weight coefficient, is a regularization function. The optimal deformation field parameters are solved by iterative optimization so that the energy function E(T) is minimized. The present application can solve the optimal deformation field parameters by the Gauss-Newton iteration method, and iteratively update the deformation field by linearizing the incremental equation until the energy function converges. The historical TSDF voxel grid is updated by the optimized deformation field, and is fused with the current frame TSDF by weight, and the establishment of corresponding point pairs, the construction of the energy function, and the minimization of the energy function are repeated until the non-rigid registration is completed. In an embodiment of the present application, when the signs of p_i and q_i in the target TSDF are inconsistent, the point pair is eliminated.
[0074] Step 204 : Separate the human body region from the overall scene, perform dynamic 3D reconstruction of the human body using SMPL, and generate a triangular mesh of the human body surface.
[0075] In the embodiment of the present application, the human body region is separated from the whole scene, including: performing single-frame 2D detection from the whole scene based on Mask R-CNN to generate an initial human body mask, reconstructing a human body point cloud by using 3D projection based on the initial human body mask, inputting the human body point cloud into a time segmentation module, and obtaining the human body region, wherein the time segmentation module adopts a graph convolution LSTM structure, extracts spatial features of the human body point cloud through graph convolution, fuses time sequence information based on LSTM, and predicts the probability of whether each point belongs to the human body. The time segmentation module can repair the errors of single-frame detection.
[0076] In the embodiment of the present application, please refer to Figure 4 , the human body is dynamically reconstructed by SMPL, including: constructing a feature pyramid network for the human body region based on compressed sparse voxel mapping, obtaining a global feature vector according to the feature pyramid network, receiving output posture parameters and shape parameters by an SMPL parameter decoder, generating a corresponding human body mesh by SMPL, and obtaining a parameterized human body mesh.
[0077] In the embodiment of the present application, the corresponding human body mesh is generated by SMPL to obtain the parameterized human body mesh, including: based on a preset differentiable optimization framework, using SMPL as a deformable template, modeling LSTM parameters time sequence dynamics, minimizing the geometric difference between the SMPL mesh and the TSDF field, and when the geometric difference between the SMPL mesh and the TSDF field is minimized, generating a corresponding human body mesh by SMPL as a parameterized human body mesh, wherein minimizing the geometric difference between the SMPL mesh and the TSDF field includes: based on a preset formula, converting human body posture estimation into an optimization problem under space-time constraints; wherein the preset formula is:
[0078]
[0079] wherein is a geometric constraint, is a time sequence constraint, is an anatomical constraint, and Θ={θ1, θ2, …, θ T} is a time sequence posture parameter, is a mapping function of the SMPL model, is a t-time TSDF data field, and based on the preset formula, there is a contraction factor that makes the Riemannian gradient descent method converge exponentially.
[0080] In an embodiment of the present application, dynamic three-dimensional reconstruction of the human body is performed through SMPL, which also includes: based on posture parameters and shape parameters, segmenting the sequence slices through the sliding window technology, extracting the time series features of the sequence slices and vectorizing them to obtain time series feature vectors, using the time series feature vectors as the input of the bidirectional LSTM network, obtaining future situation prediction values θ', β', obtaining continuous action sequences based on the future situation prediction values θ', β', and completing the time series three-dimensional reconstruction of the human body. After the three-dimensional reconstruction of the human body is completed, the continuous action sequence is reversely projected into the TSDF space to perform spatiotemporal consistency constraints on the multi-frame point cloud; wherein θ' is the posture parameter and β' is the shape parameter; based on the posture parameters and morphological parameters, the corresponding human body mesh is generated through SMPL, the vertices of the generated human body mesh are projected, and based on the projected vertices, the TSDF data consistency verification is performed to realize the alignment error feedback of the SMPL parameter decoder.
[0081] This application does not directly regress SMPL parameters, but instead builds a differentiable optimization framework. SMPL is used as a deformable template, LSTM models the time series dynamics of parameters, and minimizes the geometric difference between the SMPL grid and the TSDF field. When the geometric difference between the SMPL grid and the TSDF field is minimized, the corresponding human body mesh generated by SMPL is used as the parameterized human body mesh. For a differentiable optimization framework, please refer to Figure 5 ,The mathematical arguments are as follows: This application formalizes the problem of minimizing the geometric difference between the SMPL grid and the TSDF field as follows: Human pose estimation is essentially an optimization problem under spatiotemporal constraints, in is a geometric constraint, is a timing constraint, is the anatomical constraint, Θ = {θ1, θ2...θ T} is the timing attitude parameter, is the mapping function of the SMPL model, is the TSDF data field at time t.
[0082] Differential geometry principle of geometric consistency loss: P is the set of SMPL surface sampling points, φ(p): the SDF value at the sampling point, minimizing the distance from the point cloud to the zero isosurface, TSDF normal vector (n v ) and SMPL normal vector (n m ) is the cosine of the angle, and γ(x) is used to penalize normal misalignment.
[0083] Dynamic modeling of time-series constraints: using a second-order autoregressive model as a dynamic basis
[0084]
[0085] Take θ t ≈θ t-1 +vΔt+0.5aΔt 2 Inheriting Newtonian dynamics continuity
[0086] Σ is the covariance matrix of joint motion, which conforms to
[0087]
[0088] Anatomically constrained topology optimization: in is the current rotation, ξ is the anatomical limit rotation. dist(θ i ,Ω j ) ensures that in the feasible region Ω j Inside.
[0089] The proof of the convergence of the optimization framework is as follows: in the deterministic case, the algorithm converges locally exponentially, and in the random sampling case, it converges to o(1 / k 1 / 3 ) rate converges, the anatomically constrained projection does not destroy the convergence, and the total error is linearly controlled by the errors of data acquisition and prediction model.
[0090] Proving the convergence of the differential optimization framework should be done from the following aspects: 1. The geometric properties of the objective function (strong convexity, smoothness) 2. The convergence of the Riemannian gradient descent method (deterministic case). The objective function to be optimized is as follows:
[0091] First, we should determine the expression of the optimization problem on the Riemann manifold. The posture parameter space is the Riemann manifold, P k =SO(3) / stab(e k ), where stab(e k ) is around the joint axis e k The rotation subgroup of θ The Riemann metric is defined on P as: Among them, M k , M t is the Hessian matrix of the geometric loss function, which ensures that the update is in the direction of steepest descent.
[0092] Geometric properties of the objective function: Prove that in the feasible region, the function to be optimized is strongly convex, and the decomposition of the Hessian matrix yields:
[0093]
[0094] γ g,γ t ,λ a They are The strong convex coefficient of .
[0095] Geometric constraints: In the feasible region, the Hessian matrix satisfies:
[0096]
[0097] is positive definite when ∈<1 / 2L v When A≥2I. L v is the Lipschitz constant of TSDF.
[0098] is positive semidefinite, and since the SMPL model is a local diffeomorphism in non-singular poses, it is full rank in the feasible region.
[0099]
[0100] Only when ∈<1 / 2L v Establishment
[0101] Timing constraints:
[0102] Anatomical constraints: Anatomical constraints restrict parameters to a compact convex subset middle, The role is to approach the boundary , which produces a gradually increasing gradient:
[0103]
[0104] This ensures that the iteration point will not go beyond the feasible region. Take μ = γ g +γ t λ a +γ a λ a >0, which proves strong convexity.
[0105] The convergence of the Riemann gradient descent method proves that the Riemann gradient descent method converges locally linearly. If the Riemann gradient descent method converges locally linearly, it proves that: using iterative update:
[0106]
[0107] is the retracement mapping. There is d p (θ k ,θ * )≤Ce -μk ,in
[0108] To prove this conclusion, we need to first prove The Riemann gradient of is Lipschitz continuous
[0109]
[0110]
[0111] Upper bounds for each component:
[0112]
[0113] because Compact, there must be a supremum, that is
[0114] Assume that the energy function Along the iterative path are:
[0115]
[0116] According to the definition of the retracement mapping:
[0117] and Obviously, we can get:
[0118] The above proof In θ * Strong convexity can be obtained
[0119]
[0120] γ=γ g +γ t λ a
[0121] Exploiting the boundedness of the gradient norm We can get:
[0122]
[0123] From the above, it is easy to conclude that there is a shrinkage factor that makes the Riemannian gradient descent method converge exponentially. This conclusion leads to the construction of a differentiable optimization framework. Using SMPL as a deformable template, the LSTM modeling parameter temporal dynamics minimizes the geometric difference between the SMPL mesh and the TSDF field. When the geometric difference between the SMPL mesh and the TSDF field is minimized, using the corresponding SMPL-generated human body mesh as the parameterized human body mesh achieves higher accuracy than directly regressing the SMPL parameters.
[0124] In one possible implementation, the mathematical derivation of sparse voxel → SMPL parameters is as follows: The mathematical expression of SMPL parameters is: The human body model parameters are defined as θ∈R 72 ,β∈R 10 , where θ is the posture parameter and β is the shape parameter. The posture parameter θ is parameterized by the axis-angle rotation matrix of the 24 joints. The rotation matrix R of each joint j ∈SO(3) can be mapped to a 3D axis-angle vector ω through the Rodrigues formula j =[ω jx ,ω jy ,ω jz ] T : Total posture parameters The shape parameter β defines the body shape through a linear combination of the shape basis in is the average template, S k is the PCA shape basis.
[0125] Sparse voxel feature extraction: The input sparse voxel set can be expressed as V = {(x i ,y i ,z i ,d i )}, where {x i ,y i ,z i} is the voxel coordinate, d i It is the TSDF value (non-zero value after compression).
[0126] Multi-scale features are extracted through sparse 3D convolution. The sparse convolution kernel operation is: for each valid voxel point vi∈V, its neighborhood is defined as a 3x3x3 cube area centered on vi. The convolution output feature of the lth layer is for Where is the valid voxel in the neighborhood, is the 3D convolution kernel weight tensor, and W (l) is the front layer feature, b (l) is the bias term.
[0127] Feature pyramid construction: Downsampling is achieved through strided convolution (stride = 2) to obtain multi-scale feature maps: where N k is the number of effective voxels in layer k, C k is the number of channels, F (1) ,F (2) ,F (3) is the feature map of each layer. The features of each layer are aligned to the original resolution and spliced through sparse trilinear interpolation:
[0128] f pyramid= Concat(UpSample(F (1) ), F (2) , DownSample(F (3) ))).
[0129] Global feature aggregation and parameter mapping, attention pooling, introduce spatial attention mechanism to distinguish key regions i = σ(W a f pyramid,i + b a ), where σ is the Sigmoid function, N is the total number of voxels, and i represents the importance weight of the i-th voxel. W a , b a are parameter weights, and f attn is the refined feature after screening and fusion.
[0130] Parameter regression network, map the global feature vector to SMPL parameters, pose parameters θ generate 24 joint rotation vectors through branch network: θ = W θ · MLP θ (f attn ) + b θ , where MLP θ contains hidden layer dimensions [512, 256], and the activation function is LeakyReLU. Where MLP θ is a multi-layer perceptron, W θ is a parameter weight, and b θ is a bias. Shape parameter β: β = W β · MLP β (f attn ) + b β , W β is a parameter weight, and b β is a bias.
[0131] Differential optimization of geometric alignment constraints, the SMPL vertex set can be defined as Its optimality in the TSDF space needs to satisfy: Where π(·) is the projection of the vertex to the voxel space, TSDF(·) is the truncated signed distance function, n i is the vertex normal, and is the gradient of the TSDF at point v i .
[0132] The chain rule decomposes the derivative of the vertex position with respect to the parameters: Derivative of vertex position with respect to parameters
[0133] Hybrid skinning formula from SMPL: Among them is G k Joint transformation matrix, S(β) i is the shape deformation. The corresponding Jacobian matrix Automatic differentiation is used to calculate the derivative of the geometric loss with respect to the vertex v i (β,θ) is the final coordinate of the i-th vertex after the body shape β and posture θ are adjusted.
[0134] Use the normal vector to align the items: Among them H TSDF (v i ) is the Hessian matrix of TSDF, which is approximated by voxel center differences.
[0135] Joint loss function: The overall optimization objective is a weighted combination of parameter loss and geometric loss: where θ gt , β gt are the real posture parameters and morphological parameters, λ1, λ2, and λ3 are the key hyperparameters for controlling different losses. It is a regularization term. This application is driven by posture parameters and morphological parameters, and integrates time series prediction, human body modeling, and geometric verification to realize the process from human motion parameter input to action sequence prediction and human body mesh generation verification. In the process of path planning, the movement of the target human body can be predicted in real time, so that the target object can still achieve accurate path planning when it is blocked.
[0136] Step 205: extract features from the triangular mesh of the human body surface, obtain bone segment information of the human body, and perform bone depth mapping based on the bone segment information; perform UV mapping after aligning the triangular mesh of the human body surface with the standard acupoint model, and generate the coordinates of the human acupoints based on the UV mapping and bone depth mapping.
[0137] In an embodiment of the present application, feature extraction is performed on the triangular mesh of the human body surface, wherein the feature extraction includes multimodal feature extraction, wherein the multimodal features include multimodal features obtained by detecting anatomical landmarks based on curvature, detecting anatomical landmarks based on edges, detecting visible landmarks on the body surface, and searching for geometric constraint rules.
[0138] It should be noted that anatomical landmarks are detected based on curvature: bone protrusions are located based on curvature analysis through depth data, the Gaussian curvature or mean curvature of the triangular mesh on the human body surface is calculated, and abnormally high or low curvature areas are found as candidate anatomical landmark points. Joints or bone turning points can be located through curvature detection.
[0139] Based on edge detection of anatomical landmarks: Due to differences in structure and shape between bone segments, there are local edge features, such as shape discontinuity and obvious contour lines. This application uses an edge detection algorithm to find the boundary lines or edge points between bone segments, and uses edge points to assist in distinguishing different bone segments. In the embodiment of this application, the edge detection algorithm can use gradient detection, multi-scale edge extraction, etc.
[0140] Detection of visible surface landmarks: Anatomical landmarks that are directly observable or easily located, such as bony prominences, joints, and tendon insertions, provide auxiliary positioning information. The system uses the density, color, texture, and surface normal variations of the 3D point cloud of the human surface triangulation to identify visible landmarks. Integrating medical knowledge, the system then establishes corresponding relationships between these landmarks and bone segments. This application can also detect visible surface landmarks, such as the navel and anterior superior iliac spine, through object detection algorithms.
[0141] Geometric constraint rule search: According to the anatomical spatial relationship of human bone segments, geometric constraints are set, such as the distance range of adjacent bone segments, connection angle restrictions, and joint motion range. Candidate landmark points are verified and screened through a rule search algorithm to accurately locate the bone segment boundaries. The rule search algorithm in this application can adopt graph search based on adjacency relationships, shape matching, RANSAC, etc.
[0142] In the embodiment of the present application, feature extraction is performed on the triangular mesh of the human body surface to obtain the bone segment information of the human body, and the specific contents include:
[0143] Acquire the human body surface triangular mesh, calculate the curvature features in the human body surface triangular mesh, perform edge detection on the human body surface triangular mesh, fuse the curvature features with the edge detection information, preliminarily determine the candidate anatomical landmark points, determine the specific anatomical key landmark points based on the visible landmarks on the body surface, search by using geometric constraint rules, screen the correct bone segment boundaries based on the known bone segment spatial positioning and adjacency rules, and divide the human body surface triangular mesh into several bone segments through the key landmark points.
[0144] This application uses curvature detection to locate sudden changes in joints and other areas, providing candidates for high-curvature landmark points. Edge detection allows for bone segmentation, supplementing curvature information and enabling precise segmentation. Detection of visible surface landmarks allows for precise location of key anatomical points. Geometric constraint rule searches allow for verification of bone segment structural relationships and boundary determination. Spatial distance and angle constraints ensure reasonable segmentation.
[0145] In the process of bone mapping, the body shape differences are processed through individual adaptation and dynamic adjustment. Specifically, the body shape parameters are adjusted based on the parametric human body mesh and the bone shape is dynamically scaled. The dynamic scaling of bone shape can be expressed as: 实际 =k×inch 标准(k = height / 170 cm), wherein k is an adjustment coefficient, for example, for a person with a height of 190 cm, the actual length of the bone degree 1 inch needs to be proportionally increased by about 12%.
[0146] It should be noted that the bone degree mapping is an acupoint positioning system based on the bone degree inch method, and the bone degree mapping represents the weight of the skin surface vertex affected by each bone segment. Through the bone degree mapping, the standardized surface description can be converted into accurate mapping of the personalized human body. The standard acupoint model is a standardized human body surface model, which is labeled with the positions of standard acupoints. The human body surface triangular mesh is a personalized three-dimensional human mesh model based on the collected vertices of the target human body.
[0147] In the embodiments of the present application, bone degree mapping is performed based on bone segment information; UV mapping is performed after the human body surface triangular mesh is registered with the standard acupoint model, and based on the UV mapping and the bone degree mapping, the acupoint coordinates of the human body are generated. The specific content includes: based on the bone segment information, the influence weight of each human body surface mesh vertex relative to the bone segment is calculated, and a bone degree mapping matrix is obtained. In the embodiments of the present application, a linear blend skinning algorithm can be used, and the bone degree weight is automatically or semi-automatically calculated based on the distance, normal, anatomical constraints and the like of the vertex to the bone segment.
[0148] Based on the non-rigid registration, the acupoints on the standard model are mapped to the triangular mesh surface of the target human body, and the mapping of the two-dimensional coordinate system is performed on the triangular mesh of the human body. The coordinates of the registered acupoint model are converted to the UV plane of the human mesh, and the position of the acupoint is adjusted to conform to the differences in individual bone structures in combination with the weight information of the bone degree mapping matrix, so as to obtain the three-dimensional coordinates of each acupoint on the human body triangular mesh, that is, to obtain the acupoint coordinates of the target human body.
[0149] The present application realizes accurate positioning of acupoints from the standard anatomical model to the target human body by registering the standard bone model with the human body surface triangular mesh, combining bone degree mapping and UV development, and further realizes the problem of inaccurate positioning of acupoints of personalized human bodies.
[0150] Step 206, based on the acupoint coordinates of the human body and the meridian theory, an initial physiotherapy path is generated, a curved path fitting is performed on the initial physiotherapy path based on the constitution path weight adjustment and the danger anatomical structure avoidance, and a final real-time physiotherapy path is generated.
[0151] In an embodiment of the present application, an initial therapy path is generated based on the human body's acupuncture point coordinates and meridian theory. This includes: determining the meridian association information of each acupuncture point based on the acupuncture point coordinates; converting the meridian association information into an ordered graph structure of acupuncture point nodes, where the nodes of the graph structure are acupuncture points and the edges are meridian connections. Based on a parameterized human body mesh, the node coordinates are mapped to three-dimensional space; based on preset therapy needs, starting and target acupuncture points are selected, and a sequential acupuncture point path is generated using a graph search algorithm. The path coordinates are smoothly interpolated to achieve a continuous therapy path trajectory, thereby obtaining the initial therapy path.
[0152] It should be noted that Traditional Chinese Medicine (TCM) categorizes the human body into nine basic constitutions (e.g., balanced constitution, qi deficiency constitution, yang deficiency constitution, and phlegm-dampness constitution). Different constitutions exhibit significant differences in their responses to physical therapy and appropriate stimulation pathways. Constitution pathway weight adjustment aims to quantify constitution characteristics into "meridian weight coefficients." This, combined with the patient's real-time physiological state, dynamically optimizes the pathway planning for physical therapy (e.g., massage, moxibustion, and electrical stimulation), achieving precise, tailored intervention. The input data for the constitution trait quantification model include: The Four Diagnoses of TCM: patient questionnaires (e.g., tongue image, pulse, and self-assessment of symptoms), and AI tongue / facial analysis results; physiological signals: basal metabolic rate (BMR), infrared thermograms (distribution of cold and heat), and skin conductance (Qi and blood activity); and biochemical markers: inflammatory cytokine concentrations (IL-6, CRP), and metabolite levels (e.g., uric acid and blood glucose). The constitution type T in the constitution trait quantification model can be represented as a weighted combination of feature vectors: where imprecise features are converted to quantifiable weights of 0 to 1 using a membership function. For example, the degree of "cold or hot" can be expressed as 0.6. The multi-objective optimization of the quantitative model for constitution characteristics can be expressed as follows: Constraints: where represents the theoretical therapeutic efficacy score for the path coverage area, based on the constitution-meridian correlation. represents the risk penalty coefficient, dynamically adjusted based on the patient's sensitivity. represents the weight value for the first constitution type.
[0153] The embodiment of the present application obtains a physical characteristic vector of the human body based on a physical characteristic quantification model, and adjusts the weight of each node of the initial physical therapy path based on the physical characteristic vector of the human body, thereby highlighting key areas and avoiding areas that are not suitable for physical therapy.
[0154] Dangerous anatomical structures: During medical procedures such as acupuncture and surgical robot operations, high-risk anatomical structures such as nerves, blood vessels, and organs can be identified and avoided in real time to achieve a dynamic balance between the safety and effectiveness of physical therapy.
[0155] Based on the preset key parts of the human body, parameterized human body model and LSTM timing model, the spatial range of the human body's dangerous area is marked, and the spatial range of the human body's dangerous area is dynamically updated in real time based on the LSTM timing model to determine the real-time safety buffer zone.
[0156] The curved path fitting aims to plan a continuous, smooth and TCM treatment rule-compliant physiotherapy trajectory for the complex geometric structure of a three-dimensional human body surface. The core challenge lies in balancing geometric constraints (curvature, continuity) and physiotherapy requirements (therapeutic effect coverage, obstacle avoidance safety).
[0157] Based on the real-time safety buffer, dangerous nodes of the initial physiotherapy path are filtered to obtain screened nodes. For the screened nodes in the real-time safety buffer, a physiotherapy path is generated by path fitting based on a weighted B-spline curve, and collision detection is performed on the physiotherapy path. When collision occurs, the physiotherapy path is fine-tuned to avoid obstacles, and a real-time physiotherapy path is output.
[0158] The application obtains a smooth, comfortable and safe real-time physiotherapy path by combining weight-based node control with the limitation of a real-time safety area and by weighted curved path fitting. The application adjusts the physiotherapy path in real time, which can avoid collision problems caused by human movement in real time.
[0159] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0160] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution sequence is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.
[0161] With reference to the above Figure 6 The physiotherapy path planning system based on human body 3D reconstruction described in the embodiment includes a picture acquisition module 601 configured to acquire N pictures based on a preset trajectory, and pre-process the N pictures, wherein the pictures include a depth map and an RGB map.
[0162] The fused image acquisition module 602 is used to perform coordinate conversion on the pre-processed depth map and project it into the RGB map to obtain a fused image;
[0163] A voxel network reconstruction module 603 is configured to use the first camera pose as the origin of the world coordinate system and reconstruct the entire scene through the TSDF voxel network based on the fused image;
[0164] The human body surface triangular mesh generation module 604 is used to separate the human body area from the overall scene, perform dynamic 3D reconstruction of the human body through SMPL, and generate a human body surface triangular mesh;
[0165] The human acupoint coordinate generation module 605 is used to extract features from the human body surface triangular mesh, obtain human bone segment information, and perform bone depth mapping based on the bone segment information; after aligning the human body surface triangular mesh with the standard acupoint model, perform UV mapping, and generate human acupoint coordinates based on the UV mapping and bone depth mapping;
[0166] The real-time therapy path generation module 606 is used to generate an initial therapy path based on the acupuncture point coordinates and meridian theory of the human body, and perform surface path fitting on the initial therapy path based on the physical path weight adjustment and dangerous anatomical structure avoidance to generate the final real-time therapy path.
[0167] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.
Claims
1. A method for physical therapy path planning based on 3D reconstruction of the human body, characterized in that: include: Obtain N pictures based on the preset trajectory and preprocess the N pictures; Perform coordinate transformation on the preprocessed depth map and project it into the RGB map to obtain a fused image; The first camera pose is used as the origin of the world coordinate system, and the entire scene is reconstructed through the TSDF voxel network based on the fused image; Separate the human body area from the overall scene, perform dynamic 3D reconstruction of the human body through SMPL, and generate a triangular mesh of the human body surface; Extract features from the triangular mesh of the human body surface to obtain bone segment information, and perform bone depth mapping based on the bone segment information; perform UV mapping after aligning the triangular mesh of the human body surface with the standard acupoint model, and generate the coordinates of the human acupoints based on the UV mapping and bone depth mapping; Based on the acupuncture point coordinates and meridian theory of the human body, the initial therapy path is generated. Based on the constitution path weight adjustment and the avoidance of dangerous anatomical structures, the initial therapy path is fitted with a surface path to generate the final real-time therapy path.
2. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 1, characterized in that: Get N pictures based on the preset trajectory, including: The deep filtering module extracts spatial features from RGB-D sequences. The deep filtering module uses a convolutional LSTM structure. The convolutional layer at the front end extracts the spatial features of a single-frame depth map. The LSTM layer aggregates the spatial features in the temporal dimension, memorizes the depth features of previous depth maps through the hidden state of the LSTM, and optimizes the temporal consistency of the depth map of the current frame.
3. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 1, characterized in that: The first camera pose is used as the origin of the world coordinate system. Based on the fused image, the entire scene is reconstructed through the TSDF voxel network, including: A coarse-grained voxel grid is constructed for the scene based on the first camera pose and the fused image, with each voxel associated with an LSTM unit. As new depth maps are input, the state of each voxel is updated based on the current observation. The LSTM unit balances current observations with historically accumulated information. For stable surfaces, the LSTM increases the weight of the corresponding voxel to suppress noise. For short-lived dynamic objects, the LSTM automatically forgets their contributions through an internal gating mechanism. The state of each voxel includes the SDF value and weight.
4. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 3, characterized in that: A coarse-grained voxel grid is constructed for the scene based on the first camera pose and the fused image. This involves using an asymmetric encoder-decoder structure from a sparsity optimization model, with sparse 3D convolution as the core, to compress and reconstruct the voxel grid. Specifically, A dense voxel grid is obtained, and the encoder is used to reduce the dimension and extract sparse features of the dense voxel grid. The sparse features are compactly represented through the bottleneck layer, and the features of the compact representation are sparsely reconstructed based on the decoder to obtain a compressed sparse voxel map. The overall loss function expression used when training the sparsity optimization model is: Total Loss=Loss_BCE+0.5×Loss_Grad+0.1×Loss_TV; Where Total Loss represents the overall loss function, Loss_BCE, Loss_Grad, and Loss_TV represent the binary cross entropy loss function, 3D gradient difference loss, and total variation regularization loss, respectively.
5. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 1, characterized in that: Separate the human body area from the overall scene, including: Based on Mask R-CNN, single-frame 2D detection is performed from the entire scene to generate an initial human mask. Based on the initial human mask, the human point cloud is reconstructed by using 3D projection. The human point cloud is input into the temporal segmentation module to obtain the human body area. The temporal segmentation module adopts a graph convolution LSTM structure to extract the spatial features of the human point cloud through graph convolution. Based on the LSTM fusion of temporal information, the probability of each point belonging to the human body is predicted.
6. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 5, characterized in that: Dynamic 3D reconstruction of the human body using SMPL, including: Based on the compressed sparse voxel mapping, a feature pyramid network is constructed for the human body area. The global feature vector is obtained according to the feature pyramid network. The SMPL parameter decoder receives the output posture parameters and shape parameters, and generates the corresponding human body mesh through SMPL to obtain a parameterized human body mesh.
7. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 6, characterized in that: Generate a corresponding human body mesh through SMPL to obtain a parameterized human body mesh, including: based on a preset differentiable optimization framework, using SMPL as a deformable template, LSTM modeling parameter temporal dynamics, and minimizing the geometric difference between the SMPL mesh and the TSDF field; when the geometric difference between the SMPL mesh and the TSDF field is minimized, use the corresponding human body mesh generated by SMPL as the parameterized human body mesh; Minimizing the geometric difference between the SMPL grid and the TSDF field includes: transforming the essence of human posture estimation into an optimization problem under spatiotemporal constraints based on a preset formula; the preset formula is: in is a geometric constraint, is a timing constraint, is the anatomical constraint, Θ = {θ1, θ2...θ T } is the timing attitude parameter, is the mapping function of the SMPL model, is the TSDF data field at time t. Based on the preset formula, it is easy to obtain that there is a shrinkage factor that makes the Riemann gradient descent method converge exponentially.
8. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 6, characterized in that: Dynamic 3D reconstruction of the human body using SMPL also includes: based on posture parameters and shape parameters, using sliding window technology to obtain sequence slices, extracting the temporal features of the sequence slices and vectorizing them to obtain temporal feature vectors, using the temporal feature vectors as input to a bidirectional LSTM network, obtaining future situation prediction values θ', β', and obtaining a continuous action sequence based on the future situation prediction values θ', β' to complete the temporal 3D reconstruction of the human body.
9. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 1, characterized in that: Feature extraction is performed on the triangular mesh of the human body surface, wherein the feature extraction includes multimodal feature extraction, wherein the multimodal features include multimodal features obtained by detecting anatomical landmarks based on curvature, detecting anatomical landmarks based on edges, detecting visible landmarks on the body surface, and searching for geometric constraint rules.
10. The method for physical therapy path planning based on 3D reconstruction of the human body according to claim 1, characterized in that: Based on the acupuncture point coordinates and meridian theory of the human body, an initial treatment path is generated, including: Based on the acupoint coordinates, the meridian association information of each acupoint is determined, and the meridian association information is converted into an ordered graph structure of acupoint nodes. The nodes of the graph structure are acupoints, and the edges are meridian connections. Based on the parameterized human body grid, the node coordinates are mapped to three-dimensional space. Based on the preset physical therapy needs, the starting and target acupoints are selected, and the acupoint sequential path is generated through a graph search algorithm. The path coordinates are smoothly interpolated to achieve the continuity of the physical therapy path trajectory and obtain the initial physical therapy path.
Citation Information
Cited By
Robot environment high-fidelity three-dimensional reconstruction method and device and medium
CN122244340A
