Human mesh non-rigid registration method based on x-ray fluoroscopy images

Abstract

This application discloses a non-rigid registration method for human body meshes based on X-ray fluoroscopic images. This method is used to non-rigidly register a human body mesh model, including a surface mesh M and a skeletal mesh S, to obtain an accurate human body mesh model. The method includes extracting contours from actual X-ray image data, uniformly downsampling the contours to obtain surface point clouds and skeletal point clouds, and then reusing these points. Based on mappings f and g, a non-rigid registration energy function E is obtained, and the minimum value of A is obtained when Emin is minimized. M A S This invention achieves optimal non-rigid registration. By using X-ray image data from real human bodies, the human body mesh can be simultaneously non-rigidly registered in both the surface mesh (M) and the skeletal mesh (S), achieving pixel-level matching and significantly improving the internal and external matching degree between the human body mesh and the actual human body.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and more particularly to the field of methods for revising three-dimensional digital human models based on X-ray images, specifically to a non-rigid registration method for human body meshes based on X-ray fluoroscopic images. Background Technology

[0002] 3D digital human generation technology refers to the process of creating virtual human figures (i.e., 3D digital humans or virtual humans) using computer technology and algorithms. This technology typically involves the following steps:

[0003] Data Acquisition: Human image data is acquired through various methods, including 2D images, videos, depth images, and laser scans. Computer graphics and machine learning algorithms are used to transform the acquired data into a 3D model. This process may include generating facial features, body structure, and texture mapping. After obtaining the 3D model, animation techniques are used to give it dynamic performance, enabling it to display various expressions, movements, and interactions. By combining data from different modalities, the generated 3D digital human is optimized to make it visually more realistic and natural.

[0004] The application of 3D digital human generation technology is wide-ranging and can play an important role in multiple fields. For example, in virtual characters, high-quality virtual characters can be generated in video games, animated films, and virtual reality experiences, enhancing the user experience. Users can create personalized virtual avatars according to their preferences. Users can use virtual avatars as their representatives on social platforms to interact with others, and also use them for facial replacement and special effects enhancement in social applications. In the medical and educational fields, 3D digital human models can be used for anatomy teaching and simulated surgical practice, among other applications.

[0005] In existing technologies, 3D models can be generated from single or multiple 2D images. Common methods include deep learning and convolutional neural networks (CNNs). Additionally, video sequences are used to extract motion information and facial expressions to generate more dynamic 3D models, primarily used in animation, virtual reality (VR), and augmented reality (AR). While existing technologies can generate 3D digital human models from 2D images and videos, digital humans generated from different data sources all have some shortcomings and limitations. Their comprehensiveness, realism, and especially the accuracy of generating the internal structure of the digital human, including the skeleton, are still insufficient, and precision needs improvement. Summary of the Invention

[0006] To address the issue of insufficient accuracy in existing 3D digital human generation technologies, this application provides a non-rigid registration method for human body meshes based on X-ray perspective images. This method uses objective and realistic X-ray image data to revise and register the human body mesh of an already generated 3D digital human, thereby obtaining a 3D digital human that closely matches or overlaps with the actual human body. Depending on the specific X-ray image data used for registration and revision, the non-rigid registration includes revisions to the surface mesh and skeletal mesh, ensuring that the entire 3D digital human matches the parameters of the actual human body from the inside out, resulting in a high-precision 3D digital human body model.

[0007] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0008] A non-rigid registration method for human body meshes based on X-ray fluoroscopic images is used to non-rigidly register a human body mesh model, including a body surface mesh M and a skeletal mesh S, to obtain an accurate human body mesh model. The method includes the following steps:

[0009] Step STP100 involves extracting contours from actual X-ray image data, using a contour detection algorithm to extract the human body surface contours from each X-ray image. Simultaneously, use U-Net or Mask-R-CNN algorithms to segment the in vivo skeletal contours.

[0010] Step STP200: When the X-ray image data is a CT or MRI image, the human body surface contour obtained in step STP100 is... and the outline of the internal skeleton Uniform downsampling is performed to obtain sampling points on all images. These points are then spatially stacked according to their actual pixel size to form surface point clouds. and skeletal point cloud Among them, surface point clouds In this context, N1 represents the number of points in the body surface point cloud. It is the i-th point in the homogeneous coordinates of the body surface point cloud, and the skeletal point cloud. In this context, N2 represents the number of points in the skeletal point cloud. It is the j-th point in the homogeneous coordinates of the skeletal point cloud;

[0011] Step STP300 involves using a local geometric similarity and inconsistency correspondent diffusion pruning method to calculate the vertices and surface point clouds of the body surface mesh M in the human body mesh to be registered. The mapping between f and the vertices and bone point clouds of the skeletal mesh S in the human body mesh to be registered. The mapping g between them;

[0012] Step STP400, based on the mapping of f and g, obtains the non-rigid registration A of the body surface mesh M to the target body surface mesh M'. M , For sets or matrices with non-rigid registration, Point M represents i The transformation matrix is ​​used to obtain the non-rigid registration A of the bone mesh S to the target bone mesh S'. S , For sets or matrices with non-rigid registration, Representing point S i The transformation matrix;

[0013] Step STP500: Establish the non-rigid registration energy function E.

[0014] E(A M A S ;f,g)

[0015] =α.E points (A M A S ;f,g) / +E smooth (A M A S ;f,g)

[0016] +E orth (A M A S ;f,g)

[0017] Where α represents the weight, when the body surface point cloud and skeletal point cloud α = 1 if it exists, otherwise α = 0; E points E smooth and E orth These are point cloud loss, smoothness constraint, and orthogonality constraint; the minimum value of A when Emin is obtained is obtained. M A S That is, to achieve optimal non-rigid registration.

[0018] Preferably, when the X-ray image data in step STP200 is any one or more of CT, MRI, single or continuous DR, or CBCT images, it further includes a step of contour-based non-rigid registration, specifically including:

[0019] Step STP210: Select any point M on the body surface mesh M. i Obtain the corresponding point by transforming from the standard world coordinate system to the X-ray tube coordinate system used in X-ray image data capture.

[0020]

[0021] Among them, T n This is the transformation matrix from the world coordinate system to the X-ray tube coordinate system, where n is the number of shots.

[0022] Step STP220: Obtain the pixel coordinates (PP) of all points in the human body mesh model based on the detector imaging area size and the positioning of the X-ray tube and detector. n

[0023]

[0024] Among them, cale width scale height These represent the actual length of one pixel in the horizontal and vertical directions, respectively. The pixel coordinates of the imaging point of the ray center on the detector are (ppx, ppy).

[0025]

[0026] Where width and height are the width and height of the detector imaging area, respectively, and w and h are the width and height of the detector imaging pixels, respectively.

[0027] Step STP230: Extract the set of all points at the outermost edge to obtain the body surface contour.

[0028] Step STP240, following steps STP210-STP230, process any point S on the bone mesh. j Perform conversion to obtain skeletal outline

[0029] Step STP250: The body surface contour obtained in step STP230 is... The surface profile extracted in step STP100 Perform registration, and then register the skeletal contours obtained in step STP240. Compared with the skeletal contours extracted in step STP100 Perform registration and establish the non-rigid registration energy function E.

[0030] E(A M A S ;f,g)

[0031] =α.E points (A M A S ;f,g) / +β(E sil (A M A S ;f,g))+E smooth (A M A S ;f,g)+E orth (A M A S ;f,g)

[0032] Where α and β represent weights, when the body surface point cloud and skeletal point cloud α = 1 if it exists, otherwise α = 0; when the body surface contour and skeletal outline If it exists, β = 1; otherwise, β = 0, where E points E sil E smooth and E orth These are point cloud loss, contour loss, smoothness constraint, and orthogonality constraint.

[0033] Preferably, the point cloud loss E points Obtained through the following algorithm

[0034]

[0035] Among them, w i w represents the error weight at the body surface point. j Represents the weight of the skeleton point error. If M i If there are corresponding points on the target shape, then the weight w i The weight is 1 if S is 1, otherwise the weight is 0; j If there are corresponding points on the target shape, then the weight w j The weight is 1 if it is 1, otherwise the weight is 0.

[0036] Preferably, the smoothness constraint E smooth Obtained through the following algorithm

[0037]

[0038] Among them, M i and M j S represents the adjacent vertices of the body surface. i and S j These are the vertices of adjacent bones.

[0039] Preferably, the orthogonality constraint E orth Obtained through the following algorithm

[0040]

[0041] in, It is a 3×3 rotation matrix. It is extraction The constant 3×4 matrix of the rotation components, and make sure and It is a rotation matrix, and st represents the summation performed under the condition that ... is satisfied.

[0042] Preferably, the contour loss E silObtained through the following algorithm

[0043]

[0044] Where N3 is 0 or a positive integer.

[0045] Beneficial effects:

[0046] This invention uses X-ray image data from real human bodies to perform non-rigid registration of the human body mesh on both the surface mesh (M) and the skeletal mesh (S) simultaneously, achieving pixel-level matching and greatly improving the internal and external matching degree between the human body mesh and the actual human body.

[0047] This invention can adapt to existing mainstream X-ray imaging data, including CT, MRI images, single or continuous DR or CBCT, making it applicable to a wide range of scenarios and with low requirements for source data. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the three-dimensional digital human body generation process for multimodal human body images provided by the present invention.

[0050] Figure 2 It is a multimodal feature fusion. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0054] Example 1:

[0055] This embodiment provides a multimodal human body 3D imaging method, which generates a 3D digital human body based on multimodal human body image data input from multiple viewpoints, including the following steps:

[0056] Step STP10, Image Data Acquisition and Preprocessing: The collected image data includes any one or more of RGB images, depth images, infrared images, and X-ray fluoroscopic images; the collected image data is encoded and feature extracted; the color features, depth features, and infrared features extracted from the encoded RGB images, depth images, and infrared images are denoted as Fi, ... rgb F depth F Infrared ; Figure 1 In this context, FF represents a set of multiple features.

[0057] Step STP20, image feature fusion, combines any set of aligned color features F rgb Deep features F depth and infrared signature F Infrared The features are processed by separate convolutional neural networks, and then the sigmoid function is used to obtain a deep attention mask M1 and an infrared attention mask M2. The obtained attention masks M1 and M2 are used as the primary / secondary features and the color feature F, respectively. rgb Perform fusion to obtain fusion feature F Fused , respectively using the obtained fusion features F Fused Perform camera parameter estimation and human body parameter estimation.

[0058] Step STP30, camera parameter estimation, using the fusion features F obtained in step STP200. Fused The camera pose matrix is ​​obtained by inputting it into the MLP network, and the formula for obtaining the camera pose matrix is ​​as follows:

[0059]

[0060] Among them, F Fused-i Represents the fusion features from the i-th perspective. The pose matrix and rotation matrix R represent the camera. i Translation matrix t i N is a positive integer, F c Represents MLP networks;

[0061] Step STP40, Human Parameter Estimation, involves using the fusion features F obtained in step STP200. Fused Input into the Transformer decoder to generate shape parameters The 2.5D joints J and Φ of the human body are obtained; finally, the 2.5D joints J and Φ are fed into the inverse kinematics module IK to obtain the pose parameters of the human body parameterized model.

[0062] Step STP50 generates a 3D image of the human body, and feeds the morphological parameter β and pose parameter θ into a standard SMPL model or SKEL model to obtain the fused feature F. Fused Optimized 3D human body imaging.

[0063] Example 2:

[0064] This embodiment proposes a new method for fusing features from different modalities, based on Embodiment 1. See details below. Figure 2 As shown, in step STP10, the image data encoding and feature extraction uses convolutional neural networks with CNN, ResNet, and ViT as basic structures. Independent encoders are used to encode and extract features from RGB images, depth images, and infrared images respectively. The independent encoding is expressed as follows:

[0065] F rgb =Enc rgb (I rgb )

[0066] F depth =Enc depth (I depth )

[0067] F infrared =Enc infrared (I infrared )

[0068] Among them, Enc rgb For RGB encoders, Enc depth For depth encoders, Enc infrared It is an infrared encoder.

[0069] In this embodiment, see Figure 2 As shown, in step STP20, attention masks M1 and M2 are used as primary / secondary features and color feature F, respectively. rgb The fusion process is performed according to the number of attention masks obtained, following these steps:

[0070] When only depth features F are available depthWhen obtaining the deep attention mask M1, the fusion steps are as follows:

[0071] Step STP211: Obtain the attention mask M1. The deep attention mask M1 is derived using the sigmoid function, i.e., M1 = Sigmod(CNN(F)). Depth ));

[0072] Step STP212, using depth features F depth The main features and color features F rgb The fusion feature F obtained during fusion P1 It is obtained through the following expression:

[0073]

[0074] Step STP213, using color feature F rgb The main features and depth features F depth The fusion feature F obtained during fusion s1 It is obtained through the following expression:

[0075]

[0076] Step STP214, the fusion feature F obtained through steps STP212 and STP213 P1 and F s1 Obtain fusion feature F Fused The expression is as follows:

[0077]

[0078] in, The symbol represents a fusion operation;

[0079] When only infrared feature F Infrared When obtaining the infrared attention mask M2, the fusion steps are as follows:

[0080] Step STP221: Obtain the attention mask M2, and derive the deep attention mask M1 using the sigmoid function, i.e., M2 = Sigmod(CNN(F)). Infrared ));

[0081] Step STP222, using infrared feature F Infrared The main features and color features F rgb The fusion feature F obtained during fusion P2 It is obtained through the following expression:

[0082]

[0083] Step STP223, using color feature Frgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 It is obtained through the following expression:

[0084]

[0085] Step STP224, the fusion feature F obtained through steps STP222 and STP223 P2 and F s2 Obtain fusion feature F Fused The expression is as follows:

[0086]

[0087] in, The symbol represents a fusion operation;

[0088] When both depth features F are present depth and infrared signature F Infrared When obtaining the depth attention mask M1 and the infrared attention mask M2, the fusion steps are as follows:

[0089] Step STP231: Obtain the depth attention mask M1 and the infrared attention mask M2.

[0090] M1 = Sigmod(CNN(F) Depth ))

[0091] M2 = Sigmod(CNN(F) Infrared ))

[0092] Step STP232, using depth features F depth The main features and color features F rgb The fusion feature F obtained during fusion P1 It is obtained through the following expression:

[0093]

[0094] Step STP233, using color feature F rgb The main features and depth features F depth The fusion feature F obtained during fusion s1 It is obtained through the following expression:

[0095]

[0096] Step STP234, using infrared feature F Infrared The main features and color features F rgb The fusion feature F obtained during fusion P2 Obtained through the following expression:

[0097]

[0098] Step STP235, using color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained through the following expression:

[0099]

[0100] Step STP236, the fusion feature F obtained through steps STP232 and STP235 P1 F s1 F P2 and F s2 Obtain fusion feature F Fused The expression is as follows:

[0101]

[0102] in, The symbols represent fusion operations. It's worth noting that the more features input, the more comprehensive the fused features will be, resulting in a more realistic 3D digital human image that closely resembles the real object. The more modalities of feature information, the more comprehensive the entire model; the more feature information in a single modality, the closer the parameters affecting that modality will be to the real object.

[0103] Example 3:

[0104] This embodiment is a further optimization based on any of the above embodiments, because to output a human body mesh using an SMPL model, only the shape parameters need to be provided. and attitude parameters Here, 3k represents a multiple of 3, k represents a node, and 3 represents that each node has three angles. This is a common mathematical expression in the SMPL model and belongs to the existing technology expression. In this model, β and θ are the input parameters. β represents 10 parameters such as the height, weight, and head-to-body ratio of the human body. The β parameter is the Shape Blend Pose parameter, which can control the change of human body shape through 10 incremental templates. Specifically, the change of human body shape controlled by each parameter can be depicted through animation.

[0105] θ represents the overall human body posture and the relative angles of the 24 joints, totaling 75 parameters (24*3+3; 3 degrees of freedom per joint, plus 3 root nodes). It's worth noting that, for the entire SMPL model, β and θ represent morphological or shape parameters and posture parameters, respectively, and can be understood equivalently as part of the present invention. and However, since the computation process in this invention is directional, a vector representation is more accurate. Because directly predicting the axis angles of nodes using the network is relatively difficult, specifically, in step STP400, the 2.5D human body joint J and the joint torsion angle Φ are fed into the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model. The process is implemented using the following steps:

[0106] Step STP410: Given the position of the k-th joint in human joint J. Initial joint position The rotation matrix relationship before and after the joint pose change can be expressed as:

[0107]

[0108] p k -p pa(k) =R k (t k -t pa(k) )

[0109] Among them, R k Let be the rotation matrix before and after the pose change of the current k-th joint. This is the current attitude vector. Let pa(k) be the initial state vector, and pa(k) represent the index of the parent node of the k-th joint.

[0110] Step STP420, obtain the expression for the deformed rotation matrix:

[0111]

[0112] Among them, R pa(k) Let R be the rotation matrix of the parent node of the k-th node. pa(k),k This represents the relative rotation of the k-th joint with respect to its parent joint.

[0113] Step STP430 converts the global rotation matrix into a relative rotation matrix and then splits the relative rotation matrix into a swing and a twist matrix, resulting in:

[0114] The axis of rotation based on Swing is simultaneously perpendicular to and Then there is The oscillating rotation R is derived from Rodriguez's formula. sw R tw Thus, the relative rotation matrix R is obtained;

[0115]

[0116] in, Represents perpendicular to and The vector, where α is the swing angle and φ is the twist angle. yes A skew-symmetric matrix, yes A skew-symmetric matrix, It is a 3x3 identity matrix.

[0117] Example 4:

[0118] This embodiment is an optimized version based on any of the above embodiments. In step STP500, the shape parameters obtained in step STP400 are... and attitude parameters The process of generating a 3D human body image by inputting a standard SMPL model is implemented using the following expression:

[0119]

[0120] Where M represents the SMPL function; W represents the Skinning function; T p J represents the equation for calculating vertex coordinates from shape and pose parameters; J represents the equation for predicting joints. Represents mixed weights;

[0121] Specifically,

[0122] R 3N Human average network model, SMPL Template Mesh network, with two models: Female & Male, and the default pose is T-Pose;

[0123] R N×K Blending Weight, which is the influence weight of K joints on each vertex of the SMPL Template Mesh;

[0124] R 3N×10The PCA base for human body shape, and the PCA base responsible for body shape;

[0125] R 3N×3K Calculate the regression matrix of K joints from the standard T-pose Template mesh;

[0126] R 3N×9K The influence of human posture on shape is represented by the PCA basis of human posture, which is responsible for the influence of posture on body shape and is proportional to the coefficients of the rotation matrix.

[0127] Example 5:

[0128] This embodiment is a further refinement based on any of the above embodiments. In step STP500, the shape parameters are... and attitude parameters The SKEL model is learned using an MLP network before input. arrive The steps involved in mapping data pairs to construct and generate a 3D image of the human body specifically include:

[0129] Step STP510: Select the SMPL data pair from the official SMPL dataset;

[0130] Step STP520 utilizes the control parameters from the data pair. and Inputting the data into the standard SMPL model, regression yields the vertex and node coordinates of the SMPL.

[0131] Step STP530, initialize the SKEL model, and... Directly used as Will Initialize to 0 and input into the SKEL model to obtain the skin model skin_verts and the bone model skel_verts;

[0132] Step STP540: Classify and calculate loss;

[0133] Step STP541: Calculate the vertex loss. verts The calculation method is obtained using the following formula.

[0134]

[0135] Where fitting_msk is the label for SMPL vertices and SKEL vertices;

[0136] Step STP542: Calculate the node loss. jointsThe calculation method is obtained using the following formula.

[0137]

[0138] in, This represents the node coordinate regression matrix of the skeletal model SKEL;

[0139] Step STP543, calculate scapular loss. scapula The calculation method is obtained using the following formula.

[0140]

[0141] scapula indices ={26,27,28,36,37,38}

[0142] In this context, the number 26 represents the right scapular abduction angle; 27 represents the right scapular elevation angle; 28 represents the right scapular external rotation angle; 36 represents the left scapular abduction angle; 37 represents the left scapular elevation angle; and 38 represents the left and right scapular external rotation angles.

[0143] Step STP544: Calculate spinal cord loss. spine The calculation method is obtained using the following formula.

[0144]

[0145] spine indices ={17,18,19,20,21,22,23,24}

[0146] Among them, 17, 18, and 19 represent lumbar flexion parameters, 20, 21, and 22 represent chest extension parameters, and 22, 23, and 24 represent head twisting parameters.

[0147] Step STP545: Calculate the total loss between the SKEL model and the SMPL model. total

[0148] Loss total =loss verts +loss joints +loss scapula +loss spine

[0149] Step STP555, optimization Loss total Minimum, get and Data

[0150]

[0151] Step STP565: Construct a 3D image of the human body. Using the obtained data pairs, an MLP network can be trained to... Directly obtain

[0152]

[0153] This data is then input into the SKEL model to obtain a 3D model of the human skeleton.

[0154]

[0155] Will Input into the SKEL model to obtain a 3D image of the human body.

[0156] To better illustrate the modifications of this invention regarding skin positioning, joint points, and bone vertices, the principles will be explained in detail in the following embodiment. The inverse dynamics IK process has already been described in Embodiment 3, i.e., utilizing the obtained shape parameters... and attitude parameters Inputting the SMPL model into version 1.5.1 will yield a human body surface model. The following section describes how to recover the internal skeletal model (SKEL) from the skin model (SKIN). Similar to the SMPL model, SKEL is a parametric human body model that outputs both skin and skeletal models. The input to the SKEL model is: shape parameters. and attitude parameters The output is a skin model v skin , skeleton model v skel And the joint point J, expressed as:

[0157]

[0158] It should be noted that the above This example is for illustrative purposes only and is intended to avoid confusion with the content described in this invention, especially with the foregoing. Distinguish between them.

[0159] Skin vertices: SKEL is built upon the additive method of SMPL, from the average template mesh. To begin, add the learned displacement β·B S +B P (q), where B S It is the shape basis of PCA, while B P (q) represents the pose-dependent displacement. The skin vertices are then calculated using the following linear blending skin equation:

[0160]

[0161] Among them, W i skin It is a 6890×24 weight matrix, representing the influence weight of the skin vertices on the nodes. This weight is inherited from the SMPL model by defining a corresponding SMPL joint for each NJ=24 joint of SKEL.

[0162] It is a rigid transformation matrix, representing the rotation and translation matrix of the k-th node relative to the original pose, expressed as:

[0163]

[0164] T(R k (β),0) -1 T(0,J k (β)) -1 Represents the position around its joint T(0,J) k (β)) -1 Cancel its rotation matrix (R) k (β),0) -1 Transform the bone space in its initial state; T(R) k (β),J k (β) represents the rotation and translation matrix under the initial attitude. This represents the rotation and translation matrix under the current attitude.

[0165] Key Point: The expression for key point J is as follows:

[0166]

[0167] in, This represents the node coordinate regression matrix.

[0168] Skeletal vertices: Similar to skin vertex definitions, the expression for a skeleton vertex is:

[0169]

[0170] Among them, T B As the initial skeletal template, s(J B (β) represents the proportion factor for each bone, which defines the limb length; W i skel It is the weight matrix of bones excluding the spine and thoracic cavity, representing the weight of the bone vertices affected by the nodes, where the weight of the bottom of the spine is interpolated to 0 and the weight of the top is 1; It is a rigid transformation matrix, that is, a rotation and translation matrix relative to the initial pose. It translates and rotates the bone vertices associated with the i-th node according to the pose parameter q, and its expression is:

[0171]

[0172] Among them, T(R) k (β),J k (β) represents the rotation and translation matrix under the initial attitude. This represents the rotation and translation matrix under the current attitude.

[0173] At this point, the entire 3D imaging algorithm process is complete. By combining multiple viewpoint sensors, a 3D human body mesh in the world coordinate system is generated, including the 3D coordinates of surface vertices, internal skeletons, and internal nodes. Furthermore, the coordinate system transformation relationship between multiple camera sensors is obtained, enabling coordinate transformation.

[0174] Three-dimensional digital humans can be obtained in all of the above embodiments 1-5.

[0175] Example 6:

[0176] This embodiment provides a non-rigid registration method for human body mesh based on X-ray fluoroscopic images, which is used to further non-rigid register the existing three-dimensional digital human body mesh or the three-dimensional digital human body mesh obtained in Embodiments 1-5, thereby obtaining a more accurate human body mesh.

[0177] A non-rigid registration method for human body meshes based on X-ray fluoroscopic images is used to non-rigidly register a human body mesh model, including a body surface mesh M and a skeletal mesh S, to obtain an accurate human body mesh model. The method includes the following steps:

[0178] Step STP100 involves extracting contours from actual X-ray image data, using a contour detection algorithm to extract the human body surface contours from each X-ray image. Simultaneously, use U-Net or Mask-R-CNN algorithms to segment the in vivo skeletal contours.

[0179] Step STP200: When the X-ray image data is a CT or MRI image, the human body surface contour obtained in step STP100 is used. and the outline of the internal skeleton Uniform downsampling is performed to obtain sampling points on all images. These points are then spatially stacked according to their actual pixel size to form surface point clouds. and skeletal point cloud Among them, surface point clouds In this context, N1 represents the number of points in the body surface point cloud. It is the i-th point in the homogeneous coordinates of the body surface point cloud, and the skeletal point cloud. In this context, N2 represents the number of points in the skeletal point cloud. It is the j-th point in the homogeneous coordinates of the skeletal point cloud;

[0180] Step STP300 involves using a local geometric similarity and inconsistency correspondent diffusion pruning method to calculate the vertices and surface point clouds of the body surface mesh M in the human body mesh to be registered. The mapping between f and the vertices and bone point clouds of the skeletal mesh S in the human body mesh to be registered. The mapping g between them;

[0181] Step STP400, based on the mapping of f and g, obtains the body surface mesh M and transforms it into the target body surface mesh M. ‘ Non-rigid registration A M , For sets or matrices with non-rigid registration, Point M represents i The transformation matrix is ​​used to obtain the non-rigid registration A of the bone mesh S to the target bone mesh S'. S , For sets or matrices with non-rigid registration, Point S i The transformation matrix;

[0182] Step STP500: Establish the non-rigid registration energy function E.

[0183] E(A M A S ;f,g)

[0184] =α.E points (A M A S ;f,g) / +E smooth (A M A S ;f,g)

[0185] +E orth (A M A S ;f,g)

[0186] Where α represents the weight, when the body surface point cloud and skeletal point cloud α = 1 if it exists, otherwise α = 0; E points E smooth and E orth These are point cloud loss, smoothness constraint, and orthogonality constraint; the minimum value of A when Emin is obtained is obtained. M A S That is, to achieve optimal non-rigid registration.

[0187] Example 7:

[0188] To further optimize the registration method and increase the registration dimensions, this embodiment is a further optimization and extension based on embodiment 6. In step STP200, when the X-ray image data is any one or more of CT, MRI, single or continuous DR, or CBCT images, a step based on contour-based non-rigid registration is also included, specifically:

[0189] Step STP210: Select any point M on the body surface mesh M. i Obtain the corresponding point by transforming from the standard world coordinate system to the X-ray tube coordinate system used in X-ray image data capture.

[0190]

[0191] Among them, T n This is the transformation matrix from the world coordinate system to the X-ray tube coordinate system, where n is the number of shots.

[0192] Step STP220: Obtain the pixel coordinates (PP) of all points in the human body mesh model based on the detector imaging area size and the positioning of the X-ray tube and detector. n

[0193]

[0194] Among them, cale width scale height These represent the actual length of one pixel in the horizontal and vertical directions, respectively. The pixel coordinates of the imaging point of the ray center on the detector are (ppx, ppy).

[0195]

[0196] Where width and height are the width and height of the detector imaging area, respectively, and w and h are the width and height of the detector imaging pixels, respectively.

[0197] Step STP230: Extract the set of all points at the outermost edge to obtain the body surface contour.

[0198] Step STP240, following steps STP210-STP230, process any point S on the bone mesh. j Perform conversion to obtain skeletal outline

[0199] Step STP250: The body surface contour obtained in step STP230 is... The surface profile extracted in step STP100 Perform registration, and then register the skeletal contours obtained in step STP240. Compared with the skeletal contours extracted in step STP100 Perform registration and establish the non-rigid registration energy function E.

[0200] E(A M A S ;f,g)

[0201] =α.E points (A M A S ;f,g) / +β(E sil (A M A S ;f,g))+E smooth (A M A S ;f,g)+E orth (A M A S ;f,g)

[0202] Where α and β represent weights, when the body surface point cloud and skeletal point cloud α = 1 if it exists, otherwise α = 0; when the body surface contour and skeletal outline If it exists, β = 1; otherwise, β = 0, where E points E sil E smooth and E orth These are point cloud loss, contour loss, smoothness constraint, and orthogonality constraint.

[0203] Preferably, the point cloud loss E points Obtained through the following algorithm

[0204]

[0205] Among them, w i w represents the error weight at the body surface point. j Represents the weight of the skeleton point error. If M i If there are corresponding points on the target shape, then the weight w i The weight is 1 if S is 1, otherwise the weight is 0; j If there are corresponding points on the target shape, then the weight w j The weight is 1 if it is not 1, otherwise the weight is 0.

[0206] Example 8:

[0207] This embodiment further optimizes the smoothness constraint based on the above embodiment 6 or embodiment 7, where the smoothness constraint E smooth Obtained through the following algorithm

[0208]

[0209] Among them, M i and M j For adjacent body surface vertices, S i and S j These are the vertices of adjacent bones.

[0210] Example 9:

[0211] This embodiment further optimizes the orthogonality constraint based on the above embodiment 6 or embodiment 7, where the orthogonality constraint E orth Obtained through the following algorithm

[0212]

[0213] in, It is a 3×3 rotation matrix. It is extraction The constant 3×4 matrix of the rotation components, and make sure and It is a rotation matrix, and st represents the summation performed under the condition that ... is satisfied.

[0214] Example 10:

[0215] This embodiment further optimizes the contour loss based on the above embodiment 7, with the contour loss E... sil Obtained through the following algorithm

[0216]

[0217] Where N3 is 0 or a positive integer.

[0218] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A non-rigid registration method for human body meshes based on X-ray fluoroscopic images, used to non-rigidly register a human body mesh model including a body surface mesh M and a skeletal mesh S to obtain an accurate human body mesh model, characterized in that... Includes the following steps: Step STP100 involves extracting contours from actual X-ray image data, using a contour detection algorithm to extract the human body surface contours from each X-ray image. Simultaneously, use U-Net or Mask-R-CNN algorithms to segment the in vivo skeletal contours. Step STP200: When the X-ray image data is a CT or MRI image, the human body surface contour obtained in step STP100 is... and the outline of the internal skeleton Uniform downsampling is performed to obtain sampling points on all images. These points are then spatially stacked according to their actual pixel size to form surface point clouds. and skeletal point cloud Among them, surface point clouds In this context, N1 represents the number of points in the body surface point cloud. It is the i-th point in the homogeneous coordinates of the body surface point cloud, and the skeletal point cloud. In this context, N2 represents the number of points in the skeletal point cloud. It is the j-th point in the homogeneous coordinates of the skeletal point cloud; Step STP300 involves using a local geometric similarity and inconsistency correspondent diffusion pruning method to calculate the vertices and surface point clouds of the body surface mesh M in the human body mesh to be registered. The mapping between f and the vertices and bone point clouds of the skeletal mesh S in the human body mesh to be registered. The mapping g between them; Step STP400, based on the mapping of f and g, obtains the non-rigid registration A of the body surface mesh M to the target body surface mesh M'. M , For sets or matrices with non-rigid registration, Point M represents i The transformation matrix is ​​used to obtain the non-rigid registration A of the bone mesh S to the target bone mesh S'. S , For sets or matrices with non-rigid registration, Point S i The transformation matrix; Step STP500: Establish the non-rigid registration energy function E. E(A M ,A S ;f,g) α.E points (TO M ,TO S ;f,g) / +E smooth (TO M ,TO S (f,g) +E orth (A M ,A S ;f,g) Where α represents the weight, when the body surface point cloud and skeletal point cloud α = 1 if it exists, otherwise α = 0; E points E smooth and E orth These are point cloud loss, smoothness constraint, and orthogonality constraint; the minimum value of A when Emin is obtained is obtained. M A S That is, to achieve optimal non-rigid registration.

2. The non-rigid registration method for human body mesh based on X-ray fluoroscopic images according to claim 1, characterized in that, When the X-ray image data in step STP200 is any one or more of CT, MRI, single or continuous DR or CBCT images, it also includes a step of contour-based non-rigid registration, specifically including: Step STP210: Select any point M on the body surface mesh M. i Obtain the corresponding point by transforming from the standard world coordinate system to the X-ray tube coordinate system used in X-ray image data capture. Among them, T n This is the transformation matrix from the world coordinate system to the X-ray tube coordinate system, where n is the number of shots. Step STP220: Obtain the pixel coordinates (PP) of all points in the human body mesh model based on the detector imaging area size and the positioning of the X-ray tube and detector. n Among them, cale width scale height These represent the actual length of one pixel in the horizontal and vertical directions, respectively. The pixel coordinates of the imaging point of the ray center on the detector are (ppx, ppy). Where width and height are the width and height of the detector imaging area, respectively, and w and h are the width and height of the detector imaging pixels, respectively. Step STP230: Extract the set of all points at the outermost edge to obtain the body surface contour. Step STP240, following steps STP210-STP230, process any point S on the bone mesh. j Perform conversion to obtain skeletal outline Step STP250: The body surface contour obtained in step STP230 is... The surface profile extracted in step STP100 Perform registration, and then register the skeletal contours obtained in step STP240. Compared with the skeletal contours extracted in step STP100 Perform registration and establish the non-rigid registration energy function E. E(A M ,A S ;f,g) =α.Epoints(A M ,A S ;f,g) / +β(Esil(A M ,A S ;f,g)) +Esmooth(A M ,A S ;f,g)+Eorth(A M ,A S ;f,g) Where α and β represent weights, when the body surface point cloud and skeletal point cloud α = 1 if it exists, otherwise α = 0; when the body surface contour and skeletal outline If it exists, β = 1; otherwise, β = 0, where E points E sil E smooth and E orth These are point cloud loss, contour loss, smoothness constraint, and orthogonality constraint.

3. The non-rigid registration method for human body mesh based on X-ray fluoroscopic images according to claim 1 or 2, characterized in that, The point cloud loss E points Obtained through the following algorithm Among them, w i w represents the error weight at the body surface point. j Represents the weight of the skeleton point error. If M i If there are corresponding points on the target shape, then the weight w i The weight is 1 if S is 1, otherwise the weight is 0; j If there are corresponding points on the target shape, then the weight w j The weight is 1 if it is 1, otherwise the weight is 0.

4. The non-rigid registration method for human body mesh based on X-ray fluoroscopic images according to claim 1 or 2, characterized in that, Smoothness constraint E smooth Obtained through the following algorithm Among them, M i and M j S represents the adjacent vertices of the body surface. i and S j These are the vertices of adjacent bones.

5. The non-rigid registration method for human body mesh based on X-ray fluoroscopic images according to claim 1 or 2, characterized in that, Orthogonality constraint E orth Obtained through the following algorithm in, It is a 3×3 rotation matrix. It is extraction The constant 3×4 matrix of the rotation components, and make sure and It is a rotation matrix, and st represents the summation performed under the condition that ... is satisfied.

6. The non-rigid registration method for human body mesh based on X-ray fluoroscopic images according to claim 2, characterized in that, Contour loss E sil Obtained through the following algorithm Where N3 is 0 or a positive integer.

Images