Human body grid non-rigid registration method based on X-ray perspective image

Abstract

The invention discloses a human body grid non-rigid registration method based on an X-ray perspective image, which is used for performing non-rigid registration on a human body grid model comprising a body surface grid M and a skeleton grid S to obtain an accurate human body grid model, and comprises the following steps: extracting a contour according to actual X-ray image data; and performing uniform downsampling according to the contour to obtain a body surface point cloud # imgabs0 # and a skeleton point cloud # imgabs1 #, obtaining a non-rigid registration energy function E on the premise of establishing mapping f and g, and completing the optimal non-rigid registration by obtaining AM and AS when the value is the minimum Emin. According to the method, non-rigid registration can be carried out on the human body grids in the two aspects of the body surface grids M and the skeleton grids S at the same time through the X-ray image data from the real human body, pixel-level matching is achieved, and the internal and external matching degree of the human body grids and the real human body can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, in particular to the technical field of methods for revising three-dimensional digital human models based on X-ray images, and specifically to a human body grid non-rigid registration method based on X-ray perspective images. Background Art

[0002] 3D digital human generation technology refers to the process of creating a virtual human image (i.e. 3D digital human or virtual human) using computer technology and algorithms. This technology usually involves the following steps:

[0003] Data collection: Data on human images is obtained through various means, including 2D images, videos, depth images, laser scanning, etc. The collected data is converted into a three-dimensional model using computer graphics and machine learning algorithms. This process may include the generation of facial features, body structure, texture mapping, etc. After obtaining the three-dimensional model, it is given dynamic expression through animation technology, enabling it to show various expressions, movements and interactions. Combining data from different modalities, the generated three-dimensional digital human is optimized to make it more visually realistic and natural.

[0004] The application range of 3D digital human generation technology is wide, and it can play an important role in many fields: For example, virtual characters: in video games, animated films and virtual reality experiences, high-quality virtual characters are generated to enhance user experience. Users can create personalized virtual images according to their preferences. Users can use virtual human images as their own representatives on social platforms to interact with others. At the same time, virtual human images can be used in social applications for facial replacement and special effects enhancement. In the medical and educational fields, 3D digital human models can be used for anatomical teaching and simulated surgical practice, etc.

[0005] In the existing technology, 3D models can be generated from single or multiple 2D images. Common methods include deep learning, convolutional neural networks (CNN), etc. Of course, video sequences are also used to extract motion information and expression changes to generate more dynamic 3D models, which are mainly used in animation production, virtual reality (VR) and augmented reality (AR). Although the existing technology can generate 3D digital human models from 2D images and videos, the digital humans generated based on different data have more or less some shortcomings and one-sidedness. In terms of comprehensiveness, realism, and especially the generation of the internal structure of the digital human including the skeleton, the accuracy needs to be improved. Summary of the invention

[0006] In order to solve the problem of insufficient accuracy of three-dimensional digital human generation in the prior art, the present application provides a non-rigid registration method for human body mesh based on X-ray fluoroscopic images, which is used to revise the human body mesh of an already generated three-dimensional digital human through objective and real X-ray image data, so as to obtain a three-dimensional digital human that fits or overlaps with the actual human body as closely as possible; according to the different X-ray image data for actual registration revision, the non-rigid registration includes revision of the body surface mesh and the bone mesh, so that the entire three-dimensional digital human will match the actual human body parameters from the inside out, and obtain a high-precision three-dimensional digital human model.

[0007] In order to achieve the above purpose, the technical solution adopted in this application is:

[0008] A human body mesh non-rigid registration method based on X-ray fluoroscopic images is used to perform non-rigid registration on a human body mesh model including a body surface mesh M and a skeleton mesh S to obtain an accurate human body mesh model, comprising the following steps:

[0009] Step STP100, extracting contours based on actual X-ray image data, using contour detection algorithms to extract the human body surface contours on each X-ray image data At the same time, use U-Net or Mask-R-CNN algorithm to segment the bone contours in the body

[0010] Step STP200, when the X-ray image data is a CT or MRI image, the human body surface contour obtained in step STP100 is converted into and bone contours Uniform downsampling is performed to obtain the sampling points on all images, and spatial stacking is performed according to the actual pixel size to form a surface point cloud. and bone point cloud Among them, the surface point cloud N1 represents the number of points in the body surface point cloud. is the i-th point in the homogeneous coordinates on the body point cloud, the bone point cloud N2 represents the number of points in the skeleton point cloud. is the jth point in homogeneous coordinates on the bone point cloud;

[0011] Step STP300, using a diffusion pruning method based on local geometric similarity and inconsistent counterparts to calculate the vertices of the body surface mesh M in the human body mesh to be registered and the body surface point cloud The mapping between f and the vertices and bone point cloud of the skeleton mesh S in the human body mesh to be registered The mapping between g;

[0012] Step STP400, based on the mapping f and g, obtain the non-rigid registration A of the body surface mesh M to the target body surface mesh M' M , is the set or matrix of non-rigid registration, Indicates point M i The transformation matrix of the skeleton mesh S is obtained by transferring the non-rigid registration A of the target skeleton mesh S' S , is the set or matrix of non-rigid registration, Indicates point S i The transformation matrix of

[0013] Step STP500, establishing a non-rigid registration energy function E,

[0014] E(A M ,A S ;f,g)

[0015] =α.E points (A M ,A S ;f,g) / +E smooth (A M ,A S ;f,g)

[0016] +E orth (A M ,A S ;f,g)

[0017] Among them, α represents the weight, when the surface point cloud and bone point cloud α=1 when it exists, otherwise α=0; E points 、E smooth and E orth They are point cloud loss, smoothness constraint and orthogonality constraint respectively; A with the minimum value Emin is obtained M , A S That is, the best non-rigid registration is completed.

[0018] Preferably, when the X-ray image data is any one or more of CT, MRI, single or continuous DR or CBCT in step STP200, a step based on contour non-rigid registration is further included, specifically including:

[0019] Step STP210, any point M on the body surface grid M i Convert the standard world coordinate system to the tube coordinate system of the X-ray image data shooting state to obtain the corresponding point

[0020]

[0021] Among them, T n is the transformation matrix from the world coordinate system to the tube coordinate system, and n is the number of shots;

[0022] Step STP220, obtaining pixel coordinates PP of all points of the human body mesh model according to the size of the detector imaging area and the placement of the tube and the detector n

[0023]

[0024] Among them, width 、scale height Represent the actual length of a pixel in the horizontal and vertical directions respectively. The pixel coordinates of the imaging point of the ray center on the detector are (ppx, ppy);

[0025]

[0026] Among them, width and height are the width and height of the detector imaging area, respectively, and w and h are the width and height of the detector imaging pixel, respectively;

[0027] Step STP230, extract the set of all points on the outermost edge to obtain the body surface contour

[0028] Step STP240, according to steps STP210-STP230, any point S on the skeleton mesh j Convert to get the bone outline

[0029] Step STP250, the body surface contour obtained in step STP230 Surface profile extracted with step STP100 Perform registration and place the bone contour obtained in step STP240 The bone contour extracted in step STP100 Perform registration and establish a non-rigid registration energy function E.

[0030] E(A M ,A S ;f,g)

[0031] =α.E points (A M ,A S ; f,g) / +β(E sil (A M ,A S ;f,g))+E smooth (A M ,A S ;f,g)+E orth (A M ,A S ;f,g)

[0032] Among them, α and β represent weights. and bone point cloud When it exists, α=1, otherwise α=0; when the body surface contour and bone contours When it exists, β=1, otherwise β=0, where E points 、E sil 、E smooth and E orth They are point cloud loss, contour loss, smoothness constraint and orthogonality constraint.

[0033] Preferably, the point cloud loss E points Obtained by the following algorithm

[0034]

[0035] Among them, w i represents the body surface point error weight, w j Represents the error weight of the bone point. If M i There are corresponding points on the target shape, then the weight w i is 1, otherwise the weight is 0; if S j There are corresponding points on the target shape, then the weight w j is 1, otherwise the weight is 0.

[0036] Preferably, the smoothness constraint E smooth Obtained by the following algorithm

[0037]

[0038] Among them, M i and M j are adjacent surface vertices, S i and S j are adjacent bone vertices.

[0039] Preferably, the orthogonality constraint E orth Obtained by the following algorithm

[0040]

[0041] in, is a 3×3 rotation matrix, It is extracted The rotation component of the constant 3×4 matrix, and make sure and is the rotation matrix, and st represents the completion of the above summation under the condition that...

[0042] Preferably, the contour loss E silObtained by the following algorithm

[0043]

[0044] Wherein, N3 is 0 or a positive integer.

[0045] Beneficial effects:

[0046] The present invention uses X-ray image data from a real human body to perform non-rigid registration of the human body mesh in both the body surface mesh M and the skeleton mesh S to achieve pixel-level matching, which can greatly improve the internal and external matching between the human body mesh and the actual human body.

[0047] The present invention can adapt to existing mainstream X-ray image data, including CT, MRI images, single or continuous DR or CBCT, so that the present invention has extremely broad application scenarios and has low threshold requirements for source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 It is a flow chart of generating a three-dimensional digital human body of multimodal human body images provided by the present invention.

[0050] Figure 2 It is multimodal feature fusion. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below in combination with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0055] Embodiment 1:

[0056] This embodiment provides a multi-modal human body three-dimensional imaging method, which generates a three-dimensional digital human body based on multi-modal human body image data input under multiple viewing angles, including the following steps:

[0057] Step STP10, image data acquisition and preprocessing, the collected image data includes any one or more of RGB image, depth image, infrared image, and X-ray fluoroscopic image data; the collected image data is encoded and feature extracted, and the color features, depth features, and infrared features extracted from the encoded RGB image, depth image, and infrared image are respectively recorded as F rgb , F depth , F Infrared ; Figure 1 The FF in represents a collection of multiple features.

[0058] Step STP20, fusion of image features, any set of aligned color features F rgb , deep feature F depth and infrared signature F Infrared The deep attention mask M1 and infrared attention mask M2 are obtained by using the sigmoid function. The obtained attention masks M1 and M2 are used as the primary / secondary features and the color feature F respectively. rgb Fusion is performed to obtain the fusion feature F Fused , respectively using the obtained fusion features F Fused Perform camera parameter estimation and human body parameter estimation,

[0059] Step STP30, camera parameter estimation, the fusion feature F obtained in step STP200 Fused Input the MLP network to obtain the camera pose matrix, where the formula for obtaining the camera pose matrix is ​​as follows:

[0060]

[0061] Among them, F Fused-i represents the fusion feature under the i-th perspective, Represents the camera's attitude matrix, rotation matrix R i , translation matrix t i , N is a positive integer, F c Represents the MLP network;

[0062] Step STP40, human body parameter estimation, the fusion feature F obtained in step STP200 Fused Input into Transformer decoder to generate shape parameters The human body's 2.5D joint point J and the joint torsion angle Φ are finally sent to the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model.

[0063] Step STP50, generate a three-dimensional image of the human body, and send the morphological parameters β and posture parameters θ to the standard SMPL model or SKEL model to obtain the fused feature F Fused Optimized 3D imaging of the human body.

[0064] Embodiment 2:

[0065] This embodiment is a new method for fusing features of different modes based on Embodiment 1. Figure 2 As shown, the way of encoding and feature extraction of image data in step STP10 adopts a convolutional neural network with CNN, ResNet and ViT as the basic structure, and uses independent encoders to encode and extract features of RGB images, depth images and infrared images respectively. The expression of independent encoding is:

[0066] F rgb =Enc rgb (I rgb )

[0067] F depth =Enc depth (I depth )

[0068] F infrared =Enc infrared (I infrared )

[0069] Among them, Enc rgb For RGB encoder, Enc depth For the deep encoder, Enc infrared For infrared encoder.

[0070] In this embodiment, see Figure 2 As shown, in step STP20, attention masks M1 and M2 are used as primary / secondary features and color feature F respectively. rgb The fusion steps are performed according to the number of attention masks obtained:

[0071] When only the deep feature F depthWhen the deep attention mask M1 is obtained, the fusion steps are as follows:

[0072] Step STP211, obtain the attention mask M1, and obtain the deep attention mask M1 through the sigmoid function, that is, M1 = Sigmod(CNN(F Depth ));

[0073] Step STP212, with deep feature F depth is the main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression:

[0074]

[0075] Step STP213, using the color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression:

[0076]

[0077] Step STP214, the fusion feature F obtained by step STP212 and step STP213 P1 and F s1 Get the fusion feature F Fused The expression is as follows:

[0078]

[0079] in, The symbol represents a fused operation;

[0080] When only infrared feature F Infrared When the infrared attention mask M2 is obtained, the fusion steps are as follows:

[0081] Step STP221, obtain the attention mask M2, and obtain the deep attention mask M1 through the sigmoid function, that is, M2 = Sigmod(CNN(F Infrared ));

[0082] Step STP222, using infrared feature F Infrared is the main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression:

[0083]

[0084] Step STP223, using the color feature Frgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression:

[0085]

[0086] Step STP224, the fusion feature F obtained by step STP222 and step STP223 P2 and F s2 Get the fusion feature F Fused The expression is as follows:

[0087]

[0088] in, The symbol represents a fused operation;

[0089] When there is a deep feature F depth and infrared signature F Infrared When the obtained deep attention mask M1 and infrared attention mask M2 are obtained, the fusion steps are as follows:

[0090] Step STP231, obtain the depth attention mask M1 and the infrared attention mask M2

[0091] M1=Sigmod(CNN(F Depth ))

[0092] M2=Sigmod(CNN(F Infrared ))

[0093] Step STP232, with deep feature F depth is the main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression:

[0094]

[0095] Step STP233, using the color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression:

[0096]

[0097] Step STP234, using infrared feature F Infrared is the main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression:

[0098]

[0099] Step STP235, using the color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression:

[0100]

[0101] Step STP236, the fusion feature F obtained by step STP232 and step STP235 P1 , F s1 , F P2 and F s2 Get the fusion feature F Fused The expression is as follows:

[0102]

[0103] in, The symbol represents the fusion operation. It is worth noting that the more features are input, the more comprehensive the fused features will be, and the subsequent final three-dimensional digital human imaging will be closer to the real object or the more similar / close to the real object. The more feature information modes, the more comprehensive the entire model is, and the more feature information of a single mode, the closer the parameters affected by the mode will be to the real object.

[0104] Embodiment 3:

[0105] This embodiment is a further optimization based on any of the above embodiments. In order to use the SMPL model to output the human body Mesh, it is only necessary to provide shape parameters. and attitude parameters Here, 3k represents a multiple of 3, k represents a node, and 3 represents that each node has three angles. It is a general mathematical expression in the SMPL model and belongs to the prior art expression. In the model, β and θ are input parameters, where β represents 10 parameters of the proportions of human body height, weight, head-to-body ratio, etc.; the β parameter is the Shape Blend Pose parameter, which can control the change of human body shape through 10 incremental templates: Specifically: each parameter controls the change of human body shape, which can be portrayed by animated images.

[0106] θ is 75 (24*3+3; 3 degrees of freedom for each joint point, plus 3 root nodes) parameters representing the overall motion posture of the human body and the relative angles of the 24 joints. Of course, it is worth noting that for the entire SMPL model, β and θ represent the morphology or shape parameter and the posture parameter, respectively, which can be understood as the present invention. and However, the present invention has a direction in the calculation process, so it is more accurate to express it in vector form. Since it is relatively difficult to directly use the network to predict the axis angle of the node, specifically, in step STP400, the 2.5D joint point J of the human body and the joint torsion angle Φ are sent to the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model The process is implemented using the following steps:

[0107] Step STP410, given the position of the kth joint point in the human joint point J Initial joint position The rotation matrix relationship before and after the joint point posture changes is expressed as:

[0108]

[0109] p k -p pa(k) =R k (t k -t pa(k) )

[0110] Among them, R k is the rotation matrix before and after the posture change of the current k-th joint point, is the current posture vector, is the initial state vector, pa(k) represents the parent node index of the kth joint point;

[0111] Step STP420, obtain the rotation matrix expression of the deformation:

[0112]

[0113] Among them, R pa(k) Represents the rotation matrix of the parent node of the kth node, R pa(k),k represents the relative rotation of the kth joint relative to its parent joint,

[0114] Step STP430, convert the global rotation matrix into a relative rotation matrix and split the relative rotation matrix into swing and twist, then we have

[0115] The axis of the swing rotation is perpendicular to and Then there is According to Rodriguez formula, the swing rotation R sw , R tw , thereby obtaining the relative rotation matrix R;

[0116]

[0117] in, Represents perpendicular to and vector, α is the swing angle, φ is the torsion angle, yes The skew-symmetric matrix is yes The skew-symmetric matrix is It is the 3*3 identity matrix.

[0118] Embodiment 4:

[0119] This embodiment is based on any of the above embodiments to optimize the settings, step STP500 the shape parameters obtained in step STP400 and attitude parameters The process of generating 3D human body imaging by inputting the standard SMPL model is implemented using the following expression:

[0120]

[0121] Among them, M stands for SMPL function, W stands for Skinning function, T p represents the equation for calculating vertex coordinates from shape parameters and pose parameters; J represents the equation for predicting joint points; represents the mixing weight;

[0122] Specifically,

[0123] R 3N , human average network model, SMPL's Template Mesh template network, has two models, Female&Male, and the default Pose is T-Pose;

[0124] R N×K , Blending Weight, that is, the influence weight of K joint points on each vertex on the SMPL Template Mesh;

[0125] R 3N×10,PCA basis of human body shape, responsible for the PCA basis of body shape;

[0126] R 3N×3K , calculate the regression matrix of K joint points from the standard T-pose Template mesh;

[0127] R 3N×9K , the influence of human posture on shape, is the PCA basis of human posture, responsible for the influence of posture on body shape, which is proportional to the coefficient of the rotation matrix.

[0128] Embodiment 5:

[0129] This embodiment is a further refinement based on any of the above embodiments. In step STP500, the shape parameter and attitude parameters Use the MLP network for learning before entering the SKEL model arrive The steps of mapping data to construct and generate three-dimensional imaging of the human body include:

[0130] Step STP510, selecting SMPL data pairs in the SMPL official data set;

[0131] Step STP520, using the control parameters in the data pair and Input into the standard SMPL model and regress to obtain the vertex and node coordinates of SMPL;

[0132] Step STP530, initialize the SKEL model, Direct use Will Initialize to 0 and input into the SKEL model to get the skin model skin_verts and the skeleton model skel_verts;

[0133] Step STP540, classify and calculate loss;

[0134] Step STP541, calculate vertex loss, vertex loss loss verts The calculation method is obtained by the following formula

[0135]

[0136] Among them, fitting_msk is the mark of SMPL vertex and SKEL vertex;

[0137] Step STP542, calculate node loss, node loss loss jointsThe calculation method is obtained by the following formula

[0138]

[0139] in, It represents the node coordinate regression matrix of the skeleton model SKEL;

[0140] Step STP543, calculate scapula loss, scapula loss loss scapula The calculation method is obtained by the following formula

[0141]

[0142] scapula indices ={26,27,28,36,37,38}

[0143] Among them, the number 26 represents the right scapula abduction angle; 27 represents the right scapula elevation angle; 28 represents the right scapula external rotation angle; 36 represents the left scapula abduction angle; 37 represents the left scapula elevation angle; 38 represents the left and right scapula external rotation angles;

[0144] Step STP544, calculate spine loss, spine loss loss spine The calculation method is obtained by the following formula

[0145]

[0146] spine indices ={17,18,19,20,21,22,23,24}

[0147] Among them, 17, 18, and 19 represent lumbar flexion parameters, 20, 21, and 22 represent thoracic extension parameters, and 22, 23, and 24 represent head twist parameters;

[0148] Step STP545, calculate the total loss Loss between the SKEL model and the SMPL model total

[0149] Loss total =loss verts +loss joints +loss scapula +loss spine

[0150] Step STP555, Optimization Make Loss total Minimum, get and Data pair

[0151]

[0152] Step STP565, constructing a three-dimensional image of the human body, using the obtained data pairs, by training an MLP network, Directly get

[0153]

[0154] Then input it into the SKEL model to obtain a 3D model of the human skeleton

[0155]

[0156] Will Input into the SKEL model to obtain three-dimensional imaging of the human body.

[0157] In order to better illustrate the modification of the skin points, joint points and bone vertices of the present invention, the following embodiment will elaborate on its principle. The inverse dynamics IK process has been described in Example 3, that is, using the obtained shape parameters and attitude parameters Input into the SMPL model in 1.5.1 to get the human body surface model. The following will introduce how to restore the human body skeleton model (SKEL) from the skin model (SKIN). Similar to the SMPL model, SKEL is a parameterized human body model that can output the skin model and the skeleton model. The input of the SKEL model is: shape parameters and attitude parameters Output is skin model v skin , Skeleton Model v skel And the joint point J, expressed as:

[0158]

[0159] It should be noted that the above This example is only used to illustrate the SKEL model in this embodiment, in order to avoid confusion with the content of the present invention, especially with the aforementioned Be distinguished.

[0160] Skin Vertices: SKEL builds on SMPL's additive approach to extract vertices from an average template mesh. To begin, add the learned displacement β·B S +B P (q), where B S is the PCA shape basis, and B P (q) is the pose-dependent displacement. The skin vertices are then calculated using the following linear blended skinning equation:

[0161]

[0162] Among them, W i skin It is a 6890×24 weight matrix, which represents the weight of the skin vertex affected by the node. This weight is inherited from the SMPL model by defining a corresponding SMPL joint for each NJ=24 joint of SKEL.

[0163] is the rigid transformation matrix, which represents the rotation and translation matrix of the kth node relative to the original posture, and is expressed as:

[0164]

[0165] T(R k (β),0) -1 T(0,J k (β)) -1 Represents the position of the joint around T(0,J k (β)) -1 , cancel its rotation matrix (R k (β),0) -1 , transform the bone space in the initial state; T(R k (β),J k (β)) represents the rotation and translation matrix at the initial posture, It represents the rotation and translation matrix of the current posture.

[0166] Joint point: The expression of joint point J is as follows:

[0167]

[0168] in, It represents the node coordinate regression matrix.

[0169] Bone vertex: Similar to the skin vertex definition, the expression of the bone vertex is:

[0170]

[0171] Among them, T B is the initial bone template, s(J B (β) represents the scale factor of each bone, which defines the limb length; W i skel It is the bone weight matrix except for the spine and rib cage, which indicates the weight of the bone vertices affected by the nodes, where the weight at the bottom of the spine is interpolated to 0 and the top to 1; is the rigid transformation matrix, that is, the rotation and translation matrix relative to the initial pose, which translates and rotates the bone vertices associated with the i-th node according to the pose parameter q. The expression is:

[0172]

[0173] Among them, T(R k (β),J k (β)) represents the rotation and translation matrix at the initial posture, It represents the rotation and translation matrix of the current posture.

[0174] At this point, the entire 3D imaging algorithm process is completed. Combined with the multi-view sensor, a 3D mesh of the human body is generated in the world coordinate system, including the 3D coordinates of the body surface vertices, internal bones, and internal nodes. The coordinate system transformation relationship between multiple camera sensors is obtained, and coordinate conversion can be performed.

[0175] In the above-mentioned embodiments 1 to 5, a three-dimensional digital human can be obtained.

[0176] Embodiment 6:

[0177] This embodiment provides a non-rigid registration method for a human body mesh based on X-ray fluoroscopic images, which is used to perform further non-rigid registration on the three-dimensional digital human body mesh in the prior art or the three-dimensional digital human body mesh obtained in Embodiments 1 to 5, so as to obtain a more accurate human body mesh.

[0178] A human body mesh non-rigid registration method based on X-ray fluoroscopic images is used to perform non-rigid registration on a human body mesh model including a body surface mesh M and a skeleton mesh S to obtain an accurate human body mesh model, comprising the following steps:

[0179] Step STP100, extracting contours based on actual X-ray image data, using contour detection algorithms to extract the human body surface contours on each X-ray image data At the same time, use U-Net or Mask-R-CNN algorithm to segment the bone contours in the body

[0180] Step STP200, when the X-ray image data is a CT or MRI image, the human body surface contour obtained in step STP100 is converted into and bone contours Uniform downsampling is performed to obtain the sampling points on all images, and spatial stacking is performed according to the actual pixel size to form a surface point cloud. and bone point cloud Among them, the surface point cloud N1 represents the number of points in the body surface point cloud. is the i-th point in the homogeneous coordinates on the body point cloud, the bone point cloud N2 represents the number of points in the skeleton point cloud. is the jth point in homogeneous coordinates on the bone point cloud;

[0181] Step STP300, using a diffusion pruning method based on local geometric similarity and inconsistent counterparts to calculate the vertices of the body surface mesh M in the human body mesh to be registered and the body surface point cloud The mapping between f and the vertices and bone point cloud of the skeleton mesh S in the human body mesh to be registered The mapping between g;

[0182] Step STP400, based on the mapping of f and g, obtain the body surface grid M and transfer it to the target body surface grid M ‘ Non-rigid registration of A M , is the set or matrix of non-rigid registration, Indicates point M i The transformation matrix of the skeleton mesh S is obtained by transferring the non-rigid registration A of the target skeleton mesh S' S , is the set or matrix of non-rigid registration, Indicates point S i The transformation matrix of

[0183] Step STP500, establishing a non-rigid registration energy function E,

[0184] E(A M ,A S ;f,g)

[0185] =α.E points (A M ,A S ;f,g) / +E smooth (A M ,A S ;f,g)

[0186] +E orth (A M ,A S ;f,g)

[0187] Among them, α represents the weight, when the surface point cloud and bone point cloud α=1 when it exists, otherwise α=0; E points 、E smooth and E orth They are point cloud loss, smoothness constraint and orthogonality constraint respectively; A with the minimum value Emin is obtained M , A S That is, the best non-rigid registration is completed.

[0188] Embodiment 7:

[0189] In order to further optimize the registration method and increase the registration dimension, this embodiment is further optimized and expanded on the basis of Embodiment 6. In step STP200, when the X-ray image data is any one or more of CT, MRI, single or continuous DR or CBCT, it also includes a step of contour-based non-rigid registration, specifically including:

[0190] Step STP210, any point M on the body surface grid M i Convert the standard world coordinate system to the tube coordinate system of the X-ray image data shooting state to obtain the corresponding point

[0191]

[0192] Among them, T n is the transformation matrix from the world coordinate system to the tube coordinate system, and n is the number of shots;

[0193] Step STP220, obtaining pixel coordinates PP of all points of the human body mesh model according to the size of the detector imaging area and the placement of the tube and the detector n

[0194]

[0195] Among them, width 、scale height Represent the actual length of a pixel in the horizontal and vertical directions respectively. The pixel coordinates of the imaging point of the ray center on the detector are (ppx, ppy);

[0196]

[0197] Among them, width and height are the width and height of the detector imaging area, respectively, and w and h are the width and height of the detector imaging pixel, respectively;

[0198] Step STP230, extract the set of all points on the outermost edge to obtain the body surface contour

[0199] Step STP240, according to steps STP210-STP230, any point S on the skeleton mesh j Convert to get the bone outline

[0200] Step STP250, the body surface contour obtained in step STP230 Surface profile extracted with step STP100 Perform registration and place the bone contour obtained in step STP240 The bone contour extracted in step STP100 Perform registration and establish a non-rigid registration energy function E.

[0201] E(A M ,A S ;f,g)

[0202] =α.E points (A M ,A S ; f,g) / +β(E sil (A M ,A S ;f,g))+E smooth (A M ,A S ;f,g)+E orth (A M ,A S ;f,g)

[0203] Among them, α and β represent weights. and bone point cloud When it exists, α=1, otherwise α=0; when the body surface contour and bone contours When it exists, β=1, otherwise β=0, where E points 、E sil 、E smooth and E orth They are point cloud loss, contour loss, smoothness constraint and orthogonality constraint.

[0204] Preferably, the point cloud loss E points Obtained by the following algorithm

[0205]

[0206] Among them, w i represents the body surface point error weight, w j Represents the error weight of the bone point. If M i There are corresponding points on the target shape, then the weight w i is 1, otherwise the weight is 0; if S j There are corresponding points on the target shape, then the weight w j is 1, otherwise the weight is 0.

[0207] Embodiment 8:

[0208] This embodiment further optimizes the smoothness constraint based on the above embodiment 6 or embodiment 7. smooth Obtained by the following algorithm

[0209]

[0210] Among them, M i and M j are adjacent surface vertices, S i and S j are adjacent bone vertices.

[0211] Embodiment 9:

[0212] This embodiment further optimizes the orthogonality constraint based on the above embodiment 6 or embodiment 7. orth Obtained by the following algorithm

[0213]

[0214] in, is a 3×3 rotation matrix, It is extracted The rotation component of the constant 3×4 matrix, and make sure and is the rotation matrix, and st represents the completion of the above summation under the condition that...

[0215] Embodiment 10:

[0216] This embodiment further optimizes the contour loss based on the above embodiment 7. sil Obtained by the following algorithm

[0217]

[0218] Wherein, N3 is 0 or a positive integer.

[0219] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A human body mesh non-rigid registration method based on X-ray fluoroscopic images is used to perform non-rigid registration on a human body mesh model including a body surface mesh M and a skeleton mesh S to obtain an accurate human body mesh model, characterized in that: The following steps are involved: Step STP100, extracting contours based on actual X-ray image data, using contour detection algorithms to extract the human body surface contours on each X-ray image data At the same time, use U-Net or Mask-R-CNN algorithm to segment the bone contours in the body Step STP200, when the X-ray image data is a CT or MRI image, the human body surface contour obtained in step STP100 is converted into and bone contours Perform uniform downsampling to obtain sampling points on all images, and then spatially stack them according to the actual pixel size to form a body surface point cloud. and bone point cloud Among them, the surface point cloud N1 represents the number of points in the body surface point cloud. is the i-th point in the homogeneous coordinates on the body point cloud, the bone point cloud N2 represents the number of points in the skeleton point cloud. is the jth point in homogeneous coordinates on the bone point cloud; Step STP300, using a diffusion pruning method based on local geometric similarity and inconsistent counterparts to calculate the vertices of the body surface mesh M in the human body mesh to be registered and the body surface point cloud The mapping between f and the vertices and bone point cloud of the skeleton mesh S in the human body mesh to be registered The mapping between g; Step STP400, based on the mapping f and g, obtain the non-rigid registration A of the body surface mesh M to the target body surface mesh M' M , is the set or matrix of non-rigid registration, Indicates point M i The transformation matrix of the skeleton mesh S is obtained by transferring the non-rigid registration A of the target skeleton mesh S' S , is the set or matrix of non-rigid registration, Indicates point S i The transformation matrix of Step STP500, establishing a non-rigid registration energy function E, E(A M ,A S ;f,g) α.E points (TO M ,TO S ;f,g) / +E smooth (TO M ,TO S (f,g) +E orth (A M ,A S ;f,g) Among them, α represents the weight, when the surface point cloud and bone point cloud α=1 when it exists, otherwise α=0; E points 、E smooth and E orth They are point cloud loss, smoothness constraint and orthogonality constraint respectively; A with the minimum value Emin is obtained M , A S That is, the best non-rigid registration is completed.

2. The human body grid non-rigid registration method based on X-ray fluoroscopic images according to claim 1, characterized in that: When the X-ray image data in step STP200 is any one or more of CT, MRI, single or continuous DR or CBCT, the step of contour-based non-rigid registration is also included, specifically including: Step STP210, any point M on the body surface grid M i Convert the standard world coordinate system to the tube coordinate system of the X-ray image data shooting state to obtain the corresponding point Among them, T n is the transformation matrix from the world coordinate system to the tube coordinate system, and n is the number of shots; Step STP220, obtaining pixel coordinates PP of all points of the human body mesh model according to the size of the detector imaging area and the placement of the tube and the detector n Among them, width 、scale height Represent the actual length of a pixel in the horizontal and vertical directions respectively. The pixel coordinates of the imaging point of the ray center on the detector are (ppx, ppy); Among them, width and height are the width and height of the detector imaging area, respectively, and w and h are the width and height of the detector imaging pixel, respectively; Step STP230, extract the set of all points on the outermost edge to obtain the body surface contour Step STP240, according to steps STP210-STP230, any point S on the skeleton mesh j Convert to get the bone outline Step STP250, the body surface contour obtained in step STP230 Surface profile extracted with step STP100 Perform registration and place the bone contour obtained in step STP240 The bone contour extracted in step STP100 Perform registration and establish a non-rigid registration energy function E. E(A M ,A S ;f,g) =α.Epoints(A M ,A S ;f,g) / +β(Esil(A M ,A S ;f,g)) +Esmooth(A M ,A S ;f,g)+Eorth(A M ,A S ;f,g) Among them, α and β represent weights. and bone point cloud When it exists, α=1, otherwise α=0; when the body surface contour and bone contours When it exists, β=1, otherwise β=0, where E points 、E sil 、E smooth and E orth They are point cloud loss, contour loss, smoothness constraint and orthogonality constraint.

3. The human body grid non-rigid registration method based on X-ray fluoroscopic images according to claim 1 or 2, characterized in that: The point cloud loss E points Obtained by the following algorithm Among them, w i represents the body surface point error weight, w j Represents the error weight of the bone point. If M i There are corresponding points on the target shape, then the weight w i is 1, otherwise the weight is 0; if S j There are corresponding points on the target shape, then the weight w j is 1, otherwise the weight is 0.

4. The human body grid non-rigid registration method based on X-ray fluoroscopic images according to claim 1 or 2, characterized in that: Smoothness constraint E smooth Obtained by the following algorithm Among them, M i and M j are adjacent surface vertices, S i and S j are adjacent bone vertices.

5. The human body grid non-rigid registration method based on X-ray fluoroscopic images according to claim 1 or 2, characterized in that: Orthogonality Constraint E orth Obtained by the following algorithm in, is a 3×3 rotation matrix, It is extracted The rotation component of the constant 3×4 matrix, and make sure and is the rotation matrix, and st represents the completion of the above summation under the condition that...

6. The human body grid non-rigid registration method based on X-ray fluoroscopic images according to claim 2, characterized in that: Contour loss E sil Obtained by the following algorithm Wherein, N3 is 0 or a positive integer.

Images