Multi-modal human body three-dimensional imaging method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-modal human body three-dimensional imaging method, which is used for generating a three-dimensional digital human body based on multi-modal human body image data input under multiple visual angles, and comprises the following steps of image data acquisition and preprocessing, image feature fusion, camera parameter estimation, human body parameter estimation and human body three-dimensional imaging generation. The invention further provides a multi-mode human body three-dimensional imaging device, electronic equipment, a storage medium and a computer program. According to the method, registration of multi-modal input data can be realized, and the three-dimensional digital human body is revised in the aspects of human body form and posture parameters, so that the generated three-dimensional digital human body is more accurate and real.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, in particular to computer image data processing, and specifically to a multimodal human body three-dimensional imaging method. Background Art

[0002] 3D digital human generation technology refers to the process of creating a virtual human image (i.e. 3D digital human or virtual human) using computer technology and algorithms. This technology usually involves the following steps:

[0003] Data collection: Data on human images is obtained through various means, including 2D images, videos, depth images, laser scanning, etc. The collected data is converted into a three-dimensional model using computer graphics and machine learning algorithms. This process may include the generation of facial features, body structure, texture mapping, etc. After obtaining the three-dimensional model, it is given dynamic expression through animation technology, enabling it to show various expressions, movements and interactions. Combining data from different modalities, the generated three-dimensional digital human is optimized to make it more visually realistic and natural.

[0004] The application range of 3D digital human generation technology is wide, and it can play an important role in many fields: For example, virtual characters: in video games, animated films and virtual reality experiences, high-quality virtual characters are generated to enhance user experience. Users can create personalized virtual images according to their preferences. Users can use virtual human images as their own representatives on social platforms to interact with others. At the same time, virtual human images can be used in social applications for facial replacement and special effects enhancement. In the medical and educational fields, 3D digital human models can be used for anatomical teaching and simulated surgical practice, etc.

[0005] In the existing technology, 3D models can be generated from single or multiple 2D images. Common methods include deep learning, convolutional neural networks (CNN), etc. Of course, video sequences are also used to extract motion information and expression changes to generate more dynamic 3D models, which are mainly used in animation production, virtual reality (VR) and augmented reality (AR). Although the existing technology can generate 3D digital human models from 2D images and videos, the digital humans generated based on different data have more or less some shortcomings and one-sidedness. In terms of comprehensiveness, realism, and especially the generation of the internal structure of the digital human including the skeleton, the accuracy needs to be improved. Summary of the invention

[0006] In order to solve the problem of insufficient accuracy in the generation of three-dimensional digital humans in the prior art, the present application provides a multi-modal three-dimensional human imaging method, which can overcome at least one of the following technical problems by using data from different modalities to jointly generate digital humans:

[0007] 1. Overcome the problem that single modal data is one-sided, resulting in insufficient realism and accuracy of the model.

[0008] 2. Overcome the problem that the digital human model built based on bitmap images lacks depth information, resulting in insufficient fit with the actual human object.

[0009] 3. Overcome the problem of three-dimensional digital humans generated by a single model being biased towards standardization and lacking in personalization.

[0010] In order to achieve the above purpose, the technical solution adopted in this application is:

[0011] The multimodal human body three-dimensional imaging method generates a three-dimensional digital human body based on multimodal human body image data input under multiple viewing angles, and includes the following steps:

[0012] Step STP100, image data acquisition and preprocessing, the collected image data includes any one or more of RGB image, depth image, infrared image, and X-ray fluoroscopic image data; the collected image data is encoded and feature extracted, and the color features, depth features, and infrared features extracted from the encoded RGB image, depth image, and infrared image are respectively recorded as F rgb , F depth , F Infrared ;

[0013] Step STP200, fusion of image features, any set of aligned color features F rgb , deep feature F depth and infrared signature F Infrared The deep attention mask M1 and infrared attention mask M2 are obtained by using the sigmoid function. The obtained attention masks M1 and M2 are used as the primary / secondary features and the color feature F respectively. rgb Fusion is performed to obtain the fusion feature F Fused , respectively using the obtained fusion features F Fused Perform camera parameter estimation and human body parameter estimation,

[0014] Step STP300, camera parameter estimation, the fusion feature F obtained in step STP200 Fused Input the MLP network to obtain the camera pose matrix, where the formula for obtaining the camera pose matrix is ​​as follows:

[0015]

[0016] Among them, F Fused-i represents the fusion feature under the i-th perspective, Represents the camera's attitude matrix, rotation matrix R i , translation matrix ti , N is a positive integer, F c Represents the MLP network;

[0017] Step STP400, human body parameter estimation, the fusion feature F obtained in step STP200 Fused Input into Transformer decoder to generate shape parameters The human body's 2.5D joint point J and the joint torsion angle Φ are finally sent to the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model.

[0018] Step STP500, generate a three-dimensional image of the human body, and convert the morphological parameters and attitude parameters Send it to the standard SMPL model or SKEL model to obtain the fused feature F Fused Optimized 3D imaging of the human body.

[0019] Preferably, the method of encoding and feature extraction of image data in step STP100 adopts a convolutional neural network with CNN, ResNet and ViT as basic structures, and adopts independent encoders to encode and extract features of RGB images, depth images and infrared images respectively. The expression of independent encoding is:

[0020] F rgb =Enc rgb (I rgb )

[0021] F depth =Enc depth (I depth )

[0022] F infrared =Enc infrared (I infrared )

[0023] Among them, Enc rgb For RGB encoder, Enc depth For the deep encoder, Enc infrared For infrared encoder.

[0024] Preferably, in step STP200, the attention masks M1 and M2 are used as the primary / secondary features and the color feature F respectively. rgb The fusion steps are performed according to the number of attention masks obtained:

[0025] When only the deep feature F depth When the deep attention mask M1 is obtained, the fusion steps are as follows:

[0026] Step STP211, obtain the attention mask M1, and obtain the deep attention mask M1 through the sigmoid function, that is, M1 = Sigmod(CNN(F Depth ));

[0027] Step STP212, with deep feature F depth is the main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression:

[0028]

[0029] Step STP213, using the color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression:

[0030]

[0031] Step STP214, the fusion feature F obtained by step STP212 and step STP213 P1 and F s1 Get the fusion feature F Fused The expression is as follows:

[0032]

[0033] in, The symbol represents the multiplication operation;

[0034] When only infrared feature F Infrared When the infrared attention mask M2 is obtained, the fusion steps are as follows:

[0035] Step STP221, obtain the attention mask M2, and obtain the deep attention mask M1 through the sigmoid function, that is, M2 = Sigmod(CNN(F Infrared ));

[0036] Step STP222, using infrared feature F Infrared is the main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression:

[0037]

[0038] Step STP223, using the color feature F rgb The main features and infrared features FInfrared The fusion feature F obtained during fusion s2 Obtained by the following expression:

[0039]

[0040] Step STP224, the fusion feature F obtained by step STP222 and step STP223 P2 and F s2 Get the fusion feature F Fused The expression is as follows:

[0041]

[0042] in, The symbol represents the multiplication operation;

[0043] When there is a deep feature F depth and infrared signature F Infrared When the obtained deep attention mask M1 and infrared attention mask M2 are obtained, the fusion steps are as follows:

[0044] Step STP231, obtain the depth attention mask M1 and the infrared attention mask M2

[0045] M1=Sigmod(CNN(F Depth ))

[0046] M2=Sigmod(CNN(F Infrared ))

[0047] Step STP232, with deep feature F depth is the main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression:

[0048]

[0049] Step STP233, using the color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression:

[0050]

[0051] Step STP234, using infrared feature F Infrared is the main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression:

[0052]

[0053] Step STP235, using the color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression:

[0054]

[0055] Step STP236, the fusion feature F obtained by step STP232 and step STP235 P1 , F s1 , F P2 and F s2 Get the fusion feature F Fused The expression is as follows:

[0056]

[0057] in, The symbol represents the multiplication operation.

[0058] Preferably, in step STP400, the 2.5D joint points J of the human body and the joint torsion angle Φ are input into the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model. The process is implemented using the following steps:

[0059] Step STP410, given the position of the kth joint point in the human joint point J Initial joint position The rotation matrix relationship before and after the joint point posture changes is expressed as:

[0060]

[0061] p k -p pa(k) =R k (t k -t pa(k) )

[0062] Among them, R k is the rotation matrix before and after the posture change of the current k-th joint point, is the current posture vector, is the initial state vector, pa(k) represents the parent node index of the kth joint point;

[0063] Step STP420, obtain the rotation matrix expression of the deformation:

[0064]

[0065] Among them, R pa(k) Represents the rotation matrix of the parent node of the kth node, R pa(k)，k represents the relative rotation of the kth joint relative to its parent joint,

[0066] Step STP430, convert the global rotation matrix into a relative rotation matrix and split the relative rotation matrix into swing and twist, then we have

[0067] The axis of the swing rotation is perpendicular to and Then there is According to Rodriguez formula, the swing rotation R sw , R tw , thereby obtaining the relative rotation matrix R;

[0068]

[0069] in, Represents perpendicular to and vector, α is the swing angle, φ is the torsion angle, yes The skew-symmetric matrix is yes The skew-symmetric matrix is It is the 3*3 identity matrix.

[0070] Preferably, in step STP500, the shape parameters obtained in step STP400 are and attitude parameters The process of inputting the standard SMPL model to generate 3D human body imaging is implemented using the following expression:

[0071]

[0072] Among them, M stands for SMPL function, W stands for Skinning function, T p represents the equation for calculating vertex coordinates from shape parameters and pose parameters; J represents the equation for predicting joint points; represents the mixing weight;

[0073] Specifically,

[0074] R 3N, human average network model, SMPL's Template Mesh template network, has two models, Female&Male, and the default Pose is T-Pose;

[0075] R N×K , Blending Weight, that is, the influence weight of K joint points on each vertex on the SMPL Template Mesh;

[0076] R 3N×10 ,PCA basis of human body shape, responsible for the PCA basis of body shape;

[0077] R 3N×3K , calculate the regression matrix of K joint points from the standard T-pose Template mesh;

[0078] R 3N×9K , the influence of human posture on shape, is the PCA basis of human posture, responsible for the influence of posture on body shape, which is proportional to the coefficient of the rotation matrix.

[0079] Preferably, in step STP500, the shape parameter and attitude parameters Use the MLP network for learning before entering the SKEL model arrive The steps of mapping data to construct and generate three-dimensional imaging of the human body include:

[0080] Step STP510, selecting SMPL data pairs in the SMPL official data set;

[0081] Step STP520, using the control parameters in the data pair and Input into the standard SMPL model and regress to obtain the vertex and node coordinates of SMPL;

[0082] Step STP530, initialize the SKEL model, Direct use Will Initialize to 0 and input into the SKEL model to get the skin model skin_verts and the skeleton model skel_verts;

[0083] Step STP540, classify and calculate loss;

[0084] Step STP541, calculate vertex loss, vertex loss loss vertsThe calculation method is obtained by the following formula

[0085]

[0086] Among them, fitting_msk is the mark of SMPL vertex and SKEL vertex;

[0087] Step STP542, calculate node loss, node loss loss joints The calculation method is obtained by the following formula

[0088]

[0089] in, It represents the node coordinate regression matrix of the skeleton model SKEL;

[0090] Step STP543, calculate scapula loss, scapula loss loss scapula The calculation method is obtained by the following formula

[0091]

[0092] scapula indices =*26,27,28,36,37,38+

[0093] Among them, the number 26 represents the right scapula abduction angle; 27 represents the right scapula elevation angle; 28 represents the right scapula external rotation angle; 36 represents the left scapula abduction angle; 37 represents the left scapula elevation angle; 38 represents the left and right scapula external rotation angles;

[0094] Step STP544, calculate spine loss, spine loss loss spine The calculation method is obtained by the following formula

[0095]

[0096] spine indices =*17, 18, 19, 20, 21, 22, 23, 24+

[0097] Among them, 17, 18, and 19 represent lumbar flexion parameters, 20, 21, and 22 represent thoracic extension parameters, and 22, 23, and 24 represent head twist parameters;

[0098] Step STP545, calculate the total loss Loss between the SKEL model and the SMPL model total

[0099] Loss total =loss verts +loss joints +lossscapula +loss spine

[0100] Step STP555, Optimization Make Loss total Minimum, get and Data pair

[0101]

[0102] Step STP565, constructing a three-dimensional image of the human body, using the obtained data pairs, by training an MLP network, Directly get

[0103]

[0104] Then input it into the SKEL model to obtain a 3D model of the human skeleton

[0105]

[0106] Will and Input into the SKEL model to obtain a three-dimensional image of the human body including bones and skin.

[0107] The present invention also provides a multi-modal human body three-dimensional imaging device, comprising

[0108] An image acquisition module is used to acquire and store image data of different modalities including RGB images, depth images, infrared images, and X-ray fluoroscopic image data;

[0109] An image encoding module is used to encode the image data transmitted by the image acquisition module and perform feature extraction on the image data to obtain corresponding feature information of image data of different modalities;

[0110] An image feature fusion module is used to fuse feature information of image data of different modalities and make the fused feature information recognizable by a camera parameter estimation module and a human body parameter estimation module;

[0111] The camera parameter estimation module has a built-in shared MLP platform. After identifying the fused feature information, the camera posture estimation information corresponding to the feature information is obtained;

[0112] The human body parameter estimation module has a built-in Transformer decoder, which generates shape parameters by decoding the fused feature information The 2.5D joint point J of the human body, and the joint torsion angle φ;

[0113] Digital human generation model, including built-in SMPL model or SKEL model, by inputting different shape parameters Attitude parameters Generate a corresponding three-dimensional digital human model.

[0114] The present invention also provides a multimodal human body three-dimensional imaging electronic device, comprising at least one processor and a memory electrically or communicatively connected to the processor, wherein the memory contains at least one storage unit for storing a computer program of the multimodal human body three-dimensional imaging method, and the storage unit implements the human body three-dimensional imaging method by executing the computer program through the processor.

[0115] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to execute the multi-modal three-dimensional human body imaging method through a computer.

[0116] The present invention also provides a computer program, including a computer program, which implements a multi-modal three-dimensional human body imaging method when executed by a processor.

[0117] Beneficial effects:

[0118] 1. The present invention can realize the registration of multi-modal input data, and jointly revise the three-dimensional digital human in terms of human body shape and posture parameters, so that the generated three-dimensional digital human is more accurate and realistic.

[0119] 2. The present invention also adopts multi-model comprehensive registration optimization, including the optimization of human body surface, skeleton and organ models, so that the generated three-dimensional digital human has more specificity in static and dynamic performance than the digital human generated by a single model, reduces the standardization ratio and generalization degree, and makes it closer or infinitely close to the research object. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative labor.

[0121] Figure 1 It is a flow chart of generating a three-dimensional digital human body of multimodal human body images provided by the present invention.

[0122] Figure 2 It is multimodal feature fusion. DETAILED DESCRIPTION

[0123] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0124] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0125] Embodiment 1:

[0126] Before describing the embodiments of the present invention in detail, the existing digital human generation technology is first briefly described. The existing three-dimensional digital human generation technology is based on a deep learning network architecture and is generated using the SMPL model; wherein the SMPL model is a ready-made, standardized human model, which has extremely high generalization and standardization, but has almost no specificity or personalization, that is, the standard SMPL model is standardized and defined based on countless samples, and it is almost impossible to correspond to any specific person with clear parameters. For ease of understanding, the SMPL model can be understood as a standardized, idealized virtual human, which has every part of a person, but does not have the characteristics of an actual human body, including body shape, posture, etc. Therefore, the standardized digital human can be used for virtual human-computer interaction, or for improving the experience of virtual scenes, such as game scenes, but it is almost not applicable to the field of medical reconstruction. Because the height, weight, and body shape of each patient in the medical field are inconsistent, in order to enable the generation of a three-dimensional imaging model that is consistent with the actual human body, such as a certain patient, for subsequent diagnosis or treatment research, it is necessary to revise the parameters of the model to obtain a three-dimensional digital model that is closer to the real human body.

[0127] In the prior art, the most commonly used method is bitmap parameter revision, that is, a camera is used to obtain an object bitmap, and then the characteristic pixels in the bitmap are matched with the model to obtain a model that is relatively matched with the object. However, since the image information of a single modality has limitations, the matching accuracy of the existing three-dimensional digital human is difficult to meet medical purposes. In addition, for the three-dimensional reconstruction of a certain part, X-ray images, such as CT tomographic images, are also used for reconstruction. Since CT tomographic images contain the characteristics of the body surface edge and the bones and tissues in the body, the three-dimensional reconstruction through tomographic images has a higher accuracy, especially when the scanning layer thickness is set to a smaller value, the accuracy will be higher; but for local reconstruction, only the local organs, tissues and bones can be seen, and the state of each organ, organ, tissue and bone of the human body in different positions and postures cannot be dynamically understood. Therefore, in order to better study the changes of different parts of the human body in different postures, a more accurate three-dimensional digital human is needed, and the present invention is completed for this purpose.

[0128] See the instruction manual Figure 1 As shown, this embodiment provides a multi-modal human body three-dimensional imaging method, which generates a three-dimensional digital human body based on multi-modal human body image data input under multiple viewing angles, including the following steps:

[0129] Step STP100, image data acquisition and preprocessing, the collected image data includes any one or more of RGB image, depth image, infrared image, and X-ray fluoroscopic image data; the collected image data is encoded and feature extracted, and the color features, depth features, and infrared features extracted from the encoded RGB image, depth image, and infrared image are respectively recorded as F rgb , F depth , F Infrared ; Figure 1 The FF in represents a collection of multiple features.

[0130] Step STP200, fusion of image features, any set of aligned color features F rgb , deep feature F depth and infrared signature F Infrared The deep attention mask M1 and infrared attention mask M2 are obtained by using the sigmoid function. The obtained attention masks M1 and M2 are used as the primary / secondary features and the color feature F respectively. rgb Fusion is performed to obtain the fusion feature F Fused , respectively using the obtained fusion features F Fused Perform camera parameter estimation and human body parameter estimation,

[0131] Step STP300, camera parameter estimation, the fusion feature F obtained in step STP200 FusedInput the MLP network to obtain the camera pose matrix, where the formula for obtaining the camera pose matrix is ​​as follows:

[0132]

[0133] Among them, F Fused-i represents the fusion feature under the i-th perspective, Represents the camera's attitude matrix, rotation matrix R i , translation matrix t i , N is a positive integer, F c Represents the MLP network;

[0134] Step STP400, human body parameter estimation, the fusion feature F obtained in step STP200 Fused Input into Transformer decoder to generate shape parameters The human body's 2.5D joint point J and the joint torsion angle Φ are finally sent to the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model.

[0135] Step STP500, generate a three-dimensional image of the human body, and send the morphological parameters β and posture parameters θ to the standard SMPL model or SKEL model to obtain the fused feature F Fused Optimized 3D imaging of the human body.

[0136] Embodiment 2:

[0137] This embodiment is a new method for fusing features of different modes based on Embodiment 1. Figure 2 As shown, the method of encoding and feature extraction of image data in step STP100 adopts a convolutional neural network with CNN, ResNet and ViT as the basic structure, and uses independent encoders to encode and extract features of RGB images, depth images and infrared images respectively. The expression of independent encoding is:

[0138] F rgb =Enc rgb (I rgb )

[0139] F depth =Enc depth (I depth )

[0140] F infrared =Enc infrared (I infrared )

[0141] Among them, Enc rgbFor RGB encoder, Enc depth For the deep encoder, Enc infrared For infrared encoder.

[0142] In this embodiment, see Figure 2 As shown, in step STP200, attention masks M1 and M2 are used as primary / secondary features and color feature F respectively. rgb The fusion steps are performed according to the number of attention masks obtained:

[0143] When only the deep feature F depth When the deep attention mask M1 is obtained, the fusion steps are as follows:

[0144] Step STP211, obtain the attention mask M1, and obtain the deep attention mask M1 through the sigmoid function, that is, M1 = Sigmod(CNN(F Depth ));

[0145] Step STP212, with deep feature F depth is the main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression:

[0146]

[0147] Step STP213, using the color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression:

[0148]

[0149] Step STP214, the fusion feature F obtained by step STP212 and step STP213 P1 and F s1 Get the fusion feature F Fused The expression is as follows:

[0150]

[0151] in, The symbol represents the multiplication operation;

[0152] When only infrared feature F Infrared When the infrared attention mask M2 is obtained, the fusion steps are as follows:

[0153] Step STP221, obtain the attention mask M2, and obtain the deep attention mask M1 through the sigmoid function, that is, M2 = Sigmod(CNN(F Infrared ));

[0154] Step STP222, using infrared feature F Infrared is the main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression:

[0155]

[0156] Step STP223, using the color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression:

[0157]

[0158] Step STP224, the fusion feature F obtained by step STP222 and step STP223 P2 and F s2 Get the fusion feature F Fused The expression is as follows:

[0159]

[0160] in, The symbol represents the multiplication operation;

[0161] When there is a deep feature F depth and infrared signature F Infrared When the obtained deep attention mask M1 and infrared attention mask M2 are obtained, the fusion steps are as follows:

[0162] Step STP231, obtain the depth attention mask M1 and the infrared attention mask M2

[0163] M1=Sigmod(CNN(F Depth ))

[0164] M2=Sigmod(CNN(F Infrared ))

[0165] Step STP232, with deep feature F depth is the main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression:

[0166]

[0167] Step STP233, using the color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression:

[0168]

[0169] Step STP234, using infrared feature F Infrared is the main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression:

[0170]

[0171] Step STP235, using the color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression:

[0172]

[0173] Step STP236, the fusion feature F obtained by step STP232 and step STP235 P1 , F s1 , F P2 and F s2 Get the fusion feature F Fused The expression is as follows:

[0174]

[0175] in, The symbol represents a multiplication operation. It is worth noting that the more features are input, the more comprehensive the fused features will be, and the subsequent final three-dimensional digital human imaging will be closer to the real object or the more similar / close to the real object. The more feature information modes, the more comprehensive the entire model is, and the more feature information of a single mode, the closer the parameters affected by the mode will be to the real object.

[0176] Embodiment 3:

[0177] This embodiment is a further optimization based on any of the above embodiments. In order to use the SMPL model to output the human body Mesh, it is only necessary to provide the shape parameters. and attitude parameters Here, 3k represents a multiple of 3, k represents a node, and 3 represents that each node has three angles. It is a general mathematical expression in the SMPL model and belongs to the prior art expression. In the model, β and θ are input parameters, where β represents 10 parameters of the proportions of human body height, weight, head-to-body ratio, etc.; the β parameter is the Shape Blend Pose parameter, which can control the change of human body shape through 10 incremental templates: Specifically: each parameter controls the change of human body shape, which can be portrayed by animated images.

[0178] θ is 75 (24*3+3; 3 degrees of freedom for each joint point, plus 3 root nodes) parameters representing the overall motion posture of the human body and the relative angles of the 24 joints. Of course, it is worth noting that for the entire SMPL model, β and θ represent the morphology or shape parameter and the posture parameter, respectively, which can be understood as the present invention. and However, the present invention has a direction in the calculation process, so it is more accurate to express it in vector form. Since it is relatively difficult to directly use the network to predict the axis angle of the node, specifically, in step STP400, the 2.5D joint point J of the human body and the joint torsion angle Φ are sent to the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model The process is implemented using the following steps:

[0179] Step STP410, given the position of the kth joint point in the human joint point J Initial joint position The rotation matrix relationship before and after the joint point posture changes is expressed as:

[0180]

[0181] p k -p pa(k) =R k (t k -t pa(k) )

[0182] Among them, R k is the rotation matrix before and after the posture change of the current k-th joint point, is the current posture vector, is the initial state vector, pa(k) represents the parent node index of the kth joint point;

[0183] Step STP420, obtain the rotation matrix expression of the deformation:

[0184]

[0185] Among them, R pa(k)Represents the rotation matrix of the parent node of the kth node, R pa(k),k represents the relative rotation of the kth joint relative to its parent joint,

[0186] Step STP430, convert the global rotation matrix into a relative rotation matrix and split the relative rotation matrix into swing and twist, then we have

[0187] The axis of the swing rotation is perpendicular to and Then there is According to Rodriguez formula, the swing rotation R sw , R tw , thereby obtaining the relative rotation matrix R;

[0188]

[0189] in, Represents perpendicular to and vector, α is the swing angle, φ is the torsion angle, yes The skew-symmetric matrix is yes The skew-symmetric matrix is It is the 3*3 identity matrix.

[0190] Embodiment 4:

[0191] This embodiment is based on any of the above embodiments to optimize the settings, step STP500 the shape parameters obtained in step STP400 and attitude parameters The process of inputting the standard SMPL model to generate 3D human body imaging is implemented using the following expression:

[0192]

[0193] Among them, M stands for SMPL function, W stands for Skinning function, T p represents the equation for calculating vertex coordinates from shape parameters and pose parameters; J represents the equation for predicting joint points; represents the mixing weight;

[0194] Specifically,

[0195] R 3N, human average network model, SMPL's Template Mesh template network, has two models, Female&Male, and the default Pose is T-Pose;

[0196] R N×K , Blending Weight, that is, the influence weight of K joint points on each vertex on the SMPL Template Mesh;

[0197] R 3N×10 ,PCA basis of human body shape, responsible for the PCA basis of body shape;

[0198] R 3N×3K , calculate the regression matrix of K joint points from the standard T-pose Template mesh;

[0199] R 3N×9K , the influence of human posture on shape, is the PCA basis of human posture, responsible for the influence of posture on body shape, which is proportional to the coefficient of the rotation matrix.

[0200] Embodiment 5:

[0201] This embodiment is a further refinement based on any of the above embodiments. In step STP500, the shape parameter and attitude parameters Use the MLP network for learning before entering the SKEL model arrive The steps of mapping data to construct and generate three-dimensional imaging of the human body include:

[0202] Step STP510, selecting SMPL data pairs in the SMPL official data set;

[0203] Step STP520, using the control parameters in the data pair and Input into the standard SMPL model and regress to obtain the vertex and node coordinates of SMPL;

[0204] Step STP530, initialize the SKEL model, Direct use Will Initialize to 0 and input into the SKEL model to get the skin model skin_verts and the skeleton model skel_verts;

[0205] Step STP540, classify and calculate loss;

[0206] Step STP541, calculate vertex loss, vertex loss loss verts The calculation method is obtained by the following formula

[0207]

[0208] Among them, fitting_msk is the mark of SMPL vertex and SKEL vertex;

[0209] Step STP542, calculate node loss, node loss loss joints The calculation method is obtained by the following formula

[0210]

[0211] in, It represents the node coordinate regression matrix of the skeleton model SKEL;

[0212] Step STP543, calculate scapula loss, scapula loss loss scapula The calculation method is obtained by the following formula

[0213]

[0214] scapula indices =*26,27,28,36,37,38+

[0215] Among them, the number 26 represents the right scapula abduction angle; 27 represents the right scapula elevation angle; 28 represents the right scapula external rotation angle; 36 represents the left scapula abduction angle; 37 represents the left scapula elevation angle; 38 represents the left and right scapula external rotation angles;

[0216] Step STP544, calculate spine loss, spine loss loss spine The calculation method is obtained by the following formula

[0217]

[0218] spine indices =*17,18,19,20,21,22,23,24+

[0219] Among them, 17, 18, and 19 represent lumbar flexion parameters, 20, 21, and 22 represent thoracic extension parameters, and 22, 23, and 24 represent head twist parameters;

[0220] Step STP545, calculate the total loss Loss between the SKEL model and the SMPL model total

[0221] Loss total=loss verts +loss joints +loss scapula +loss spine

[0222] Step STP555, Optimization Make Loss total Minimum, get and Data pair

[0223]

[0224] Step STP565, constructing a three-dimensional image of the human body, using the obtained data pairs, by training an MLP network, Directly get

[0225]

[0226] Then input it into the SKEL model to obtain a 3D model of the human skeleton

[0227]

[0228] Will and Input into the SKEL model to obtain a three-dimensional image of the human body including bones and skin.

[0229] In order to better illustrate the modification of the skin points, joint points and bone vertices of the present invention, the following embodiment will elaborate on its principle. The inverse dynamics IK process has been described in Example 3, that is, using the obtained shape parameters and attitude parameters Input into the SMPL model in 1.5.1 to get the human body surface model. The following will introduce how to restore the human body skeleton model (SKEL) from the skin model (SKIN). Similar to the SMPL model, SKEL is a parameterized human body model that can output the skin model and the skeleton model. The input of the SKEL model is: shape parameters and attitude parameters Output is skin model v skin , Skeleton Model v skel And the joint point J, expressed as:

[0230]

[0231] It should be noted that the above This example is only used to illustrate the SKEL model in this embodiment, in order to avoid confusion with the content of the present invention, especially with the aforementioned Skin vertices: SKEL builds on the additive method of SMPL and extracts the skin vertices from the average template mesh. To begin, add the learned displacement β·B S +B P (q), where B S is the PCA shape basis, and B P (q) is the pose-dependent displacement. The skin vertices are then calculated using the following linear blended skinning equation:

[0232]

[0233] in, It is a 6890×24 weight matrix, which represents the weight of the skin vertex affected by the node. This weight is inherited from the SMPL model by defining a corresponding SMPL joint for each NJ=24 joint of SKEL.

[0234] is the rigid transformation matrix, which represents the rotation and translation matrix of the kth node relative to the original posture, and is expressed as:

[0235]

[0236] T(R k (β), 0) -1 T(0,J k (β)) -1 Represents the position of the joint around T(0, J k (β)) -1 , cancel its rotation matrix (R k (β), 0) -1 , transform the bone space in the initial state; T(R k (β), J k (β)) represents the rotation and translation matrix at the initial posture, It represents the rotation and translation matrix of the current posture.

[0237] Joint point: The expression of joint point J is as follows:

[0238]

[0239] in, It represents the node coordinate regression matrix.

[0240] Bone vertex: Similar to the skin vertex definition, the expression of the bone vertex is:

[0241]

[0242] Among them, T Bis the initial bone template, s(J B (β) represents the scaling factor of each bone, which defines the limb length; It is the bone weight matrix except for the spine and rib cage, which indicates the weight of the bone vertices affected by the nodes, where the weight at the bottom of the spine is interpolated to 0 and the top to 1; is the rigid transformation matrix, that is, the rotation and translation matrix relative to the initial pose, which translates and rotates the bone vertices associated with the i-th node according to the pose parameter q. The expression is:

[0243]

[0244] Among them, T(R k (β),J k (β)) represents the rotation and translation matrix at the initial posture, It represents the rotation and translation matrix of the current posture.

[0245] At this point, the entire 3D imaging algorithm process is completed. Combined with the multi-view sensor, a 3D mesh of the human body is generated in the world coordinate system, including the 3D coordinates of the body surface vertices, internal bones, and internal nodes. The coordinate system transformation relationship between multiple camera sensors is obtained, and coordinate conversion can be performed.

[0246] Embodiment 6:

[0247] The present invention also provides a multimodal human body three-dimensional imaging device, including an image acquisition module for acquiring and storing image data of different modalities including RGB images, depth images, infrared images, and X-ray perspective image data;

[0248] An image encoding module is used to encode the image data transmitted by the image acquisition module and perform feature extraction on the image data to obtain corresponding feature information of image data of different modalities;

[0249] An image feature fusion module is used to fuse feature information of image data of different modalities and make the fused feature information recognizable by a camera parameter estimation module and a human body parameter estimation module;

[0250] The camera parameter estimation module has a built-in shared MLP platform. After identifying the fused feature information, the camera posture estimation information corresponding to the feature information is obtained;

[0251] The human body parameter estimation module has a built-in Transformer decoder, which generates shape parameters β, human body 2.5D joint points J, and joint torsion angles φ by decoding the fused feature information;

[0252] The digital human generation model, including the built-in SMPL model or SKEL model, generates the corresponding three-dimensional digital human model by inputting different shape parameters β and posture parameters θ.

[0253] Embodiment 7:

[0254] The present invention also provides a multimodal human body three-dimensional imaging electronic device, comprising at least one processor and a memory electrically or communicatively connected to the processor, wherein the memory contains at least one storage unit for storing a computer program of the multimodal human body three-dimensional imaging method described in the above-mentioned embodiments 1 to 5, and the storage unit implements the human body three-dimensional imaging method by executing the computer program through the processor.

[0255] Embodiment 8:

[0256] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to execute the multimodal human body three-dimensional imaging method described in Examples 1 to 5 through a computer.

[0257] Embodiment 9:

[0258] The present invention also provides a computer program, including a computer program, which, when executed by a processor, implements the multimodal human body three-dimensional imaging method described in Examples 1 to 5.

[0259] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal human body three-dimensional imaging method, which generates a three-dimensional digital human body based on multimodal human body image data input from multiple perspectives, characterized in that: The steps include: Step STP100, image data acquisition and preprocessing, the collected image data includes any one or more of RGB image, depth image, infrared image, X-ray fluoroscopic image data; the collected image data is encoded and feature extracted, and the color features, depth features and infrared features extracted from the encoded RGB image, depth image and infrared image are respectively recorded as F rgb , F depth , F Infrared ; Step STP200, fusion of image features, any set of aligned color features F rgb , deep feature F depth and infrared signature F Infrared Input the independent convolutional neural network for feature processing, and then use the sigmoid function to obtain the depth attention mask M1 and infrared attention mask M2. The obtained attention masks M1 and M2 are used as the main / secondary features and color feature F respectively. rgb Perform fusion to obtain fusion feature F Fused , respectively using the obtained fusion features F Fused Perform camera parameter estimation and human body parameter estimation, Step STP300, camera parameter estimation, the fusion feature F obtained in step STP200 Fused Input the MLP network to obtain the camera pose matrix, where the formula for obtaining the camera pose matrix is ​​as follows: Among them, F Fused-i represents the fusion features under the i-th perspective, Represents the camera's attitude matrix, rotation matrix R i , translation matrix t i , N is a positive integer, F c Represents the MLP network; Step STP400, human body parameter estimation, the fusion feature F obtained in step STP200 Fused Input into the Transformer decoder to generate shape parameters The human body's 2.5D joint point J and joint torsion angle Φ are finally fed into the inverse kinematics module IK to obtain the posture parameters of the human body parametric model. Step STP500, generate a three-dimensional image of the human body, and convert the morphological parameters and posture parameters Send it to the standard SMPL model or SKEL model to obtain the fused feature F Fused Optimized 3D imaging of the human body.

2. The multimodal human body three-dimensional imaging method according to claim 1, characterized in that: In step STP100, the image data is encoded and feature extracted using a convolutional neural network with CNN, ResNet, and ViT as the basic structure. Independent encoders are used to encode and extract features for RGB images, depth images, and infrared images respectively. The expression of independent encoding is as follows: F rgb =Enc rgb (I rgb ) F depth =Enc depth (I depth ) F infrared =Enc infrared (I infrared ) Among them, Enc rgb For RGB encoder, Enc depth For the deep encoder, Enc infrared For infrared encoder.

3. The multimodal human body three-dimensional imaging method according to claim 1, wherein: In step STP200, attention masks M1 and M2 are used as primary / secondary features and color feature F respectively. rgb The fusion steps are as follows: When only the deep feature F depth When the deep attention mask M1 is obtained, the fusion steps are as follows: Step STP211, obtain the attention mask M1, and obtain the deep attention mask M1 through the sigmoid function, that is, M1 = Sigmod(CNN(F Depth )); Step STP212, using the depth feature F depth The main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression: Step STP213, using color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression: Step STP214, the fusion feature F obtained by step STP212 and step STP213 P1 and F s1 Get the fusion feature F Fused The expression is as follows: in, The symbol represents the multiplication operation; When only infrared feature F Infrared When the infrared attention mask M2 is obtained, the fusion steps are as follows: Step STP221, obtain the attention mask M2, and obtain the deep attention mask M1 through the sigmoid function, that is, M2 = Sigmod(CNN(F Infrared )); Step STP222, using infrared feature F Infrared The main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression: Step STP223, using color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression: Step STP224, the fusion feature F obtained by step STP222 and step STP223 P2 and F s2 Get the fusion feature F Fused The expression is as follows: in, The symbol represents the multiplication operation; When there is a deep feature F depth and infrared signature F Infrared When the obtained depth attention mask M1 and infrared attention mask M2 are obtained, the fusion steps are as follows: Step STP231, obtain the depth attention mask M1 and infrared attention mask M2 M1=Sigmod(CNN(F Depth )) M2=Sigmod(CNN(F Infrared )) Step STP232, using the depth feature F depth The main feature and color feature F rgb The fusion feature F obtained during fusion P1 Obtained by the following expression: Step STP233, using color feature F rgb is the main feature and deep feature F depth The fusion feature F obtained during fusion s1 Obtained by the following expression: Step STP234, using infrared feature F Infrared The main feature and color feature F rgb The fusion feature F obtained during fusion P2 Obtained by the following expression: Step STP235, using color feature F rgb The main features and infrared features F Infrared The fusion feature F obtained during fusion s2 Obtained by the following expression: Step STP236, the fusion feature F obtained by step STP232 and step STP235 P1 , F s1 , F P2 and F s2 Get the fusion feature F Fused The expression is as follows: in, The symbol represents the multiplication operation.

4. The multimodal human body three-dimensional imaging method according to claim 1, wherein: In step STP400, the 2.5D joint point J of the human body and the joint torsion angle Φ are input into the inverse kinematics module IK to obtain the posture parameters of the human body parameterized model. The process is implemented using the following steps: Step STP410, given the position of the kth joint point in the human body joint point J Initial joint position The rotation matrix relationship before and after the joint posture changes is expressed as: p k -p pa(k) =R k (t k -t pa(k) ) Among them, R k is the rotation matrix before and after the current k-th joint posture change, is the current posture vector, is the initial state vector, pa(k) represents the parent node index of the k-th joint point; Step STP420, obtain the deformation rotation matrix expression: Among them, R pa(k) Represents the rotation matrix of the parent node of the kth node, R pa(k),k represents the relative rotation of the kth joint relative to its parent joint, Step STP430, convert the global rotation matrix into a relative rotation matrix and split the relative rotation matrix into swing and twist, then we have The axis of the swing rotation is perpendicular to and Then there is According to the Rodriguez formula, the swing rotation R sw , R tw , thereby obtaining the relative rotation matrix R; in, Represents perpendicular to and The vector of α is the swing angle, φ is the torsion angle, yes The skew-symmetric matrix, yes The skew-symmetric matrix, It is a 3*3 identity matrix.

5. The multimodal human body three-dimensional imaging method according to claim 1, characterized in that: In step STP500, the shape parameters obtained in step STP400 are and posture parameters The process of generating a 3D human body image by inputting the standard SMPL model is implemented using the following expression: Among them, M stands for SMPL function, W stands for Skinning function, T p represents the equation for calculating vertex coordinates from shape parameters and pose parameters; J represents the equation for predicting joint points; represents the mixing weight; Specific location, R 3N , human average network model, SMPL's Template Mesh template network, has two models: Female & Male, and the default Pose is T-Pose; R N×K , Blending Weight, that is, the influence weight of K joint points on each vertex on the SMPL Template Mesh; R 3N×10 ,PCA basis of human body shape, responsible for the PCA basis of body shape; R 3N×3K , calculate the regression matrix of K joint points from the standard T-pose Template mesh; R 3N×9K ,The influence of human posture on shape is the PCA basis of human posture, which is responsible for the influence of posture on body shape, and is proportional to the coefficient of the rotation matrix.

6. The multimodal human body three-dimensional imaging method according to claim 1, characterized in that: In step STP500, the shape parameters and posture parameters Use MLP network to learn before entering SKEL model arrive The steps of constructing data pairs and generating three-dimensional imaging of the human body by mapping include: Step STP510, selecting SMPL data pairs in the SMPL official dataset; Step STP520, using the control parameters of the data and Input into the standard SMPL model and regress to obtain the vertex and node coordinates of SMPL; Step STP530, initialize the SKEL model, Direct use Will Initialize it to 0 and input it into the SKEL model to get the skin model skin_verts and the skeleton model skel_verts; Step STP540, classify and calculate loss; Step STP541, calculate vertex loss, vertex loss loss verts The calculation method is obtained using the following formula Among them, fitting_msk is the mark of SMPL vertex and SKEL vertex; Step STP542, calculate node loss, node loss loss joints The calculation method is obtained using the following formula in, It represents the node coordinate regression matrix of the skeleton model SKEL; Step STP543, calculate scapula loss, scapula loss loss scapula The calculation method is obtained using the following formula scapula indices ={26,27,28,36,37,38} Among them, the number 26 represents the right scapula abduction angle; 27 represents the right scapula elevation angle; 28 represents the right scapula external rotation angle; 36 represents the left scapula abduction angle; 37 represents the left scapula elevation angle; 38 represents the left and right scapula external rotation angles; Step STP544, calculate spine loss, spine loss loss spine The calculation method is obtained using the following formula spine indices ={17,18,19,20,21,22,23,24} Among them, 17, 18, and 19 represent lumbar flexion parameters, 20, 21, and 22 represent thoracic extension parameters, and 22, 23, and 24 represent head twist parameters; Step STP545, calculate the total loss Loss between the SKEL model and the SMPL model total Loss total =loss verts +loss joints +loss scapula +loss spine Step STP555, optimization Make Loss total Minimum, get and Data pair Step STP565, constructing a three-dimensional image of the human body, using the obtained data pairs, by training an MLP network, you can get Directly obtain Then input it into the SKEL model to obtain a 3D model of the human skeleton M SKEL (i skel ,b skel ): Will and Input into the SKEL model to obtain a three-dimensional image of the human body including bones and skin.

7. A multimodal three-dimensional human body imaging device, characterized in that: include Image acquisition module, used to acquire and store image data of different modalities including RGB images, depth images, infrared images, and X-ray fluoroscopy data; The image encoding module is used to encode the image data transmitted by the image acquisition module and perform feature extraction on the image data to obtain corresponding feature information of image data of different modalities; An image feature fusion module is used to fuse feature information of image data of different modalities and enable the fused feature information to be recognized by the camera parameter estimation module and the human body parameter estimation module; The camera parameter estimation module has a built-in shared MLP platform. After identifying the fused feature information, it obtains the estimated camera pose information corresponding to the feature information. The human body parameter estimation module has a built-in Transformer decoder that generates shape parameters by decoding the fused feature information The human body's 2.5D joint point J and joint torsion angle Φ; Digital human generation model, including built-in SMPL model or SKEL model, by inputting different shape parameters Posture parameters Generate the corresponding three-dimensional digital human model.

8. Multimodal human body three-dimensional imaging electronic device, characterized in that: The invention comprises at least one processor and a memory electrically or communicatively connected to the processor, wherein the memory contains at least one storage unit for storing a computer program of the method according to any one of claims 1 to 7, and the storage unit implements a three-dimensional imaging method of a human body by executing the computer program through the processor.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that The computer instructions are used to execute the method according to any one of claims 1 to 7 by a computer.

10. A computer program, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images