Multimodal human three-dimensional imaging methods, devices, electronic equipment and storage media

Abstract

This application discloses a multimodal human body 3D imaging method. Based on multimodal human body image data input from multiple perspectives, it generates a 3D digital human body, including the following steps: image data acquisition and preprocessing, image feature fusion, camera parameter estimation, human body parameter estimation, and generating a 3D human body image. It also includes a multimodal human body 3D imaging device, electronic equipment, storage medium, and computer program. This invention can achieve registration of multimodal input data, jointly revising the 3D digital human body's morphology and posture parameters, making the generated 3D digital human body more accurate and realistic.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more particularly to computer image data processing, specifically to a multimodal human three-dimensional imaging method. Background Technology

[0002] 3D digital human generation technology refers to the process of creating virtual human figures (i.e., 3D digital humans or virtual humans) using computer technology and algorithms. This technology typically involves the following steps:

[0003] Data Acquisition: Human image data is acquired through various methods, including 2D images, videos, depth images, and laser scans. Computer graphics and machine learning algorithms are used to transform the acquired data into a 3D model. This process may include generating facial features, body structure, and texture mapping. After obtaining the 3D model, animation techniques are used to give it dynamic performance, enabling it to display various expressions, movements, and interactions. By combining data from different modalities, the generated 3D digital human is optimized to make it visually more realistic and natural.

[0004] The application of 3D digital human generation technology is wide-ranging and can play an important role in multiple fields. For example, in virtual characters, high-quality virtual characters can be generated in video games, animated films, and virtual reality experiences, enhancing the user experience. Users can create personalized virtual avatars according to their preferences. Users can use virtual avatars as their representatives on social platforms to interact with others, and also use them for facial replacement and special effects enhancement in social applications. In the medical and educational fields, 3D digital human models can be used for anatomy teaching and simulated surgical practice, among other applications.

[0005] In existing technologies, 3D models can be generated from single or multiple 2D images. Common methods include deep learning and convolutional neural networks (CNNs). Additionally, video sequences are used to extract motion information and facial expressions to generate more dynamic 3D models, primarily used in animation, virtual reality (VR), and augmented reality (AR). While existing technologies can generate 3D digital human models from 2D images and videos, digital humans generated from different data sources all have some shortcomings and limitations. Their comprehensiveness, realism, and especially the accuracy of generating the internal structure of the digital human, including the skeleton, are still insufficient, and precision needs improvement. Summary of the Invention

[0006] To address the issue of insufficient accuracy in existing 3D digital human generation technologies, this application provides a multimodal 3D human imaging method. By using data from different modalities to generate the digital human, this method can overcome at least one of the following technical problems:

[0007] 1. Overcome the problem of insufficient realism and accuracy in models due to the one-sidedness of single-modal data.

[0008] 2. Overcome the problem that digital human models built based on bitmap images lack depth information, resulting in insufficient fit with actual human objects.

[0009] 3. Overcome the problem that 3D digital humans generated using a single model tend to be standardized and lack a high degree of personalization.

[0010] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0011] A multimodal human body 3D imaging method, based on multimodal human body image data input from multiple viewpoints, generates a 3D digital human body, including the following steps:

[0012] Step STP100: Image data acquisition and preprocessing. The collected image data includes any one or more of RGB images, depth images, infrared images, and X-ray fluoroscopic images. The collected image data is encoded and feature extracted. The color features, depth features, and infrared features extracted from the encoded RGB images, depth images, and infrared images are denoted as follows: , , ;

[0013] Step STP200, image feature fusion, combines any set of aligned color features Deep features and infrared features The features are processed by separate convolutional neural networks, and then the sigmoid function is used to obtain a deep attention mask M1 and an infrared attention mask M2. The obtained attention masks M1 and M2 are used as the primary / secondary features and the color feature, respectively. Perform fusion to obtain fusion characteristics Utilizing the obtained fusion features respectively Perform camera parameter estimation and human body parameter estimation.

[0014] Step STP300, camera parameter estimation, using the fused features obtained in step STP200. The camera pose matrix is ​​obtained by inputting it into the MLP network, and the formula for obtaining the camera pose matrix is ​​as follows:

[0015]

[0016] in, Represents the fusion features from the i-th perspective. The pose matrix and rotation matrix represent the camera. Translation matrix N is a positive integer. Represents MLP networks;

[0017] Step STP400, Human Parameter Estimation, involves fusing the features obtained in step STP200. Input into the Transformer decoder to generate shape parameters Human body 2.5D joint point J, and joint torsion angle Finally, the 2.5D joint point J of the human body, as well as the joint torsion angle, are determined. The parameters are fed into the inverse kinematics module (IK) to obtain the pose parameters of the human parametric model. ;

[0018] Step STP500 generates a 3D image of the human body, and sets the shape parameters... and attitude parameters The fused features are obtained by feeding them into a standard SMPL model or SKEL model. Optimized 3D human body imaging.

[0019] Preferably, in step STP100, the encoding and feature extraction of image data employs a convolutional neural network with CNN, ResNet, and ViT as its basic structures. Independent encoders are used to encode and extract features from the RGB image, depth image, and infrared image respectively. The independent encoding is expressed as follows:

[0020]

[0021]

[0022]

[0023] in, It is an RGB encoder. For depth encoders, It is an infrared encoder.

[0024] Preferably, in step STP200, attention masks M1 and M2 are used as primary / secondary features and color features, respectively. The fusion process is performed according to the number of attention masks obtained, following these steps:

[0025] When only depth features are available When obtaining the deep attention mask M1, the fusion steps are as follows:

[0026] Step STP211: Obtain the attention mask M1, and derive the deep attention mask M1 using the sigmoid function. ;

[0027] Step STP212, using depth features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0028]

[0029] Step STP213, using color features The main features and depth features Fusion features obtained during fusion It is obtained through the following expression:

[0030]

[0031] Step STP214, the fusion features obtained through steps STP212 and STP213 and Obtain fusion features The expression is as follows:

[0032]

[0033] in, The symbol represents the multiplication operation;

[0034] When only infrared features are available When obtaining the infrared attention mask M2, the fusion steps are as follows:

[0035] Step STP221: Obtain the attention mask M2, and derive the depth attention mask M1 using the sigmoid function. ;

[0036] Step STP222, using infrared features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0037]

[0038] Step STP223, using color features The main features and infrared characteristics Fusion features obtained during fusion It is obtained through the following expression:

[0039]

[0040] Step STP224, the fusion features obtained through steps STP222 and STP223 and Obtain fusion features The expression is as follows:

[0041]

[0042] in, The symbol represents the multiplication operation;

[0043] When both depth features are present and infrared features When obtaining the depth attention mask M1 and the infrared attention mask M2, the fusion steps are as follows:

[0044] Step STP231: Obtain the depth attention mask M1 and the infrared attention mask M2.

[0045]

[0046]

[0047] Step STP232, using depth features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0048]

[0049] Step STP233, using color features The main features and depth features Fusion features obtained during fusion It is obtained through the following expression:

[0050]

[0051] Step STP234, using infrared features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0052]

[0053] Step STP235, using color features The main features and infrared characteristics Fusion features obtained during fusion It is obtained through the following expression:

[0054]

[0055] Step STP236, the fusion features obtained through steps STP232 and STP235 , , and Obtain fusion features The expression is as follows:

[0056]

[0057] in, The symbol represents a multiplication operation.

[0058] Preferably, in step STP400, the human body 2.5D joint point J and the joint torsion angle are used. The parameters are fed into the inverse kinematics module (IK) to obtain the pose parameters of the human parametric model. The process is implemented using the following steps:

[0059] Step STP410: Given the position of the k-th joint in human joint J. Initial joint position The rotation matrix relationship before and after the joint pose change can be expressed as:

[0060]

[0061]

[0062] in, Let be the rotation matrix before and after the pose change of the current k-th joint. This is the current attitude vector. Let be the initial state vector. The index of the parent node of the k-th joint;

[0063] Step STP420, obtain the expression for the deformed rotation matrix:

[0064]

[0065] in, This represents the rotation matrix of the parent node of the k-th node. This represents the relative rotation of the k-th joint with respect to its parent joint. , ;

[0066] Step STP430 converts the global rotation matrix into a relative rotation matrix and then splits the relative rotation matrix into a swing and a twist matrix, resulting in: ;

[0067] The axis of rotation based on Swing is simultaneously perpendicular to and Then there is , , Based on Rodriguez's formula, the oscillation and rotation are derived respectively. , Thus, the relative rotation matrix R is obtained;

[0068]

[0069]

[0070] in, Represents perpendicular to and The vector, The swing angle, For the twist angle, yes A skew-symmetric matrix, yes A skew-symmetric matrix, It is a 3x3 identity matrix.

[0071] Preferably, in step STP500, the shape parameters obtained in step STP400 are... and attitude parameters The process of generating a 3D human body image by inputting a standard SMPL model is implemented using the following expression:

[0072]

[0073] in, This represents the SMPL function. This represents the skinning function. The equation representing the calculation of vertex coordinates using shape and pose parameters; Represents the predicted joint equation; Represents mixed weights;

[0074] }, specifically,

[0075] Human average network model, SMPL Template Mesh network, with two models: Female & Male, and the default pose is T-Pose;

[0076] Skin weighting, i.e. The influence weight of each key point on each vertex in the SMPL Template Mesh;

[0077] PCA bases for human body shape; PCA bases responsible for body shape.

[0078] Calculate the regression matrix of K joints from the standard T-pose Template mesh;

[0079] The influence of human posture on shape is represented by the PCA basis of human posture, which is responsible for the influence of posture on body shape and is proportional to the coefficients of the rotation matrix.

[0080] Preferably, in step STP500, the shape parameters are... and attitude parameters The SKEL model is learned using an MLP network before input. arrive The steps involved in mapping data pairs to construct and generate a 3D image of the human body specifically include:

[0081] Step STP510: Select the SMPL data pair from the official SMPL dataset;

[0082] Step STP520 utilizes the control parameters from the data pair. and The values ​​are input into the standard SMPL model, and regression is used to obtain the vertex and node coordinates of the SMPL.

[0083] Step STP530, initialize the SKEL model, and... Directly used as ,Will Initialize to 0 and input into the SKEL model to obtain the skin model. and skeletal model ;

[0084] Step STP540: Classify and calculate loss;

[0085] Step STP541, calculate vertex loss, vertex loss The calculation method is obtained using the following formula.

[0086]

[0087] in, Labels for SMPL vertices and SKEL vertices;

[0088] Step STP542, calculate the node loss, node loss The calculation method is obtained using the following formula.

[0089]

[0090]

[0091] in, This represents the node coordinate regression matrix of the skeletal model SKEL;

[0092] Step STP543, calculate scapular loss, scapular loss The calculation method is obtained using the following formula.

[0093]

[0094]

[0095] In this context, the number 26 represents the right scapular abduction angle; 27 represents the right scapular elevation angle; 28 represents the right scapular external rotation angle; 36 represents the left scapular abduction angle; 37 represents the left scapular elevation angle; and 38 represents the left and right scapular external rotation angles.

[0096] Step STP544, calculate spinal cord injury, spinal cord injury The calculation method is obtained using the following formula.

[0097]

[0098]

[0099] in, Represents lumbar curvature parameters. Represents chest stretch parameters. Represents the head twisting parameter;

[0100] Step STP545: Calculate the total loss between the SKEL model and the SMPL model.

[0101]

[0102] Step STP555, optimization Make Minimum, get and Data pairs

[0103]

[0104] Step STP565: Construct a 3D image of the human body. Using the obtained data pairs, an MLP network can be trained to... Directly obtain

[0105]

[0106] This data is then input into the SKEL model to obtain a 3D model of the human skeleton.

[0107]

[0108] Will and Input into the SKEL model to obtain a 3D human body image including bones and skin.

[0109] The present invention also provides a multimodal human three-dimensional imaging device, including

[0110] The image acquisition module is used to acquire and store image data of different modalities, including RGB images, depth images, infrared images, and X-ray fluoroscopic images.

[0111] The image encoding module is used to encode the image data transmitted by the image acquisition module and extract features from the image data to obtain the corresponding feature information of image data of different modalities.

[0112] The image feature fusion module is used to fuse the feature information of image data from different modalities, and to enable the fused feature information to be recognized by the camera parameter estimation module and the human parameter estimation module.

[0113] The camera parameter estimation module has a built-in shared MLP platform. After identifying the fused feature information, it obtains the camera pose prediction information corresponding to the feature information.

[0114] The human body parameter estimation module has a built-in Transformer decoder, which generates shape parameters by decoding the fused feature information. Human body 2.5D joint J, and joint torsion angle ϕ;

[0115] Digital human generation models, including built-in SMPL or SKEL models, are generated by inputting different shape parameters. Attitude parameters Generate the corresponding 3D digital human model.

[0116] The present invention also provides a multimodal human body three-dimensional imaging electronic device, including at least one processor and a memory electrically or communicatively connected to the processor. The memory contains at least one storage unit for storing a computer program of the above-mentioned multimodal human body three-dimensional imaging method. The storage unit executes the computer program through the processor to realize the human body three-dimensional imaging method.

[0117] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for executing a multimodal human three-dimensional imaging method via a computer.

[0118] The present invention also provides a computer program, including a computer program that, when executed by a processor, implements a multimodal human body three-dimensional imaging method.

[0119] Beneficial effects:

[0120] 1. This invention can achieve registration of multimodal input data, jointly revising the human form and posture parameters of the 3D digital human, making the generated 3D digital human more accurate and realistic.

[0121] 2. This invention also employs multi-model integrated registration optimization, including optimization of human body surface, skeletal and organ models, so that the generated 3D digital human has more specificity in both static and dynamic performance compared to digital human generated by a single model, reducing the proportion of standardization and generalization, and making it closer to or infinitely closer to the research object. Attached Figure Description

[0122] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0123] Figure 1 This is a flowchart of the three-dimensional digital human body generation process for multimodal human body images provided by the present invention.

[0124] Figure 2 It is a multimodal feature fusion. Detailed Implementation

[0125] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0126] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0127] Example 1:

[0128] Before describing the detailed embodiments of the present invention, a brief explanation of existing digital human generation technologies is provided. Existing 3D digital human generation technologies are based on deep learning network architectures and utilize the SMPL model. The SMPL model is a readily available, standardized human body model with extremely high generalization and standardization, but almost no specificity or personalization. In other words, the standard SMPL model is defined based on countless samples and is almost impossible to correspond to any specific person with clearly defined parameters. For ease of understanding, the SMPL model can be understood as a standardized, idealized virtual human. It possesses every part of a human body but lacks the characteristics of an actual human, including body shape and posture. Therefore, standardized digital humans can be used for virtual human-computer interaction or to enhance the experience of virtual scenes, such as game scenes. However, it is almost unsuitable for the field of medical reconstruction. Because the height, weight, and body shape of each patient in the medical field are inconsistent, in order to generate a 3D imaging model that closely resembles the actual human body, such as a specific patient, for subsequent diagnostic or treatment research, the model parameters need to be revised to obtain a 3D digital model that more closely approximates the real human body.

[0129] In existing technologies, the most commonly used method is bitmap parameter revision. This involves acquiring a bitmap of the object using a camera and then matching the feature pixels in the bitmap with the model to obtain a model that matches the object. However, due to the limitations of single-modal image information, the matching accuracy of existing 3D digital humans is insufficient for medical applications. Additionally, for 3D reconstruction of a specific body part, X-ray images, such as CT scans, are used. Since CT scans include features of the body surface edges and internal bones and tissues, 3D reconstruction using CT scans has high accuracy, especially when the slice thickness is small. However, local reconstruction only shows the condition of local organs, tissues, and bones, and cannot dynamically reveal the state of these organs, tissues, and bones in different body positions and postures. Therefore, to better study the changes in different parts of the human body under different postures, a more accurate 3D digital human is needed, leading to this invention.

[0130] See the instruction manual appendix Figure 1 As shown, this embodiment provides a multimodal human body 3D imaging method, which generates a 3D digital human body based on multimodal human body image data input from multiple viewpoints, including the following steps:

[0131] Step STP100: Image data acquisition and preprocessing. The collected image data includes any one or more of RGB images, depth images, infrared images, and X-ray fluoroscopic images. The collected image data is encoded and feature extracted. The color features, depth features, and infrared features extracted from the encoded RGB images, depth images, and infrared images are denoted as follows: , , ; Figure 1 In this context, FF represents a set of multiple features.

[0132] Step STP200, image feature fusion, combines any set of aligned color features Deep features and infrared features The features are processed by separate convolutional neural networks, and then the sigmoid function is used to obtain a deep attention mask M1 and an infrared attention mask M2. The obtained attention masks M1 and M2 are used as the primary / secondary features and the color feature, respectively. Perform fusion to obtain fusion characteristics Utilizing the obtained fusion features respectively Perform camera parameter estimation and human body parameter estimation.

[0133] Step STP300, camera parameter estimation, using the fused features obtained in step STP200. The camera pose matrix is ​​obtained by inputting it into the MLP network, and the formula for obtaining the camera pose matrix is ​​as follows:

[0134]

[0135] in, Represents the fusion features from the i-th perspective. The pose matrix and rotation matrix represent the camera. Translation matrix N is a positive integer. Represents MLP networks;

[0136] Step STP400, Human Parameter Estimation, involves fusing the features obtained in step STP200. Input into the Transformer decoder to generate shape parameters Human body 2.5D joint point J, and joint torsion angle Finally, the 2.5D joint point J of the human body, as well as the joint torsion angle, are determined. The parameters are fed into the inverse kinematics module (IK) to obtain the pose parameters of the human parametric model. ;

[0137] Step STP500 generates a 3D image of the human body, and feeds the shape parameter β and pose parameter θ into a standard SMPL model or SKEL model to obtain fused features. Optimized 3D human body imaging.

[0138] Example 2:

[0139] This embodiment proposes a new method for fusing features from different modalities, based on Embodiment 1. See details below. Figure 2 As shown, in step STP100, the encoding and feature extraction of image data employs convolutional neural networks with CNN, ResNet, and ViT as their basic structures. Independent encoders are used to encode and extract features from RGB images, depth images, and infrared images respectively. The expression for independent encoding is as follows:

[0140]

[0141]

[0142]

[0143] in, It is an RGB encoder. For depth encoders, It is an infrared encoder.

[0144] In this embodiment, see Figure 2 As shown, in step STP200, attention masks M1 and M2 are used as primary / secondary features and color features, respectively. The fusion process is performed according to the number of attention masks obtained, following these steps:

[0145] When only depth features are available When obtaining the deep attention mask M1, the fusion steps are as follows:

[0146] Step STP211: Obtain the attention mask M1, and derive the deep attention mask M1 using the sigmoid function. ;

[0147] Step STP212, using depth features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0148]

[0149] Step STP213, using color features The main features and depth features Fusion features obtained during fusion It is obtained through the following expression:

[0150]

[0151] Step STP214, the fusion features obtained through steps STP212 and STP213 and Obtain fusion features The expression is as follows:

[0152]

[0153] in, The symbol represents the multiplication operation;

[0154] When only infrared features are available When obtaining the infrared attention mask M2, the fusion steps are as follows:

[0155] Step STP221: Obtain the attention mask M2, and derive the depth attention mask M1 using the sigmoid function. ;

[0156] Step STP222, using infrared features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0157]

[0158] Step STP223, using color features The main features and infrared characteristics Fusion features obtained during fusion It is obtained through the following expression:

[0159]

[0160] Step STP224, the fusion features obtained through steps STP222 and STP223 and Obtain fusion features The expression is as follows:

[0161]

[0162] in, The symbol represents the multiplication operation;

[0163] When both depth features are present and infrared features When obtaining the depth attention mask M1 and the infrared attention mask M2, the fusion steps are as follows:

[0164] Step STP231: Obtain the depth attention mask M1 and the infrared attention mask M2.

[0165]

[0166]

[0167] Step STP232, using depth features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0168]

[0169] Step STP233, using color features The main features and depth features Fusion features obtained during fusion It is obtained through the following expression:

[0170]

[0171] Step STP234, using infrared features The main features and color features Fusion features obtained during fusion It is obtained through the following expression:

[0172]

[0173] Step STP235, using color features The main features and infrared characteristics Fusion features obtained during fusion It is obtained through the following expression:

[0174]

[0175] Step STP236, the fusion features obtained through steps STP232 and STP235 , , and Obtain fusion features The expression is as follows:

[0176]

[0177] in, The symbol represents a multiplication operation. It's worth noting that the more features input, the more comprehensive the fused features will be, resulting in a more realistic 3D digital human image that closely resembles the real object. The more modalities of feature information, the more comprehensive the entire model; the more feature information in a single modality, the closer the parameters affecting that modality will be to the real object.

[0178] Example 3:

[0179] This embodiment is a further optimization based on any of the above embodiments, because to output a human body mesh using an SMPL model, only the shape parameters need to be provided. and attitude parameters Here, 3k represents a multiple of 3, k represents a node, and 3 represents that each node has three angles. This is a common mathematical expression in the SMPL model and belongs to the existing technology expression. In this model, β and θ are the input parameters. β represents 10 parameters such as the height, weight, and head-to-body ratio of the human body. The β parameter is the Shape Blend Pose parameter, which can control the change of human body shape through 10 incremental templates. Specifically, the change of human body shape controlled by each parameter can be depicted through animation.

[0180] θ represents the overall human body posture and the relative angles of the 24 joints, totaling 75 parameters (24*3+3; 3 degrees of freedom per joint, plus 3 root nodes). It's worth noting that, for the entire SMPL model, β and θ represent morphological or shape parameters and posture parameters, respectively, and can be understood equivalently as part of the present invention. and However, since the calculation process in this invention is directional, it is more accurate to express it using a vector method. Because directly predicting the axis angle of a node using the network is relatively difficult, specifically, in step STP400, the 2.5D joint J of the human body and the joint torsion angle are used. The parameters are fed into the inverse kinematics module (IK) to obtain the pose parameters of the human parametric model. The process is implemented using the following steps:

[0181] Step STP410: Given the position of the k-th joint in human joint J. Initial joint position The rotation matrix relationship before and after the joint pose change can be expressed as:

[0182]

[0183]

[0184] in, Let be the rotation matrix before and after the pose change of the current k-th joint. This is the current attitude vector. Let be the initial state vector. The index of the parent node of the k-th joint;

[0185] Step STP420, obtain the expression for the deformed rotation matrix:

[0186]

[0187] in, This represents the rotation matrix of the parent node of the k-th node. This represents the relative rotation of the k-th joint with respect to its parent joint. , ;

[0188] Step STP430 converts the global rotation matrix into a relative rotation matrix and then splits the relative rotation matrix into a swing and a twist matrix, resulting in: ;

[0189] The axis of rotation based on Swing is simultaneously perpendicular to and Then there is , , Based on Rodriguez's formula, the oscillation and rotation are derived respectively. , Thus, the relative rotation matrix R is obtained;

[0190]

[0191]

[0192] in, Represents perpendicular to and The vector, The swing angle, For the twist angle, yes A skew-symmetric matrix, yes A skew-symmetric matrix, It is a 3x3 identity matrix.

[0193] Example 4:

[0194] This embodiment is an optimized version based on any of the above embodiments. In step STP500, the shape parameters obtained in step STP400 are... and attitude parameters The process of generating a 3D human body image by inputting a standard SMPL model is implemented using the following expression:

[0195]

[0196] in, This represents the SMPL function. This represents the skinning function. The equation representing the calculation of vertex coordinates using shape and pose parameters; Represents the predicted joint equation; Represents mixed weights;

[0197] }, specifically,

[0198] Human average network model, SMPL Template Mesh network, with two models: Female & Male, and the default pose is T-Pose;

[0199] Skin weighting, i.e. The influence weight of each key point on each vertex in the SMPL Template Mesh;

[0200] PCA bases for human body shape; PCA bases responsible for body shape.

[0201] Calculate the regression matrix of K joints from the standard T-pose Template mesh;

[0202] The influence of human posture on shape is represented by the PCA basis of human posture, which is responsible for the influence of posture on body shape and is proportional to the coefficients of the rotation matrix.

[0203] Example 5:

[0204] This embodiment is a further refinement based on any of the above embodiments. In step STP500, the shape parameters are... and attitude parameters The SKEL model is learned using an MLP network before input. arrive The steps involved in mapping data pairs to construct and generate a 3D image of the human body specifically include:

[0205] Step STP510: Select the SMPL data pair from the official SMPL dataset;

[0206] Step STP520 utilizes the control parameters from the data pair. and The values ​​are input into the standard SMPL model, and regression is used to obtain the vertex and node coordinates of the SMPL.

[0207] Step STP530, initialize the SKEL model, and... Directly used as ,Will Initialize to 0 and input into the SKEL model to obtain the skin model. and skeletal model ;

[0208] Step STP540: Classify and calculate loss;

[0209] Step STP541, calculate vertex loss, vertex loss The calculation method is obtained using the following formula.

[0210]

[0211] in, Labels for SMPL vertices and SKEL vertices;

[0212] Step STP542, calculate the node loss, node loss The calculation method is obtained using the following formula.

[0213]

[0214]

[0215] in, This represents the node coordinate regression matrix of the skeletal model SKEL;

[0216] Step STP543, calculate scapular loss, scapular loss The calculation method is obtained using the following formula.

[0217]

[0218]

[0219] In this context, the number 26 represents the right scapular abduction angle; 27 represents the right scapular elevation angle; 28 represents the right scapular external rotation angle; 36 represents the left scapular abduction angle; 37 represents the left scapular elevation angle; and 38 represents the left and right scapular external rotation angles.

[0220] Step STP544, calculate spinal cord injury, spinal cord injury The calculation method is obtained using the following formula.

[0221]

[0222]

[0223] in, Represents lumbar curvature parameters. Represents chest stretch parameters. Represents the head twisting parameter;

[0224] Step STP545: Calculate the total loss between the SKEL model and the SMPL model.

[0225]

[0226] Step STP555, optimization Make Minimum, get and Data pairs

[0227]

[0228] Step STP565: Construct a 3D image of the human body. Using the obtained data pairs, an MLP network can be trained to... Directly obtain

[0229]

[0230] This data is then input into the SKEL model to obtain a 3D model of the human skeleton.

[0231]

[0232] Will and Input into the SKEL model to obtain a 3D human body image including bones and skin.

[0233] To better illustrate the modifications of this invention regarding skin positioning, joint points, and bone vertices, the principles will be explained in detail in the following embodiment. The inverse dynamics IK process has already been described in Embodiment 3, i.e., utilizing the obtained shape parameters... and attitude parameters Inputting these parameters into the SMPL model in version 1.5.1 will yield a human body surface model. The following section will explain how to recover the internal skeletal model (SKEL) from the skin model (SKIN). Similar to the SMPL model, SKEL is a parametric human body model that can output skin and skeletal models. The input to the SKEL model is: shape parameters. and attitude parameters The output is a skin model. skeletal model and key points , expressed as:

[0234]

[0235] It should be noted that the above This example is for illustrative purposes only and is intended to avoid confusion with the content described in this invention, especially with the foregoing. Distinguish between them.

[0236] Skin vertices: SKEL is built upon the additive method of SMPL, from the average template mesh. Begin by adding the learned displacements. ,in It is the basis of the PCA shape, and This refers to pose-dependent displacement. The skin vertices are then calculated using the following linear blending skin equation:

[0237]

[0238] in, It is a 6890×24 weight matrix, representing the influence weight of the skin vertices on the nodes. This weight is inherited from the SMPL model by defining a corresponding SMPL joint for each 𝑁𝐽=24 joint of SKEL.

[0239] It is a rigid transformation matrix, representing the rotation and translation matrix of the k-th node relative to the original pose, expressed as:

[0240]

[0241] Indicates the position around its joint. Cancel its rotation matrix Transform the bone space in its initial state; This represents the rotation and translation matrix under the initial orientation. This represents the rotation and translation matrix under the current attitude.

[0242] Joints: The expression for joint 𝐽 is as follows:

[0243]

[0244] in, = , This represents the node coordinate regression matrix.

[0245] Skeletal vertices: Similar to skin vertex definitions, the expression for a skeleton vertex is:

[0246]

[0247] in, As the initial skeletal template, This represents the proportion factor of each bone, which defines the limb length; It is the weight matrix of bones excluding the spine and thoracic cavity, representing the weight of the bone vertices affected by the nodes, where the weight of the bottom of the spine is interpolated to 0 and the weight of the top is 1; It is a rigid transformation matrix, that is, a rotation and translation matrix relative to the initial pose, which is based on the attitude parameters. To translate and rotate the bone vertex associated with the i-th node, the expression is:

[0248]

[0249] in, This represents the rotation and translation matrix under the initial orientation. This represents the rotation and translation matrix under the current attitude.

[0250] At this point, the entire 3D imaging algorithm process is complete. By combining multiple viewpoint sensors, a 3D human body mesh in the world coordinate system is generated, including the 3D coordinates of surface vertices, internal skeletons, and internal nodes. Furthermore, the coordinate system transformation relationship between multiple camera sensors is obtained, enabling coordinate transformation.

[0251] Example 6:

[0252] The present invention also provides a multimodal human three-dimensional imaging device, including an image acquisition module for acquiring and storing image data of different modalities, including RGB images, depth images, infrared images, and X-ray fluoroscopic images;

[0253] The image encoding module is used to encode the image data transmitted by the image acquisition module and extract features from the image data to obtain the corresponding feature information of image data of different modalities.

[0254] The image feature fusion module is used to fuse the feature information of image data from different modalities, and to enable the fused feature information to be recognized by the camera parameter estimation module and the human parameter estimation module.

[0255] The camera parameter estimation module has a built-in shared MLP platform. After identifying the fused feature information, it obtains the camera pose prediction information corresponding to the feature information.

[0256] The human body parameter estimation module has a built-in Transformer decoder, which generates shape parameter β, human body 2.5D joint J, and joint torsion angle ϕ by decoding the fused feature information.

[0257] Digital human generation models, including built-in SMPL or SKEL models, are generated by inputting different shape parameters β and pose parameters. A corresponding 3D digital human model is generated. This is specifically implemented using the method described in Example 1, which will not be elaborated upon here.

[0258] Example 7:

[0259] The present invention also provides a multimodal human three-dimensional imaging electronic device, including at least one processor and a memory electrically or communicatively connected to the processor. The memory contains at least one storage unit for storing a computer program of the multimodal human three-dimensional imaging method described in Embodiments 1-5 above. The storage unit executes the computer program through the processor to realize the human three-dimensional imaging method.

[0260] Example 8:

[0261] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for executing the multimodal human three-dimensional imaging method described in Examples 1-5 via a computer.

[0262] Example 9:

[0263] The present invention also provides a computer program, which, when executed by a processor, implements the multimodal human three-dimensional imaging method described in Embodiments 1-5.

[0264] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-modal human three-dimensional imaging method, based on multi-view multi-modal human image data input, generating a three-dimensional digital human, characterized in that, Comprising the following steps: Step STP100, image data collection and preprocessing, the collected image data includes RGB image, depth image, infrared image, X-ray perspective image data; the collected image data is encoded and feature extraction is performed, and the color features, depth features and infrared features extracted from the encoded RGB image, depth image and infrared image are respectively denoted as , , ; Step STP200, fusion of image features, any set of aligned color features , depth features and infrared features are input into independent convolutional neural networks for feature processing, and then sigmoid functions are used to obtain depth attention mask M1 and infrared attention mask M2. The obtained attention masks M1 and M2 are used as primary / secondary features, respectively, to fuse with color features to obtain fused features . Camera parameter estimation and human body parameter estimation are performed using the obtained fused features , respectively. Step STP300, camera parameter estimation, fusing features obtained in step STP200 The camera pose matrix is obtained in the input MLP network, where the camera pose matrix is obtained by the following formula: wherein, represents the fused feature under the i-th view, represents a pose matrix of the camera, a rotation matrix , a translation matrix N is a positive integer, represents an MLP network; Step STP400, Human Parameter Estimation, involves fusing the features obtained in step STP200. Input into the Transformer decoder to generate shape parameters Human body 2.5D joint point J, and joint torsion angle Finally, the 2.5D joint point J of the human body, as well as the joint torsion angle, are determined. The parameters are fed into the inverse kinematics module (IK) to obtain the pose parameters of the human parametric model. ; Step STP500, generating a human 3D imaging, sending shape parameters and pose parameters into a standard SMPL model or SKEL model to obtain a human 3D imaging optimized with fused features .

2. The multi-modal three-dimensional human imaging method of claim 1, wherein: The encoding and feature extraction of the image data in step STP100 adopts a convolutional neural network with CNN, ResNet and ViT as the basic structure, and an independent encoder is used to encode and extract features of the RGB image, the depth image and the infrared image respectively. The expression mode of the independent encoding is: wherein, RGB is an RGB encoder, Depth is a depth encoder, IR is an infrared encoder.

3. The multi-modal three-dimensional human imaging method of claim 1, wherein: The attention masks M1, M2 are applied in step STP200 to the primary / secondary features and color features, respectively The steps of fusing are performed according to the number of attention masks obtained, respectively, as follows: When only depth features The fusion step is as follows when the depth attention mask M1 is derived: Step STP211, obtain the attention mask M1, and derive the deep attention mask M1 through a sigmoid function, that is ; Step STP212, with deep features as primary features and color features fusion features obtained at fusion obtained by the following expression: Step STP213, with color features as primary features and depth features fusion features obtained at fusion time obtained by the following expression: Step STP214, obtaining fusion features from step STP212 and step STP213 and obtaining fusion features The expression is as follows: wherein the symbol represents a multiplication operation; When only the infrared features The fusion step is as follows when the infrared attention mask M2 is derived: Step STP221, obtain the attention mask M2, and obtain the deep attention mask M1 through a sigmoid function, that is ; Step STP222, with infrared features as primary features and color features fusion features obtained at fusion obtained by the following expression: Step STP223, to color features as primary features and infrared features fusion features obtained at fusion time obtained by the following expression: Step STP224, obtaining the fused feature from step STP222 and step STP223 and obtaining the fused feature The expression is as follows: wherein the symbol represents a multiplication operation; When both depth features and infrared features are available, the fusion step is as follows: In step STP231, the depth attention mask M1 and the infrared attention mask M2 are obtained Step STP232, to obtain the deep feature as the main feature and the color feature fusion feature obtained at the time of fusion obtained by the following expression: Step STP233, to color features as primary features and depth features fusion features obtained at fusion time obtained by the following expression: Step STP234, with infrared features as the main features and color features fusion features obtained at fusion time obtained by the following expression: Step STP235 to color features as primary features with infrared features fusion features obtained at fusion time obtained by the following expression: Step STP236, obtaining the fused feature from the step STP232 and the step STP235 , , and obtaining the fused feature The expression of the fused feature is as follows: wherein The symbol represents a multiplication operation.

4. The multi-modal three-dimensional human imaging method of claim 1, wherein: The process of step STP400 is implemented by the following steps: The process of step STP400 is implemented by the following steps: The process of step STP400 is implemented by the following steps: Step STP410, given the position of the kth joint J in the human body , initial joint position The rotation matrix relationship before and after the change of the joint posture is expressed as: wherein, is the rotation matrix before and after the change of the current kth joint pose, is the current pose vector, is the initial state vector, represents the parent node index of the kth joint. In step STP420, the expression of the deformed rotation matrix is obtained: wherein, Rk represents the rotation matrix of the parent node of the kth node, Rk represents the relative rotation of the kth joint with respect to its parent joint, , ; Step STP430, converting the global rotation matrix into a relative rotation matrix and splitting the relative rotation matrix into Swing and Twist, then ; The axis of rotation of the swing is perpendicular to the axis of rotation of the roll at the same time and then , , The swing rotation is derived according to the Rodrigues formula , so as to obtain the relative rotation matrix R; wherein represents a vector perpendicular to and , is the swing angle, is the twist angle, is the skew-symmetric matrix of , is the skew-symmetric matrix of , is the 3*3 identity matrix.

5. The multi-modal three-dimensional human imaging method of claim 1, wherein: The shape parameters obtained in step STP400 are input into a standard SMPL model in step STP500 and pose parameters The process of generating a three-dimensional image of a human body from the standard SMPL model is implemented using the following expression: wherein, represents a SMPL function; represents a Skinning function; represents an equation for calculating vertex coordinates from shape parameters and pose parameters; represents a predicted joint equation; represents a blending weight; in particular, , Human average network model, Template Mesh template network of SMPL, two models of Female & Male, default Pose is T-Pose; Blending Weight, i.e. the influence weight of each joint on each vertex of the SMPL Template Mesh; , PCA basis of human body shape, PCA basis responsible for body shape; compute a regression matrix of K joints from the standard T-pose Template mesh; , the human pose influence on shape, the PCA basis for the human pose, responsible for the pose influence on body shape, proportional to the coefficients of the rotation matrix.

6. The multi-modal three-dimensional human imaging method of claim 1, wherein: The shape parameters and pose parameters are input into the SKEL model to perform data pair construction and generate a human three-dimensional imaging step, specifically comprising: In step STP510, the SMPL data pair is selected from the official SMPL data set; Step STP520, using the control parameters of the data pair and , input into the standard SMPL model, and the vertex and node coordinates of SMPL are obtained by regression; Step STP 530, initialize the SKEL model, to be used directly as , to be initialized to 0 and input into the SKEL model to obtain a skin model and a skeleton model ; In step STP540, the classification loss loss is calculated; Step STP 541, calculate vertex loss, vertex loss The calculation is obtained using the following formula wherein, is a label for SMPL vertices and SKEL vertices; Step STP 542, compute node loss, node loss The computation is obtained using the following formula wherein, denotes the node coordinate regression matrix of the skeleton model SKEL; Step STP 543, calculate scapula loss, scapula loss The calculation is obtained using the following formula wherein the number 26 represents the right scapular abduction angle; 27 represents the right scapular elevation angle; 28 represents the right scapular external rotation angle; 36 represents the left scapular abduction angle; 37 represents the left scapular elevation angle; 38 represents the left and right scapular external rotation angle; Step STP 544, calculate the spinal column loss, Spinal Column Loss The calculation is obtained using the following formula wherein, represents a lumbar flexion parameter, represents a thoracic extension parameter, represents a head twist parameter; Step STP545, compute total loss between SKEL model and SMPL model Step STP555, optimize such that min, obtain data pairs with ​ Step STP 565, building a three-dimensional image of the human body, using the data obtained pairs, by training a MLP network, namely from directly Then input into the SKEL model to obtain the human body skeleton 3D model will be and input into the SKEL model to obtain a three-dimensional imaging of the human body including the skeleton and the skin.

7. A multi-modal three-dimensional human imaging apparatus, characterized by, Comprising An image acquisition module is configured to acquire and pre-process image data, and the collected image data includes RGB images, depth images, infrared images and X-ray perspective image data; An image coding module is configured to code the collected image data and extract features, and the coded RGB image, depth image, and infrared image are extracted color features, depth features, and infrared features, respectively, denoted as , , ; an image feature fusion module configured to input any one of the aligned color features, depth features and infrared features into an independent convolutional neural network for feature processing, and obtain a depth attention mask M1 and an infrared attention mask M2 by using a sigmoid function, and fuse the obtained attention masks M1 and M2 with the color features as primary / secondary features to obtain fused features perform camera parameter estimation and human body parameter estimation, respectively.​​​​​ a camera parameter estimation module configured to obtain a camera pose matrix from the fused features obtained in step STP200 obtaining a camera pose matrix in an input MLP network, wherein the camera pose matrix is obtained according to the following formula: wherein, represents the fused feature under the i-th view, represents a pose matrix of the camera, a rotation matrix , a translation matrix N belongs to a positive integer, represents an MLP network; The human body parameter estimation module integrates the fused features obtained in step STP200. Input into the Transformer decoder to generate shape parameters Human body 2.5D joint point J, and joint torsion angle Finally, the 2.5D joint point J of the human body, as well as the joint torsion angle, are determined. The parameters are fed into the inverse kinematics module (IK) to obtain the pose parameters of the human parametric model. ; Digital human generation model, including built-in SMPL model or SKEL model, obtains optimized three-dimensional imaging of human body by sending shape parameters and pose parameters into standard SMPL model or SKEL model to obtain fused features .

8. Multi-modal three-dimensional imaging electronic device of a human body, characterized in that, The computer program is stored in the memory, and the computer program is executed by the processor to realize the method of any one of claims 1-7.

9. A non-transitory computer readable storage medium having stored thereon computer instructions, characterized in that, The computer program is stored in the memory, and the computer program is executed by the processor to realize the method of any one of claims 1-7.

10. Computer program product, characterized in that, The computer program is stored in the memory, and the computer program is executed by the processor to realize the method of any one of claims 1-7.

Images