Three-dimensional human body reconstruction method and device
Through a single-belt RGB depth camera and automatic rotation platform technology, rapid and accurate three-dimensional human body reconstruction is achieved, solving the problems of high accuracy and cost in the existing technology, applicable to multiple application scenarios, and improving the practicality and market of the product Competitiveness.
Patent Information
- Application Number
- CN202411311676.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-12-06
AI Technical Summary
The existing three-dimensional human body reconstruction technology uses a separate RGB camera or multiple RGB+ depth cameras to have accuracy problems and high cost problems, especially when 360-degree shooting and multi-camera calibration are required, which is time-consuming and inaccurate.
A single-belt RGB depth camera is used to combine automatic detection of uniform rotation platform to directly generate a three-dimensional human body model by calculating the external parameters of the depth camera and high-precision data acquisition, avoiding the steps of feature matching and calibration, and achieving rapid and accurate three-dimensional human body reconstruction.
It improves the integrity and accuracy of three-dimensional reconstruction, reduces time and cost, is suitable for health management, exercise analysis and other fields, and provides more comprehensive and accurate human data support.
Smart Images

Figure CN119091053A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of computer vision, and specifically to a three-dimensional human body reconstruction method and device. Background Art
[0002] Anthropometry technology is gradually being widely used in many fields such as health management, sports assessment, and human morphology modeling. In the prior art, a single RGB (red, green, and blue) camera or multiple RGB depth cameras are usually used to reconstruct a 3D human body.
[0003] Reconstructing with a single RGB camera or a single RGB depth camera requires taking 360-degree photos of the human body, extracting feature points from each image, and performing feature point matching to obtain the inaccurate camera spatial position. This is not only time-consuming, but also causes accuracy issues in reconstructing the human body.
[0004] Using multiple RGB+depth cameras has a cost issue. The more detailed the human body reconstruction, the more cameras are needed. Secondly, the intrinsic parameters of multiple cameras are obviously different, and additional calibration steps are required to obtain the camera spatial position (external parameters). Summary of the invention
[0005] The present disclosure provides a three-dimensional human body reconstruction method, apparatus, device, storage medium and computer program product.
[0006] According to a first aspect of the present disclosure, a three-dimensional human body reconstruction method is provided, comprising: calculating the external parameters of a depth camera based on the center point coordinates of a rotating platform, a normal line, and a rotation angle of a human body relative to the depth camera in a depth image; converting the depth image into point cloud data, and splicing the point cloud data into a human body point cloud model based on the external parameters; extracting two-dimensional key points from the depth image and converting them into three-dimensional key points; generating a three-dimensional human body model based on the three-dimensional key points; and fitting the three-dimensional human body model to the human body point cloud model to obtain a reconstructed three-dimensional human body model.
[0007] According to a second aspect of the present disclosure, a three-dimensional human body modeling system is provided, characterized in that it includes: a host; a depth camera connected to the host and used to photograph a human body standing on a rotating platform; a rotating platform connected to the host and for the human body to stand on; the host includes a processor and a memory, the memory stores a computer program, and the computer program, when executed by the processor, implements the method described in any one of the first aspects.
[0008] According to a third aspect of the present disclosure, a three-dimensional human body reconstruction device is provided, comprising: a calculation unit, configured to calculate the external parameters of a depth camera according to the coordinates of the center point of a rotating platform, a normal line and a rotation angle of a human body relative to the depth camera in a depth image; a stitching unit, configured to convert the depth image into point cloud data, and stitch the point cloud data into a human body point cloud model according to the external parameters; a conversion unit, configured to extract two-dimensional key points from the depth image and convert them into three-dimensional key points; a generation unit, configured to generate a three-dimensional human body model based on the three-dimensional key points; and a reconstruction unit, configured to fit the three-dimensional human body model to the human body point cloud model to obtain a reconstructed three-dimensional human body model.
[0009] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect.
[0010] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any one of the methods according to the first aspect.
[0011] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements any one of the methods of the first aspect when executed by a processor.
[0012] The three-dimensional human body reconstruction method and device provided by the embodiments of the present disclosure use a single RGB depth camera to automatically detect a uniform rotating platform to quickly obtain accurate external parameters without feature matching and calibration. By combining the precise control of the uniform rotating plate and the high-precision data acquisition capability of the depth camera, a full range of high-precision scanning and reconstruction of the human body is achieved. This method not only improves the integrity and accuracy of three-dimensional reconstruction, but also completes data acquisition and processing in a shorter time, and is suitable for various human body measurement and modeling application scenarios.
[0013] This application can be applied to situations where accurate three-dimensional modeling of the human body is required, such as health management, motion analysis, virtual reality (VR) and other fields. Through this method, the product can provide users with more comprehensive and accurate human body data support, greatly improving the practicality and market competitiveness of the product.
[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0016] Figure 1 is a system architecture diagram of an embodiment of the three-dimensional human body modeling system disclosed herein;
[0017] Figure 2 is a flow chart of an embodiment of a three-dimensional human body modeling method according to the present disclosure;
[0018] Figure 3a-3d is a schematic diagram of an application scenario of the three-dimensional human body modeling method according to the present disclosure;
[0019] Figure 4 is a flow chart of another embodiment of a three-dimensional human body modeling method according to the present disclosure;
[0020] Figure 5 is a structural schematic diagram of an embodiment of a three-dimensional human body modeling device according to the present disclosure;
[0021] Figure 6 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0023] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the 3D human body modeling method or 3D human body modeling apparatus of the present disclosure can be applied.
[0024] like Figure 1 As shown, the system architecture 100 may include a host 101, a depth camera 102, and a rotating platform 103. The network is used to provide a medium for a communication link between the host 101, the depth camera 102, and the rotating platform 103. The network may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.
[0025] The host 101 can control the depth camera 102 and the rotating platform 103. The user can directly send start and stop instructions to the rotating platform 103 and the depth camera 102 through a remote control or voice. The user can also send start and stop instructions to the host 101, and then forward the user's instructions to the rotating platform 103 and the depth camera 102. The rotating platform 103 starts rotating and stops rotating according to the user's instructions, and the depth camera 102 starts taking pictures and stops taking pictures according to the user's instructions. The host 101 can determine that sufficient modeling data has been collected, and send a stop operation instruction to the depth camera 102 and the rotating platform 103. The depth camera 102 can send the collected image to the host 101, and the host 101 performs three-dimensional human body modeling. The host 101 includes a processor and a memory, and the memory stores a computer program. When the computer program is executed by the processor, it implements the methods described in processes 200 and 400.
[0026] The depth camera 102 can collect RGB images containing depth information. This system uses only one depth camera. The depth camera 102 is connected to the host 101 and is used to receive control instructions from the host to shoot a human body standing on the rotating platform.
[0027] The rotating platform 103 can rotate at a constant speed, and the shape of the rotating platform is not limited, and can be circular or other shapes. The rotating platform 103 is connected to the host 101 to receive control instructions from the host.
[0028] It should be noted that the 3D human body modeling method provided in the embodiments of the present disclosure is generally executed by the host 101 , and accordingly, the 3D human body modeling device is generally disposed in the host 101 .
[0029] Continue to refer Figure 2 , shows a process 200 of an embodiment of a 3D human body modeling method according to the present disclosure. The 3D human body modeling method comprises the following steps:
[0030] Step 201 , calculating the external parameters of the depth camera according to the center point coordinates of the rotating platform, the normal line and the rotation angle of the human body in the depth image relative to the depth camera.
[0031] In this embodiment, the execution subject of the 3D human body modeling method (eg Figure 1The host shown in the figure can obtain the center point coordinates and normal of the rotating platform in advance through an unloaded rotating platform. During the uniform rotation of the rotating platform, the depth camera also collects human body images at a uniform speed. Based on the pre-obtained three-dimensional center and normal, the rotation angle of the human body relative to the camera in each frame is calculated. Furthermore, using these rotation angles, the relative external parameters of the camera relative to the human body are reversely calculated. It can be regarded as the human body is stationary, and the depth camera rotates relative to the human body, so that each depth image will correspond to an external parameter of the depth camera. The external parameters of the camera can be determined by geometric and mathematical principles.
[0032] Step 202: convert the depth image into point cloud data, and stitch the point cloud data into a human body point cloud model according to external parameters.
[0033] In this embodiment, after obtaining the baseline data of the rotating platform without a human body, the rotating platform with a human body is photographed. The acquired depth image with RGB information is also converted into point cloud data with color. The depth image contains depth information, and the image can be converted into point cloud data based on the depth information. The point cloud data may include RGB information. The point cloud data can be spliced into a human point cloud model based on external parameters using the ICP (Iterative Closest Point) method.
[0034] Step 203: extract two-dimensional key points from the depth image and convert them into three-dimensional key points.
[0035] In this embodiment, two-dimensional key points can be extracted from the depth image by a pre-trained human key point detection model. Three-dimensional key points can be calculated based on the two-dimensional key points by least squares. The two-dimensional coordinates can be converted to coordinates in three-dimensional space by operations such as translation and rotation of the image. This usually involves the use of image processing libraries, such as OpenCV, by constructing an affine matrix and performing an affine transformation. Figure 3a As shown, the two images on the far left are human body images, the two images in the middle are two-dimensional human body key point images, and the two images on the far right are depth images.
[0036] Step 204: Generate a 3D human body model based on the 3D key points.
[0037] In this embodiment, a 3D human body model can be generated based on 3D key points by using a Hybrik (hybrid inverse kinematics) algorithm or an openpose algorithm. The 3D human body model can be an SMPL (Skinned Multi-Person Linear Model). The human body parameters in the SMPL model include the initial posture parameters (pose) and initial body shape parameters (beta) of the human body. Figure 3b As shown, the left side is the human body point cloud, the middle is the 3D human body key points, and the right side is the initial SMPL human body model.
[0038] Step 205 , fitting the three-dimensional human body model to the human body point cloud model to obtain a reconstructed three-dimensional human body model.
[0039] In this embodiment, the parameters of the 3D human body model are adjusted so that the 3D human body model fits the human body point cloud model. For example, the posture parameters and body shape parameters are adjusted so that the distance between the closest points of the two models is minimized. Figure 3c As shown in the figure, the network represents SMPL. The parameters of SMPL are adjusted to make the distance between the two models the shortest. The parameters of SMPL can be adjusted from the front and back directions for fitting. The final reconstructed 3D human body model is shown in Figure 3d shown.
[0040] The method provided by the above embodiment of the present disclosure realizes high-precision reconstruction of a 3D human body, and the generated 3D human body model has accurate geometric shape, providing reliable data support for subsequent applications. The measurement accuracy and data integrity of the product are significantly improved, and the user's body data can be captured more accurately. Compared with traditional methods, this solution reduces the impact caused by incomplete data or modeling errors, ensuring high accuracy and consistency of human body measurement.
[0041] In addition, the method's efficient data processing capability has significantly improved the real-time performance of the product, allowing users to obtain measurement results in a shorter time, thereby improving user experience and satisfaction. By applying this technical solution, the product performs more stably and reliably when facing complex human body measurement tasks, enhancing the product's competitiveness and laying a solid foundation for expanding application scenarios.
[0042] In some optional implementations of this embodiment, the method further includes: converting the depth image of the rotating platform without a human body into unloaded point cloud data; separating the point cloud data of the rotating platform from the unloaded point cloud data; and obtaining the center point coordinates and normal of the rotating platform by fitting the point cloud data of the rotating platform. Before shooting the human body, first record a video of the rotating platform in an unmanned state. Use a depth camera with RGB to capture images, and convert these depth images with RGB information into colored point cloud data. By analyzing these point cloud data, the positions of the ground and the rotating platform are identified. Using point cloud segmentation technology, the point cloud of the rotating platform is separated from the overall scene. Then, by fitting the point cloud data of the rotating platform, the three-dimensional center, normal direction and radius of the rotating platform are calculated. These parameters provide a basis for subsequent image processing with human bodies.
[0043] In some optional implementations of the present embodiment, separating the point cloud data of the rotating platform from the unloaded point cloud data includes: filtering the point cloud data of the rotating platform from the unloaded point cloud data according to the color of the rotating platform. Since the point cloud data includes color information, the point cloud data of the rotating platform can be segmented by using the difference between the color of the rotating platform and the color of the surrounding environment. For example, if it is known that the rotating platform is white, the white point cloud data can be filtered out from the unloaded point cloud data as the point cloud data of the rotating platform. Pre-separating the point cloud data of the rotating platform and then fitting the point cloud data of the rotating platform can increase the speed of data fitting.
[0044] In some optional implementations of the present embodiment, separating the point cloud data of the rotating platform from the unloaded point cloud data includes: filtering the point cloud data of the rotating platform from the unloaded point cloud data according to the height of the rotating platform. Since the point cloud data includes height information, the point cloud data of the rotating platform can be filtered out when the height of the rotating platform is known. For example, if the height of the rotating platform is 10 cm, the point cloud data with a height of 10 cm can be filtered out from the unloaded point cloud data as the point cloud data of the rotating platform. Pre-separating the point cloud data of the rotating platform and then fitting the point cloud data of the rotating platform can increase the speed of data fitting.
[0045] In some optional implementations of this embodiment, the method further includes: assigning the color information of the surface of the human body point cloud model to the reconstructed three-dimensional human body model. The nearest neighbor query technology can be used to match the point cloud surface texture with the SMPL model surface, thereby obtaining complete three-dimensional human body model surface color information. The three-dimensional human body model generated in this way has an accurate geometric shape and realistic surface texture, providing reliable data support for subsequent applications.
[0046] In some optional implementations of this embodiment, the method further includes: calculating the rotation angle by a given rotation step and a time difference between adjacent depth images. The rotation step refers to the rotation speed of the rotating platform, for example, 1 degree per second. The time difference between adjacent depth images refers to the image acquisition frequency, for example, 1000 frames per second.
[0047] In some optional implementations of this embodiment, the external parameters of the depth camera are calculated based on the center point coordinates of the rotating platform, the normal and the rotation angle of the human body relative to the depth camera in the depth image, including: calculating the first rotation matrix from the normal to the Y axis by the Rodriguez rotation formula, and combining the first rotation matrix and the unit matrix into a second rotation matrix; combining the center point coordinates and the unit matrix into a translation matrix; combining the second rotation matrix and the translation matrix into a first transformation matrix; combining the first transformation matrix with a pre-set rotation matrix to obtain a second transformation matrix; using the axis-angle rotation matrix algorithm, using the rotation angle to generate a third rotation matrix around the Y axis; combining the second transformation matrix with the third rotation matrix to obtain the external parameters of the depth camera. By using the method of geometric analysis to calculate the external parameters of the depth camera, a single RGB depth camera can be used to automatically detect a uniform rotating platform to quickly obtain accurate external parameters without feature matching and calibration, thereby achieving high-quality reconstruction of a three-dimensional human body.
[0048] In some optional implementations of the present embodiment, extracting two-dimensional key points from the depth image and converting them into three-dimensional key points includes: extracting two-dimensional key points from the depth image; converting the two-dimensional key points into three-dimensional key points by using the intrinsic and extrinsic parameters of the depth camera. In each frame of the image, two-dimensional human body posture estimation is performed to calculate the bounding box and key points of the human body in the image. Then, using the intrinsic parameters and relative extrinsic parameters of the camera, these two-dimensional key points are converted into corresponding three-dimensional key points. The accuracy is higher and the speed is faster than directly detecting three-dimensional key points from a human point cloud model.
[0049] In some optional implementations of this embodiment, generating a 3D human body model based on the 3D key points includes: generating a 3D human body model based on the 3D key points by a hybrid inverse kinematics method. Converting the 3D key point positions to a 3D human body mesh (i.e., SMPL) for end-to-end training. A closed loop is formed between the 3D skeleton and the parameterized model. At the same time, the alignment problem of the current model-based method and the problem of unreal human body structure of the key point estimation method are solved.
[0050] In some optional implementations of this embodiment, fitting the 3D human body model to the human body point cloud model to obtain a reconstructed 3D human body model includes: calculating a loss value by iterating the closest point between the 3D human body model and the human body point cloud model; and adjusting the posture parameters and body shape parameters of the 3D human body model in the direction where the loss value converges. This can speed up the fitting process and obtain an accurate 3D human body model.
[0051] Further references Figure 4, which shows a process 400 of another embodiment of a three-dimensional human body modeling method. The process 400 of the three-dimensional human body modeling method includes the following steps:
[0052] Step 401 , calculating the external parameters of the depth camera according to the center point coordinates of the rotating platform, the normal line and the rotation angle of the human body in the depth image relative to the depth camera.
[0053] Step 402 , converting the depth image into point cloud data, and splicing the point cloud data into a human body point cloud model according to external parameters.
[0054] Steps 401-402 are substantially the same as steps 201-202, and thus will not be described in detail.
[0055] Step 403: extract two-dimensional key points from the depth image and convert them into three-dimensional key points, and segment the human body point cloud model into a front point cloud model and a back point cloud model.
[0056] In this embodiment, two-dimensional key points can be extracted from the depth image by a pre-trained human key point detection model. Three-dimensional key points can be calculated based on the two-dimensional key points by least squares. The orientation of the human point cloud model can be determined according to the two-dimensional key points or the three-dimensional key points, and the human point cloud model can be divided into a front point cloud model and a back point cloud model according to the orientation. For example, the depth image with the longest distance between the key points of both eyes is determined as the front image, the point cloud data generated by the front image is spliced into the front point cloud model, and the remaining point cloud data in the human point cloud model is spliced into the back point cloud model.
[0057] In some optional implementations of the present embodiment, the human body point cloud model is divided into a front point cloud model and a back point cloud model, including: determining the orientation of the human body in the depth image based on the two-dimensional key points; determining the point cloud data portion of the human body point cloud model corresponding to the depth image with the front side facing the depth camera as the front point cloud model; and determining the point cloud data portion of the human body point cloud model corresponding to the depth image with the back side facing the depth camera as the back point cloud model.
[0058] In some optional implementations of this embodiment, determining the orientation of a human body in a depth image based on the two-dimensional key points includes: in response to detecting that the left hand key point of the human body is on the right and the right hand key point is on the left, determining the depth image with the longest distance between the left hand key point and the right hand key point as facing the front of the depth camera; in response to detecting that the left hand key point of the human body is on the left and the right hand key point is on the right, determining the depth image with the longest distance between the left hand key point and the right hand key point as facing the back of the depth camera. The orientation of the human body can be determined quickly and accurately.
[0059] Step 404: Generate a 3D human body model based on the 3D key points, and segment the 3D human body model into a front human body model and a back human body model.
[0060] In this embodiment, the three-dimensional human body model can also determine the human body orientation according to the key points, and then be divided into a front human body model and a back human body model.
[0061] Step 405 , fitting the front human body model to the front point cloud model, and fitting the back human body model to the back point cloud model.
[0062] In this embodiment, in the final stage of 3D human reconstruction, a unified body shape parameter (beta) is shared, and different pose parameters (PoseF and PoseB) are set for the front point cloud model (PCF) and the back point cloud model (PCB). By iterating the closest point (ICP) loss function, the fitting degree of the SMPL model on the front human body model (SMPL_mesh F) and the back human body model (SMPL_mesh B) is optimized respectively, and finally the optimal body shape parameter BestBeta is obtained. The nearest neighbor query technology is used to match the surface texture of the point cloud with the surface of the SMPL model, so as to obtain the complete surface color information of the 3D human body model.
[0063] Since the human body may have slight movements during the shooting process, the reconstructed point cloud is divided into two parts: the front point cloud model (PCF) and the back point cloud model (PCB) for processing to ensure the accuracy of reconstruction.
[0064] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a three-dimensional human body modeling device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0065] like Figure 5 As shown, the three-dimensional human body modeling device 500 of this embodiment includes: a calculation unit 501, a splicing unit 502, a conversion unit 503, a generation unit 504 and a reconstruction unit 505. Among them, the calculation unit 501 is configured to calculate the external parameters of the depth camera according to the center point coordinates of the rotating platform, the normal and the rotation angle of the human body relative to the depth camera in the depth image. The splicing unit 502 is configured to convert the depth image into point cloud data, and splice the point cloud data into a human body point cloud model according to the external parameters. The conversion unit 503 is configured to extract two-dimensional key points from the depth image and convert them into three-dimensional key points. The generation unit 504 is configured to generate a three-dimensional human body model based on the three-dimensional key points. The reconstruction unit 505 is configured to fit the three-dimensional human body model to the human body point cloud model to obtain a reconstructed three-dimensional human body model.
[0066] In this embodiment, the specific processing of the calculation unit 501, the splicing unit 502, the conversion unit 503, the generation unit 504 and the reconstruction unit 505 of the 3D human body modeling device 500 can be referred to. Figure 2 Corresponding to step 201, step 202, step 203, step 204 and step 205 in the embodiment.
[0067] In some optional implementations of this embodiment, the device 500 further includes a preprocessing unit (not shown in the drawings), configured to: convert the depth image of the rotating platform without a human body into unloaded point cloud data, separate the point cloud data of the rotating platform from the unloaded point cloud data, and obtain the center point coordinates and normal of the rotating platform by fitting the point cloud data of the rotating platform.
[0068] In some optional implementations of this embodiment, the pre-processing unit is further configured to: filter out the point cloud data of the rotating platform from the unloaded point cloud data according to the color of the rotating platform.
[0069] In some optional implementations of this embodiment, the pre-processing unit is further configured to: filter out the point cloud data of the rotating platform from the unloaded point cloud data according to the height of the rotating platform.
[0070] In some optional implementations of this embodiment, the device 500 further includes a coloring unit (not shown in the drawings) configured to assign color information of the surface of the human body point cloud model to the reconstructed three-dimensional human body model.
[0071] In some optional implementations of this embodiment, the calculation unit 501 is further configured to: calculate the rotation angle by using a given rotation step and a time difference between adjacent depth images.
[0072] In some optional implementations of this embodiment, the calculation unit 501 is further configured to: calculate the first rotation matrix from the normal to the Y axis by the Rodriguez rotation formula, and combine the first rotation matrix and the unit matrix into a second rotation matrix. The center point coordinates and the unit matrix form a translation matrix. The second rotation matrix and the translation matrix form a first transformation matrix. The first transformation matrix is combined with a pre-set rotation matrix to obtain a second transformation matrix. Through the axis-angle rotation matrix algorithm, the rotation angle is used to generate a third rotation matrix around the Y axis. The second transformation matrix is combined with the third rotation matrix to obtain the external parameters of the depth camera.
[0073] In some optional implementations of this embodiment, the conversion unit 503 is further configured to: extract two-dimensional key points from the depth image, and convert the two-dimensional key points into three-dimensional key points using intrinsic parameters and extrinsic parameters of the depth camera.
[0074] In some optional implementations of this embodiment, the generating unit 504 is further configured to: generate a three-dimensional human body model based on the three-dimensional key points by using a hybrid inverse kinematics device.
[0075] In some optional implementations of this embodiment, the reconstruction unit 505 is further configured to: calculate the loss value by iterating the closest point between the 3D human body model and the human body point cloud model, and adjust the posture parameters and body shape parameters of the 3D human body model in the direction where the loss value converges.
[0076] In some optional implementations of this embodiment, the reconstruction unit 505 is further configured to: segment the human body point cloud model into a front point cloud model and a back point cloud model; segment the three-dimensional human body model into a front human body model and a back human body model; fit the front human body model to the front point cloud model; and fit the back human body model to the back point cloud model.
[0077] In some optional implementations of this embodiment, the front human body model and the back human body model share unified body shape parameters.
[0078] In some optional implementations of this embodiment, the reconstruction unit 505 is further configured to: determine the orientation of the human body in the depth image according to the two-dimensional key points. Determine the point cloud data portion of the human body point cloud model corresponding to the depth image with the front side facing the depth camera as the front point cloud model. Determine the point cloud data portion of the human body point cloud model corresponding to the depth image with the back side facing the depth camera as the back point cloud model.
[0079] In some optional implementations of this embodiment, the reconstruction unit is further configured to: in response to detecting that the left hand key point of the human body is on the right and the right hand key point is on the left, determine that the front of the human body in the depth image is facing the depth camera. In response to detecting that the left hand key point of the human body is on the left and the right hand key point is on the right, determine that the back of the human body in the depth image is facing the depth camera.
[0080] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0081] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0082] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in process 200 or 400.
[0083] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in process 200 or 400.
[0084] A computer program product includes a computer program, wherein the computer program implements the method described in flow 200 or 400 when executed by a processor.
[0085] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0086] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0087] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0088] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a three-dimensional human body modeling method. For example, in some embodiments, the three-dimensional human body modeling method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the three-dimensional human body modeling method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the three-dimensional human body modeling method in any other appropriate manner (e.g., by means of firmware).
[0089] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0090] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0091] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0093] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0094] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0095] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0096] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A three-dimensional human body reconstruction method, comprising: The external parameters of the depth camera are calculated based on the coordinates of the center point of the rotating platform, the normal line, and the rotation angle of the human body relative to the depth camera in the depth image; Converting the depth image into point cloud data, and splicing the point cloud data into a human body point cloud model according to the external parameters; Extracting two-dimensional key points from the depth image and converting them into three-dimensional key points; Generate a three-dimensional human body model based on the three-dimensional key points; The three-dimensional human body model is fitted to the human body point cloud model to obtain a reconstructed three-dimensional human body model.
2. The method according to claim 1, wherein: The method further comprises: Convert the depth image of the rotating platform without a human body into unloaded point cloud data; Separating the point cloud data of the rotating platform from the unloaded point cloud data; The center point coordinates and normal line of the rotating platform are obtained by fitting the point cloud data of the rotating platform.
3. The method according to claim 2, wherein: The step of separating the point cloud data of the rotating platform from the unloaded point cloud data comprises: The point cloud data of the rotating platform is filtered out from the unloaded point cloud data according to the color of the rotating platform.
4. The method according to claim 2, wherein: The step of separating the point cloud data of the rotating platform from the unloaded point cloud data comprises: The point cloud data of the rotating platform is filtered out from the unloaded point cloud data according to the height of the rotating platform.
5. The method according to claim 1, wherein: The method further comprises: The color information of the surface of the human body point cloud model is assigned to the reconstructed three-dimensional human body model.
6. The method according to claim 1, wherein: The method further comprises: The rotation angle is calculated by the given rotation step size and the time difference between adjacent depth images.
7. The method according to claim 1, wherein: The step of calculating the external parameters of the depth camera according to the center point coordinates of the rotating platform, the normal line, and the rotation angle of the human body relative to the depth camera in the depth image includes: Calculate a first rotation matrix from the normal to the Y axis using the Rodriguez rotation formula, and combine the first rotation matrix and the unit matrix into a second rotation matrix; The center point coordinates and the unit matrix form a translation matrix; Combining the second rotation matrix and the translation matrix into a first transformation matrix; Combining the first transformation matrix with a preset rotation matrix to obtain a second transformation matrix; Generate a third rotation matrix around the Y axis using the rotation angle through an axis-angle rotation matrix algorithm; The second transformation matrix is combined with the third rotation matrix to obtain the external parameters of the depth camera.
8. The method according to claim 1, wherein: The step of extracting two-dimensional key points from the depth image and converting them into three-dimensional key points includes: Extracting two-dimensional key points from the depth image; The two-dimensional key points are converted into three-dimensional key points by using the intrinsic parameters and extrinsic parameters of the depth camera.
9. The method according to claim 1, wherein: The generating of the three-dimensional human body model based on the three-dimensional key points comprises: A three-dimensional human body model is generated based on the three-dimensional key points by a hybrid inverse kinematics method.
10. The method according to claim 1, wherein: The step of fitting the three-dimensional human body model to the human body point cloud model to obtain a reconstructed three-dimensional human body model comprises: Calculate the loss value by iterating the closest point between the three-dimensional human body model and the human body point cloud model; The posture parameters and body shape parameters of the three-dimensional human body model are adjusted in the direction of convergence of the loss value.
11. The method according to claim 1, wherein: The step of fitting the three-dimensional human body model to the human body point cloud model comprises: Segmenting the human body point cloud model into a front point cloud model and a back point cloud model; Segmenting the three-dimensional human body model into a front human body model and a back human body model; Fitting the front human body model to the front point cloud model; The back human body model is fitted to the back point cloud model.
12. The method according to claim 11, wherein: The front human body model and the back human body model share unified body shape parameters.
13. The method according to claim 11, wherein the step of segmenting the human body point cloud model into a front point cloud model and a back point cloud model comprises: Determining the orientation of a human body in the depth image according to the two-dimensional key points; Determine the point cloud data portion corresponding to the depth image facing the depth camera in the human body point cloud model as a front point cloud model; A point cloud data portion corresponding to the depth image with the back side facing the depth camera in the human body point cloud model is determined as a back side point cloud model.
14. The method according to claim 13, wherein determining the orientation of the human body in the depth image according to the two-dimensional key points comprises: In response to detecting that the left hand key point of the human body is on the right side and the right hand key point is on the left side, determining the depth image with the longest distance between the left hand key point and the right hand key point as facing the depth camera; In response to detecting that the left hand keypoint of the human body is on the left and the right hand keypoint is on the right, the depth image with the longest distance between the left hand keypoint and the right hand keypoint is determined as facing the back side of the depth camera.
15. A three-dimensional human body modeling system, characterized in that: include: Host; A depth camera, connected to the host, for photographing a human body standing on the rotating platform; A rotating platform, connected to the host, for the human body to stand on; The host comprises a processor and a memory, wherein the memory stores a computer program, and the computer program implements the method according to any one of claims 1 to 14 when executed by the processor.
16. A three-dimensional human body reconstruction device, comprising: A calculation unit configured to calculate the external parameters of the depth camera according to the coordinates of the center point of the rotating platform, the normal line and the rotation angle of the human body relative to the depth camera in the depth image; a stitching unit, configured to convert the depth image into point cloud data, and stitch the point cloud data into a human body point cloud model according to the external parameters; A conversion unit, configured to extract two-dimensional key points from the depth image and convert them into three-dimensional key points; A generating unit, configured to generate a three-dimensional human body model based on the three-dimensional key points; The reconstruction unit is configured to fit the three-dimensional human body model to the human body point cloud model to obtain a reconstructed three-dimensional human body model.
17. The device according to claim 16, wherein: The device also includes a pre-processing unit configured to: Convert the depth image of the rotating platform without a human body into unloaded point cloud data; Separating the point cloud data of the rotating platform from the unloaded point cloud data; The center point coordinates and normal line of the rotating platform are obtained by fitting the point cloud data of the rotating platform.
18. The device according to claim 17, wherein: The pre-processing unit is further configured to: The point cloud data of the rotating platform is filtered out from the unloaded point cloud data according to the color of the rotating platform.
19. The device according to claim 17, wherein: The pre-processing unit is further configured to: The point cloud data of the rotating platform is filtered out from the unloaded point cloud data according to the height of the rotating platform.
20. The device according to claim 16, wherein: The device also includes a coloring unit configured to: The color information of the surface of the human body point cloud model is assigned to the reconstructed three-dimensional human body model.
21. The device according to claim 16, wherein: The computing unit is further configured to: The rotation angle is calculated by the given rotation step size and the time difference between adjacent depth images.
22. The device according to claim 16, wherein: The computing unit is further configured to: Calculate a first rotation matrix from the normal to the Y axis using the Rodriguez rotation formula, and combine the first rotation matrix and the unit matrix into a second rotation matrix; The center point coordinates and the unit matrix form a translation matrix; Combining the second rotation matrix and the translation matrix into a first transformation matrix; Combining the first transformation matrix with a preset rotation matrix to obtain a second transformation matrix; Generate a third rotation matrix around the Y axis using the rotation angle through an axis-angle rotation matrix algorithm; The second transformation matrix is combined with the third rotation matrix to obtain the external parameters of the depth camera.
23. The device according to claim 16, wherein: The conversion unit is further configured to: Extracting two-dimensional key points from the depth image; The two-dimensional key points are converted into three-dimensional key points by using the intrinsic parameters and extrinsic parameters of the depth camera.
24. The device according to claim 16, wherein: The generating unit is further configured to: A three-dimensional human body model is generated based on the three-dimensional key points through a hybrid inverse kinematics device.
25. The device according to claim 16, wherein: The reconstruction unit is further configured to: Calculate the loss value by iterating the closest point between the three-dimensional human body model and the human body point cloud model; The posture parameters and body shape parameters of the three-dimensional human body model are adjusted in the direction of convergence of the loss value.
26. The device according to claim 16, wherein: The reconstruction unit is further configured to: Segmenting the human body point cloud model into a front point cloud model and a back point cloud model; Segmenting the three-dimensional human body model into a front human body model and a back human body model; Fitting the front human body model to the front point cloud model; The back human body model is fitted to the back point cloud model.
27. The device according to claim 26, wherein: The front human body model and the back human body model share unified body shape parameters.
28. The apparatus according to claim 26, wherein the reconstruction unit is further configured to: Determining the orientation of a human body in the depth image according to the two-dimensional key points; Determine the point cloud data portion corresponding to the depth image facing the depth camera in the human body point cloud model as a front point cloud model; A point cloud data portion corresponding to the depth image with the back side facing the depth camera in the human body point cloud model is determined as a back side point cloud model.
29. The apparatus according to claim 28, wherein the reconstruction unit is further configured to: In response to detecting that the left hand key point of the human body is on the right side and the right hand key point is on the left side, determining the depth image with the longest distance between the left hand key point and the right hand key point as facing the depth camera; In response to detecting that the left hand keypoint of the human body is on the left and the right hand keypoint is on the right, the depth image with the longest distance between the left hand keypoint and the right hand keypoint is determined as facing the back side of the depth camera.
30. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.
31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-14.
32. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.
