System and method for obtaining a three-dimensional body model

Abstract

The present disclosure relates to a system and method for obtaining a three-dimensional human body model. Human body model recovery can be implemented with pre-trained artificial neural networks. A first neural network can be trained to determine body key points based on a human body image. A second neural network can be trained to predict pose parameters associated with the human body based on the body key points. A third neural network can be trained to predict body shape parameters associated with the human body based on a human body depth image. A 3D human body model can then be generated based on the pose and body shape parameters predicted by the second and third neural networks, respectively. The training of the second neural network can be performed using synthetically generated body key points, and the training of the third neural network can be performed using normal maps. The pose and body shape parameters predicted by the second and third neural networks can be further optimized through an iterative optimization process.

Description

Technical Field

[0001] This application relates to the field of medical applications, specifically to models related to human posture. Background Technology

[0002] Human body models (such as 3D human body models, e.g., meshes) that realistically reflect the shape and pose of an individual patient can be used in a variety of medical applications, including patient localization, surgical navigation, and unified medical record analysis. For example, in radiotherapy and medical imaging, success often depends on the ability to position and maintain the patient in the desired pose so that treatment or scans can be performed with precision and accuracy. Real-time knowledge of the individual patient's physical characteristics (such as shape and pose) in these situations can be beneficial, including, for example, faster and more accurate patient localization based on scan or treatment protocols, and more consistent results. In other example cases, such as during surgery, information about the individual patient's shape can provide insight and guidance for both treatment planning and execution. This information can, for example, be used for localization and navigation around the patient's treatment site. When presented in real-time visualization, this information can also allow for monitoring of the patient's status during the procedure.

[0003] Existing deep learning-based techniques for human model reconstruction (HMR) require annotated 3D training data, which is difficult (if not impossible) to obtain. The results produced by these techniques also lack accuracy, especially when parts of the target human body are covered, obscured, or otherwise invisible. Therefore, there is a strong desire for 3D HMR systems and methods capable of accurately reconstructing 3D human models, even without 3D annotations and even when the target human body is not fully visible. Summary of the Invention

[0004] This document describes systems, methods, and apparatuses associated with reconstructing three-dimensional (3D) human body models based on one or more images of the human body (e.g., two-dimensional (2D) images). The system, method, and / or apparatus may utilize one or more processors configured to: determine human body keypoints using a first neural network at least based on a first human body image (e.g., a color image); determine at least a plurality of first parameters associated with human body pose using a second neural network based on the human body keypoints determined by the first neural network; and generate a three-dimensional (3D) human body model representing at least the human body pose based on at least the plurality of first parameters. The second neural network may be trained by a training process comprising: providing the second neural network with a first set of body keypoints (e.g., synthetically generated body keypoints); causing the second neural network to predict pose parameters based on the first set of body keypoints; generating a preliminary 3D human body model at least based on the pose parameters predicted by the second neural network; inferring a second set of body keypoints from the preliminary 3D human body model; and adjusting the execution parameters of the second neural network based on the differences between the first and second sets of body keypoints.

[0005] In the example, the systems, methods, and / or apparatuses described herein may also utilize one or more processors to determine, using a third neural network, at least based on a second human body image (e.g., a depth image), a plurality of second parameters associated with human body shape. The one or more processors may then generate the aforementioned 3D human body model based on the plurality of first parameters determined by the second neural network and the plurality of second parameters determined by the third neural network, wherein the 3D human body model may also represent human body shape.

[0006] In the example, one or more processors can be configured to determine a normal map based on a second image and to determine multiple second parameters based on the normal map. The second and third neural networks can be trained (e.g., together) via a training process that includes: causing the second neural network to predict pose parameters associated with a human body based on a first set of body keypoints associated with the human body; causing the third neural network to predict body shape parameters associated with the human body based on a depth image of the human body (e.g., based on a first normal map derived from the depth image); and generating a preliminary 3D human body model based on the pose parameters predicted by the second neural network and the body shape parameters predicted by the third neural network. The training process may also include: inferring the second set of body keypoints and the second normal map at least based on the preliminary 3D human body model; adjusting the execution parameters of the second neural network based on the differences between the first and second sets of body keypoints; and adjusting the execution parameters of the third neural network based on the differences between the first and second normal maps.

[0007] In the example, the system, method, and / or apparatus described herein may also utilize one or more processors to optimize the 3D human model after generating it based on a plurality of first parameters and a plurality of second parameters by: inferring at least a second set of body keypoints and a second normal map from the 3D human model; adjusting the plurality of first parameters based on the difference between the second set of body keypoints and body keypoints determined by a first neural network; adjusting the plurality of second parameters based on the difference between the second normal map and a normal map determined based on a second image; and adjusting the 3D human model based on the adjusted plurality of first parameters and the adjusted plurality of second parameters.

[0008] In the example, the device configured to perform the tasks described herein may include a first sensing device configured to capture a first image of a human body and a second sensing device configured to capture a second image of a human body. Attached Figure Description

[0009] The examples disclosed herein can be understood in more detail from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0010] Figure 1 This is a diagram illustrating an example system for generating 3D human body models using the methods and apparatus disclosed herein.

[0011] Figure 2 This diagram illustrates how a 3D human body model can be reconstructed from a 2D human body image.

[0012] Figure 3 This is a diagram illustrating an example of an iterative process for progressively improving a 3D human body model.

[0013] Figure 4 This is a diagram illustrating example operations that can be associated with training a neural network for body keypoint detection.

[0014] Figure 5 This is a diagram illustrating example operations that can be associated with training a pose regression neural network.

[0015] Figure 6 This is a diagram illustrating example operations that can be associated with training a body shape regression neural network.

[0016] Figure 7 This is a diagram illustrating an example device that can be used to reconstruct a 3D human body model. Detailed Implementation

[0017] The present disclosure is illustrated by way of example rather than limitation in the figures.

[0018] Figure 1A diagram illustrates an example system 100 that can be used to generate human body models such as 3D human body models (e.g., 3D meshes) using the methods and apparatus disclosed herein.

[0019] As shown in the figure, system 100 can be configured to use medical scanner 102 (e.g., computed tomography (CT) scanner, magnetic resonance imaging (MRI) machine, positron emission tomography (PET) scanner, X-ray machine, etc.) to perform medical scanning or imaging procedures, but system 100 can also be adapted to provide other types of healthcare services, including, for example, radiation therapy, surgery, etc.

[0020] System 100 may include one or more sensing devices 104 (e.g., image capture devices) configured to capture images (e.g., 2D images) of a patient 106, such as the patient lying on a scanning bed or treatment bed in front of a medical scanner 102. Sensing devices 104 may include one or more sensors, including one or more cameras (e.g., digital color cameras), one or more red, green, and blue (RGB) sensors, one or more depth sensors, one or more RGB-D sensors, one or more thermal sensors, such as infrared (FIR) or near-infrared (NIR) sensors, etc. Depending on the type of sensor used, the images captured by sensing devices 104 may include, for example, one or more photographs taken by a camera (e.g., one or more 2D color photographs of the patient 106), one or more depth images generated by a depth sensor, etc. In example embodiments, sensing devices 104 may be mounted or placed in various different locations within system 100, and sensing devices 104 may have different viewpoints (e.g., fields of view) facing the patient 106.

[0021] One or more sensing devices 104 may include corresponding processors configured to process images of patient 106 captured by the sensors described herein. Additionally or alternatively, system 100 may include processing device 108 communicatively coupled to sensing device 104 and configured to process images of patient 106 captured by sensing device 104. Processing device 108 may be coupled to sensing device 104, for example, via a communication network 110 (e.g., coupled to sensors included in sensing device 104), which may be a wired or wireless communication network. In response to receiving an image of patient 106, sensing device 104 and / or processing device 108 may analyze the image (e.g., at the pixel level) to determine various body characteristics of patient 106 (e.g., body shape, posture, etc.). For example, in response to obtaining an image of patient 106, sensing device 104 and / or processing device 108 may utilize one or more neural networks to generate (e.g., construct) a human model such as a 3D human model of patient 106 based on the patient's image, as described more fully below with reference to the accompanying drawings. Human models may include parametric models, such as skinned multi-person linear (SMPL) models that indicate the body shape, posture and / or other anatomical features of patient 106.

[0022] The human body model generated by sensing device 104 and / or processing device 108 can be used to facilitate multiple downstream medical applications and services, including, for example, patient positioning, medical protocol design, unified or related diagnosis and treatment, patient monitoring, surgical navigation, etc. For example, processing device 108 can determine, based on the 3D human body model, whether the pose and / or posture of patient 106 meets the requirements of a predetermined protocol (e.g., when patient 106 is standing in front of medical scanner 102 or lying on a scanning table), and (e.g., via display device 112) provide real-time confirmation or adjustment instructions to help patient 106 (or medical scanner 102) enter the desired pose and / or posture. As another example, sensing device 104 and / or processing device 108 can be coupled to a medical record library 114 configured to store patient medical records, including scanned images of patient 106 obtained through other imaging modalities (e.g., CT, MR, X-ray, SPECT, PET, etc.). Processing device 108 can use a 3D human body model as a reference to analyze the medical records of patient 106 stored in library 114 in order to gain a comprehensive understanding of the patient's physical condition. For example, processing device 108 can align scanned images of patient 106 from library 114 with the 3D human body model to allow for the presentation (e.g., via display device 112) and analysis of scanned images with reference to the anatomical features (e.g., body shape and / or posture) of patient 106 as indicated by the 3D human body model.

[0023] Figure 2 A simplified block diagram 200 is shown, illustrating how it can be based on the human body (e.g., Figure 1 One or more images (e.g., 2D images) 202 of the patient 106 are used to reconstruct a 3D human model 216.

[0024] Image 202 may include, for example, a human body color image (e.g., an RGB image) and a depth image (e.g., an image created by...). Figure 1 The corresponding sensing device 104 shown captures the image. As shown, based on a human RGB image, multiple body keypoints 206 can be extracted from the image, for example, using a first neural network 204 (e.g., a body keypoint extraction network). The extracted body keypoints 206 (e.g., in the form of one or more heatmaps representing body keypoints 206) can be provided to a second neural network 208 (e.g., a pose regression network), which can be configured to determine at least multiple pose parameters (e.g., θ) 210 based on the extracted body keypoints 206. In the example, the second neural network 208 can also be configured to determine multiple body shape parameters (e.g., preliminary or baseline body shape parameters, which are not captured by the initial or baseline body shape parameters) based on the extracted body keypoints 206. Figure 2 (as shown in the image). When used in this document, body key points can refer to body parts and / or joints that can indicate human posture and / or skeletal structure. These body key points may include, but are not limited to, for example, the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, etc.

[0025] Further as Figure 2 As shown, from a human body depth image (e.g., a depth map), a third neural network 212 (e.g., a body shape regression network) can be used to determine multiple body shape parameters (e.g., β) 214 based on depth information extracted from the depth image. The depth information can be extracted in the form of a normal map and can be provided to the third neural network 212 to determine the multiple body shape parameters β. When referenced herein, a normal map can include any form of representation that can indicate a direction perpendicular to the human body surface at each pixel of the depth image. For example, a normal map can be generated by determining the corresponding normal vector at each pixel location of the depth image by obtaining a height value associated with the pixel location from the depth image and calculating a vector tangent to the pixel location based on the derivative (e.g., partial derivative) or rate of change of the height value at that pixel location in the X, Y, and Z directions. Once multiple pose parameters θ and multiple body shape parameters β are determined, a 3D human body model 216 can be generated based on the determined pose and body shape parameters, for example by determining multiple vertices associated with an individual body (e.g., 6890 vertices based on 82 body shape and pose parameters), connecting vertices with edges to form polygons (e.g., triangles), connecting multiple polygons to form surfaces, using multiple surfaces to determine 3D shapes, and applying textures and / or shadows to surfaces and / or shapes.

[0026] The first neural network 204, the second neural network 208, and / or the third neural network 212 can be implemented using one or more processors and one or more storage devices. The storage devices can be configured to store instructions that, when executed by one or more processors, cause the processors to implement one or more neural networks (e.g., the first neural network 204, the second neural network 208, and / or the third neural network 212). Each of the first, second, and third neural networks can include a convolutional neural network (CNN) and / or a fully connected neural network (FNN). The one or more processors can include a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a reduced instruction set computer (RISC) processor, an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a physical processing unit (PPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or a combination thereof. The one or more storage devices can include volatile or non-volatile memories, such as semiconductor memories (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.), flash memory, and mass storage devices (e.g., disks, such as internal hard disks, removable disks, magneto-optical disks, CD-ROMs, or DVD-ROMs, etc.).

[0027] Each of the first neural network 204, the second neural network 208, and the third neural network 212 may include multiple layers, such as an input layer, one or more convolutional layers (e.g., followed by corresponding linear or non-linear activation functions), one or more pooling layers, and / or one or more fully connected layers. Each layer may correspond to multiple filters (e.g., kernels) designed to detect (e.g., learn) features that can represent human pose and / or body shape from the input image 202. The filters may be associated with corresponding weights, which, when applied to the input, produce an output indicating whether certain visual features or patterns have been detected. The weights associated with the filters of the neural network may be learned through a corresponding training process, which will be described in more detail below.

[0028] Figure 2 The illustrated deep learning-based techniques can offer many advantages over conventional human model restoration (HRM) systems and methods. For example, conventional HMR systems and methods may require training deep neural networks with 3D annotations, which can be very difficult to obtain and costly. In contrast, Figure 2The illustrated technique divides the HRM task into multiple steps, including, for example, using a first neural network 204 to detect human keypoints based on a human image, and subsequently using a second neural network 206 to infer at least pose parameters of the human body based on the body keypoints detected by the first neural network 204. As will be described in more detail below, the first neural network 204 can be trained using annotated 2D training data, which is easier and more efficient to obtain, and the second neural network 208 can be trained using synthetically generated body keypoints, which can eliminate (e.g., substantially reduce) the need for 3D annotation. Furthermore, the accuracy of the human body model can be improved by using a third neural network 212 to infer body shape parameters based on a human depth image and constructing a 3D human body model 216 based on the body shape and pose parameters obtained using the techniques described herein.

[0029] In the example, the 3D human model generated using the above techniques can be further optimized through an iterative process (e.g., by optimizing modules). This is because separate networks (e.g., ...) can be used. Figure 2 The second neural network 208 and the third neural network 212 are used to estimate the pose parameters and body shape parameters used to construct the 3D human body model, and it can be seen that the body key points and / or body surface normal maps determined based on the 3D human body model may be different from the originally determined body key points (e.g., by...). Figure 2 The first neural network 204 detected) and / or the original derived normal map (e.g., by...). Figure 2 The third neural network (derived from 212) is inconsistent (e.g., may be misaligned), so optimization can be performed. These inaccuracies and / or inconsistencies can be corrected, for example, by iteratively adjusting the pose and body shape parameters associated with the 3D human model after training and / or deploying the neural network. Figure 3 A diagram (e.g., 300) illustrates an example of such an iterative optimization process (e.g., which may be implemented by an optimization module).

[0030] like Figure 3 As shown, the iterative optimization process may include one or more of the following steps. From a 3D human model 316 generated using the techniques described herein (e.g., Figure 2 From the 3D human body model 216, a set of body key points 302 and a normal map 308 associated with the human body surface can be inferred (e.g., consistent with the viewpoint of the sensing device or camera used to generate the 3D human body model). The inferred body key points 302 can be compared with the originally extracted body key points (e.g., Figure 2The difference 304 between the body keypoints 206 is calculated. Similarly, the difference 310 between the inferred normal map 308 and the original normal map used to construct the 3D human body model can also be calculated. Then, the pose parameters (e.g., θ) 306 and / or body shape parameters (e.g., β) 312 can be adjusted based on the calculated differences 304 and 310 (e.g., adjusting a predetermined amount or step size), and the adjusted pose parameters 306 and / or adjusted body shape parameters 312 can be used to generate an adjusted (e.g., more optimized) 3D human body model 314. The adjusted 3D human body model 314 can then be used as the starting point for the next optimization iteration.

[0031] The optimization process described above can be performed on multiple iterations (e.g., after the deployment of the neural network) until the 3D human model 314 meets certain criteria. For example, the criteria can be considered met if the differences 304 and 310 fall below a threshold or if a predetermined number of iterations have been completed. The number of iterations may depend on the initial state of the pose and body shape parameters, the amount of adjustment made during each iteration (e.g., step size), etc. Progressive improvements made on multiple iterations can lead to a 3D human model that more realistically reflects (e.g., better fits) the pose and body shape of the individual depicted in the input image.

[0032] The neural networks described in this article (e.g., Figure 2 The first neural network 204, the second neural network 208, and / or the third neural network 212 can be trained together, for example, in an end-to-end manner, or the neural networks can be based on, for example, a... Figure 2 The corresponding inputs and outputs of the illustrated neural network are trained separately.

[0033] Figure 4 This illustrates how a body keypoint detection neural network can be trained using annotated 2D training images (e.g., RGB images). Figure 2The flowchart illustrates example operations associated with the first neural network (204). Operations can begin at 402 with the initialization of network parameters (e.g., weights associated with the various layers of the neural network). For example, the neural network parameters can be initialized by sampling from one or more probability distributions or based on parameter values ​​from another neural network with a similar architecture. At 404, the body keypoint detection neural network can receive a 2D training image, such as an RGB image, as input, and at 406, the body keypoint detection neural network can extract features from the 2D training image and predict a set of body keypoints based on the extracted features. The body keypoint detection neural network can then (e.g., at 408) compare the predicted body keypoints with annotated body keypoints corresponding to the input 2D training image and determine a loss associated with the prediction. This loss can be determined using a suitable loss function, such as a loss function configured to compute the mean squared error (MSE), L1 norm, L2 norm, etc., between the predicted body keypoints and the annotated body keypoints (e.g., the gold standard). At 410, a determination can be made regarding whether one or more training termination criteria are met. For example, a training termination criterion can be met if a predetermined number of training iterations have been completed, or if the change in the loss value between two training iterations is less than a predetermined threshold. If it is determined at 410 that the training termination criterion is not met, then at 412 the loss determined at 408 can be used to adjust the parameters (e.g., weights) of the body keypoint detection neural network, for example, by backpropagating the loss (e.g., gradient descent of the loss) through the neural network. Training can then return to 406. If it is determined at 410 that the training termination criterion is met, then training can end.

[0034] Figure 5 This illustrates how pose regression neural networks can be used (e.g., Figure 2 The flowchart illustrates an example operation associated with training the second neural network 208. As shown, the training operation can begin at 502 with the initialization of network parameters (e.g., weights associated with the various layers of the neural network). The neural network parameters can be initialized, for example, by sampling the neural network parameters from one or more probability distributions or based on parameter values ​​from another neural network with a similar architecture. At 504, the pose regression neural network can receive training data including synthetically generated body keypoints. These keypoints can be obtained using various techniques, such as via computer simulation, by manual labeling, or using a pre-trained body keypoint detection neural network (e.g., [missing information]). Figure 2The first neural network 204, etc. For example, a parametric 3D model of a human can be obtained, and multiple keypoints (e.g., joints) can be defined on the 3D model. The 3D model (along with the defined keypoints) can be used (e.g., using a suitable computer graphics engine) to render digital 3D human bodies with different poses and / or body shapes. The keypoints on these digital 3D human bodies can then be projected onto corresponding 2D images and used as gold standard annotations for training pose regression neural networks.

[0035] In 506, a pose regression neural network can predict multiple parameters associated with human pose and / or a preliminary set of parameters associated with human body shape based on received body keypoints. The pose regression neural network can also estimate a 3D human model, for example, based at least on pose parameters and body shape parameters predicted by the pose regression neural network, or at least on pose parameters predicted by the pose regression neural network and parameters derived from, for example... Figure 2 The third neural network 212, a body shape regression neural network, predicts a set of body shape parameters (e.g., if both neural networks are trained together). Subsequently (e.g., at 508), the pose regression neural network can infer a set of body keypoints from the estimated 3D human model and can compare the inferred body keypoints with body keypoints included in the training data (e.g., at 510) to determine the loss associated with the pose parameter prediction and / or model estimation. The loss can be determined, for example, using a suitable loss function, such as a loss function based on the MSE, L1 norm, L2 norm, etc., between the inferred body keypoints and the body keypoints included in the training data. At 512, the pose regression neural network can determine whether one or more training termination criteria are met. For example, the pose regression neural network can determine that the training termination criteria are met if the pose regression neural network has completed a predetermined number of training iterations, or if the change in the calculated loss between two training iterations has fallen below a predetermined threshold. If the training termination criterion is not met as determined in 512, then in 514, the pose regression neural network can adjust its execution parameters (e.g., weights) based on the loss determined in 510, for example, by backpropagating the loss (e.g., gradient descent of the loss) through the neural network. Training can then return to 506. If the training termination criterion is met as determined in 512, then training can end.

[0036] As can be seen from the above operations, pose regression neural networks (e.g., Figure 2The training of the second neural network (208) can utilize synthetically generated body keypoints and is performed by having the neural network predict at least one set of pose parameters and / or a 3D human model based on the synthetically generated body keypoints. Furthermore, the performance of the neural network (e.g., execution parameters) can be evaluated and optimized by inferring body keypoints from the predicted 3D human model and forcing the neural network to minimize the differences between the inferred body keypoints and those included in the training data. Therefore, the training of the pose regression neural network can be performed in an unsupervised manner and / or without requiring annotated 3D data, thereby allowing deep learning-based techniques to be applied to human model recovery.

[0037] Figure 6 This illustrates what can be done with a body shape regression neural network (e.g., Figure 2 The flowchart illustrates an example operation associated with the training of the third neural network (212). As shown, the training operation can begin at 602 with the initialization of network parameters (e.g., weights associated with the various layers of the neural network). The neural network parameters can be initialized, for example, by sampling the neural network parameters from one or more probability distributions or based on the parameter values ​​of another neural network with a similar architecture. At 604, the body shape regression neural network can receive a human depth image and can extract depth information associated with the human surface from the depth image. For example, the body shape regression neural network can determine the corresponding normal vector at each pixel location in the depth image by obtaining the height value associated with the pixel location from the depth image and calculating the vector tangent to the pixel location based on the derivative (e.g., partial derivative) or rate of change of the height value at that pixel location (e.g., in the X, Y, and Z directions). The body shape regression neural network can then generate a normal map corresponding to the depth image based on the corresponding normal vectors associated with the pixels of the depth image.

[0038] In 606, a body shape regression neural network can predict multiple parameters associated with human body shape based on a normal map derived from an input depth image. The body shape regression neural network can be based at least on the body shape parameters predicted by the body shape regression neural network and can use parameters such as... Figure 2The pose regression neural network of the second neural network 208 predicts pose parameters to further estimate the 3D human model (e.g., if both neural networks are trained together). Subsequently (e.g., at 608), the body shape regression neural network can infer a normal map from the estimated 3D human model and can compare the inferred normal map with the normal map determined at 604 (e.g., at 610) to determine the loss associated with the body shape parameter prediction and / or model estimation. The loss can be determined, for example, using a suitable loss function, such as a function based on the MSE, L1 norm, L2 norm, etc., between the inferred normal map and the normal map derived at 606. The body shape regression neural network can then determine at 612 whether one or more training termination criteria are met. For example, if the body shape regression neural network has completed a predetermined number of training iterations, or if the body shape regression neural network determines that the change in the calculated loss between two training iterations has fallen below a predetermined threshold, the body shape regression neural network can determine that the training termination criteria are met. If the training termination criterion is not met as determined in 612, then in 614, the body shape regression neural network can adjust its execution parameters (e.g., weights) based on the loss determined in 610, for example, by backpropagating the loss (e.g., gradient descent of the loss) through the neural network. Training can then return to 606. If the training termination criterion is met as determined in 612, then training can end.

[0039] In the examples, training of all or a subset of the neural networks described herein (e.g., first neural network 204, second neural network 208, and / or third neural network 212) can also be combined (e.g., at least partially) to further optimize the parameters of the neural networks. For example, in accordance with Figure 5 Training pose regression neural networks (e.g., Figure 2 The second neural network 208) and according to Figure 6 Training a body shape regression neural network (e.g., Figure 2When using the third neural network (212), the pose parameters predicted by the pose regression neural network (e.g., in 506) and the body shape parameters predicted by the body shape regression neural network (e.g., in 606) can be used together to estimate a 3D human body model. A set of body keypoints and normal maps can then be inferred from the estimated 3D human body model and compared with the corresponding body keypoints and normal maps used to generate the 3D human body model to determine the corresponding losses associated with the predictions made by the pose regression neural network and the body shape regression neural network. The losses can be determined using a loss function as described herein and can be backpropagated through the neural network (e.g., individually or as a combined loss, such as an average loss) to adjust the corresponding parameters of the neural network, for example, with the aim of minimizing the differences between the inferred body keypoints and the originally determined body keypoints and / or the differences between the inferred normal maps and the originally determined normal maps.

[0040] For the sake of simplicity, the system (e.g., Figure 1 The operations of System 100 are depicted and described herein in a specific order. However, it should be understood that these operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations capable of being performed by the system are depicted in the accompanying drawings and are not described herein. It should also be noted that not all illustrated operations require performance by the system.

[0041] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 7 This is a block diagram illustrating an example device 700 that can be configured to reconstruct a human model according to one or more embodiments described herein. As shown, device 700 may include a processor (e.g., one or more processors) 702, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 700 may also include communication circuitry 704, memory 706, mass storage device 708, input device 710, and / or communication link 712 (e.g., communication bus) through which one or more components shown in the figure may exchange information.

[0042] Communication circuitry 704 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 706 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 702 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 708 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 702. The input device 710 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from the device 700.

[0043] It should be noted that device 700 can operate as a standalone device or can be connected to other computing devices (e.g., networked or clustered) to perform the functions described herein. And even in Figure 7 Only one example of each component is shown in the figure, and those skilled in the art will understand that the device 700 may include multiple instances of one or more components shown in the figure.

[0044] The methods, components, and features described herein may be implemented by discrete hardware components or integrated into the functionality of other hardware components such as ASICs, FPGAs, DSPs, or similar devices. Alternatively, the methods, components, and features may be implemented by firmware modules or functional circuitry within a hardware device. Furthermore, the methods, components, and features may be implemented as any combination of hardware devices and computer program components, or as a computer program.

[0045] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data representing physical (e.g., electronic) quantities within the registers and memories of the computer system into other data representing physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0046] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art after reading and understanding the above description.

Claims

1. An apparatus for obtaining a three-dimensional human body model, comprising: One or more processors, which are configured as follows: The first neural network is used to determine the key points of the human body based at least on a first image of the human body; A second neural network is used to determine at least a plurality of first parameters associated with the human posture based on the human body key points determined by the first neural network; as well as A 3D human body model representing at least the human body's posture is generated based on at least the plurality of first parameters; The one or more processors are further configured to use a third neural network to determine a plurality of second parameters associated with the human body shape based at least on a second human body image, including: the one or more processors being configured to determine a normal map based on the second image and to determine the plurality of second parameters based on the normal map; wherein the 3D human body model is generated based on the plurality of first parameters determined by the second neural network and the plurality of second parameters determined by the third neural network, and wherein the 3D human body model also represents the human body shape; After generating the 3D human body model based on the plurality of first parameters and the plurality of second parameters, the one or more processors are further configured to optimize the 3D human body model by at least the following operations: inferring a second set of body key points and a second normal map from the 3D human body model; adjusting the plurality of first parameters based on the difference between the second set of body key points and the body key points determined by the first neural network; adjusting the plurality of second parameters based on the difference between the second normal map and the normal map determined based on the second image; and adjusting the 3D human body model based on the adjusted plurality of first parameters and the adjusted plurality of second parameters.

2. The device according to claim 1, wherein, The second neural network is trained through a training process, which includes: The second neural network is provided with a first set of body key points; The second neural network is then used to predict pose parameters based on the first set of body key points. A preliminary 3D human body model is generated based at least on the pose parameters predicted by the second neural network; Inferring a second set of key body points from the preliminary 3D human model; and The execution parameters of the second neural network are adjusted based on the differences between the first set of body key points and the second set of body key points.

3. The device according to claim 2, wherein, The first set of body key points was synthesized.

4. The device according to claim 1, wherein, The second neural network and the third neural network are trained through a training process, which includes: The second neural network is used to predict posture parameters associated with the human body based on a first set of body key points associated with the human body. The third neural network is configured to predict body shape parameters associated with the human body based on a depth image of the human body, wherein a first normal map is derived from the depth image and used to predict the body shape parameters; A preliminary 3D human body model is generated based on the pose parameters predicted by the second neural network and the body shape parameters predicted by the third neural network. The second set of body key points and the second normal map are inferred based at least on the preliminary 3D human body model. The execution parameters of the second neural network are adjusted based on the differences between the first set of body key points and the second set of body key points; and The execution parameters of the third neural network are adjusted based on the difference between the first normal map and the second normal map.

5. The device according to claim 1, wherein, The first image includes the color image of the human body, and the second image includes the depth image of the human body.

6. The device of claim 5, further comprising a first sensing device configured to capture the first image of the human body and a second sensing device configured to capture the second image of the human body.

7. A method for obtaining a three-dimensional human body model, the method comprising: The first neural network is used to determine the key points of the human body based at least on a first image of the human body; A second neural network is used to determine at least a plurality of first parameters associated with the human posture based on the human body key points determined by the first neural network; as well as A 3D human body model representing at least the human body's posture is generated based on at least the plurality of first parameters; Using a third neural network to determine a plurality of second parameters associated with human body shape based at least on a second human body image, comprising: one or more processors being configured to determine a normal map based on the second image and to determine the plurality of second parameters based on the normal map; wherein the 3D human body model is generated based on the plurality of first parameters determined by the second neural network and the plurality of second parameters determined by the third neural network, and wherein the 3D human body model also represents the human body shape; After generating the 3D human body model based on the plurality of first parameters and the plurality of second parameters, the 3D human body model is optimized by at least the following operations: inferring a second set of body key points and a second normal map from the 3D human body model; adjusting the plurality of first parameters based on the difference between the second set of body key points and the body key points determined by the first neural network; adjusting the plurality of second parameters based on the difference between the second normal map and the normal map determined based on the second image; and adjusting the 3D human body model based on the adjusted plurality of first parameters and the adjusted plurality of second parameters.

Images