Human body three-dimensional model construction method, device, robot and storage medium

Abstract

The present application relates to a method, device, robot and storage medium for constructing a three-dimensional model of a human body. The method comprises: obtaining a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information; obtaining a three-dimensional model to be calibrated of the object using a skinned multi-person linear model (SMPL) according to the two-dimensional image; obtaining a skeleton model of the object containing depth information according to the two-dimensional image and the depth information; and performing depth calibration on the three-dimensional model to be calibrated according to the skeleton model containing depth information to obtain a target three-dimensional model. The method can improve the accuracy of three-dimensional model construction.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, robot and storage medium for constructing a three-dimensional human body model. Background Art

[0002] Three-dimensional model reconstruction is a task in computer graphics and computer vision. Specifically, three-dimensional human body reconstruction aims to automatically generate a human body surface model based on human body images through algorithms to accurately restore the human body's position, posture, surface shape and other geometric information in the three-dimensional world.

[0003] In the current 3D model construction method, features are extracted from images and the model is constructed based on two-dimensional projection fitting. However, objects at different positions, postures or surfaces may present the same two-dimensional projection when they are at different distances from the camera, resulting in depth errors in the fitted 3D model, affecting the accuracy of the constructed 3D model. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, robot and storage medium for constructing a three-dimensional model of the human body that can improve the accuracy of three-dimensional model construction in order to address the above technical problems.

[0005] A method for constructing a three-dimensional human body model, the method comprising:

[0006] Obtaining a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information;

[0007] According to the two-dimensional image, a skinned multi-person linear model (SMPL) is used to obtain a three-dimensional model of the object to be calibrated;

[0008] Obtaining a skeleton model of the object including depth information according to the two-dimensional image and the depth information; and

[0009] The depth of the 3D model to be calibrated is calibrated according to the skeleton model containing depth information to obtain a target 3D model.

[0010] A device for constructing a three-dimensional model of a human body, the device comprising:

[0011] An image acquisition module, used to acquire a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information;

[0012] The model to be calibrated module is used to obtain the 3D model to be calibrated of the object according to the 2D image by using a skinned multi-person linear model (SMPL);

[0013] A depth information processing module, used to obtain a skeleton model of the object including depth information according to the two-dimensional image and the depth information; and

[0014] The depth calibration module is used to perform depth calibration on the 3D model to be calibrated according to the skeleton model containing depth information to obtain the target 3D model.

[0015] A robot comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0016] Obtaining a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information;

[0017] According to the two-dimensional image, a skinned multi-person linear model (SMPL) is used to obtain a three-dimensional model of the object to be calibrated;

[0018] Obtaining a skeleton model of the object including depth information according to the two-dimensional image and the depth information; and

[0019] The depth of the 3D model to be calibrated is calibrated according to the skeleton model containing depth information to obtain a target 3D model.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0021] Obtaining a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information;

[0022] According to the two-dimensional image, a skinned multi-person linear model (SMPL) is used to obtain a three-dimensional model of the object to be calibrated;

[0023] Obtaining a skeleton model of the object including depth information according to the two-dimensional image and the depth information; and

[0024] The depth of the 3D model to be calibrated is calibrated according to the skeleton model containing depth information to obtain a target 3D model.

[0025] The above-mentioned human body three-dimensional model construction method, device, robot and storage medium obtain the three-dimensional model to be calibrated through a skinned multi-person linear model based on the two-dimensional image of the object containing the three-dimensional model to be constructed, obtain a skeleton model containing depth information based on the two-dimensional image of the object and the corresponding depth information, and perform depth calibration on the three-dimensional model to be calibrated based on the skeleton model containing the depth information to obtain a target three-dimensional model corresponding to the object. In the process of constructing a three-dimensional model for a human body object, the depth of the three-dimensional model to be calibrated is calibrated based on the skeleton model including the depth information of the object, thereby reducing the depth error of the three-dimensional model and improving the accuracy of the three-dimensional model construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A diagram of an application environment of a method for constructing a three-dimensional human body model in one embodiment;

[0027] Figure 2 A schematic diagram of a process of constructing a three-dimensional human body model in one embodiment;

[0028] Figure 3 is a schematic diagram of relative depth error in one embodiment;

[0029] Figure 4 A schematic diagram of a flow chart of a method for constructing a three-dimensional human body model in another embodiment;

[0030] Figure 5 is a schematic diagram of absolute depth error in one embodiment;

[0031] Figure 6 is a schematic diagram of an uncalibrated three-dimensional human body model in one embodiment;

[0032] Figure 7 In one embodiment Figure 6 Schematic diagram of the relative depth calibration of the three-dimensional human body model;

[0033] Figure 8 A schematic diagram of a three-dimensional human body mesh model constructed after correction in one embodiment;

[0034] Fig. 9 A structural block diagram of a device for constructing a three-dimensional human body model in one embodiment;

[0035] Fig.10 1 is a diagram showing the internal structure of a robot in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] The method for constructing a three-dimensional human body model provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 can communicate with the server 104 through the network. The terminal 102 can shoot the human object to obtain a two-dimensional image of the human body containing the three-dimensional model to be constructed, such as an RGB (RGB color mode) image. In some embodiments, an image containing only grayscale information can also be used. The terminal 102 sends the two-dimensional image of the human body to the server 104. The server 104 obtains the three-dimensional model to be calibrated based on the two-dimensional image using a skinned multi-person linear model, obtains a skeleton model containing depth information based on the two-dimensional image of the object and the corresponding depth information, and performs depth calibration on the three-dimensional model to be calibrated based on the skeleton model containing depth information to obtain a target three-dimensional model corresponding to the object. The server 104 can display the target three-dimensional model by connecting to a display device, or feed it back to the terminal 102 for display. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, portable wearable devices and other devices that can store pictures. The terminal 102 can also be an electronic device that can shoot, such as a camera, a video camera, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers. The server 104 can display the constructed target three-dimensional model by connecting to a display device.

[0038] In one embodiment, Figure 2 As shown, a method for constructing a three-dimensional human body model is provided. Figure 1 The server in the example is used to illustrate the following steps:

[0039] Step S202: Acquire a two-dimensional image of an object including a three-dimensional model to be constructed and corresponding depth information.

[0040] Among them, the two-dimensional image can be a picture taken by a camera containing an object to be constructed of a three-dimensional model, and the object to be constructed of a three-dimensional model is an object corresponding to the construction of the three-dimensional model, and can specifically be a human body, an object, etc. The two-dimensional image can be a picture of different channel formats, such as an RGB picture, which is a picture encoded by a method of describing the color of a pixel point by the relative intensity of three basic colors (red, green, and blue). The two-dimensional image is a plane image that does not contain depth information. Specifically, the two-dimensional image can be obtained by photographing the corresponding human object by the terminal, and sent to the server for human body three-dimensional model construction processing. For the construction of a three-dimensional model during the movement of the object, the number of two-dimensional images can be multiple frames, and each frame of the two-dimensional image has a corresponding time sequence relationship, corresponding to the shooting time sequence of the two-dimensional image, and based on multiple frames of time-series two-dimensional images, the movement process of the object can be constructed into a three-dimensional model to obtain the movement process of the corresponding three-dimensional model of the object.

[0041] Step S204: According to the two-dimensional image, a skinned multi-person linear model (SMPL) is used to obtain a three-dimensional model of the object to be calibrated.

[0042] The server constructs a three-dimensional model based on the obtained two-dimensional image, and specifically uses the Skinned Multi-Person Linear model (SMPL) to obtain the object's calibrated three-dimensional model. Among them, SMPL is a mathematical model that can output human mesh and skeleton of different body shapes in different postures. It can describe human bodies of different body shapes and postures with 10 body shape parameters and 72 posture parameters; the Mesh model is a human mesh model, which is a model that collects sample points on the surface of an object and describes the geometric shape of the surface of the object with the link relationship between adjacent sample points and the three-dimensional coordinates of the sample points. The body shape parameters represent the proportions of a person's height, weight, head-to-body ratio, etc.; the posture parameters can represent the overall motion posture of the human body and 75 parameters of the relative angles of 24 joints. According to the posture parameters, the Skeleton model, that is, the human skeleton model, can be obtained, which is the human skeleton obtained by connecting the key points of the human body (such as the calf, thigh, upper arm, lower arm, etc.). The human skeleton model can reflect the movements and postures of the human body.

[0043] Among them, the three-dimensional model to be calibrated is obtained based on the two-dimensional image of the object using a skinned multi-person linear model, and the two-dimensional image includes the two-dimensional information of the object. The posture of the three-dimensional model to be calibrated may have a depth error with the real posture of the object, and it needs to be calibrated in combination with the depth information of the object to ensure the accuracy of the constructed three-dimensional model.

[0044] Step S206: obtaining a skeleton model of the object including depth information according to the two-dimensional image and the depth information.

[0045] Among them, the depth information can characterize the geometric shape of the visible surface of the object. The depth information can specifically be the distance from the object at a specific coordinate to the object closest to the camera plane to the plane in the field of view of the depth sensor, that is, relative to the two-dimensional image captured by the camera, the depth information of the object can reflect the distance characteristics between the object and the camera. The depth information can be determined from the depth image of the object. The depth image is also called a distance image, which refers to an image that uses the distance (depth) from the image collector to each point in the scene as a pixel value. The depth image can be taken by a depth camera, such as Intel Realsense, and can be specifically obtained based on a laser radar depth imaging method, computer stereoscopic vision imaging, a coordinate measuring machine method, a moiré fringe method, a structured light method, and the like. The skeleton model is a model that reflects the human skeleton obtained by connecting the key points of the object. For example, when the object is a human body, the key points can be various joints of the human body and points on the human body such as eyes and nose that transmit language information; for example, it can specifically be the right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, top of the head and neck, etc. A skeleton model of the object containing depth information is obtained according to the two-dimensional image and the depth information. The skeleton model carries the depth information of the object, that is, each key point in the skeleton model carries a depth feature, which can be the three-dimensional coordinates of each key point in space.

[0046] In this embodiment, the server obtains the depth information of the object, which can be specifically extracted from the depth image of the object. The depth image can be captured simultaneously with the two-dimensional image by a depth camera. A skeleton model containing the depth information of the object is obtained based on the two-dimensional image of the object and the corresponding depth information. Specifically, the depth information of each key point of the object can be determined, and the key points are connected based on the depth information of each key point to obtain the skeleton model carrying the depth information of the object.

[0047] Step S208 , performing depth calibration on the 3D model to be calibrated according to the skeleton model containing depth information to obtain a target 3D model.

[0048] After obtaining the three-dimensional model to be calibrated and the skeleton model containing the depth information of the object, the server performs depth calibration on the three-dimensional model to be calibrated based on the skeleton model containing the depth information to obtain the target three-dimensional model after depth calibration. Specifically, the server can first determine the distance between the skeleton model containing the depth information and the three-dimensional model to be calibrated. The distance can be determined based on the distance between the corresponding vectors composed of the corresponding key points in the skeleton model and the three-dimensional model to be calibrated. The distance between the skeleton model and the three-dimensional model to be calibrated reflects the depth error between the two models, specifically the relative depth error. Among them, the relative depth error is the relative depth error caused by the fact that when the three-dimensional model of the object is fitted by a two-dimensional image, different actual postures of the object can correspond to the same two-dimensional image projection, that is, different two-dimensional images may be fitted to the same three-dimensional model, or the same two-dimensional image may be fitted to different three-dimensional models.

[0049] like Figure 3 As shown in the figure, in the construction of a human body three-dimensional model, when the traditional projection based on the projection of the human body two-dimensional image fits the human body contour, the two-dimensional RGB image cannot provide the depth information of the human body surface points, and the coordinates of different parts of the human body in the generated model relative to other parts on the z-axis (i.e., the depth direction) may have errors. For example, the hands, feet, and limbs may present the same projection in the two-dimensional image at different depths (from the camera). In the figure, A and D are different human body postures, A is the front posture, and D is the back posture. The projections of the corresponding two-dimensional images of A and D are both B, and the two-dimensional skeleton models are both C. When the projection of the human body image is B and the skeleton model is C, it is impossible to accurately fit A or D, which brings relative depth errors and affects the accuracy of the three-dimensional model construction.

[0050] In this embodiment, the server performs depth calibration on the three-dimensional model to be calibrated using a skeleton model containing depth information, thereby calibrating the relative depth error of the three-dimensional model to be calibrated, and obtaining a target three-dimensional model with the relative depth error eliminated. The target three-dimensional model can effectively reflect the three-dimensional features of the object, such as posture, movement, etc., thereby ensuring the accuracy of the target three-dimensional model.

[0051] In the above-mentioned human body three-dimensional model construction method, a three-dimensional model to be calibrated is obtained through a skinned multi-person linear model based on a two-dimensional image of an object containing a three-dimensional model to be constructed, a skeleton model containing depth information is obtained based on the two-dimensional image of the object and the corresponding depth information, and depth calibration is performed on the three-dimensional model to be calibrated based on the skeleton model containing the depth information to obtain a target three-dimensional model corresponding to the object. In the process of constructing a three-dimensional model for a human body object, the depth of the three-dimensional model to be calibrated is calibrated based on the skeleton model including the depth information of the object, thereby reducing the depth error of the three-dimensional model and improving the accuracy of the three-dimensional model construction.

[0052] In one embodiment, Figure 4 As shown, a method for constructing a three-dimensional human body model is provided, comprising:

[0053] Step S402: Acquire a two-dimensional image of an object including a three-dimensional model to be constructed and corresponding depth information.

[0054] The process of obtaining the two-dimensional image of the object containing the three-dimensional model to be constructed in step S402 is similar to step S202, and will not be repeated here. As for the process of obtaining the depth information corresponding to the two-dimensional image of the object containing the three-dimensional model to be constructed, the server can specifically obtain the depth image corresponding to the two-dimensional image, and the depth image and the two-dimensional image are obtained by shooting at the same time. Specifically, the object can be shot by a depth camera to obtain the two-dimensional image and the depth image at the same time, and the depth image includes the depth information of the object.

[0055] Step S404: According to the two-dimensional image, a skinned multi-person linear model (SMPL) is used to obtain a three-dimensional model of the object to be calibrated.

[0056] Step S404 is similar to step S204 and will not be described again.

[0057] Step S406: obtaining a skeleton model of the object including depth information according to the two-dimensional image and the depth information.

[0058] In this embodiment, step S406 specifically includes: determining the two-dimensional coordinates and confidence levels of multiple key points of the object based on the two-dimensional image; taking each key point whose confidence level is greater than a preset confidence threshold as a valid key point; determining the three-dimensional coordinates of each valid key point based on the two-dimensional coordinates and depth image of each valid key point; and obtaining a skeleton model of the object containing depth information based on the three-dimensional coordinates of the valid key points.

[0059] When constructing a skeleton model containing depth information in this embodiment, the server determines the depth information of each effective key point of the object based on the depth image and the two-dimensional image of the object containing the three-dimensional model to be constructed, and constructs a skeleton model containing depth information corresponding to the object based on the depth information of each effective key point. Specifically, when constructing the skeleton model of the object, the server determines the corresponding two-dimensional coordinates and confidences of multiple key points of the object based on the two-dimensional image. The two-dimensional image is a two-dimensional color image, such as an RGB image. Based on the two-dimensional image, the two-dimensional information of each key point of the object in the two-dimensional image, such as the two-dimensional coordinates and the corresponding confidence, can be determined. The confidence reflects the reliability of the two-dimensional coordinates corresponding to each key point. The higher the confidence, the higher the reliability of the two-dimensional coordinates, that is, the more accurate the two-dimensional coordinates of the key point. Specifically, the key point recognition of the object can be performed on the two-dimensional image to identify multiple key points of the object from the two-dimensional image, and based on the position of each key point in the two-dimensional image, the corresponding two-dimensional information of the multiple key points is obtained, specifically including the two-dimensional coordinates and confidence of the multiple key points. In specific implementation, the two-dimensional coordinates and confidence of multiple key points of the object in the two-dimensional image can be determined based on the AlphaPose algorithm. Based on the confidence of each key point, the server regards each key point with a confidence greater than a preset confidence threshold as a valid key point, that is, a key point with high confidence and high reliability of two-dimensional coordinates as a valid key point. Among them, the confidence threshold can be pre-set according to actual needs. The higher the confidence of the valid key point, the higher the accuracy of its two-dimensional coordinates. The subsequent three-dimensional model construction based on the valid key point can ensure the accuracy of the three-dimensional model construction.

[0060] Furthermore, the server can determine the depth information corresponding to each valid key point, such as the three-dimensional coordinates of each valid key point, based on the two-dimensional information corresponding to each valid key point, such as the two-dimensional coordinates, and the depth image. Specifically, each valid key point determined from the two-dimensional image can be mapped to the position corresponding to the depth image to obtain the three-dimensional coordinates corresponding to each valid key point. Based on the three-dimensional coordinates corresponding to each valid key point, a skeleton model containing the depth information of the object is obtained. For example, each valid key point can be connected based on the corresponding three-dimensional coordinates to obtain a skeleton model containing the depth information, and the skeleton model contains the real depth information of the object.

[0061] In this embodiment, the three-dimensional coordinates corresponding to each effective key point of the object are determined through the two-dimensional image of the object and the corresponding depth image, and a skeleton model containing the real depth information of the object is obtained based on the three-dimensional coordinates. Based on the skeleton model, depth calibration can be performed on the three-dimensional model to be calibrated to improve the accuracy of three-dimensional model construction.

[0062] Step S408 , performing depth calibration on the 3D model to be calibrated according to the skeleton model containing depth information to obtain a target 3D model.

[0063] In this embodiment, the 3D model to be calibrated includes an uncalibrated mesh model of the object and a corresponding uncalibrated skeleton model; step S408 performs depth calibration, specifically including:

[0064] Step S4082: the three-dimensional model to be calibrated includes an uncalibrated mesh model of the object and a corresponding uncalibrated skeleton model; and the three-dimensional coordinates of a plurality of comparison key points corresponding to a plurality of valid key points in the uncalibrated skeleton model are obtained.

[0065] In this embodiment, the posture parameters of the object in the uncalibrated mesh model are corrected according to the distance between the uncalibrated skeleton model in the three-dimensional model to be calibrated and the skeleton model containing depth information, so as to realize the depth calibration of the three-dimensional model to be calibrated and obtain the target three-dimensional model. Specifically, the three-dimensional model to be calibrated includes the uncalibrated mesh model of the object and the corresponding uncalibrated skeleton model, the mesh model is constructed by the posture parameter θ and the body shape parameter β, the posture parameter θ represents 75 parameters of the overall motion posture of the human body and the relative angles of 24 joints, and the body shape parameter β represents 10 parameters of the proportion of the height, fatness, thinness, head-to-body ratio, etc. of the human body; the skeleton model can be obtained according to the posture parameter θ, which is a human skeleton (such as calf, thigh, upper arm, lower arm, etc.) obtained by connecting the key points of the human body, and the skeleton model can reflect the movement and posture of the human body. For the uncalibrated skeleton model in the three-dimensional model to be calibrated, it is obtained from the two-dimensional image, the three-dimensional information of each key point in the skeleton model is unreliable, and there is a depth error.

[0066] The server determines a plurality of comparison key points in the uncalibrated skeleton model that correspond to a plurality of valid key points in the skeleton model containing depth information, and determines the three-dimensional coordinates of the plurality of comparison key points. The comparison key points correspond to the valid key points, and specifically can be key points representing the same human body. For example, if the valid key point A is a key point representing the upper arm of the human body in the skeleton model containing depth information, then the comparison key point A' corresponding to the valid key point A is the key point representing the upper arm of the human body in the uncalibrated skeleton model. The three-dimensional coordinates of the plurality of comparison key points are obtained based on the two-dimensional image of the object, lacking reliable depth information, and the three-dimensional coordinates of each comparison key point have depth errors, and depth calibration is required.

[0067] Step S4084, determining the distance between the uncalibrated skeleton model and the skeleton model containing the depth information according to the three-dimensional coordinates corresponding to the multiple valid key points and the multiple comparison key points.

[0068] After obtaining the three-dimensional coordinates of the comparison key points corresponding to each valid key point in the skeleton model containing depth information, the server obtains the distance between the uncalibrated skeleton model and the skeleton model containing depth information based on the three-dimensional coordinates of the multiple valid key points and the three-dimensional coordinates of the corresponding multiple comparison key points. Specifically, the server can combine the multiple valid key points in pairs to obtain multiple first vectors, and determine the first cosine distance between the first vectors; similarly, the server determines the second cosine distance corresponding to the multiple comparison key points, that is, the server combines the multiple comparison key points in pairs to obtain multiple second vectors, and determines the second cosine distance between the second vectors, and then determines the distance between the uncalibrated skeleton model and the skeleton model containing depth information based on the first cosine distance and the second cosine distance.

[0069] Specifically, step S4084 includes: determining multiple first vectors based on at least a part of the multiple valid key points; and determining multiple first cosine distances between the multiple first vectors; determining multiple second vectors corresponding to the multiple first vectors based on at least a part of the multiple comparison key points; and determining multiple second cosine distances between the multiple second vectors; and determining the distance between the uncalibrated skeleton model and the skeleton model containing depth information based on the sum of the squares of the differences between each first cosine distance and the corresponding second cosine distance.

[0070] Among them, the first vector is obtained according to at least a part of the multiple valid key points. For example, a certain number of first vector key points can be selected from each valid key point, and multiple first vector key points are combined in pairs to construct multiple first vectors. After the server constructs multiple first vectors, it determines the first cosine distance between the multiple first vectors. Specifically, the server can pair the multiple first vectors in pairs to form first vector pairs, and calculate the cosine distance of each first vector pair to obtain the first cosine distance between the multiple first vectors. The number of first cosine distances is the same as the number of constructed first vector pairs. Similar to the process of obtaining the first cosine distance according to multiple valid key points, the server obtains the second cosine distances corresponding to multiple comparison key points. Specifically, the server selects the same number of second vector key points as the first vector key points from each comparison key point, and constructs multiple second vectors by combining multiple second vector key points in pairs. The number of second vectors is the same as the number of first vectors. After constructing multiple second vectors, the server determines the second cosine distances between the multiple second vectors. Specifically, the server can pair the multiple second vectors in pairs to form second vector pairs, where the number of second vector pairs is the same as the number of first vector pairs, and calculate the cosine distances of each second vector pair to obtain the second cosine distances between the multiple second vectors, where the number of second cosine distances is the same as the number of constructed second vector pairs.

[0071] After obtaining the first cosine distances between multiple first vectors and the second cosine distances between multiple second vectors, the distance between the uncalibrated skeleton model and the skeleton model containing depth information is obtained based on each first cosine distance and each second cosine distance. Specifically, the server determines the sum of the squares of the distance differences between each first cosine distance and each second cosine distance, and uses the sum of the squares of the distance differences as the distance between the uncalibrated skeleton model and the skeleton model containing depth information.

[0072] Step S4086, with the goal of minimizing the distance between the uncalibrated skeleton model and the skeleton model containing depth information, correct the posture parameters of the object in the uncalibrated mesh model to obtain a target three-dimensional model.

[0073] After obtaining the distance between the uncalibrated skeleton model and the skeleton model containing the depth information, the posture parameters of the uncalibrated mesh model in the three-dimensional model to be calibrated are corrected based on the distance to obtain the posture parameters. Specifically, the server uses the distance between the uncalibrated skeleton model and the skeleton model containing the depth information as the objective function, takes the minimum value of the distance as the goal, and corrects the posture parameters of the object in the uncalibrated mesh model based on the parameters corresponding to the minimum value of the distance to obtain the target three-dimensional model.

[0074] Step S410 , calculating translation parameters between the uncalibrated skeleton model and the skeleton model containing depth information.

[0075] After depth calibration of the 3D model to be calibrated based on the skeleton model containing depth information, the relative depth error in the 3D model to be calibrated can be eliminated, and a target 3D model with the relative depth error overcome can be obtained. The skeleton model containing depth information can be further used to eliminate the absolute depth error in the target 3D model. The absolute depth error is the absolute depth of the 3D model in the camera coordinate system, which is caused by the ambiguity and error of the absolute depth of the 3D model in the camera coordinate system, when the 3D model of the object is fitted by the 2D image, when the object is at any distance from the camera, that is, at any depth, the object contour in the 2D image can be well fitted.

[0076] like Figure 5 As shown, in the traditional human body 3D model construction method, the degree of fit between the 2D projection of the human body model and the real human body contour in the RGB image is maximized. However, only the size of the human body model needs to be adjusted, and the human body contour in the 2D image can be well fitted at any depth (distance from the camera), resulting in ambiguity or error in the absolute depth of the human body model in the camera coordinate system. Figure 5 In the figure, although objects P1 and P2 have different absolute depths, that is, different distances from the camera, they appear to be the same size in the camera. That is, the absolute depth of the object cannot be effectively determined based on the two-dimensional image obtained by the camera, resulting in absolute depth error.

[0077] In this embodiment, for the absolute depth error in the 3D model to be calibrated, the server further performs absolute depth calibration on the obtained target skeleton model through the translation parameters between the skeleton model containing depth information and the uncalibrated skeleton model, thereby calibrating the absolute depth error in the target skeleton model, and obtaining a target 3D model with the object corresponding to the real depth, in which the absolute depth error is eliminated. The target 3D model with the real depth eliminates both the relative depth error and the absolute depth error, and can effectively reflect the 3D features of the object, such as posture, action, etc., thereby ensuring the accuracy of the construction of the 3D model of the human body.

[0078] Specifically, after performing depth calibration on the 3D model to be calibrated based on the skeleton model containing depth information to obtain the target 3D model, the server calculates the translation parameter between the uncalibrated skeleton model and the skeleton model containing depth information. The translation parameter reflects the absolute error between the skeleton model containing depth information and the 3D model to be calibrated, and can be obtained based on the centroid vector obtained from the centroids of the skeleton model containing depth information and the 3D model to be calibrated.

[0079] Further, step S410 includes:

[0080] Step S4102 , respectively determining a first skeleton centroid of the uncalibrated skeleton model and a second skeleton centroid of the skeleton model containing depth information.

[0081] In this embodiment, the target three-dimensional model is absolutely depth calibrated based on the skeleton model containing depth information, specifically, the absolute depth calibration is performed based on the translation vector determined by the center of gravity vector obtained from the respective centers of gravity of the skeleton model containing depth information and the uncalibrated skeleton model, so as to further eliminate the absolute depth error of the target three-dimensional model. Specifically, after the server obtains the target three-dimensional model after relative error calibration, the server determines the first skeleton center of gravity of the uncalibrated skeleton model and the second skeleton center of gravity of the skeleton model containing depth information respectively. The first skeleton center of gravity can be determined based on each key point in the uncalibrated skeleton model, and the second skeleton center of gravity can be determined based on each key point in the skeleton model containing depth information.

[0082] Step S4104, obtaining a translation vector according to the first skeleton centroid and the second skeleton centroid.

[0083] After obtaining the first skeleton center of gravity and the second skeleton center of gravity, the server generates a translation vector according to the first skeleton center of gravity and the second skeleton center of gravity. The translation vector can be used as a translation parameter, which reflects the absolute depth error between the skeleton model containing depth information and the uncalibrated skeleton model. By calibrating the absolute depth error of the target 3D model through the translation vector, the absolute depth error in the target 3D model can be eliminated, thereby improving the accuracy of 3D model construction.

[0084] Step S412: according to the translation parameters, the target three-dimensional model is translated to the real depth in the camera coordinate system to obtain the target three-dimensional model corresponding to the object and including the real depth.

[0085] After obtaining the translation parameters, the server translates the target 3D model to the real depth in the camera coordinate system according to the translation parameters, thereby obtaining a target 3D model corresponding to the object containing the real depth. The target 3D model containing the real depth eliminates the absolute depth error.

[0086] Further, step S412 includes:

[0087] Step S4122: The target three-dimensional model is translated to the real depth in the camera coordinate system according to the translation vector, so as to obtain the target three-dimensional model corresponding to the object and including the real depth.

[0088] Specifically, after obtaining the translation vector according to the first skeleton center of gravity and the second skeleton center of gravity, the server translates the target three-dimensional model according to the obtained translation vector to translate the target three-dimensional model to the real depth in the camera coordinate system, thereby performing an absolute depth check on the target three-dimensional model, eliminating the absolute depth error of the target three-dimensional model, and obtaining a target three-dimensional model containing the real depth of the object corresponding to the object with the relative depth error and the absolute depth error eliminated. Among them, the camera coordinate system is a three-dimensional rectangular coordinate system with the camera optical center as the origin, the optical axis as the z-axis, and the x and y axes parallel to the wide and high sides of the picture; the real depth is the depth information that overcomes the absolute depth error.

[0089] The absolute depth of the target 3D model is verified by the translation vector obtained from the center of gravity of the uncalibrated skeleton model and the skeleton model containing depth information, thereby eliminating the absolute depth error of the target 3D model and improving the accuracy of 3D model construction.

[0090] Step S414, obtaining a preset number of frames of a plurality of previous two-dimensional images and corresponding plurality of previous depth images of the object to be constructed containing the three-dimensional model; wherein the shooting sequence of the previous two-dimensional images and the previous depth images is before the two-dimensional images and the depth images.

[0091] In this embodiment, taking into account that in actual applications, a three-dimensional model is constructed for the object's motion process, and when a three-dimensional model is constructed for a single-frame image, there is a lack of historical image information during the motion process, which leads to a large variation in the three-dimensional model construction in the time series. The calibrated target three-dimensional model is further smoothed in combination with the historical frames, and the posture parameters of the target three-dimensional model containing the true depth are updated specifically according to the posture parameters of the three-dimensional model corresponding to the historical frames, thereby further improving the accuracy of the three-dimensional model construction.

[0092] Specifically, after obtaining the target three-dimensional model including the real depth, the server obtains several previous depth images corresponding to several previous two-dimensional images of a preset number of frames, wherein the previous two-dimensional image corresponds to the previous depth image, and can be specifically obtained by simultaneously shooting with a depth camera, the previous two-dimensional image and the previous depth image both contain the object to be constructed into the three-dimensional model, and the shooting sequence of the previous two-dimensional image and the previous depth image is before the two-dimensional image and the depth image, that is, the previous two-dimensional image is shot before the two-dimensional image, and the previous depth image is shot after the previous depth image. The preset number of frames can be set according to actual needs, such as 2 frames.

[0093] Step S416, determining corrected posture parameters of objects in a plurality of previous two-dimensional images and a plurality of previous depth images.

[0094] After obtaining a number of previous two-dimensional images and a number of corresponding previous depth images, the server determines the corrected posture parameters of the corresponding objects in the number of previous two-dimensional images and the number of previous depth images. The corrected posture parameters of the objects in the number of previous two-dimensional images and the number of previous depth images can be used as a historical reference for the current target three-dimensional model including the real depth. Based on the historical reference, the target three-dimensional model including the real depth can be smoothed to reduce the variation range of the three-dimensional model construction in the time series, and further improve the accuracy of the three-dimensional model construction.

[0095] Step S418, smoothing the corrected posture parameters in the target three-dimensional model including the real depth according to the corrected posture parameters of the objects in the previous two-dimensional images and the previous depth images, to obtain a smoothed target three-dimensional model.

[0096] After obtaining the corrected posture parameters of the objects in the several previous two-dimensional images and the several previous depth images, the server performs smoothing processing on the corrected posture parameters in the target three-dimensional model containing the real depth based on the corrected posture parameters of the objects in the several previous two-dimensional images and the several previous depth images. Specifically, the server can determine the smoothed posture parameters based on the corrected posture parameters of the objects in the several previous two-dimensional images and the several previous depth images, and the corrected posture parameters in the target three-dimensional model containing the real depth, such as performing weighted processing to obtain the smoothed posture parameters. The corrected posture parameters in the target three-dimensional model containing the real depth are adjusted to the smoothed posture parameters, thereby achieving smoothing processing of the corrected posture parameters in the target three-dimensional model containing the real depth, and obtaining the target three-dimensional model after smoothing. Specifically, the corrected posture parameters in the target three-dimensional model containing the real depth can be updated based on the smoothed posture parameters after smoothing, thereby obtaining the target three-dimensional model after smoothing.

[0097] Based on the obtained smoothed posture parameters, the corrected posture parameters in the target three-dimensional model containing the true depth are smoothed, thereby reducing the variation amplitude of the constructed three-dimensional model in the time series and improving the accuracy of the three-dimensional model construction.

[0098] It should be noted that steps 414 , 416 , and 418 may be performed before step 408 to further correct the pose parameters of the object in the uncalibrated mesh model in step 4086 .

[0099] In one embodiment, the human body three-dimensional model construction method provided by the present application is applied to the human body three-dimensional model reconstruction process, and the constructed human body mesh model, that is, the three-dimensional human body model, can be applied to scenes such as machine-human interaction, sports analysis, and virtual fitting. At present, the traditional three-dimensional human body reconstruction method is mainly based on the SMPL human body model - using traditional methods or deep neural networks to extract features from RGB pictures to predict SMPL model parameters, and finally calculate the three-dimensional human body mesh model, that is, the three-dimensional human body Mesh model. Among them, SMPL is a mathematical model that can output human body meshes and skeletons of human bodies of different body shapes in different postures, which can describe human bodies of different body shapes and postures with 10 body shape parameters and 72 posture parameters; Mesh model is a human body mesh model, which is a model that collects sample points on the surface of an object and describes the geometric shape of the surface of an object with the link relationship between adjacent sample points and the three-dimensional coordinates of the sample points.

[0100] However, although the projection of the 3D model generated by the existing method in the 2D image can accurately fit the contour of the human body in the image, since the RGB image cannot provide the depth information of the human body surface points, the coordinates of different parts of the human body in the generated model on the z-axis (i.e., the depth direction) may have errors relative to other parts. For example, the hands, feet, and limbs may have the same projection in the 2D image at different depths (from the camera). Figure 3 On the other hand, the existing method aims to maximize the fit between the two-dimensional projection of the human body model in the RGB image and the real human body contour, and only needs to adjust the size of the human body model. It is possible to better fit the human body contour in the two-dimensional image at any depth (distance from the camera), resulting in ambiguity or error in the absolute depth of the human body model in the camera coordinate system. Figure 5The absolute depth error shown. In this embodiment, based on the three-dimensional human body Mesh model obtained by the existing Human Mesh Recovery (HMR, human mesh reconstruction) algorithm, the absolute and relative depth errors of the Mesh model are calibrated using the collected depth image. Among them, the HMR algorithm is an algorithm based on deep learning that can generate a three-dimensional mesh model of the human body from an RGB image. The three-dimensional human body Mesh model is constructed by posture parameters θ and body shape parameters β. The posture parameters θ represent 75 parameters of the overall motion posture of the human body and the relative angles of 24 joints. The body shape parameters β represent 10 parameters of the proportion of a person's height, fatness, and head-to-body ratio. According to the posture parameters θ, a Skeleton model can be obtained, that is, a human skeleton model, which is a human skeleton (such as calf, thigh, upper arm, lower arm, etc.) obtained by connecting the key points of the human body. The human skeleton model can reflect the movements and postures of the human body. The camera coordinate system is a three-dimensional rectangular coordinate system with the optical center of the camera as the origin, the optical axis as the z-axis, and the x and y axes parallel to the wide and high sides of the picture.

[0101] Specifically, a depth camera, such as Intel Realsense, is used to collect time-series RGB images and depth images containing the human body. The RGB image is used as an image containing the object to be constructed as a three-dimensional model; the depth image is a picture collected by the depth camera, which is the same size as the RGB image, and each pixel contains the distance from the camera to the point on the object to which the pixel belongs in the corresponding RGB image. The AlphaPose algorithm is used to obtain the two-dimensional information of the key points of the human body in each RGB image, that is, the two-dimensional coordinates and confidence of each key point, and then the key points with confidence greater than the confidence threshold are mapped to the corresponding positions of the corresponding depth image to obtain the corresponding depth information of each key point, specifically the three-dimensional coordinates, and thus obtain the human skeleton model B containing the real depth. GT Among them, the AlphaPose algorithm is an algorithm based on deep learning that can generate two-dimensional key points of the human body from RGB images; the human skeleton model, namely the Skeleton model, is a human skeleton obtained by connecting key points of the human body, such as the calf, thigh, upper arm, lower arm, etc. On the other hand, the Human Mesh Recovery algorithm is used to obtain an uncalibrated three-dimensional human mesh model and a corresponding uncalibrated human skeleton model B.

[0102] When performing relative depth calibration on the uncalibrated human skeleton model B, the distance D = f(B(θ, β), B GT), the uncalibrated human skeleton B obtained by the Human Mesh Recovery algorithm is determined by the posture parameter θ and the body shape parameter β. The body shape does not change with the change of posture, so we only need to find the posture parameter θ* that can minimize the distance, and then use β and θ* to regenerate the calibrated 3D human mesh model using the SMPL model. Specifically, with B(θ,β) and B GT represents the set of key points of a 3D human body, where B(θ,β)={(x1,y1,z1),…,(xK,yK,zK)}, B GT ={(x1 gt ,y1 gt ,z1 gt ),…,(x K gt ,y K gt ,z K gt )}, K represents the number of key points. Connecting any two key points in the same set can obtain a set of vectors, that is, V = {(x i -x j ,y i -y j ,z i -z j )|all i and j} and V={(x i -x j ,y i -y j ,z i -z j )|all i and j}. For any two vectors v in V m and v n , and V GT Any two vectors v in m gt and v n gt , calculate their cosine distance d = 1- <v m ,v n > and d gt =1- <v m gt ,v n gt > (i.e. 1 – cosine similarity). Determine the model distance D = f(B(θ, β), B GT ), which is the square of the difference between the two distances (d–d gt ) 2 The sum D over all pairs of vectors = ∑(d m,n –d m,n gt )2 Using the gradient descent algorithm, with D as the objective function, find the θ* parameter that can minimize D. Using the SMPL model to regenerate the human skeleton B(θ*,β)={(x1,y1,z1),…,(x K ,y K ,z K )} and the three-dimensional human body Mesh model, thereby realizing the calibration of the relative depth error of the three-dimensional human body Mesh model.

[0103] like Figure 6 and Figure 7 As shown, in one embodiment, after relative depth calibration, the arms of the three-dimensional human body Mesh model are Figure 6 The posture shown is corrected to Figure 7 The drooping posture shown.

[0104] After the relative depth error calibration, the absolute depth calibration is performed. This can be done by calculating the skeleton model B containing the real depth. GT The translation vector between the calibrated 3D human body mesh model and the uncalibrated skeleton model B is then used to move the calibrated 3D human body mesh model parallel to the real depth in the camera coordinate system. Specifically, the regenerated human body skeleton B(θ*,β)={(x1,y1,z1),…,(x K ,y K ,z K )} and the human skeleton B containing the true depth GT ={(x1 gt ,y1 gt ,z1 gt ),…,(x K gt ,y K gt ,z K gt )}'s centroid (c1,c 2, c3) and (c1 gt ,c2 gt ,c3 gt ), we can get the vector v=(c1 gt -c1,c2 gt -c2,c3 gt -c3). The 3D human body Mesh model regenerated after relative depth error is translated along the direction and length of v to obtain the 3D human body Mesh model calibrated after absolute depth error.

[0105] In addition, considering that the Human Mesh Recovery algorithm only relies on the information of a certain frame of RGB image, the output 3D human Mesh model may vary greatly in time series.i-2 ,θ i-1 ,θ i Respectively represent the human posture parameters of the 3D human Mesh model after relative depth error calibration and absolute error calibration of the RGB images of the i-2, i-1, and i frames. The smoothed parameter θ of the i-th frame i '=θ i-2 *w i-2 +θ i-1 *w i-1 +θ i *w i , which is the weighted average of the parameters of the first two frames and the i-th frame, and then the final calibrated and smoothed 3D human body Mesh model is generated through SMPL.

[0106] like Figure 8 In FIG. 1 , the three mutually perpendicular line segments on the left represent the camera position and the camera coordinate system, and the right side is a schematic diagram of a three-dimensional human body mesh model constructed in an embodiment, which can be a three-dimensional human body mesh model obtained after relative depth error calibration and absolute depth error calibration. Figure 7 and Figure 8 As shown, in one embodiment, after absolute depth calibration, the three-dimensional human body Mesh model is composed of Figure 8 The situation where the base point of the camera coordinate system is located is corrected to the real position relative to the camera coordinate system. Figure 8 Skeleton model in 3D human body Mesh model Figure 6 and Figure 7 The skeleton model in is the same.

[0107] In this embodiment, the absolute depth error and relative depth error of the three-dimensional human body Mesh model are calibrated by using the depth information of the human body in the acquired depth image, thereby improving the accuracy of constructing the three-dimensional human body Mesh model.

[0108] It should be understood that although Figure 2 , 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 4 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0109] In one embodiment, Fig. 9 As shown, a human body three-dimensional model construction device 900 is provided, comprising: an image acquisition module 902, a model module to be calibrated 904, a depth information processing module 906 and a depth calibration module 908, wherein:

[0110] An image acquisition module 902 is used to acquire a two-dimensional image of an object to be built into a three-dimensional model and corresponding depth information;

[0111] The model to be calibrated module 904 is used to obtain the 3D model to be calibrated of the object according to the 2D image by using a skinned multi-person linear model (SMPL);

[0112] A depth information processing module 906, configured to obtain a skeleton model of the object including depth information according to the two-dimensional image and the depth information; and

[0113] The depth calibration module 908 is used to perform depth calibration on the 3D model to be calibrated according to the skeleton model containing depth information to obtain a target 3D model.

[0114] In one embodiment, the depth information processing module 906 includes a key point determination module, a valid key point module, a depth image acquisition module, a three-dimensional coordinate determination module and a skeleton model construction module; wherein: the key point determination module is used to determine the two-dimensional coordinates and confidence of multiple key points of an object based on a two-dimensional image; the valid key point module is used to take each key point whose confidence is greater than a preset confidence threshold as a valid key point; the depth image acquisition module is used to acquire a depth image corresponding to the two-dimensional image; the three-dimensional coordinate determination module is used to determine the three-dimensional coordinates of each valid key point based on the two-dimensional coordinates and depth image of each valid key point; and the skeleton model construction module is used to obtain a skeleton model of the object containing depth information based on the three-dimensional coordinates of the valid key points.

[0115] In one embodiment, the three-dimensional model to be calibrated includes an uncalibrated mesh model of the object and a corresponding uncalibrated skeleton model, and the depth calibration module 908 includes a comparison key point module, a model distance determination module and a posture parameter correction module; wherein: the comparison key point module is used to obtain the three-dimensional coordinates of multiple comparison key points corresponding to multiple valid key points in the uncalibrated skeleton model; the model distance determination module is used to determine the distance between the uncalibrated skeleton model and the skeleton model containing depth information according to the three-dimensional coordinates corresponding to the multiple valid key points and the multiple comparison key points respectively; and the posture parameter correction module is used to correct the posture parameters of the object in the uncalibrated mesh model with the goal of minimizing the distance between the uncalibrated skeleton model and the skeleton model containing depth information to obtain the target three-dimensional model.

[0116] In one embodiment, the model distance determination module includes a first distance determination module, a second distance determination module and a model distance acquisition module; wherein: the first distance determination module is used to determine multiple first vectors based on at least a part of multiple valid key points; and determine the first cosine distance between the multiple first vectors; the second distance determination module is used to determine multiple second vectors corresponding to the multiple first vectors based on at least a part of multiple comparison key points; and determine the second cosine distance between the multiple second vectors; and the model distance acquisition module is used to determine the distance between the uncalibrated skeleton model and the skeleton model containing depth information based on the sum of the squares of the distance differences between each first cosine distance and each second cosine distance.

[0117] In one embodiment, it also includes a translation parameter calculation module and a model translation module; wherein: the translation parameter calculation module is used to calculate the translation parameters between an uncalibrated skeleton model and a skeleton model containing depth information; and the model translation module is used to translate the target three-dimensional model to the real depth in the camera coordinate system according to the translation parameters, so as to obtain the target three-dimensional model corresponding to the object containing the real depth.

[0118] In one embodiment, the translation parameter calculation module includes a skeleton center of gravity determination module and a translation vector determination module; wherein: the translation parameter calculation module is used to respectively determine the first skeleton center of gravity of an uncalibrated skeleton model and the second skeleton center of gravity of a skeleton model containing depth information; and the model translation module is used to obtain the translation vector based on the first skeleton center of gravity and the second skeleton center of gravity; the translation parameters include the translation vector; the model translation module is also used to translate the target three-dimensional model to the real depth in the camera coordinate system according to the translation vector, so as to obtain the target three-dimensional model corresponding to the object containing the real depth.

[0119] In one embodiment, it also includes a prior image acquisition module, a prior parameter determination module and a smoothing processing module; wherein: the prior image acquisition module is used to acquire a preset number of frames of a plurality of prior two-dimensional images and a corresponding plurality of prior depth images of the object to be constructed with a three-dimensional model; wherein the shooting sequence of the prior two-dimensional image and the prior depth image is before the two-dimensional image and the depth image; the prior parameter determination module is used to determine the corrected posture parameters of the object in the plurality of prior two-dimensional images and the plurality of prior depth images; and the smoothing processing module is used to smooth the corrected posture parameters in the target three-dimensional model containing the real depth according to the corrected posture parameters of the object in the plurality of prior two-dimensional images and the plurality of prior depth images, so as to obtain the target three-dimensional model after smoothing.

[0120] For the specific definition of the human body three-dimensional model construction device, please refer to the definition of the human body three-dimensional model construction method above, which will not be repeated here. Each module in the above-mentioned human body three-dimensional model construction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the robot in the form of hardware, or can be stored in the memory of the robot in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0121] In one embodiment, a robot 1000 is provided. The robot may include a server. The internal structure of the server may be as shown in FIG. Fig.10 As shown. The robot 1000 includes a processor 1002, a memory and a network interface 1016 connected via a system bus 1004. Among them, the processor 1002 of the robot 1000 is used to provide computing and control capabilities. The memory of the robot 1000 includes a non-volatile storage medium 1006 and an internal memory 1014. The non-volatile storage medium 1006 stores an operating system 1008, a computer program 1010 and a database 1012. The internal memory 1014 provides an environment for the operation of the operating system 1008 and the computer program 1010 in the non-volatile storage medium 1006. The database 1012 of the robot 1000 is used to store data. The network interface 1016 of the robot 1000 is used to communicate with an external terminal through a network connection. When the computer program 1010 is executed by the processor 1002, a method for constructing a three-dimensional model of a human body is implemented.

[0122] Those skilled in the art will understand that Fig.10 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the robot to which the scheme of the present application is applied. The specific robot may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In one embodiment, a robot is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method for constructing a three-dimensional human body model in any of the above embodiments is implemented.

[0124] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for constructing a three-dimensional human body model in any of the above embodiments is implemented.

[0125] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a three-dimensional human body model, characterized in that: The method comprises: Obtaining a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information; According to the two-dimensional image, a skinned multi-person linear model (SMPL) is used to obtain a three-dimensional model to be calibrated of the object; the three-dimensional model to be calibrated includes an uncalibrated mesh model of the object and a corresponding uncalibrated skeleton model; Determine two-dimensional coordinates and confidence levels of a plurality of key points of the object according to the two-dimensional image; Taking each key point whose confidence level is greater than a preset confidence threshold as a valid key point; Obtaining a skeleton model of the object including depth information based on the two-dimensional coordinates of each of the valid key points and the depth information; Obtaining three-dimensional coordinates of a plurality of comparison key points corresponding to a plurality of valid key points in the uncalibrated skeleton model; Determining a distance between the uncalibrated skeleton model and the skeleton model containing depth information according to the three-dimensional coordinates respectively corresponding to the multiple valid key points and the multiple comparison key points; and Based on the distance between the uncalibrated skeleton model and the skeleton model containing depth information, the posture parameters of the object in the uncalibrated mesh model are corrected to obtain a target three-dimensional model.

2. The method according to claim 1, characterized in that: The obtaining of the two-dimensional image containing the object to be constructed of the three-dimensional model and the corresponding depth information comprises: obtaining a depth image corresponding to the two-dimensional image; the two-dimensional image containing the object to be constructed of the three-dimensional model; The step of obtaining a skeleton model of the object containing depth information based on the two-dimensional coordinates of each of the valid key points and the depth information comprises: Determining the three-dimensional coordinates of each of the valid key points based on the two-dimensional coordinates of each of the valid key points and the depth image; and A skeleton model of the object including depth information is obtained according to the three-dimensional coordinates of the valid key points.

3. The method according to claim 1, characterized in that The step of correcting the posture parameters of the object in the uncalibrated mesh model based on the distance between the uncalibrated skeleton model and the skeleton model containing the depth information to obtain the target three-dimensional model comprises: With the goal of minimizing the distance between the uncalibrated skeleton model and the skeleton model containing depth information, the posture parameters of the object in the uncalibrated mesh model are corrected to obtain a target three-dimensional model.

4. The method according to claim 3, characterized in that: The step of determining the distance between the uncalibrated skeleton model and the skeleton model containing depth information according to the three-dimensional coordinates respectively corresponding to the multiple valid key points and the multiple comparison key points comprises: Determine a plurality of first vectors according to at least a portion of the plurality of valid key points; and determine a plurality of first cosine distances between the plurality of first vectors; Determine a plurality of second vectors corresponding to the plurality of first vectors according to at least a portion of the plurality of comparison key points; and determine a plurality of second cosine distances between the plurality of second vectors; and The distance between the uncalibrated skeleton model and the skeleton model containing depth information is determined according to the sum of squares of differences between each of the first cosine distances and the corresponding each of the second cosine distances.

5. The method according to claim 3, characterized in that: Also includes: Calculating a translation parameter between the uncalibrated skeleton model and the skeleton model containing depth information; as well as According to the translation parameters, the target three-dimensional model is translated to the real depth in the camera coordinate system to obtain the target three-dimensional model corresponding to the object and including the real depth.

6. The method according to claim 5, characterized in that The step of calculating the translation parameters between the uncalibrated skeleton model and the skeleton model containing depth information comprises: respectively determining a first skeleton center of gravity of the uncalibrated skeleton model and a second skeleton center of gravity of the skeleton model containing depth information; and Obtaining a translation vector according to the first skeleton center of gravity and the second skeleton center of gravity; The step of translating the target three-dimensional model to the real depth in the camera coordinate system according to the translation parameter to obtain the target three-dimensional model corresponding to the object and including the real depth comprises: The target three-dimensional model is translated to the real depth in the camera coordinate system according to the translation vector, so as to obtain the target three-dimensional model corresponding to the object and including the real depth.

7. The method according to claim 3, characterized in that The method further comprises: Acquire a preset number of frames of a plurality of previous two-dimensional images and a corresponding plurality of previous depth images of the object to be constructed containing the three-dimensional model; wherein the shooting sequence of the previous two-dimensional images and the previous depth images is before the shooting sequence of the two-dimensional images and the depth images; determining corrected pose parameters of the object in the plurality of prior two-dimensional images and the plurality of prior depth images; and According to the corrected posture parameters of the object in the plurality of previous two-dimensional images and the plurality of previous depth images, the corrected posture parameters in the target three-dimensional model are smoothed to obtain a smoothed target three-dimensional model.

8. A human body three-dimensional model construction device, characterized in that: The device comprises: An image acquisition module, used to acquire a two-dimensional image of an object containing a three-dimensional model to be constructed and corresponding depth information; A model module to be calibrated is used to obtain a three-dimensional model to be calibrated of the object according to the two-dimensional image by using a skinned multi-person linear model (SMPL); the three-dimensional model to be calibrated includes an uncalibrated mesh model of the object and a corresponding uncalibrated skeleton model; a depth information processing module, configured to determine the two-dimensional coordinates and confidences of a plurality of key points of the object according to the two-dimensional image; to take each of the key points whose confidences are greater than a preset confidence threshold as a valid key point; and to obtain a skeleton model of the object including the depth information based on the two-dimensional coordinates of each of the valid key points and the depth information; and A depth calibration module is used to obtain the three-dimensional coordinates of multiple comparison key points corresponding to the multiple valid key points in the uncalibrated skeleton model; determine the distance between the uncalibrated skeleton model and the skeleton model containing depth information according to the three-dimensional coordinates corresponding to the multiple valid key points and the multiple comparison key points respectively; and based on the distance between the uncalibrated skeleton model and the skeleton model containing depth information, correct the posture parameters of the object in the uncalibrated mesh model to obtain a target three-dimensional model.

9. A robot comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images