Model training method and device, computer equipment and storage medium

Abstract

The invention discloses a model training method and device, computer equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a first sample set, wherein a first sample image in the first sample set comprises a human body; inputting the first sample image into a parameter prediction model to obtain a first human hand parameter and a first human body parameter of the human body; according to the first human hand parameter of the human body, the first human body parameter and a pre-trained human body reconstruction model, performing three-dimensional reconstruction on the human body to obtain a three-dimensional human body model corresponding to the human body; and performing iterative training on the parameter prediction model based on the difference degree between the prediction coordinate information and the labeled coordinate information and the difference degree between the first hand parameter and the second hand parameter to obtain a three-dimensional reconstruction parameter prediction model. Therefore, the three-dimensional human body model with more accurate and natural connection between the human body and the human hand can be reconstructed based on the parameters for three-dimensional reconstruction output by the final three-dimensional reconstruction parameter prediction model.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a model training method, apparatus, computer equipment and storage medium. Background Art

[0002] In recent years, the task of three-dimensional reconstruction of the human body has made great progress and has been widely used in tasks such as virtual equipment, virtual dressing and motion capture.

[0003] In the related art, a human body parameter regression model and a hand parameter regression model are generally trained separately to obtain SMPL human body parameters and MANO hand parameters, and finally the human body parameters and hand parameters are fused at the parameter layer, and the human body 3D reconstruction is performed based on the fused human body and hand parameters. In this way, the connection between the hand and the human body of the 3D human body model generated by 3D reconstruction is unnatural, that is, the effect of 3D reconstruction of the human body is not good. Summary of the invention

[0004] The present application proposes a model training method, apparatus, computer equipment and storage medium to improve the naturalness of three-dimensional reconstruction of the human body.

[0005] In a first aspect, an embodiment of the present application provides a model training method, the method comprising: obtaining a first sample set, wherein a first sample image in the first sample set comprises a human body, and the first sample image carries annotated coordinate information of joint points of the human body; inputting the first sample image into a parameter prediction model to obtain first hand parameters and first human body parameters of the human body; performing three-dimensional reconstruction of the human body according to the first hand parameters of the human body, the first human body parameters and a pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body; based on the three-dimensional human body model, obtaining the coordinate information of the joint points as predicted coordinate information of the joint points; based on the degree of difference between the predicted coordinate information and the annotated coordinate information, and the degree of difference between the first hand parameters and the second hand parameters, iteratively training the parameter prediction model to obtain a three-dimensional reconstruction parameter prediction model, wherein the second hand parameters are predicted based on the first human body parameters.

[0006] In a second aspect, an embodiment of the present application provides a model training device, the device comprising: a training sample acquisition module, a parameter prediction module, a three-dimensional reconstruction module, a coordinate acquisition module and a model training module. The training sample acquisition module is used to acquire a first sample set, wherein the first sample image in the first sample set includes a human body, and the first sample image carries the annotated coordinate information of the joint points of the human body; the parameter prediction module is used to input the first sample image into a parameter prediction model to obtain the first hand parameter and the first human body parameter of the human body; the three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the human body according to the first hand parameter of the human body, the first human body parameter and a pre-trained human body reconstruction model, and obtain a three-dimensional human body model corresponding to the human body; the coordinate acquisition module is used to acquire the coordinate information of the joint point based on the three-dimensional human body model as the predicted coordinate information of the joint point; the model training module is used to iteratively train the parameter prediction model based on the difference between the predicted coordinate information and the annotated coordinate information, and the difference between the first human hand parameter and the second human hand parameter, and obtain a three-dimensional reconstruction parameter prediction model, wherein the second human hand parameter is predicted based on the first human body parameter.

[0007] In a third aspect, an embodiment of the present application provides a computer device, comprising: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the above method.

[0009] In the solution provided by the present application, a first sample set is obtained, wherein the first sample image in the first sample set includes a human body, and the first sample image carries the annotated coordinate information of the joint points of the human body; the first sample image is input into a parameter prediction model to obtain the first hand parameter and the first human body parameter of the human body; the human body is three-dimensionally reconstructed according to the first hand parameter, the first human body parameter and the pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body; based on the three-dimensional human body model, the coordinate information of the joint points is obtained as the predicted coordinate information of the joint points; based on the difference between the predicted coordinate information and the annotated coordinate information, and the difference between the first hand parameter and the second hand parameter, the parameter prediction model is iteratively trained to obtain a three-dimensional reconstruction parameter prediction model, and the second hand parameter is predicted based on the first human body parameter. In this way, the hand parameters used for three-dimensional reconstruction of the human body and the prediction task of the hand parameters are placed in the same parameter prediction model for training. Moreover, in the process of iterative training of the parameter prediction model, the first human body parameter predicted by the parameter prediction model is also used to infer the human hand parameter to obtain the second human hand parameter; thus, based on the degree of difference between the first human body parameter and the second human hand parameter, the parameter prediction model is iteratively trained as a constraint condition for model training to obtain the final three-dimensional reconstruction parameter prediction model. In this way, it is equivalent to using the relationship between the human body and the human hand to constrain the parameter prediction model, so that based on the parameters for three-dimensional reconstruction output by the final three-dimensional reconstruction parameter prediction model in actual application, a three-dimensional human body model with a more accurate and natural connection between the human body and the human hand is reconstructed; in other words, the prediction of the parameters for three-dimensional reconstruction using the three-dimensional reconstruction parameter prediction model of this application greatly improves the accuracy and naturalness of three-dimensional reconstruction of the human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A flow chart of a model training method provided in one embodiment of the present application is shown.

[0012] Figure 2 A schematic diagram showing the positions of joint points in a human body model provided in an embodiment of the present application is shown.

[0013] Figure 3 A flow chart of the three-dimensional reconstruction of a human body provided by an embodiment of the present application is shown.

[0014] Figure 4A flow chart of a model training method provided in another embodiment of the present application is shown.

[0015] Figure 5 A model architecture diagram of a parameter prediction model provided in an embodiment of the present application is shown.

[0016] Figure 6 Shows Figure 4 A schematic flow chart of the sub-steps of step S220 in one implementation manner.

[0017] Figure 7 Shows Figure 4 A flowchart of the sub-steps of step S260 in one implementation manner.

[0018] Figure 8 A flow chart of a model training method provided in yet another embodiment of the present application is shown.

[0019] Fig. 9 It is a block diagram of a model training device provided according to an embodiment of the present application.

[0020] Fig.10 It is a block diagram of a computer device according to an embodiment of the present application for executing the model training method according to an embodiment of the present application.

[0021] Fig.11 It is a storage unit of an embodiment of the present application for storing or carrying program code for implementing the model training method according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0023] It should be noted that in some processes described in the specification, claims and the above-mentioned drawings of the present application, multiple operations that appear in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The sequence numbers of operations such as S110, S120, etc. are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. And, the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0024] The inventor proposes a model training method, device, computer equipment and storage medium. The model training method provided in the embodiment of the present application is described in detail below.

[0025] Please refer to Figure 1 , Figure 1 A flow chart of a model training method provided in one embodiment of the present application. Figure 1 The model training method provided in the embodiment of the present application is described in detail. The model training method may include the following steps:

[0026] Step S110: Acquire a first sample set, wherein a first sample image in the first sample set includes a human body, and the first sample image carries annotated coordinate information of joint points of the human body.

[0027] In this embodiment, the first sample set includes a plurality of first sample images, each of which includes a human body, and each of which carries the labeled coordinate information of key points of the human body. The joints of the human body can be understood as representative points for posture recognition on the human body, and the joints can include head joints, neck joints, shoulder joints, elbow joints, wrist joints, knee joints, hip joints, and finger joints.

[0028] For example, see Figure 2 , Figure 2 There are 24 joint points shown in Figure 2There are 24 joints numbered from 0 to 23. Among them, a human hand generally includes 15 finger joints. Figure 3 Not shown.

[0029] Step S120: inputting the first sample image into a parameter prediction model to obtain a first hand parameter and a first human body parameter of the human body.

[0030] Among them, the parameter prediction model can be used to extract the human body features and hand features of the human body in the first sample image, and predict the first human body parameters for three-dimensional reconstruction based on the human body features, and also predict the first hand features for three-dimensional reconstruction based on the hand features.

[0031] In this example, see Figure 3 , Figure 3 The process architecture diagram of human body 3D reconstruction is shown in FIG. Before inputting the first sample image into the parameter prediction model, human body detection can be performed on the first sample image until the human body detection frame (i.e. Figure 3 In order to avoid that some limbs of the human body are not included in the human body detection frame, the human body detection frame can be expanded to obtain an extended human body frame (i.e. Figure 3 ). Further, the image area occupied by the extended human body frame can be cropped from the first sample image to obtain the human body image area corresponding to the first sample image. Then, the human body image area is input into the parameter prediction model, so that the parameter prediction model can output human body reconstruction parameters. Among them, the human body reconstruction parameters include posture parameters, shape parameters and camera parameters, and the posture parameters include the first human hand parameters and the first human body parameters.

[0032] Step S130: performing three-dimensional reconstruction of the human body according to the first human hand parameter of the human body, the first human body parameter and a pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body.

[0033] The pre-trained human reconstruction model may be an SMPL-H model, which is a model based on a Skinned Multi-Person Linear (SMPL) model with hand joints added. Figure 3 As shown, by inputting the human body reconstruction parameters into the SMPL-H model, a corresponding three-dimensional human body model can be obtained. The movements and body shape of the three-dimensional human body model are similar to those of the human body in the first sample image.

[0034] Step S140: Based on the three-dimensional human body model, the coordinate information of the joint points is obtained as the predicted coordinate information of the joint points.

[0035] After three-dimensional reconstruction of the human body in the sample image, the coordinate information of the aforementioned joint points in the reconstructed three-dimensional human body model can be obtained as the predicted coordinate information of the joint points.

[0036] The predicted coordinate information may be three-dimensional coordinate information, or two-dimensional coordinate information after mapping the three-dimensional coordinate information, or may include both three-dimensional coordinate information and two-dimensional coordinate information.

[0037] Step S150: Based on the difference between the predicted coordinate information and the labeled coordinate information, and the difference between the first hand parameters and the second hand parameters, the parameter prediction model is iteratively trained to obtain a three-dimensional reconstruction parameter prediction model, wherein the second hand parameters are predicted based on the first human body parameters.

[0038] It can be understood that the human hand is connected to the human body through the wrist. Obviously, the human hand parameters are actually related to the human body parameters. Therefore, based on the first human body parameters, the human hand parameters can be inferred to obtain the second human hand parameters. Based on this, if the prediction ability of the parameter prediction model is accurate enough, the second human hand parameters inferred based on the first human body parameters should not be different from the first human hand parameters directly predicted by the parameter prediction model, or the difference is small. If the difference between the two is large, it indicates that the prediction ability of the parameter prediction model is not accurate enough. Therefore, in the process of iteratively training the parameter prediction model using the first sample image, in addition to the difference between the predicted coordinate information and the annotated coordinate information, the difference between the first hand parameters and the second hand parameters inferred by inversion will be combined to jointly perform constrained iterative training on the parameter prediction model until the target training conditions are met to obtain the final three-dimensional reconstruction parameter prediction model.

[0039] Optionally, if the annotated coordinate information only includes two-dimensional coordinate information, it is necessary to map the three-dimensional coordinate information after obtaining the three-dimensional coordinate information of the key point to obtain two-dimensional coordinate information as the predicted coordinate information of the joint point. In this way, even if the first sample image with three-dimensional annotated information is not obtained, iterative training of the parameter prediction model can be achieved.

[0040] Optionally, if the annotated coordinate information only includes three-dimensional coordinate information, the acquired three-dimensional coordinate information of the joint point can be directly used as the predicted coordinate information of the aforementioned joint point.

[0041] Optionally, if the labeled coordinate information contains both two-dimensional coordinate information and three-dimensional coordinate information, it is necessary to map the three-dimensional coordinate information after acquiring the three-dimensional coordinate information of the key point to obtain the two-dimensional coordinate information; and use the two-dimensional coordinate information and the three-dimensional coordinate information as the predicted coordinate information of the joint point at the same time.

[0042] In this embodiment, the hand parameters and hand parameter prediction tasks used for three-dimensional reconstruction of the human body are placed in the same parameter prediction model for training. In addition, in the process of iterative training of the parameter prediction model, the first human body parameters predicted by the parameter prediction model are also used to infer the hand parameters to obtain the second human hand parameters; thus, based on the degree of difference between the first human body parameters and the second human hand parameters, the parameter prediction model is iteratively trained as a constraint condition for model training to obtain the final three-dimensional reconstruction parameter prediction model. In this way, it is equivalent to using the relationship between the human body and the human hand to constrain the parameter prediction model, so that based on the parameters for three-dimensional reconstruction output by the final three-dimensional reconstruction parameter prediction model in actual application, a three-dimensional human body model with a more accurate and natural connection between the human body and the human hand is reconstructed; in other words, the prediction of the parameters for three-dimensional reconstruction using the three-dimensional reconstruction parameter prediction model of this application greatly improves the accuracy and naturalness of three-dimensional reconstruction of the human body.

[0043] Please refer to Figure 4 , Figure 4 A flow chart of a model training method provided in another embodiment of the present application. Figure 4 The model training method provided in the embodiment of the present application is described in detail. The model training method may include the following steps:

[0044] Step S210: Acquire a first sample set, wherein a first sample image in the first sample set includes a human body, and the first sample image carries annotated coordinate information of joint points of the human body.

[0045] In this embodiment, the specific implementation of step S210 can refer to the content of the above-mentioned embodiment, which will not be repeated here.

[0046] Step S220: inputting the first sample image into a parameter prediction model to obtain a first hand parameter and a first human body parameter of the human body.

[0047] In some embodiments, see Figure 5 , the parameter prediction model may include a human body feature extraction unit, a feature fusion unit, a human body parameter regression unit, a human hand feature extraction unit and a human hand parameter regression unit.

[0048] In this mode, see Figure 6 , step S220 may include the following steps S221 to S225:

[0049] Step S221: downsampling the first sample image to obtain a first target image, wherein the image resolution of the first target image is smaller than the image resolution of the first sample image.

[0050] Since the first sample image is generally a high-resolution image, if human features are directly extracted based on the first sample image, the amount of data will be large, which will lead to excessive calculations during model training, reducing the efficiency of model training and may also cause model overfitting. Therefore, the first sample image can be downsampled first to obtain a first target image with an image resolution smaller than the original image resolution.

[0051] Step S222: Utilize the human feature extraction unit to extract the first human feature of the first target image and the heat map corresponding to the hand area of ​​the human body.

[0052] Among them, the human feature extraction unit can be an encoder-decoder network structure. The human feature extraction unit can first extract key points of the first target image, and based on the extracted key points, visualize the key points to obtain heat maps corresponding to the human key points and heat maps corresponding to the human hand area. The heat map corresponding to the hand area includes a human hand frame width and height heat map and a human hand frame center point heat map. Further feature processing can be performed based on the aforementioned generated heat map to obtain the aforementioned first human feature.

[0053] Step S223: Based on the heat map corresponding to the hand area, a hand area image is obtained from the first sample image, and the first hand feature of the hand area image is extracted using the hand feature extraction unit.

[0054] Furthermore, since the image resolution of the heat map is the same as that of the first target image, after obtaining the human hand width and height heat map and the heat of the center point of the human hand frame, the position of the hand area in the first target image can be known, and then the hand area can be mapped to the first sample image based on the proportional relationship between the image resolutions of the first target image and the first sample image. That is, based on the heat map corresponding to the hand area, the hand area image can be cropped from the first sample image. The hand area image is then input into the human hand feature extraction unit for feature extraction, thereby obtaining the first human hand feature of the hand area image.

[0055] Obviously, since the hand region image is cropped from the high-resolution first sample image, it can be seen that the hand region image is also a high-resolution image. Therefore, the extracted first human hand feature will have more detailed features of the human hand.

[0056] Step S224: using the feature fusion unit to fuse the first human body feature and the first human hand feature to obtain a second human body feature.

[0057] Considering that the first human body feature is extracted by the human body feature extraction unit from the low-resolution first target image, and the area occupied by the human hand in the first target image is small, resulting in fewer human hand features contained in the first human body feature. Therefore, the first human body feature and the first human hand feature can be fused. Specifically, the first human hand feature can be concatenated and fused with the human body feature in the channel dimension after passing through a convolution layer, so as to obtain the above-mentioned second human body feature.

[0058] Step S225: using the hand parameter regression unit to perform parameter regression on the first hand feature to obtain the first hand parameter, and using the body parameter regression unit to perform parameter regression on the second body feature to obtain the first body parameter.

[0059] Finally, the extracted first hand feature is input into the hand parameter regression unit, and the hand parameter regression unit performs parameter regression on the first hand feature to obtain the first hand parameter. Since the first hand feature is extracted from the high-resolution hand region image, the first hand parameter output by the hand parameter regression unit can be more accurate.

[0060] At the same time, the fused second human body features are input into the human body parameter regression unit, which performs parameter regression on the second human body features to obtain the first human body parameters. Since the second human body features contain features containing more details of human hands, the first human body parameters output by the human body parameter regression unit can be more accurate.

[0061] Among them, the first hand parameters may include the absolute rotation parameters of the wrist joint point, the relative rotation parameters of the finger joint points and the hand camera parameters. The absolute rotation parameters can be understood as the rotation parameters of the joint points compared to the corresponding root node, and the relative rotation parameters can be understood as the rotation parameters of the joint points compared to the corresponding parent node.

[0062] The second human hand parameters may include relative rotation parameters of human body joints, human body shape parameters, and human body camera parameters, and the human body joints include wrist joints.

[0063] Optionally, the hand parameter regression unit and the body parameter regression unit may be a multilayer perceptron (MLP) network, and may be capable of performing at least one MLP operation to obtain body parameters and hand parameters for three-dimensional reconstruction of the human body.

[0064] Step S230: performing three-dimensional reconstruction of the human body according to the first human hand parameter of the human body, the first human body parameter and a pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body.

[0065] Step S240: Based on the three-dimensional human body model, the coordinate information of the joint points is obtained as the predicted coordinate information of the joint points.

[0066] In some embodiments, based on the three-dimensional human body model, the three-dimensional coordinate information of the joint points is obtained to obtain the three-dimensional predicted coordinates of the joint points; the three-dimensional predicted coordinates of the joint points are mapped to a preset plane to obtain the two-dimensional predicted coordinates of the joint points.

[0067] Step S250: Based on the relative rotation parameters of the wrist joint point and the associated joint points, predict the rotation parameters of the wrist key point compared to the root node, and obtain the predicted absolute rotation parameters of the wrist joint point as the second hand parameter.

[0068] In this embodiment, the associated joint points include joint points associated with the wrist joint point among the joint points of the human body. Specifically, the product of the relative rotation parameter of the wrist joint point and the relative rotation parameter of the associated joint point can be obtained to obtain the rotation parameter of the wrist joint point compared with the root node, that is, the predicted absolute rotation parameter of the wrist joint point. The associated joint points can also be understood as all ancestor joint points of the wrist joint point.

[0069] Exemplarily, the predicted absolute rotation parameters of the wrist joint point can be obtained by the following forward kinematic chain function FK:

[0070] FK(θ rw ,A(θ rw ))

[0071] Among them, θ rw Characterizes the relative rotation parameter of the wrist joint, A(θ rw ) represents the relative rotation parameter of the associated joint point, FW(θ rw ,A(θ rw )) is the calculated predicted absolute rotation parameter of the wrist joint point.

[0072] Step S260: Determine a first loss value based on the degree of difference between the predicted coordinate information and the labeled coordinate information.

[0073] In some embodiments, the annotation coordinate information includes two-dimensional annotation coordinates and three-dimensional annotation coordinates, see Figure 7 , step S260 may include the following steps:

[0074] Step S261: Determine a three-dimensional prediction loss value based on the degree of difference between the three-dimensional predicted coordinates and the three-dimensional annotated coordinates.

[0075] Specifically, the function value of the three-dimensional loss function can be calculated using the three-dimensional predicted coordinates and the three-dimensional annotated coordinates, and the calculated function value can be used as the three-dimensional predicted loss value. The three-dimensional loss function can be a square loss function of the three-dimensional coordinates, an exponential loss function, etc., which is not limited in this embodiment.

[0076] Exemplarily, the three-dimensional prediction loss value can be calculated by the following three-dimensional loss function:

[0077]

[0078]

[0079] Among them, L b-3d Characterizes the three-dimensional prediction loss value, p b-3d Represents the three-dimensional annotation coordinates, Represents the 3D predicted coordinates of the joint points output by the 3D human body model (i.e., SMPL-H), θ rb Represents the relative rotation parameters of the joint points of the human body output by the human body parameter regression unit, s b Represents the human body shape parameters output by the human body parameter regression unit.

[0080] Step S262: Determine a two-dimensional prediction loss value based on the degree of difference between the two-dimensional prediction coordinates and the two-dimensional annotation coordinates.

[0081] Specifically, the function value of the two-dimensional loss function can be calculated using the two-dimensional prediction coordinates and the two-dimensional annotation coordinates, and the calculated function value can be used as the two-dimensional prediction loss value. The two-dimensional loss function can be a square loss function of the two-dimensional coordinates, an exponential loss function, etc., which is not limited in this embodiment.

[0082] Exemplarily, the three-dimensional prediction loss value can be calculated by the following three-dimensional loss function:

[0083]

[0084]

[0085] Among them, L b-2d Characterizes the two-dimensional prediction loss value, p b-2d represents the two-dimensional annotation coordinates, Π represents the two-dimensional projection function, The 2D predicted coordinates obtained by mapping the 3D predicted coordinates of the joint points output by the 3D human body model (i.e., SMPL-H) are represented. b Characterizing human camera parameters.

[0086] Step S263: Determine the first loss value according to the three-dimensional predicted loss value and the two-dimensional predicted loss value.

[0087] In some implementations, the three-dimensional predicted loss value and the two-dimensional predicted loss value may be weighted and summed to obtain a first loss value. For example, the first loss value L b It can be expressed as: b =λ 1 L b-2d +λ 2 L b-3d Among them, the weight coefficient λ 1 and λ 2 The value of can be set according to actual needs, and this embodiment does not limit this.

[0088] In some other implementations, the sum of the three-dimensional predicted loss value and the two-dimensional predicted loss value may also be obtained as the first loss value. b It can also be expressed as: L b =L b-2d +L b-3d .

[0089] In other embodiments, since the two-dimensional coordinates of the joints of the human body in the first sample image are easier to annotate, that is, there may be a first sample image in the first sample set whose annotated coordinate information only includes two-dimensional annotated coordinates. Based on this, the three-dimensional coordinate information of the joints can be obtained based on the three-dimensional human body model to obtain the three-dimensional predicted coordinates of the joints; the three-dimensional predicted coordinates of the joints are mapped to a preset plane to obtain the two-dimensional predicted coordinates of the joints; based on the degree of difference between the two-dimensional predicted coordinates and the two-dimensional annotated coordinates, a two-dimensional prediction loss value is determined; finally, the first loss value is determined based on the two-dimensional prediction loss value. In this way, even if there are no corresponding three-dimensional coordinates for the joints of the human body in the first sample image, such sample images can be used to perform model training of the parameter prediction model.

[0090] Step S270: Determine a second loss value based on the degree of difference between the absolute rotation parameter of the wrist joint point and the predicted absolute rotation parameter.

[0091] Furthermore, the following self-supervised loss function can be used to obtain the second loss value:

[0092]

[0093] Among them, θ gw Characterizes the absolute rotation parameter of the wrist joint point output by the hand parameter prediction unit, FK(θ rw ,A(θ rw )) represents the predicted absolute rotation parameter of the wrist joint, L s Characterizes the second loss value.

[0094] Step S280: Obtain a total loss value according to the first loss value and the second loss value.

[0095] It can be understood that the rotation angles of various joints of the human body are limited. In order to prevent the three-dimensional reconstructed human body model from having body deformities, the third loss value can be calculated based on the relative rotation parameters of the human body's joints and using the following regularization loss function.

[0096]

[0097] Among them, L b-r Characterized as the regularization loss value for preventing body deformity, that is, the third loss value.

[0098] Furthermore, the first loss value, the second loss value, and the third loss value may be weightedly summed to obtain a total loss value.

[0099] For example, the total loss value L t It can be expressed as: t =α 1 L s +α 2 L b-r +α 3 L b . Among them, the weight coefficient α 1 , α 2 , α 3 The value of can be set according to actual needs, and this embodiment does not limit this.

[0100] It can be understood that how to calculate the total loss value specifically is that no matter what type of first sample set is used, the above-mentioned regularization term loss needs to be calculated, and the other loss parts are based on the annotation information carried by each first sample image itself. For example, if the first sample image carries the annotation information of the hand frame, then when calculating the total loss value, it is also necessary to calculate the prediction loss between the hand frame predicted by the human feature extraction unit and the annotated hand frame. This embodiment will not be repeated here.

[0101] Step S290: According to the total loss value, the parameter prediction model is iteratively trained until the target training condition is met to obtain the 3D reconstruction parameter prediction model.

[0102] Finally, according to the total loss value of the first sample image input in each round, the parameters of the human feature extraction unit, feature fusion unit, human parameter regression unit, hand feature extraction unit and hand parameter regression unit in the parameter prediction model are iteratively optimized until the target training conditions are met, and a converged parameter prediction model is obtained as the above-mentioned 3D reconstruction parameter prediction model.

[0103] Among them, the target training conditions can be: the total detection loss value is less than the preset value, the total detection loss value no longer changes, or the number of training times reaches the preset number of times, etc. It can be understood that after the parameter prediction model is iteratively trained for multiple training cycles according to the first sample set, wherein each training cycle includes multiple iterative trainings, the parameters in the parameter prediction model are continuously optimized, so that the above-mentioned total detection loss value becomes smaller and smaller, and finally becomes a fixed value, or is less than the above preset value. At this time, it means that the parameter prediction model has converged; of course, it can also be determined that the parameter prediction model has converged after the number of training times reaches the preset number of times. At this time, the converged parameter prediction model can be used as the above-mentioned three-dimensional reconstruction parameter prediction model. Among them, the preset value and the preset number of times are pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not limit this.

[0104] After step S290, the target image containing the target human body can be input into the 3D reconstruction parameter prediction model to obtain the target human hand parameters and target human body parameters of the target human body; the target human body is 3D reconstructed according to the target human body parameters and the pre-trained human body reconstruction model to obtain a target 3D human body model corresponding to the target human body. In this way, using the aforementioned trained 3D reconstruction parameter prediction model, a target 3D human body model with a more accurate and natural wrist connection can be generated.

[0105] In this embodiment, the hand parameters used for three-dimensional reconstruction of the human body and the prediction task of the hand parameters are placed in the same parameter prediction model for training. In addition, in the process of iterative training of the parameter prediction model, the relative rotation parameters of the wrist joint points predicted by the human body parameter regression unit are used to infer the absolute rotation parameters of the wrist joint points to obtain the predicted absolute rotation parameters; thus, based on the difference between the absolute rotation parameters of the wrist joint points output by the hand parameter regression unit and the inferred predicted absolute rotation parameters, the self-supervised loss value is determined. The self-supervised loss value is used as one of the constraints of the model training to iteratively train the parameter prediction model to obtain the final three-dimensional reconstruction parameter prediction model. In this way, it is equivalent to using the relationship between the human body and the human hand to constrain the parameter prediction model, so that based on the parameters used for three-dimensional reconstruction output by the final three-dimensional reconstruction parameter prediction model in actual applications, a three-dimensional human body model with a more accurate and natural connection between the human body and the human hand can be reconstructed; in other words, using the three-dimensional reconstruction parameter prediction model of the present application to predict the parameters of three-dimensional reconstruction can greatly improve the accuracy and naturalness of three-dimensional reconstruction of the human body, and the reconstruction accuracy of the joint reconstruction of the human body and the human hand can be greatly improved, especially in application scenarios such as users using monocular RGB motion capture, the effect of simultaneous motion capture of the human body and the human hand can be greatly improved, thereby improving the user experience.

[0106] Please refer to Figure 8 , Figure 8 A flow chart of a model training method provided in another embodiment of the present application. Figure 8 The model training method provided in the embodiment of the present application is described in detail. The model training method may include the following steps:

[0107] Step S301: obtaining a second sample set, wherein second sample images in the second sample set all include a human body, and the second sample images carry two-dimensional annotated coordinate information of joints of the human body and a hand annotated frame of the human body.

[0108] In this embodiment, the parameter prediction model may include a human feature extraction unit, a feature fusion unit, a human parameter regression unit, a hand feature extraction unit, and a hand parameter regression unit. In order to improve the training efficiency of iteratively training the parameter prediction model as a whole using the first sample set, the human feature extraction unit may be pre-trained in advance.

[0109] Specifically, a second sample set can be obtained, which includes multiple second sample images, each of which includes a human body, and the second sample images carry two-dimensional annotated coordinate information of joints of the human body and position information of hand annotation frames of the human body.

[0110] Step S302: downsampling the second sample image to obtain a second target image.

[0111] The image resolution of the second target image is smaller than that of the second sample image. Of course, the downsampling factor used to downsample the second sample image is the same as the downsampling factor used to downsample the first sample image in the aforementioned embodiment.

[0112] Step S303: using the first initial feature extraction unit, extracting a first human body joint point heat map and a first hand joint point heat map in the second target image.

[0113] Step S304: Based on the two-dimensional labeled coordinate information of the human body joints and the hand labeling frame, a labeled human body joint point heat map and a labeled hand joint point heat map are generated.

[0114] Among them, the network structure of the first initial feature extraction unit is also an encoder-decoder network structure. The first hand joint point heat map is similar to the heat map corresponding to the hand area mentioned in the above embodiment, and the first hand joint point heat map also includes the first person hand frame center point heat map and the first person hand frame width and height heat map.

[0115] Furthermore, the two-dimensional annotated coordinate information of the joints of the human body and the hand annotated frame carried in the second sample image are visualized to generate an annotated human joint heat map and an annotated hand joint heat map. The annotated hand joint heat map includes an annotated human hand frame center heat map and an annotated human hand frame width and height heat map.

[0116] Step S305: Based on the degree of difference between the first human body joint point heat map and the labeled human body joint point heat map, and the degree of difference between the first hand joint point heat map and the labeled hand joint point heat map, the first initial feature extraction unit is iteratively trained to obtain the human body feature extraction unit.

[0117] In this embodiment, the fifth loss value can be obtained based on the difference between the first human joint point heat map and the labeled human joint point heat map; specifically, the fifth loss value can be obtained by the following fifth loss function:

[0118]

[0119] Among them, L f-2d Characterize the fifth loss value, f 2d Characterize and annotate the heat map of human joints. Characterize the heat map of the first human joint point.

[0120] And, based on the difference between the heat map of the center point of the first person's hand frame and the heat map of the center point of the labeled person's hand frame, a sixth loss value is obtained; specifically, the sixth loss value can be obtained by the following sixth loss function:

[0121]

[0122] Among them, L f-2d Characterize the sixth loss value, f bc Characterize the heat map of the center point of the hand frame of the annotator, Characterizes the heat map of the center point of the first person's hand frame.

[0123] And, based on the difference between the first person's hand frame width and height heat map and the annotated person's hand frame width and height heat map, a seventh loss value is obtained; specifically, the seventh loss value can be obtained by the following seventh loss function:

[0124]

[0125] Among them, L f-bwh Characterize the seventh loss value, f bwh Characterize the width and height heat map of the hand frame of the annotator, Characterize the first person's hand frame width and height heat map.

[0126] Finally, based on the fifth loss value, the sixth loss value and the seventh loss value, a first total loss value is obtained. f It can be expressed by the following formula:

[0127] L f =L f-2d +L f-bc +L f-bwh

[0128] Of course, the fifth loss value, the sixth loss value, and the seventh loss value may be weightedly summed to obtain the first total loss value, which will not be described in detail in this embodiment.

[0129] Based on this, the first initial feature extraction unit can be iteratively trained based on the first total loss value until the first training condition is met, so as to obtain a converged first initial feature extraction unit as the aforementioned human hand feature extraction unit.

[0130] Among them, the first training condition can be: detecting that the first total loss value is less than a preset value, detecting that the first total loss value no longer changes, or the number of training times reaches a preset number of times, etc. It can be understood that after the first initial feature extraction unit is iteratively trained for multiple training cycles according to the second sample set, wherein each training cycle includes multiple iterative trainings, the parameters in the first initial feature extraction unit are continuously optimized, so that the above-mentioned first total loss value becomes smaller and smaller, and finally becomes a fixed value, or is less than the above-mentioned preset value. At this time, it means that the first initial feature extraction unit has converged; of course, it can also be determined that the first initial feature extraction unit has converged after the number of training times reaches the preset number of times. At this time, the converged first initial feature extraction unit can be used as the above-mentioned human hand feature extraction unit. Among them, the preset value and the preset number of times are both pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not limit this.

[0131] Step S306: input the third sample images in the third sample set into the second initial feature extraction unit to obtain the second hand feature, wherein the third sample images in the third sample set all include human hands, and the third sample images carry the annotated coordinate information of the hand joints of the human hands.

[0132] In this embodiment, before using the first sample set to perform overall training on the parameter prediction model, the hand feature extraction unit and the hand parameter regression unit in the parameter prediction model may be trained first, thereby improving the subsequent training efficiency of the parameter prediction model using the first sample set.

[0133] Specifically, a third sample set is obtained, the third sample set includes a plurality of third sample images, each of the third sample images includes a human hand, and each of the third sample images carries the annotated coordinate information of the hand joint points of the human hand.

[0134] Optionally, the labeled coordinate information of the hand joint points may only include two-dimensional labeled coordinate information.

[0135] Optionally, the labeled coordinate information of the hand joint points may also include two-dimensional labeled coordinate information and three-dimensional labeled coordinate information.

[0136] Among them, the second initial feature extraction unit adopts the network structure of the encoder.

[0137] Step S307: input the second hand feature into a second initial parameter regression unit to obtain a third hand parameter.

[0138] In this embodiment, the second initial parameter regression unit may be an MLP network structure, and may be capable of performing at least one MLP operation to obtain a third human hand parameter for three-dimensional reconstruction of the human hand.

[0139] The third hand parameters may include hand camera parameters, absolute rotation parameters of wrist joints, and relative rotation parameters of finger joints.

[0140] Step S308: performing three-dimensional reconstruction of the human hand according to the third human hand parameter and the pre-trained human hand reconstruction model to obtain a three-dimensional hand model corresponding to the human hand.

[0141] In practical applications, the hand model with articulated and non-rigid defOrmations (MANO) model is compatible with the hand of the SMPL-H model, so the MANO model can be used as the pre-trained hand reconstruction model. Based on this, after obtaining the third hand parameters, the pre-trained hand reconstruction model can be used to achieve 3D reconstruction of the hand in the third sample image, and obtain the 3D hand model corresponding to the hand.

[0142] Step S309: Based on the three-dimensional human hand model, the coordinate information of the hand joint points is obtained as the predicted coordinate information of the hand joint points.

[0143] Furthermore, after performing three-dimensional reconstruction of the human hand in the third sample image and obtaining the corresponding three-dimensional hand model of the human hand, the three-dimensional predicted coordinate information of the hand joint points on the predicted three-dimensional hand model can be obtained; and the three-dimensional coordinate information of the hand joint points can be further two-dimensionally mapped to obtain the two-dimensional predicted coordinate information of the hand joint points.

[0144] Optionally, if the aforementioned annotated coordinate information of the hand joint points includes both two-dimensional annotated coordinate information and three-dimensional annotated coordinate information of the hand joint points, the two-dimensional predicted coordinate information and three-dimensional predicted coordinate information of the hand joint points can be used as the predicted coordinate information of the hand joint points.

[0145] Optionally, if the aforementioned annotated coordinate information of the hand joint points only includes two-dimensional annotated coordinate information of the hand joint points, the aforementioned two-dimensional predicted coordinate information of the hand joint points may be used as the predicted coordinate information of the hand joint points.

[0146] Step S310: Based on the difference between the predicted coordinate information of the hand joint points and the labeled coordinate information of the hand joint points, the second initial feature extraction unit and the second initial parameter regression unit are iteratively trained to obtain the hand feature extraction unit and the hand parameter regression unit.

[0147] In some implementations, the eighth loss value may be determined based on the degree of difference between the predicted coordinate information of the hand joint points and the labeled coordinate information of the hand joint points.

[0148] Optionally, if the aforementioned annotated coordinate information of the hand joints only includes the two-dimensional annotated coordinate information of the hand joints, the eighth loss value only includes the two-dimensional prediction loss value. The two-dimensional prediction loss value can be calculated by the following loss function:

[0149]

[0150]

[0151]

[0152] Among them, L h-2d Represents the two-dimensional prediction loss value for the hand joint point, p h-2d Represents the two-dimensional annotation coordinate information of the hand joint points, Π represents the two-dimensional projection function, The 2D predicted coordinate information obtained by mapping the 3D predicted coordinate information of the finger joint points output by the 3D human hand model (ie, MANO model) is represented. h Characterize the hand camera parameters, θ rf Characterizes the relative rotation parameters of the finger joint points output by the hand parameter prediction unit, θ gw Characterizes the absolute rotation parameters of the wrist joint point output by the hand parameter prediction unit.

[0153] Optionally, if the aforementioned annotated coordinate information of the hand joints includes both the two-dimensional annotated coordinate information and the three-dimensional annotated coordinate information of the hand joints, then the eighth loss value includes not only the aforementioned two-dimensional prediction loss but also the three-dimensional prediction loss value. The three-dimensional prediction loss value can be calculated by the following loss function:

[0154]

[0155] Among them, L h-3d Characterizes the three-dimensional prediction loss value of the hand joint point, p h-3d Characterize the three-dimensional coordinate information of the hand joint points, Characterizes the three-dimensional predicted coordinate information of the hand joint points output by the three-dimensional human hand model.

[0156] In addition, considering that the rotation angles of each finger joint of the human hand are limited, in order to prevent the problem of hand deformity in the three-dimensional reconstructed hand model, the ninth loss value can be calculated based on the relative rotation parameters of the joints of the human body and using the following regularization loss function.

[0157]

[0158] Among them, L h-r It is characterized by the regularization loss value for preventing hand deformity, that is, the ninth loss value.

[0159] Finally, the second total loss value can be determined according to the eighth loss value and the ninth loss value. And according to the second total loss value, the second initial feature extraction unit and the second initial parameter regression unit are iteratively trained until the second training condition is met, and a converged second initial feature extraction unit is obtained as the aforementioned human hand feature extraction unit, and a converged second initial parameter regression unit is obtained as the aforementioned human hand parameter regression unit.

[0160] Exemplarily, the second total loss value L h It can be expressed by the following formula:

[0161] L h =L h-2d +L h-3d +L h-r

[0162] Among them, the second training condition can be: detecting that the second total loss value is less than the preset value, detecting that the second total loss value no longer changes, or the number of training times reaches the preset number of times, etc. It can be understood that after the second initial feature extraction unit and the second initial parameter regression unit are iteratively trained for multiple training cycles according to the third sample set, wherein each training cycle includes multiple iterative trainings, the parameters in the second initial feature extraction unit and the second initial parameter regression unit are continuously optimized, so that the above-mentioned second total loss value becomes smaller and smaller, and finally becomes a fixed value, or is less than the above-mentioned preset value. At this time, it means that the second initial feature extraction unit and the second initial parameter regression unit have converged; of course, it can also be determined that the second initial feature extraction unit and the second initial parameter regression unit have converged after the training times reach the preset number of times. At this time, the converged second initial feature extraction unit can be used as the aforementioned human hand feature extraction unit, and the converged second initial parameter regression unit can be used as the aforementioned human hand parameter regression unit. Among them, the preset value and the preset number of times are both pre-set, and their values ​​can also be adjusted according to different application scenarios. This embodiment does not limit this.

[0163] In this embodiment, after the hand feature extraction unit, the hand parameter regression unit and the human body feature extraction unit in the parameter prediction model are obtained by pre-training through steps S301 to S310, the parameter prediction model can be trained by using the first sample set through the model training method in any of the above embodiments, so as to obtain a three-dimensional reconstruction parameter prediction model. Among them, the specific implementation method of training the entire parameter prediction model can refer to the content of any of the above embodiments, and will not be repeated here. It can be understood that the first sample set can be a sample set different from the second sample set and the third sample set, and the first sample set can also be a sample set composed of the second sample set and the third sample set. That is to say, when training all units in the parameter prediction model, each round of input batch can include the second sample image in the second sample set and the third sample image in the third sample set, so that the final three-dimensional reconstruction parameter prediction model can output more accurate hand parameters and human body parameters, so that the wrist connection of the three-dimensional human body model reconstructed based on the predicted output hand parameters and human body parameters is more natural.

[0164] In this embodiment, before the first sample set is used to perform overall training on the parameter prediction model, the second sample set can be used in advance to perform iterative training on the first initial feature extraction unit, so that the converged first initial feature extraction unit is used as the human feature extraction unit in the parameter prediction model. In addition, the third sample set is also used in advance to perform iterative training on the second initial feature extraction unit and the second initial parameter regression unit, and the converged second initial feature extraction unit is used as the aforementioned human hand feature extraction unit, and the converged second initial parameter regression unit is used as the aforementioned human hand parameter regression unit. In this way, it is equivalent to pre-improving the feature extraction capability of the human body feature extraction unit in the parameter prediction model, and improving the feature extraction capability of the human hand feature extraction unit and the parameter regression capability of the human hand parameter regression unit in the parameter prediction model; thereby, the training speed and training efficiency of the overall iterative training of the parameter prediction model using the first sample set can be greatly improved. At the same time, the sample set with only human body annotations and the sample set with only hand annotations are fully utilized.

[0165] Please refer to Fig. 9 , which shows a structural block diagram of a model training device 400 provided in an embodiment of the present application. The device 400 may include: a training sample acquisition module 410, a parameter prediction module 420, a three-dimensional reconstruction module 430, a coordinate acquisition module 440 and a model training module 450.

[0166] The training sample acquisition module 410 is used to acquire a first sample set, wherein a first sample image in the first sample set includes a human body, and the first sample image carries the labeled coordinate information of the joint points of the human body.

[0167] The parameter prediction module 420 is used to input the first sample image into a parameter prediction model to obtain a first hand parameter and a first human body parameter of the human body.

[0168] The three-dimensional reconstruction module 430 is used to perform three-dimensional reconstruction on the human body according to the first human hand parameter of the human body, the first human body parameter and a pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body.

[0169] The coordinate acquisition module 440 is used to acquire the coordinate information of the joint points based on the three-dimensional human body model as the predicted coordinate information of the joint points.

[0170] The model training module 450 is used to iteratively train the parameter prediction model based on the degree of difference between the predicted coordinate information and the labeled coordinate information, and the degree of difference between the first hand parameters and the second hand parameters, to obtain a three-dimensional reconstruction parameter prediction model, wherein the second hand parameters are predicted based on the first human body parameters.

[0171] In some embodiments, the first human body parameters include relative rotation parameters of a wrist joint point and associated joint points, the relative rotation parameters are rotation parameters of a joint point compared to a corresponding parent node, and the associated joint points include joint points of the human body that are associated with the wrist joint point. The model training device 400 may also include: a second human hand parameter acquisition module. The second human hand parameter acquisition module may be used to iteratively train the parameter prediction model based on the degree of difference between the predicted coordinate information and the annotated coordinate information, and the degree of difference between the first human hand parameter and the second human hand parameter, and before obtaining the three-dimensional reconstruction parameter prediction model, predict the rotation parameters of the wrist key point compared to the root node based on the relative rotation parameters of the wrist joint point and the associated joint points, and obtain the predicted absolute rotation parameters of the wrist joint point as the second human hand parameter.

[0172] In this manner, the first hand parameter includes the absolute rotation parameter of the wrist joint point, and the absolute rotation parameter is the rotation parameter of the joint point compared to the corresponding root node. The model training module 450 may include: a first loss acquisition unit, a second loss acquisition unit, a total loss acquisition unit, and an iterative training unit. Among them, the first loss acquisition unit can be used to determine the first loss value based on the degree of difference between the predicted coordinate information and the labeled coordinate information. The second loss acquisition unit can be used to determine the second loss value based on the degree of difference between the absolute rotation parameter of the wrist joint point and the predicted absolute rotation parameter. The total loss acquisition unit can be used to obtain the total loss value based on the first loss value and the second loss value. The iterative training unit can be used to iteratively train the parameter prediction model according to the total loss value until the target training condition is met to obtain the three-dimensional reconstruction parameter prediction model.

[0173] In this manner, the annotated coordinate information includes two-dimensional annotated coordinates and three-dimensional annotated coordinates; the coordinate acquisition module 440 can be specifically used to acquire the three-dimensional coordinate information of the joint point based on the three-dimensional human body model to obtain the three-dimensional predicted coordinates of the joint point; the three-dimensional predicted coordinates of the joint point are mapped to a preset plane to obtain the two-dimensional predicted coordinates of the joint point. The first loss acquisition unit can be specifically used to: determine the three-dimensional predicted loss value based on the degree of difference between the three-dimensional predicted coordinates and the three-dimensional annotated coordinates; determine the two-dimensional predicted loss value based on the degree of difference between the two-dimensional predicted coordinates and the two-dimensional annotated coordinates; determine the first loss value based on the three-dimensional predicted loss value and the two-dimensional predicted loss value.

[0174] In some embodiments, the parameter prediction model includes a human feature extraction unit, a feature fusion unit, a human parameter regression unit, a hand feature extraction unit, and a hand parameter regression unit. The parameter prediction module 420 can be specifically used to: downsample the first sample image to obtain a first target image, wherein the image resolution of the first target image is smaller than the image resolution of the first sample image; extract the first human feature of the first target image and the heat map corresponding to the hand region of the human body by using the human feature extraction unit; obtain a hand region image from the first sample image based on the heat map corresponding to the hand region, and extract the first hand feature of the hand region image by using the human hand feature extraction unit; fuse the first human feature and the first hand feature by using the feature fusion unit to obtain the second human feature; perform parameter regression on the first hand feature by using the hand parameter regression unit to obtain the first hand parameter, and perform parameter regression on the second human feature by using the human parameter regression unit to obtain the first human parameter.

[0175] In this manner, the model training device 400 may further include: a first pre-training module. The first pre-training module may be used to obtain a second sample set before inputting the first sample image into the parameter prediction model to obtain the first human hand parameter and the first human body parameter of the human body, wherein the second sample images in the second sample set all include the human body, and the second sample images carry the two-dimensional annotated coordinate information of the joints of the human body and the hand annotated frame of the human body; down-sample the second sample image to obtain a second target image; extract the first human body joint point heat map and the first hand joint point heat map in the second target image using the first initial feature extraction unit; generate annotated human body joint point heat map and annotated hand joint point heat map based on the two-dimensional annotated coordinate information of the joints of the human body and the hand annotated frame; iteratively train the first initial feature extraction unit based on the difference between the first human body joint point heat map and the annotated human body joint point heat map, and the difference between the first hand joint point heat map and the annotated hand joint point heat map to obtain the human body feature extraction unit.

[0176] In this manner, the model training device 400 may further include: a second pre-training module. The second pre-training module may be used to input the third sample image in the third sample set into the second initial feature extraction unit to obtain the second hand feature before inputting the first sample image into the parameter prediction model to obtain the first human hand parameter and the first human body parameter of the human body, wherein the third sample images in the third sample set all include human hands, and the third sample images carry the annotated coordinate information of the hand joints of the human hands; input the second hand feature into the second initial parameter regression unit to obtain the third hand parameter; perform three-dimensional reconstruction of the human hand according to the third hand parameter and the pre-trained human hand reconstruction model to obtain the three-dimensional human hand model corresponding to the human hand; obtain the coordinate information of the hand joints based on the three-dimensional human hand model as the predicted coordinate information of the hand joints; perform iterative training on the second initial feature extraction unit and the second initial parameter regression unit based on the difference between the predicted coordinate information of the hand joints and the annotated coordinate information of the hand joints to obtain the hand feature extraction unit and the hand parameter regression unit.

[0177] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0178] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0179] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0180] In summary, the hand parameters and hand parameter prediction tasks used for three-dimensional reconstruction of the human body are placed in the same parameter prediction model for training. In addition, in the process of iterative training of the parameter prediction model, the relative rotation parameters of the wrist joint predicted by the human body parameter regression unit are used to infer the absolute rotation parameters of the wrist joint to obtain the predicted absolute rotation parameters; thus, the self-supervised loss value is determined based on the difference between the absolute rotation parameters of the wrist joint output by the hand parameter regression unit and the inverse predicted absolute rotation parameters. And the self-supervised loss value is used as one of the constraints of model training to iteratively train the parameter prediction model to obtain the final three-dimensional reconstruction parameter prediction model. In this way, it is equivalent to using the relationship between the human body and the human hand to constrain the parameter prediction model, so that the parameters for three-dimensional reconstruction output by the final three-dimensional reconstruction parameter prediction model in actual application can be used to reconstruct a three-dimensional human body model with a more accurate and natural connection between the human body and the human hand; in other words, the prediction of the parameters of three-dimensional reconstruction using the three-dimensional reconstruction parameter prediction model of the present application has greatly improved the accuracy and naturalness of three-dimensional reconstruction of the human body.

[0181] The following will be combined Fig.10 A computer device provided by the present application is described.

[0182] Reference Fig.10 , Fig.10 The structural block diagram of a computer device 500 provided in an embodiment of the present application is shown, and the above method provided in an embodiment of the present application can be executed by the computer device 500. The computer device can be an electronic terminal with a data processing function, and the electronic terminal includes but is not limited to a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) player, and smart home devices, etc.; of course, the computer device can also be a server, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0183] The computer device 500 in the embodiment of the present application may include one or more of the following components: a processor 501, a memory 502, and one or more applications, wherein the one or more applications may be stored in the memory 502 and configured to be executed by one or more processors 501, and the one or more programs are configured to execute the method as described in the aforementioned method embodiment.

[0184] The processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts of the entire computer device 500, and executes various functions and processes data of the computer device 500 by running or executing instructions, programs, code sets or instruction sets stored in the memory 502, and calling data stored in the memory 502. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 501 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem can also be integrated into the processor 501 and implemented separately through a communication chip.

[0185] The memory 502 may include a random access memory (RAM) or a read-only memory (ROM). The memory 502 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as the various corresponding relationships described above) created by the computer device 500 during use.

[0186] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0187] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0188] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0189] Please refer to Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 600 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0190] The computer readable storage medium 600 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer readable storage medium 600 has storage space for program code 610 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 610 can be compressed, for example, in an appropriate form.

[0191] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the steps in the above-mentioned method embodiments.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a first sample set, wherein a first sample image in the first sample set includes a human body, and the first sample image carries annotated coordinate information of joint points of the human body; Inputting the first sample image into a parameter prediction model to obtain a first human hand parameter and a first human body parameter of the human body; Reconstructing the human body in three dimensions according to the first human hand parameter of the human body, the first human body parameter, and a pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body; Based on the three-dimensional human body model, acquiring coordinate information of the joint points as predicted coordinate information of the joint points; Based on the difference between the predicted coordinate information and the marked coordinate information, and the difference between the first hand parameters and the second hand parameters, the parameter prediction model is iteratively trained to obtain a three-dimensional reconstruction parameter prediction model, wherein the second hand parameters are predicted based on the first human body parameters.

2. The method according to claim 1, characterized in that The first human body parameters include relative rotation parameters of a wrist joint point and associated joint points, the relative rotation parameters are rotation parameters of a joint point compared to a corresponding parent node, and the associated joint points include joint points associated with the wrist joint point among the joint points of the human body; Before iteratively training the parameter prediction model based on the difference between the predicted coordinate information and the annotated coordinate information and the difference between the first hand parameter and the second hand parameter to obtain the 3D reconstruction parameter prediction model, the method further includes: Based on the relative rotation parameters of the wrist joint point and the associated joint points, the rotation parameters of the wrist key point compared to the root node are predicted to obtain the predicted absolute rotation parameters of the wrist joint point as the second hand parameter.

3. The method according to claim 2, characterized in that The first human hand parameter includes an absolute rotation parameter of the wrist joint point, where the absolute rotation parameter is a rotation parameter of the joint point compared to the corresponding root node; Based on the difference between the predicted coordinate information and the annotated coordinate information, and the difference between the first hand parameter and the second hand parameter, iteratively training the parameter prediction model to obtain a 3D reconstruction parameter prediction model includes: Determining a first loss value based on a degree of difference between the predicted coordinate information and the annotated coordinate information; determining a second loss value based on a degree of difference between the absolute rotation parameter of the wrist joint point and the predicted absolute rotation parameter; Obtaining a total loss value according to the first loss value and the second loss value; According to the total loss value, the parameter prediction model is iteratively trained until a target training condition is met, thereby obtaining the three-dimensional reconstruction parameter prediction model.

4. The method according to claim 3, characterized in that The step of acquiring the coordinate information of the joint points based on the three-dimensional human body model as the predicted coordinate information of the joint points includes: Based on the three-dimensional human body model, acquiring three-dimensional coordinate information of the joint points to obtain the three-dimensional predicted coordinates of the joint points; Mapping the three-dimensional predicted coordinates of the joint point to a preset plane to obtain the two-dimensional predicted coordinates of the joint point; The annotated coordinate information includes two-dimensional annotated coordinates and three-dimensional annotated coordinates, and determining a first loss value based on a difference between the predicted coordinate information and the annotated coordinate information includes: Determining a three-dimensional prediction loss value based on a degree of difference between the three-dimensional predicted coordinates and the three-dimensional annotated coordinates; Determining a two-dimensional prediction loss value based on a degree of difference between the two-dimensional prediction coordinates and the two-dimensional annotated coordinates; The first loss value is determined according to the three-dimensional predicted loss value and the two-dimensional predicted loss value.

5. The method according to claim 1, characterized in that The parameter prediction model has a human feature extraction unit, a feature fusion unit, a human parameter regression unit, a hand feature extraction unit and a hand parameter regression unit; The step of inputting the first sample image into a parameter prediction model to obtain a first human hand parameter and a first human body parameter of the human body includes: Downsampling the first sample image to obtain a first target image, wherein the image resolution of the first target image is smaller than the image resolution of the first sample image; Utilizing the human feature extraction unit, extracting a first human feature of the first target image and a heat map corresponding to a hand region of the human body; Based on the heat map corresponding to the hand area, a hand area image is acquired from the first sample image, and a first hand feature of the hand area image is extracted by using the hand feature extraction unit; Using the feature fusion unit, the first human body feature and the first human hand feature are fused to obtain a second human body feature; The hand parameter regression unit is used to perform parameter regression on the first hand feature to obtain the first hand parameter, and the body parameter regression unit is used to perform parameter regression on the second body feature to obtain the first body parameter.

6. The method according to claim 5, characterized in that Before inputting the first sample image into the parameter prediction model to obtain the first hand parameter and the first human body parameter of the human body, the method further includes: Acquire a second sample set, wherein second sample images in the second sample set all include a human body, and the second sample images carry two-dimensional annotated coordinate information of joint points of the human body and a hand annotated frame of the human body; Downsampling the second sample image to obtain a second target image; Using the first initial feature extraction unit, extracting a first human joint point heat map and a first hand joint point heat map in the second target image; Based on the two-dimensional labeled coordinate information of the joints of the human body and the hand labeling frame, generating a heat map of labeled human body joints and a heat map of labeled hand joints; Based on the degree of difference between the first human body joint point heat map and the labeled human body joint point heat map, and the degree of difference between the first hand joint point heat map and the labeled hand joint point heat map, the first initial feature extraction unit is iteratively trained to obtain the human body feature extraction unit.

7. The method according to claim 5, characterized in that Before inputting the first sample image into the parameter prediction model to obtain the first hand parameter and the first human body parameter of the human body, the method further includes: Inputting the third sample images in the third sample set into the second initial feature extraction unit to obtain a second human hand feature, wherein the third sample images in the third sample set all include human hands, and the third sample images carry the annotated coordinate information of the hand joint points of the human hands; Inputting the second hand feature into a second initial parameter regression unit to obtain a third hand parameter; Reconstructing the human hand in three dimensions according to the third human hand parameter and a pre-trained human hand reconstruction model to obtain a three-dimensional human hand model corresponding to the human hand; Based on the three-dimensional human hand model, obtaining coordinate information of the hand joint points as predicted coordinate information of the hand joint points; Based on the difference between the predicted coordinate information of the hand joint points and the labeled coordinate information of the hand joint points, the second initial feature extraction unit and the second initial parameter regression unit are iteratively trained to obtain the human hand feature extraction unit and the human hand parameter regression unit.

8. A model training device, characterized in that: The device comprises: A training sample acquisition module, used to acquire a first sample set, wherein a first sample image in the first sample set includes a human body, and the first sample image carries annotated coordinate information of joint points of the human body; A parameter prediction module, used for inputting the first sample image into a parameter prediction model to obtain a first human hand parameter and a first human body parameter of the human body; A three-dimensional reconstruction module, used to perform three-dimensional reconstruction of the human body according to the first human hand parameter of the human body, the first human body parameter and a pre-trained human body reconstruction model to obtain a three-dimensional human body model corresponding to the human body; A coordinate acquisition module, used for acquiring the coordinate information of the joint points based on the three-dimensional human body model as the predicted coordinate information of the joint points; A model training module is used to iteratively train the parameter prediction model based on the degree of difference between the predicted coordinate information and the labeled coordinate information, and the degree of difference between the first hand parameters and the second hand parameters, so as to obtain a three-dimensional reconstruction parameter prediction model, wherein the second hand parameters are predicted based on the first human body parameters.

9. A computer device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images