Three-dimensional human body model reconstruction method, device, electronic device and readable medium

Abstract

The present disclosure provides a method, device, electronic device and readable medium for reconstructing a three-dimensional human body model, wherein the method for reconstructing a three-dimensional human body model comprises: obtaining an original RGB two-dimensional image including a human body image and a silhouette of the human body image; inputting the original RGB two-dimensional image and the silhouette into an SMPL prediction network to construct an SMPL human body parameter model; rendering the SMPL human body parameter model with a front image and a back image of the human body based on the silhouette to obtain a normal map and texture features of a human body without clothing; and reconstructing a three-dimensional human body model according to the normal map and texture features of the human body. Through the embodiments of the present disclosure, the dependence on training data can be reduced, the accuracy and reliability of normal map generation can be improved, and the surface texture reconstruction effect of the three-dimensional human body model can be further improved.

Description

Background Art

[0002] Currently, most existing reconstruction algorithms are based on single-image 3D human reconstruction. This reconstruction method has a limited observable part of the input image and lacks information about the occluded or reverse side of the person, so the recovery accuracy of the invisible area in the reconstructed structure is low.

[0003] In the related art, the positive / negative normal map is predicted for the original image, and then the normal map and the original image are used as inputs of the 3D human body reconstruction network.

[0004] Although the 3D human body reconstruction scheme in the related art can improve the reconstruction accuracy of invisible areas to a certain extent, it still has at least the following technical problems:

[0005] (1) The normal map prediction is obtained by passing the RGB image through the Pix2Pix network. However, to achieve sufficient generalization in different postures, a large amount of training data is required, and the accuracy of the normal map prediction obtained through training is limited.

[0006] (2) The normal map and the original image are used as the input of the 3D human body reconstruction network, which may result in poor detail accuracy of the reconstructed 3D human body model.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] The purpose of the present disclosure is to provide a three-dimensional human body model reconstruction method, device, electronic device and readable medium, which are used to overcome the problems of poor reliability of three-dimensional human body model reconstruction caused by limitations and defects of related technologies to at least a certain extent.

[0009] According to a first aspect of an embodiment of the present disclosure, a method for reconstructing a three-dimensional human body model is provided, comprising: obtaining an original RGB two-dimensional image including a human body image and a silhouette of the human body image; inputting the original RGB two-dimensional image and the silhouette into an SMPL prediction network to construct an SMPL human body parameter model; rendering a human body front image and a human body back image of the SMPL human body parameter model based on the silhouette to obtain a human body normal map and texture features without clothing; and reconstructing a three-dimensional human body model according to the human body normal map and the texture features.

[0010] In an exemplary embodiment of the present disclosure, it further includes:

[0011] Determining voxel features, normal features and texture features of the human body normal map;

[0012] The voxel features, the normal features and the texture features are fused into the reconstructed three-dimensional human body model based on an attention mechanism.

[0013] In an exemplary embodiment of the present disclosure, determining the voxel features, the normal features and the texture features of the human body normal map includes:

[0014] voxelize the human body normal map to determine the voxel features;

[0015] Extracting a texture map and a normal map of a dressed human body from the original RGB two-dimensional image;

[0016] Feature extraction is performed on the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the texture feature.

[0017] In an exemplary embodiment of the present disclosure, extracting features from the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the texture feature includes:

[0018] Projecting the human body normal map onto a two-dimensional plane based on a bilinear interpolation algorithm;

[0019] Extracting the normal feature and the texture feature according to the projection result of the two-dimensional plane;

[0020] Determine the correspondence between any sampling point on the human body normal map and the normal feature, and the correspondence between the sampling point and the texture feature.

[0021] In an exemplary embodiment of the present disclosure, fusing the voxel feature, the normal feature, and the texture feature into the reconstructed three-dimensional human body model based on an attention mechanism includes:

[0022] Based on the attention mechanism, the voxel feature, the normal feature and the texture feature are normalized and assigned a preset weight value;

[0023] Decoding the human body normal map into an SMPL mesh to determine vertex coordinates, β and θ;

[0024] Substituting the vertex coordinates, the β, the θ, the preset weight value and the corresponding feature vector into a body fitting loss function, the feature vector including the voxel feature, the normal feature and the texture feature;

[0025] Based on the magnitude relationship between the calculation result of the body fitting loss function and the result of the preset loss function, the weight value of the feature vector is adjusted.

[0026] In an exemplary embodiment of the present disclosure, adjusting the weight value of the feature vector based on the magnitude relationship between the calculation result of the body fitting loss function and the result of the preset loss function includes:

[0027] Determining whether a calculated result of the body fitting loss function is greater than a result of the preset loss function;

[0028] If it is determined that the calculation result of the body fitting loss function is greater than the preset loss function result, the weight value is adjusted until the calculation result is less than or equal to the preset loss function result.

[0029] In an exemplary embodiment of the present disclosure, the expression of the body fitting loss function includes:

[0030]

[0031]

[0032] Among them, the v j represents the jth vertex, the n s represents the total number of vertices, the c(v j ) represents the eigenvector corresponding to the vertex, and F(c(v j )) represents the occupancy value of the vertex, and the λ R is a constant, the β init With the θ init is the initial parameter, and η represents the preset coefficient.

[0033] According to a second aspect of an embodiment of the present disclosure, a 3D human body model reconstruction device is provided, comprising:

[0034] An acquisition module, configured to acquire an original RGB two-dimensional image including a human body image and a silhouette of the human body image;

[0035] A construction module, configured to input the original RGB two-dimensional image and the silhouette into a SMPL prediction network to construct a SMPL human body parameter model;

[0036] A rendering module is configured to render a front image and a back image of a human body of the SMPL human body parameter model based on the silhouette to obtain a normal map and texture features of a human body without clothing;

[0037] A reconstruction module is configured to reconstruct a three-dimensional human body model according to the human body normal map and the texture features.

[0038] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0039] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the three-dimensional human body model reconstruction method as described in any one of the above items is implemented.

[0040] In the disclosed embodiment, by acquiring an original RGB two-dimensional image including a human body image and a silhouette of the human body image, and inputting the original RGB two-dimensional image and the silhouette into a SMPL prediction network, the SMPL prediction network can perform prediction training on the human body image area and the background based on the silhouette to construct a SMPL human body parameter model, and render the SMPL human body parameter model with a front human body image and a back human body image based on the silhouette to obtain a normal map and texture features of a human body without clothing, and finally reconstruct a three-dimensional human body model according to the normal map and the texture features, which can reduce the dependence on training data, improve the accuracy and reliability of normal map generation, and further improve the surface texture reconstruction effect of the three-dimensional human body model.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0043] Figure 1 A schematic diagram showing a reconstruction scheme of a three-dimensional human body model in the related art;

[0044] Figure 2 is a flow chart of a method for reconstructing a three-dimensional human body model in an exemplary embodiment of the present disclosure;

[0045] Figure 3 is a flow chart of another three-dimensional human body model reconstruction method in an exemplary embodiment of the present disclosure;

[0046] Figure 4 is a flow chart of another three-dimensional human body model reconstruction method in an exemplary embodiment of the present disclosure;

[0047] Figure 5is a flow chart of another three-dimensional human body model reconstruction method in an exemplary embodiment of the present disclosure;

[0048] Figure 6 is a flow chart of another three-dimensional human body model reconstruction method in an exemplary embodiment of the present disclosure;

[0049] Figure 7 is a flow chart of another method for reconstructing a three-dimensional human body model in an exemplary embodiment of the present disclosure;

[0050] Figure 8 is a schematic diagram of the architecture of a three-dimensional human body model reconstruction solution in an exemplary embodiment of the present disclosure;

[0051] Fig. 9 is a schematic diagram of the architecture of another three-dimensional human body model reconstruction solution in an exemplary embodiment of the present disclosure;

[0052] Fig.10 is a schematic diagram of the architecture of another three-dimensional human body model reconstruction solution in an exemplary embodiment of the present disclosure;

[0053] Fig.11 is a block diagram of a three-dimensional human body model reconstruction device in an exemplary embodiment of the present disclosure;

[0054] Fig.12 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0056] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0057] Figure 1 A schematic diagram showing a reconstruction scheme of a three-dimensional human body model in the related art.

[0058] like Figure 1 As shown, the reconstruction scheme of the three-dimensional human body model in the related art includes the following steps:

[0059] Step S102, input image.

[0060] Step S104, the image is further input into the Pix2Pix network.

[0061] Step S106: The Pix2Pix network generates a normal graph.

[0062] Step S108, determining three-dimensional human body parameters through SMPL parameter prediction.

[0063] Step S110, outputting SMPL parameters corresponding to the three-dimensional human body parameters.

[0064] Step S112, performing three-dimensional feature extraction.

[0065] Step S114, extracting two-dimensional features according to the normal map and the input image.

[0066] Step S116, obtaining sampling points of the three-dimensional human body model.

[0067] Step S118, determining the characteristics of the sampling points.

[0068] Step S120, obtaining an implicit function for reconstructing a three-dimensional human body model.

[0069] Step S122, predicting the sampling point positions.

[0070] Step S124, reconstructing a three-dimensional human body model based on the prediction result.

[0071] In the process of obtaining a three-dimensional human body model based on the above implementation method, the prediction of the normal map is obtained by passing the RGB image through the Pix2Pix network. However, in order to achieve sufficient generalization in different postures, a large amount of training data is required, and the accuracy of the normal map prediction is limited, and the accuracy of the texture details of the three-dimensional human body model is poor.

[0072] In view of the defects of the related art, the present invention proposes a new three-dimensional human body model reconstruction scheme. Figure 2-12 Exemplary embodiments of the present disclosure are described in detail.

[0073] Figure 2 is a flow chart of a method for reconstructing a three-dimensional human body model in an exemplary embodiment of the present disclosure.

[0074] refer to Figure 2 , the three-dimensional human body model reconstruction method may include:

[0075] Step S202, obtaining an original RGB two-dimensional image including a human body image and a silhouette of the human body image.

[0076] Step S204: input the original RGB two-dimensional image and the silhouette into the SMPL prediction network to construct an SMPL human body parameter model.

[0077] Step S206, rendering a front view image and a back view image of the human body of the SMPL human body parameter model based on the silhouette to obtain a normal map and texture features of the human body without clothing.

[0078] Step S208: reconstructing a three-dimensional human body model according to the human body normal map and the texture features.

[0079] In the disclosed embodiment, by acquiring an original RGB two-dimensional image including a human body image and a silhouette of the human body image, and inputting the original RGB two-dimensional image and the silhouette into a SMPL prediction network, the SMPL prediction network can perform prediction training on the human body image area and the background based on the silhouette to construct a SMPL human body parameter model, and render the SMPL human body parameter model with a front human body image and a back human body image based on the silhouette to obtain a normal map and texture features of a human body without clothing, and finally reconstruct a three-dimensional human body model according to the normal map and the texture features, which can reduce the dependence on training data, improve the accuracy and reliability of normal map generation, and further improve the surface texture reconstruction effect of the three-dimensional human body model.

[0080] Among them, the RGB two-dimensional image is a two-dimensional image of the three primary colors of red, green and blue.

[0081] In addition, the SMPL mode is a parametric human body model that can be used for arbitrary human body modeling and animation driving. β represents 10 parameters such as individual height, weight, head-to-body ratio, etc., and θ represents 75 parameters representing the overall motion posture of the human body and the relative angles of 24 joints.

[0082] Next, each step of the three-dimensional human body model reconstruction method is described in detail.

[0083] In an exemplary embodiment of the present disclosure, Figure 3 As shown, the three-dimensional human body model reconstruction method also includes:

[0084] Step S302, determining the voxel features, normal features and texture features of the human body normal map.

[0085] Step S304: Based on the attention mechanism, the voxel feature, the normal feature and the texture feature are integrated into the reconstructed three-dimensional human body model.

[0086] In an exemplary embodiment of the present disclosure, Figure 4 As shown, determining the voxel features, normal features and texture features of the human body normal map includes:

[0087] Step S402: voxelize the human body normal map to determine the voxel features.

[0088] Step S404: extracting a texture map and a normal map of the dressed human body from the original RGB two-dimensional image.

[0089] Step S406: extract features from the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the texture feature.

[0090] In an exemplary embodiment of the present disclosure, the voxel feature is a vector feature of each sampling point.

[0091] In an exemplary embodiment of the present disclosure, Figure 5 As shown, the feature extraction network is used to extract features from the texture map and the normal map of the dressed human body to obtain the normal features and the texture features, including:

[0092] Step S502: projecting the human body normal map onto a two-dimensional plane based on a bilinear interpolation algorithm.

[0093] In an exemplary embodiment of the present disclosure, a bilinear interpolation algorithm is used to compensate for the loss of the human body normal map projected onto a two-dimensional plane.

[0094] Step S504: extracting the normal feature and the texture feature according to the projection result of the two-dimensional plane.

[0095] Step S506, determining the correspondence between any sampling point on the human body normal map and the normal feature, and the correspondence between the sampling point and the texture feature.

[0096] In an exemplary embodiment of the present disclosure, Figure 6As shown, fusing the voxel feature, the normal feature and the texture feature into the reconstructed three-dimensional human body model based on the attention mechanism includes:

[0097] Step S602: normalize the voxel features, the normal features and the texture features based on the attention mechanism, and assign preset weight values.

[0098] In an exemplary embodiment of the present disclosure, the initial preset weight value may be set to 1 / 3.

[0099] Step S604: decode the human body normal map into an SMPL mesh to determine vertex coordinates, β and θ.

[0100] Step S606, substituting the vertex coordinates, the β, the θ, the preset weight value and the corresponding feature vector into the body fitting loss function, the feature vector including the voxel feature, the normal feature and the texture feature.

[0101] Step S608: adjusting the weight value of the feature vector based on the size relationship between the calculation result of the body fitting loss function and the result of the preset loss function.

[0102] In an exemplary embodiment of the present disclosure, Figure 7 As shown, based on the size relationship between the calculation result of the body fitting loss function and the result of the preset loss function, adjusting the weight value of the feature vector includes:

[0103] Step S702, determining whether the calculated result of the body fitting loss function is greater than the preset loss function result.

[0104] Step S704: If it is determined that the calculation result of the body fitting loss function is greater than the preset loss function result, the weight value is adjusted until the calculation result is less than or equal to the preset loss function result.

[0105] In an exemplary embodiment of the present disclosure, the expression of the body fitting loss function includes:

[0106]

[0107]

[0108] Among them, the v j represents the jth vertex, the n s represents the total number of vertices, the c(v j ) represents the eigenvector corresponding to the vertex, and F(c(v j )) represents the occupancy value of the vertex, and the λR is a constant, the β init With the θ init is the initial parameter, and η represents the preset coefficient.

[0109] In an exemplary embodiment of the present disclosure, Figure 8 As shown, the framework 800 of the progressive reconstruction strategy guides the generation of the normal map based on the three-dimensional prior knowledge of the human body, so that the predicted front and back normal vector maps are more consistent with the human body characteristics. At the same time, since the visible front area and the invisible back area of ​​the human body share a silhouette map, the rough normal map without clothing is deformed and refined based on the input RGB image and the two-dimensional contour map of the human body, which can further improve the generation effect of the normal map and the texture map. The implementation process is as follows:

[0110] 1) According to the original RGB image, the corresponding SMPL parameters are generated based on the existing SMPL parameter prediction network 900, and the SMPL human body parameter model is constructed;

[0111] 2) Render the front and back sides of the SMPL human body parametric model based on the silhouette differentiable renderer to obtain the normal map of the human body without clothing;

[0112] 3) The normal map of the unclothed human body, the original RGB map, and the silhouette map corresponding to the RGB map are sent to the normal map prediction network to generate the front and back normal maps and the back texture map of the clothed human body.

[0113] In an exemplary embodiment of the present disclosure, Fig.10 As shown, the implicit reconstruction prediction framework 1000 determines whether the sampling point is located on the three-dimensional target surface based on the sampling point feature, and performs implicit prediction by fusing voxel features, normal features and texture features, which can further refine the surface texture reconstruction details. The implementation process is as follows:

[0114] 1) The SMPL human body parameter model is voxelized and voxel features are extracted based on a 3D convolutional network. For any sampling point P, the voxel features of the sampling point can be obtained based on multi-scale trilinear interpolation;

[0115] 2) The original RGB image is sent to the feature extraction network as the front texture map and the front normal map of the dressed body to obtain the corresponding texture features and normal features. The reverse texture map and normal map are processed in the same way as above;

[0116] 3) Any sampling point P is projected onto the image plane and it is determined whether it is located on the front or back side of the human body;

[0117] 4) If the projection is located on the front of the human body, bilinear interpolation is performed at the projection pixel coordinate Π(P) to extract pixel texture features and normal features respectively. If the projection is located on the back of the human body, the same process is performed;

[0118] 5) Normalize the texture features, normal features, and voxel features respectively, and assign the same initial attention weight to obtain fused features, perform implicit prediction, and obtain the initial 3D human body model;

[0119] 6) Decode the initial 3D human body model into SMPL mesh and obtain the corresponding vertex, β, and θ parameters;

[0120] 7) The predicted vertices, β, θ and their corresponding features are introduced into the body fitting loss function as shown below:

[0121]

[0122]

[0123] Among them, v j represents the jth vertex, n s represents the total number of vertices, c(v j ) represents the eigenvector corresponding to the point, F(c(v j )) represents the occupancy value of the point, λ R is a constant, obtained experimentally, β init With θ init The initial parameters generated by the SMPL parameter prediction network, wherein η represents a preset coefficient.

[0124] 8) If L B If it is greater than the threshold, the attention weight is updated and the above steps 5) 6) 7) are continued.

[0125] Corresponding to the above method embodiments, the present disclosure also provides a three-dimensional human body model reconstruction device, which can be used to execute the above method embodiments.

[0126] Fig.11 It is a block diagram of a 3D human body model reconstruction device in an exemplary embodiment of the present disclosure.

[0127] refer to Fig.11 , the 3D human body model reconstruction device 1100 may include:

[0128] The acquisition module 1102 is configured to acquire an original RGB two-dimensional image including a human body image and a silhouette of the human body image.

[0129] The construction module 1104 is configured to input the original RGB two-dimensional image and the silhouette into the SMPL prediction network to construct the SMPL human body parameter model.

[0130] The rendering module 1106 is configured to render the front image and the back image of the human body of the SMPL human body parameter model based on the silhouette to obtain a normal map and texture features of the human body without clothing.

[0131] The reconstruction module 1108 is configured to reconstruct a three-dimensional human body model according to the human body normal map and the texture features.

[0132] In an exemplary embodiment of the present disclosure, the 3D human body model reconstruction device 1100 is further used for:

[0133] Determining voxel features, normal features and texture features of the human body normal map;

[0134] The voxel features, the normal features and the texture features are fused into the reconstructed three-dimensional human body model based on an attention mechanism.

[0135] In an exemplary embodiment of the present disclosure, the 3D human body model reconstruction device 1100 is further used for:

[0136] voxelize the human body normal map to determine the voxel features;

[0137] Extracting a texture map and a normal map of a dressed human body from the original RGB two-dimensional image;

[0138] Feature extraction is performed on the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the texture feature.

[0139] In an exemplary embodiment of the present disclosure, the 3D human body model reconstruction device 1100 is further used for:

[0140] Projecting the human body normal map onto a two-dimensional plane based on a bilinear interpolation algorithm;

[0141] Extracting the normal feature and the texture feature according to the projection result of the two-dimensional plane;

[0142] Determine the correspondence between any sampling point on the human body normal map and the normal feature, and the correspondence between the sampling point and the texture feature.

[0143] In an exemplary embodiment of the present disclosure, the 3D human body model reconstruction device 1100 is further used for:

[0144] Based on the attention mechanism, the voxel feature, the normal feature and the texture feature are normalized and assigned a preset weight value;

[0145] Decoding the human body normal map into an SMPL mesh to determine vertex coordinates, β and θ;

[0146] Substituting the vertex coordinates, the β, the θ, the preset weight value and the corresponding feature vector into a body fitting loss function, the feature vector including the voxel feature, the normal feature and the texture feature;

[0147] Based on the magnitude relationship between the calculation result of the body fitting loss function and the result of the preset loss function, the weight value of the feature vector is adjusted.

[0148] In an exemplary embodiment of the present disclosure, the 3D human body model reconstruction device 1100 is further used for:

[0149] Determining whether a calculated result of the body fitting loss function is greater than a result of the preset loss function;

[0150] If it is determined that the calculation result of the body fitting loss function is greater than the preset loss function result, the weight value is adjusted until the calculation result is less than or equal to the preset loss function result.

[0151] In an exemplary embodiment of the present disclosure, the expression of the body fitting loss function includes:

[0152]

[0153]

[0154] Among them, the v j represents the jth vertex, the n s represents the total number of vertices, the c(v j ) represents the eigenvector corresponding to the vertex, and F(c(v j )) represents the occupancy value of the vertex, and the λ R is a constant, the β init With the θ init is the initial parameter, and η represents the preset coefficient.

[0155] Since the functions of the device 1100 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0156] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0157] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0158] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0159] Refer to the following Fig.12 The electronic device 1200 according to this embodiment of the present invention is described. Fig.12 The electronic device 1200 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0160] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include but are not limited to: at least one processing unit 1210, at least one storage unit 1220, and a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210).

[0161] The storage unit stores program codes, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 1210 can perform the method shown in the embodiment of the present disclosure.

[0162] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 12201 and / or a cache storage unit 12202 , and may further include a read-only storage unit (ROM) 12203 .

[0163] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0164] Bus 1230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0165] The electronic device 1200 may also communicate with one or more external devices 1240 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0166] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0167] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0168] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0169] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0170] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0171] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0172] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0173] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0174] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A three-dimensional human body model reconstruction method, characterized in that: include: Acquire an original RGB two-dimensional image including a human body image and a silhouette of the human body image; Inputting the original RGB two-dimensional image and the silhouette into the SMPL prediction network to construct an SMPL human body parameter model; Rendering a front image and a back image of a human body of the SMPL human body parameter model based on the silhouette to obtain a normal map of a human body without clothing and texture features of a human body without clothing; Reconstructing a three-dimensional human body model according to the human body normal map and the unclothed texture features; Determining voxel features, normal features, and clothing texture features of the human body normal map includes: voxelize the human body normal map to determine the voxel features; Extracting a texture map and a normal map of a dressed human body from the original RGB two-dimensional image; Performing feature extraction on the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the clothing texture feature; The voxel features, the normal features and the clothing texture features are fused into the reconstructed three-dimensional human body model based on an attention mechanism.

2. The three-dimensional human body model reconstruction method according to claim 1, characterized in that: Performing feature extraction on the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the clothing texture feature includes: Projecting the normal map of the dressed human body onto a two-dimensional plane based on a bilinear interpolation algorithm; Extracting the normal feature and the clothing texture feature according to the projection result of the two-dimensional plane; Determine the correspondence between any sampling point on the normal map of the dressed human body and the normal feature, and the correspondence between the sampling point and the clothing texture feature.

3. The three-dimensional human body model reconstruction method according to claim 1 or 2, characterized in that: The voxel feature, the normal feature and the clothing texture feature are integrated into the reconstructed three-dimensional human body model based on the attention mechanism, comprising: Based on the attention mechanism, the voxel feature, the normal feature and the clothing texture feature are normalized and assigned a preset weight value; Decoding the normal map of the dressed human body into an SMPL mesh to determine vertex coordinates, β and θ, wherein the β and θ are parameters of the SMPL prediction network; Substituting the vertex coordinates, the β, the θ, the preset weight value and the corresponding feature vector into a body fitting loss function, the feature vector including the voxel feature, the normal feature and the clothing texture feature; Based on the magnitude relationship between the calculation result of the body fitting loss function and the result of the preset loss function, the weight value of the feature vector is adjusted.

4. The three-dimensional human body model reconstruction method according to claim 3, characterized in that: Based on the magnitude relationship between the calculation result of the body fitting loss function and the result of the preset loss function, adjusting the weight value of the feature vector includes: Determining whether a calculated result of the body fitting loss function is greater than a result of the preset loss function; If it is determined that the calculation result of the body fitting loss function is greater than the preset loss function result, the weight value is adjusted until the calculation result is less than or equal to the preset loss function result.

5. The three-dimensional human body model reconstruction method according to claim 3, characterized in that: The expression of the body fitting loss function includes: Among them, the v j represents the jth vertex, the n s represents the total number of vertices, the c(v j ) represents the eigenvector corresponding to the vertex, and F(c(v j )) represents the occupancy value of the vertex, and the λ R is a constant, the β init With the θ init is the initial parameter, and η represents the preset coefficient.

6. A three-dimensional human body model reconstruction device, characterized in that: include: An acquisition module, configured to acquire an original RGB two-dimensional image including a human body image and a silhouette of the human body image; A construction module, configured to input the original RGB two-dimensional image and the silhouette into a SMPL prediction network to construct a SMPL human body parameter model; A rendering module is configured to render a front image and a back image of a human body of the SMPL human body parameter model based on the silhouette to obtain a normal map of a human body without clothing and texture features of a human body without clothing; A reconstruction module, configured to reconstruct a three-dimensional human body model according to the human body normal map and the unclothed texture feature; Determining voxel features, normal features, and clothing texture features of the human body normal map includes: voxelize the human body normal map to determine the voxel features; Extracting a texture map and a normal map of a dressed human body from the original RGB two-dimensional image; Performing feature extraction on the texture map and the normal map of the dressed human body through a feature extraction network to obtain the normal feature and the clothing texture feature; The voxel features, the normal features and the clothing texture features are fused into the reconstructed three-dimensional human body model based on an attention mechanism.

7. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the three-dimensional human body model reconstruction method according to any one of claims 1 to 5 based on instructions stored in the memory.

8. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the three-dimensional human body model reconstruction method according to any one of claims 1 to 5 is implemented.

Images