Three-dimensional human body reconstruction method, device, equipment and storage medium

Abstract

The application is suitable for the field of computer vision, and provides a three-dimensional human body reconstruction method, device, equipment and storage medium, wherein the method comprises the following steps: obtaining a target human body image; extracting a human body joint point from the target human body image; obtaining a first human body grid feature map according to the human body joint point by using a preset three-dimensional reconstruction model, performing vertex augmentation on the first human body grid feature map to obtain a second human body grid feature map, and obtaining a first three-dimensional human body feature model according to the second human body grid feature map. According to the scheme, the uncertainty of two-dimensional-to-three-dimensional mapping in human body feature processing of a single-view three-dimensional human body reconstruction method can be reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a three-dimensional human body reconstruction method, device, equipment and storage medium. BACKGROUND

[0002] Traditional three-dimensional human body reconstruction collects multiple images and various human body features through devices such as multi-view cameras, depth cameras, laser scanners or inertial sensors, and implements subsequent feature processing on this basis to realize three-dimensional human body reconstruction. This scheme has the advantage of high three-dimensional reconstruction accuracy, but also leads to high cost and complex operation of three-dimensional human body reconstruction, and also causes more noise to be post-processed.

[0003] To balance this shortcoming, the prior art adopts a three-dimensional human body reconstruction method based on single view, which can reduce cost, does not require complex operation, and is more widely used in practical application scenarios. However, the three-dimensional human body reconstruction method based on single view will have uncertainty in two-dimensional to three-dimensional mapping in human body feature processing. SUMMARY

[0004] Embodiments of the present application provide a data processing method, device, equipment and storage medium to solve the problem of uncertainty in two-dimensional to three-dimensional mapping in human body feature processing in the prior art three-dimensional human body reconstruction method based on single view.

[0005] A first aspect of embodiments of the present application provides a three-dimensional human body reconstruction method, comprising:

[0006] obtaining a target human body image;

[0007] extracting a human body joint node from the target human body image;

[0008] adopting a preset three-dimensional reconstruction model, obtaining a first human body mesh feature map according to the human body joint node, performing vertex augmentation on the first human body mesh feature map to obtain a second human body mesh feature map, and obtaining a first three-dimensional human body feature model according to the second human body mesh feature map.

[0009] A second aspect of embodiments of the present application provides a three-dimensional human body reconstruction device, comprising:

[0010] an acquisition module configured to obtain a target human body image;

[0011] an extraction module configured to extract a human body joint node from the target human body image;

[0012] a three-dimensional reconstruction module configured to adopt a preset three-dimensional reconstruction model, obtain a first human body mesh feature map according to the human body joint node, perform vertex augmentation on the first human body mesh feature map to obtain a second human body mesh feature map, and obtain a first three-dimensional human body feature model according to the second human body mesh feature map.

[0013] A third aspect of the embodiments of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to the first aspect when executing the computer program.

[0014] A fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the steps of the method according to the first aspect.

[0015] A fifth aspect of the present application provides a computer program product, which, when executed on a computer device, causes the computer device to perform the steps of the method according to the first aspect.

[0016] As can be seen from the above, in the embodiments of the present application, by acquiring a target human body image, extracting a human body joint point from the target human body image, using a preset three-dimensional reconstruction model, obtaining a first human body mesh feature map according to the human body joint point, performing vertex augmentation on the first human body mesh feature map to obtain a second human body mesh feature map, and obtaining a three-dimensional human body feature model according to the second human body mesh feature map, the process obtains the human body mesh feature map based on the human body joint point through the preset three-dimensional reconstruction model, and only uses the human body joint point as a point feature to implement human body three-dimensional reconstruction, which reduces the cost and does not require complex operations, and at the same time, the number and feature dimension of the pixel points corresponding to the human body joints in the image are expanded through vertex augmentation of the human body network image, thereby ensuring the effectiveness of the two-dimensional feature to three-dimensional mapping of the human body. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a three-dimensional human body reconstruction method provided by the embodiments of the present application;

[0019] Figure 2 is a data flow diagram of three-dimensional human body reconstruction provided by the embodiments of the present application;

[0020] Figure 3 is a structure diagram of a three-dimensional reconstruction model provided by the embodiments of the present application;

[0021] Figure 4 is a structure diagram of an input layer provided by the embodiments of the present application;

[0022] Figure 5 is an image processing schematic diagram of a progressive layer provided by an embodiment of the present application;

[0023] Figure 6 is another image processing schematic diagram of a progressive layer provided by an embodiment of the present application;

[0024] Figure 7 is still another image processing schematic diagram of a progressive layer provided by an embodiment of the present application;

[0025] Figure 8 is still another image processing schematic diagram of a progressive layer provided by an embodiment of the present application;

[0026] Figure 9 is still another image processing schematic diagram of a progressive layer provided by an embodiment of the present application;

[0027] Figure 10 is an image processing schematic diagram of an output layer of a progressive layer provided by an embodiment of the present application.

[0028] Figure 11 is a structural diagram of a three-dimensional human body reconstruction device provided by an embodiment of the present application;

[0029] Figure 12 is a structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc. in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the embodiments of the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the embodiments of the present application with unnecessary detail.

[0031] It is to be understood that the terminology "includes", "has", "holds", "contains" used in the specification and in the following claims indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0033] It should also be further understood that the term "and / or" as used in the specification and in the claims, means any one of the associated listed items, as well as all possible combinations of the items, and includes all possible combinations.

[0034] As used in the specification and in the claims, the term "if' can be interpreted as meaning "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [the recited condition or event] is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [the recited condition or event]" or "in response to detecting [the recited condition or event]" depending on the context.

[0035] In particular implementations, the computer device described in the embodiments of the present application includes, but is not limited to, other portable devices such as a mobile phone, a laptop computer, or a tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0036] In the following discussion, a computer device including a display and a touch-sensitive surface is described. It should be understood, however, that the computer device can include one or more other physical user-interface devices, such as a physical keyboard, a mouse and / or a joystick.

[0037] The computer device supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disk authoring application, a spreadsheet application, a game application, a telephone application, a video conferencing application, an e-mail application, an instant messaging application, a workout support application, a photo management application, a digital camera application, a digital camcorder application, a web browsing application, a digital music player application, and / or a digital video player application.

[0038] The various applications that can be executed on the computer device can use at least one common physical user-interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface, as well as the corresponding information displayed on the display, can be adjusted and / or changed by the user through the various applications. Thus, the user can tailor a common physical architecture to suit a particular application. By way of another example, the user can change the configuration of a touch-sensitive surface for an e-mail application to include a keyboard input mode, a telephone input mode, and an alphanumeric input mode. By way of another example, in another implementation, the user can change the configuration of the touch-sensitive surface for an e-mail application to include a numeric input mode for a calendar application that lists dates and times.

[0039] It should be understood that the size of the serial number of each step in the embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0040] In order to illustrate the technical solutions described in the present application, the following will be described by specific embodiments.

[0041] Referring to Figure 1 , Figure 1 is a flow of a three-dimensional human body reconstruction method provided by the embodiment of the present application Figure 1 . As shown in Figure 1 , a three-dimensional human body reconstruction method, the method comprising the following steps:

[0042] Step 101, obtaining a target human body image.

[0043] Specifically, the target human body image is an RGB image containing a human body. Based on the image as an operating object, the human body is implemented three-dimensional reconstruction.

[0044] Step 102, extracting a human body joint point from the target human body image.

[0045] Specifically, the pose predictor is used to predict the pose of the target human body from the target human body image, and the human body joint point is extracted from the pose. Among them, based on the neural network architecture, the pose predictor can be constructed by using open source projects such as Openpose, Alphpose and HRNet.

[0046] In the implementation process, the position information of the human body joint point in the image predicted from the target human body image can be taken as the pose feature, denoted as P 2D ∈R J×2 , where J=19or17 is the number of human body joint points.

[0047] In one example, the extraction of the human body joint point specifically extracts the joint position coordinates of the human body joint in the target human body image, forms the human body joint point information, and finally extracts a set of position coordinates of multiple human body joints in the target human body image.

[0048] In addition, in one example, the human body joint point is a two-dimensional human body joint point. Alternatively, the human body joint point is a three-dimensional human body joint point, which is specifically mapped to a three-dimensional space based on the extracted two-dimensional human body joint point to form the three-dimensional human body joint point. Alternatively, the human body joint point contains both the two-dimensional human body joint point and the three-dimensional human body joint point obtained by mapping. Herein, only exemplary description is made, and it is not limited thereto.

[0049] Specifically, in combination with Figure 2 as shown, Figure 2The left part of the body image is first subjected to pose prediction, and the predicted 2D pose includes a two-dimensional body joint. The 2D pose can be mapped to a three-dimensional space to form a 3D pose, and the obtained 3D pose includes a three-dimensional body joint.

[0050] In one example, the two-dimensional body joint or the three-dimensional body joint can be taken as an input of a subsequent preset three-dimensional reconstruction model, or the two-dimensional body joint and the three-dimensional body joint can be stacked in a feature dimension to be taken as an input of a subsequent preset three-dimensional reconstruction model.

[0051] The two-dimensional body joint and the three-dimensional body joint are different in spatial dimension, and the position coordinates of the corresponding body joints are different in dimension.

[0052] In step 103, a preset three-dimensional reconstruction model is used to obtain a first body mesh feature map according to the body joint, perform vertex augmentation on the first body mesh feature map to obtain a second body mesh feature map, and obtain a first three-dimensional body feature model according to the second body mesh feature map.

[0053] The preset three-dimensional reconstruction model is used to process the body joint to generate a body mesh feature map, and generate a three-dimensional body feature model based on the processing result after performing vertex augmentation on the body mesh feature map, thereby realizing three-dimensional reconstruction of the human body.

[0054] Optionally, the preset three-dimensional reconstruction model can be a designed progressive quadric graph convolutional network (PQ-GCN). By designing the progressive quadric graph convolutional network, the body joint in the human body pose is converted into a 3D body mesh feature map.

[0055] The body joint is taken as an initial input of the preset three-dimensional reconstruction model, and only has sparse points. By using the template topology structure from low resolution to high resolution constructed in the preset three-dimensional reconstruction model, the vertex augmentation of the body mesh feature map is gradually and progressively performed, and then a three-dimensional body feature model is finally generated, thereby realizing three-dimensional reconstruction of the human body.

[0056] In one example, the vertex augmentation of the first body mesh feature map can be a step-by-step processing operation on the first body mesh feature map. The number of vertices in the first body mesh feature map is increased after each processing. The processing image of the current processing operation is the output image of the previous processing operation, thereby realizing step-by-step augmentation of the vertices of the body mesh feature map.

[0057] Differently, in one example, the vertex augmentation of the first human mesh feature map can consider two aspects, one is to augment the number of vertices, and the other is to augment the feature dimension of the vertices.

[0058] When the feature dimension of the vertices is augmented, it can be step-by-step augmentation or first increase and then decrease augmentation. In this process, the number of fixed points in the second human mesh feature map obtained by the vertex augmentation of the first human mesh feature map is greater than the number of vertices in the first human mesh feature map, and the feature dimension of each vertex in the second human mesh feature map is greater than the feature dimension of the vertex in the first human mesh feature map.

[0059] By augmenting the vertices of the first human mesh feature map, the number and feature dimension of the pixel points corresponding to the human joints in the image are augmented, ensuring the effectiveness of the two-dimensional human feature to three-dimensional mapping.

[0060] In the process of obtaining the first three-dimensional human feature model according to the second human mesh feature map, since the number of vertices corresponding to the human joints in the second human mesh feature map has been augmented to the target value and the feature dimension has also been increased to the set dimension, at this time, the feature dimension corresponding to each vertex is reduced to three dimensions, and a three-dimensional human feature model is obtained by augmenting the two-dimensional human joint to the three-dimensional human vertex, realizing the three-dimensional reconstruction of the human body.

[0061] In the embodiments of the present application, by obtaining a target human image, extracting human joint points from the target human image, using a preset three-dimensional reconstruction model, obtaining a first human mesh feature map according to the human joint points, augmenting the vertices of the first human mesh feature map, obtaining a second human mesh feature map, and obtaining a three-dimensional human feature model according to the second human mesh feature map, the process obtains a human mesh feature map based on human joint points through a preset three-dimensional reconstruction model, and only uses the point feature of the human joint to implement three-dimensional reconstruction of the human body. At the same time of reducing the cost and without complex operation, the number and feature dimension of the pixel points corresponding to the human joints in the image are augmented by augmenting the vertices of the human mesh image, ensuring the effectiveness of the two-dimensional human feature to three-dimensional mapping.

[0062] Further, as shown in Figure 3 In one example, the preset three-dimensional reconstruction model can include an input layer, a progressive layer and an output layer.

[0063] Specifically, the three-dimensional reconstruction model includes an input layer, which is used to obtain the first human mesh feature map according to the human joint points. The input layer includes at least one first graph convolution unit, a first data dimension conversion layer, a full connection layer and a second data dimension conversion layer connected in sequence.

[0064] The first data dimension conversion layer is configured to convert the first data dimension of the input feature map to a second data dimension matched with the fully connected layer, and the second data dimension conversion layer is configured to convert the second data dimension matched with the fully connected layer to the first data dimension.

[0065] The generation of the mesh feature map can be mapping the feature points (in this example, the human joint points) into a mesh, and the feature points can be further expanded by the grid points of the mesh by using the trained model. The finer the division of the mesh is, the better the expansion effect of the feature points is.

[0066] The first data dimension conversion layer is configured to convert the first data dimension of the input feature map to a second data dimension matched with the fully connected layer, and the second data dimension conversion layer is configured to convert the second data dimension matched with the fully connected layer to the first data dimension.

[0067] In one specific application, the first data dimension conversion layer is configured to convert the first data dimension of the input feature map to a second data dimension matched with the fully connected layer, and the second data dimension conversion layer is configured to convert the second data dimension matched with the fully connected layer to the first data dimension. Figure 4 As shown, the input layer is formed of the pose feature P 2D ∈R J×2 as input. The input layer is composed of three graph convolution units, two reshaping layers, and a fully connected layer. The main function is to encode the human joint position information P 2D into the feature map F4∈R 27×64 of the lowest resolution human mesh. The three graph convolution units are used to increase the vertex feature dimension (from 2D to 64D, i.e., to increase the feature dimension of each vertex to 64D), and the reshaping layers and the fully connected layer are used to up-sample the number of vertices (from J=17 to 27).

[0068] Further, the three-dimensional reconstruction model includes a progressive layer, which is configured to expand the number of vertices and adjust the vertex feature dimension of the first human mesh feature map to obtain the second human mesh feature map.

[0069] The progressive layer includes N graph convolution layers connected in sequence, and N is an integer greater than 1.

[0070] Each of the first N-1 graph convolution layers is configured to increase the feature dimension of each vertex in the input third feature map to a preset dimension and expand the number of vertices in the third feature map by a preset multiple, and the Nth graph convolution layer is configured to increase the feature dimension of each vertex in the input fourth feature map to a preset dimension.

[0071] The third feature map input in the first graph convolutional layer is the first human body mesh feature map, and the feature map output by the Nth graph convolutional layer is the second human body mesh feature map.

[0072] In one specific application, the main function of the progressive layer is to obtain the lowest resolution mesh feature map F4 e R 27×64 First, the vertices of the mesh feature map are gradually upsampled, and the number of vertices is gradually upsampled to 6890. The gradual change rule is 27→108→431→1723→6890, and the feature dimension of each vertex is considered to affect the memory consumption of the model. The higher the resolution of the mesh feature map, the more the number of mesh vertices in the image, and if the feature dimension of each vertex is also high, the resource consumption will be higher. Therefore, the feature dimension of each vertex in the mesh feature map is designed to increase first and then decrease to 128 dimensions.

[0073] Further, optionally, the network structure of the progressive layer is composed of five graph convolutional layers and four upsample layers. Each graph convolutional layer is composed of a residual structure of two graph convolutional units for learning and changing the feature dimension of the vertex, and the upsample layer uses the barycentric coordinate interpolation algorithm to realize the upsample of the number of vertices. Further, in one example, each of the first N-1 graph convolutional layers includes a graph convolution module and an upsample layer, and the graph convolution module includes M second graph convolution units connected in series, and M is an integer greater than 1.

[0074] The third feature map input in the graph convolutional layer is input into the graph convolution module, and the graph convolution module is used to increase the feature dimension of each vertex in the third feature map to a preset dimension to obtain an output feature map of the graph convolution module. The third feature map input in the graph convolutional layer and the output feature map of the graph convolution module are superimposed to obtain an intermediate feature map.

[0075] The intermediate feature map is input into the upsample layer, and the upsample layer is used to expand the number of vertices in the intermediate feature map by a preset multiple to obtain an output feature map of the graph convolutional layer.

[0076] During the processing of the human body mesh feature map by the progressive layer, the image feature changes as follows:

[0077] Among them, in combination with Figure 5 The input mesh feature map F4 e R 27×64 A residual structure of a graph convolutional layer in combination with the upsample layer based on the barycentric coordinate interpolation algorithm upsampled the number of vertices to 108, and the feature dimension of the vertex was increased to 128 dimensions. The generated feature map F3 is the initial feature of G3.

[0078] Further, as shown in Figure 6 When F3∈R 108×128 , a similar network structure is used, i.e., a residual structure of a graph convolution layer combined with an upsampling layer, to upsample the number of vertices to 431 and increase the vertex feature to 256 dimensions, to obtain a feature map F2 as the initial feature of G2.

[0079] Further, as shown in Figure 7 When F2∈R 431×256 , a similar network structure is used, i.e., a residual structure of a graph convolution layer combined with an upsampling layer. The number of vertices is upsampled to 1723, and the vertex feature dimension is kept at 256 dimensions, to obtain a feature map F1 as the initial feature of G1.

[0080] As shown in Figure 8 Finally, the initial feature F0∈R 1723×256 of G0 is obtained from F1∈R 6890 through network learning. ×128 .

[0081] As shown in Figure 9 When the initial feature F0∈R 6890×128 of G0 is obtained, a residual graph convolution layer is used to re-learn the feature map of the layer without changing the vertex feature dimension.

[0082] It can be seen that the purpose of the input layer and the progressive layer is to learn the final feature map F0∈R 2D of G0 from the human pose feature P J×2 ∈R 6890×128 , that is, to encode P 2D into F0∈R 6890×128 through a neural network, to try to learn the corresponding 3D information from the 2D human pose information through a neural network and encode it into a 128-dimensional feature, and finally to decode the 128-dimensional feature of each vertex into 3 dimensions through an output layer to complete the 3D reconstruction of the human body, i.e., to predict the coordinates of each vertex.

[0083] Further, as shown in Figure 2 and Figure 3 The three-dimensional reconstruction model further comprises at least one decoding module corresponding to at least one of the N graph convolution layers;

[0084] The decoding module is configured to obtain a second three-dimensional human feature model according to the intermediate feature map output by the corresponding graph convolution layer.

[0085] The decoding module comprises X third graph convolution units and a multilayer perceptron connected in series, and X is an integer greater than 1.

[0086] X of the graph convolution units are configured to reduce the feature dimension of the intermediate feature map to three dimensions.

[0087] The multi-layer perceptron is configured to expand the number of vertices of the reduced intermediate feature map to obtain the second three-dimensional human feature model.

[0088] The above process enables the reconstruction process to be terminated in advance by the increased intermediate decoding module, and directly completes three-dimensional reconstruction. By early termination, the computational complexity can be greatly reduced to meet different processing requirements, and early decoding can also achieve good performance in certain cases.

[0089] Further, the model comprises an output layer configured to obtain the first three-dimensional human feature model according to the second human mesh feature map.

[0090] The output layer comprises Y fourth graph convolution units connected in sequence, and Y is an integer greater than 1.

[0091] Y of the graph convolution units are configured to reduce the feature dimension of the second human mesh feature map to three dimensions to obtain the first three-dimensional human feature model.

[0092] In combination Figure 10 As shown, the role of the output layer is to reduce the vertex feature dimension to 3 to complete the three-dimensional human reconstruction task. The network structure is two graph convolution units.

[0093] The embodiment of the present application is based on human two-dimensional pose features, and attempts to reconstruct a human three-dimensional mesh from human two-dimensional pose features. Most previous methods are based on image features (which may contain various additional information such as color, environmental color, etc.). Feature extractors, ResNet or HRNet, are used to extract image features, and the parameters of a human parameterized model or the vertex coordinates of a human model are regressed based on the image features. Compared with image features, human 2D pose features are simpler and more pure, without additional information, and are more lightweight to operate, and ensure three-dimensional reconstruction effect.

[0094] Referring to Figure 11 , Figure 11 is a structure diagram of a three-dimensional human reconstruction device provided by an embodiment of the present application. Only parts related to the embodiments of the present application are shown for ease of description.

[0095] The three-dimensional human reconstruction device 11 comprises an acquisition module 1101, an extraction module 1102, and a three-dimensional reconstruction module 1103.

[0096] The acquisition module 1101 is configured to acquire a target human image.

[0097] ​The extraction module 1102 is configured to extract a human joint point from the target human body image.

[0098] The three-dimensional reconstruction module 1103 is configured to obtain a first human body mesh feature map according to the human joint point by using a preset three-dimensional reconstruction model, perform vertex augmentation on the first human body mesh feature map to obtain a second human body mesh feature map, and obtain a first three-dimensional human body feature model according to the second human body mesh feature map.

[0099] The three-dimensional reconstruction model includes an input layer configured to obtain the first human body mesh feature map according to the human joint point.

[0100] The input layer includes at least one first graph convolution unit, a first data dimension conversion layer, a fully connected layer, and a second data dimension conversion layer connected in sequence. The at least one first graph convolution unit is configured to output a feature map after increasing the feature dimension of the human joint point. The fully connected layer is configured to perform joint point augmentation on the input first feature map. The first data dimension conversion layer and the second data dimension conversion layer are configured to perform dimension conversion on the input second feature map. The second data dimension conversion layer outputs the first human body mesh feature map.

[0101] The three-dimensional reconstruction model includes a progressive layer configured to perform vertex number augmentation and vertex feature dimension adjustment on the first human body mesh feature map to obtain the second human body mesh feature map.

[0102] The progressive layer includes N graph convolution layers connected in sequence, where N is an integer greater than 1. Each of the first N-1 graph convolution layers is configured to increase the feature dimension of each vertex in the input third feature map to a preset dimension and expand the number of vertices in the third feature map by a preset multiple. The Nth graph convolution layer is configured to increase the feature dimension of each vertex in the input fourth feature map to a preset dimension. The first graph convolution layer inputs the first human body mesh feature map, and the Nth graph convolution layer outputs the second human body mesh feature map.

[0103] Each of the first N-1 graph convolution layers includes a graph convolution module and an upsampling layer. The graph convolution module includes M second graph convolution units connected in sequence, where M is an integer greater than 1.

[0104] The third feature map input in the graph convolution layer is input into the graph convolution module, the graph convolution module is used for increasing the feature dimension of each vertex in the third feature map to a preset dimension to obtain an output feature map of the graph convolution module, and after the third feature map input in the graph convolution layer and the output feature map of the graph convolution module are superimposed, an intermediate feature map is obtained; the intermediate feature map is input into the up-sampling layer, the up-sampling layer is used for expanding the number of vertices in the intermediate feature map by a preset multiple to obtain an output feature map of the graph convolution layer.

[0105] The three-dimensional reconstruction model further includes at least one decoding module corresponding to at least one of the N graph convolution layers respectively.

[0106] The decoding module is configured to obtain a second three-dimensional human feature model according to the intermediate feature map output by the corresponding graph convolution layer.

[0107] The decoding module includes X third graph convolution units and a multilayer perceptron connected in sequence, and X is an integer greater than 1.

[0108] The X third graph convolution units are configured to reduce the feature dimension of the intermediate feature map to three dimensions.

[0109] The multilayer perceptron is configured to expand the number of vertices of the reduced intermediate feature map to obtain the second three-dimensional human feature model.

[0110] The model includes an output layer configured to obtain the first three-dimensional human feature model according to the second human mesh feature map.

[0111] The output layer includes Y fourth graph convolution units connected in sequence, and Y is an integer greater than 1.

[0112] The Y fourth graph convolution units are configured to reduce the feature dimension of the second human mesh feature map to three dimensions to obtain the first three-dimensional human feature model.

[0113] The three-dimensional human reconstruction device provided by the embodiments of the present application can realize the processes of the embodiments of the three-dimensional human reconstruction method described above and achieve the same technical effects. To avoid repetition, details are not repeated here.

[0114] Figure 12 is a structural diagram of a computer device provided by an embodiment of the present application. As shown in the diagram, the computer device 12 of the embodiment includes at least one processor 120 (for example, a central processing unit (CPU), a microprocessor, a microcomputer, a microcontroller, or the like), a memory 130, a communication interface 140, and a bus 150. Figure 12(Only one is shown in the diagram), memory 121, and computer program 122 stored in said memory 121 and executable on said at least one processor 120, wherein said processor 120 executes said computer program 122 to implement the steps in any of the above method embodiments.

[0115] The computer device 12 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 12 may include, but is not limited to, a processor 120 and a memory 121. Those skilled in the art will understand that... Figure 12 This is merely an example of computer device 12 and does not constitute a limitation on computer device 12. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0116] The processor 120 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0117] The memory 121 can be an internal storage unit of the computer device 12, such as a hard disk or RAM of the computer device 12. The memory 121 can also be an external storage device of the computer device 12, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device 12. Furthermore, the memory 121 can include both internal and external storage units of the computer device 12. The memory 121 is used to store the computer program and other programs and data required by the computer device. The memory 121 can also be used to temporarily store data that has been output or will be output.

[0118] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0119] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0120] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0121] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0122] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0123] In addition, each of the function units in each of the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0124] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be implemented by a computer program instructing related hardware to complete, and the computer program can be stored in a computer-readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include appropriate contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0125] The above-mentioned embodiment methods can also be implemented by a computer program product, which, when running on a computer device, causes the computer device to execute the steps of the above-mentioned embodiment methods.

[0126] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for three-dimensional human body reconstruction, characterized in that, include: Acquire target human body image; Extract human joint points from the target human image; Using a preset three-dimensional reconstruction model, a first human body mesh feature map is obtained based on the human body joints. Vertex augmentation is performed on the first human body mesh feature map to obtain a second human body mesh feature map. A first three-dimensional human body feature model is obtained based on the second human body mesh feature map. The three-dimensional reconstruction model includes an input layer, which is used to obtain the first human body mesh feature map based on the human body joint points; The input layer includes at least one first graph convolutional unit, a first data dimension transformation layer, a fully connected layer, and a second data dimension transformation layer connected in series. At least one first graph convolutional unit is used to increase the feature dimension of the human body joints and output a feature map. The fully connected layer is used to augment the input first feature map with joints. The first data dimension conversion layer and the second data dimension conversion layer are used to perform dimension conversion on the input second feature map. The second data dimension conversion layer outputs the first human body mesh feature map.

2. The method according to claim 1, characterized in that, The three-dimensional reconstruction model includes a progressive layer, which is used to increase the number of vertices and adjust the vertex feature dimensions of the first human body mesh feature map to obtain the second human body mesh feature map. The progressive layer comprises N graph convolutional layers connected in series, where N is an integer greater than 1; Each of the first N-1 graph convolutional layers is used to increase the feature dimension of each vertex in the input third feature map to a preset dimension, and to increase the number of vertices in the third feature map by a preset multiple. The Nth graph convolutional layer is used to increase the feature dimension of each vertex in the input fourth feature map to a preset dimension; The third feature map input to the first graph convolutional layer is the first human body mesh feature map, and the feature map output by the Nth graph convolutional layer is the second human body mesh feature map.

3. The method according to claim 2, characterized in that, Each of the first N-1 graph convolutional layers contains a graph convolutional module and an upsampling layer. The graph convolutional module includes M second graph convolutional units connected in series, where M is an integer greater than 1. The third feature map input in the graph convolutional layer is input into the graph convolutional module. The graph convolutional module is used to increase the feature dimension of each vertex in the third feature map to a preset dimension to obtain the output feature map of the graph convolutional module. The third feature map input in the graph convolutional layer and the output feature map of the graph convolutional module are superimposed to obtain an intermediate feature map. The intermediate feature map is input into the upsampling layer, which is used to amplify the number of vertices in the intermediate feature map by a preset factor to obtain the output feature map of the graph convolutional layer.

4. The method according to claim 3, characterized in that, The three-dimensional reconstruction model also includes at least one decoding module corresponding to at least one of the N graph convolutional layers; The decoding module is used to obtain a second three-dimensional human feature model based on the intermediate feature map output by the corresponding graph convolutional layer.

5. The method according to claim 4, characterized in that, The decoding module includes X third-graph convolutional units and a multilayer perceptron connected in series, where X is an integer greater than 1; X third graph convolutional units are used to reduce the feature dimension of the intermediate feature map to three dimensions; The multilayer perceptron is used to increase the number of vertices in the dimensionality-reduced intermediate feature map to obtain the second three-dimensional human feature model.

6. The method according to claim 1, characterized in that, The model includes an output layer, which is used to obtain the first three-dimensional human feature model based on the second human mesh feature map. The output layer contains Y fourth graph convolutional units connected in series, where Y is an integer greater than 1; Y fourth graph convolutional units are used to reduce the feature dimension of the second human body mesh feature map to three dimensions to obtain the first three-dimensional human body feature model.

7. A three-dimensional human body reconstruction device, characterized in that, include: The acquisition module is used to acquire images of the target human body. The extraction module is used to extract human joint points from the target human image; The three-dimensional reconstruction module is used to obtain a first human body mesh feature map based on the human body joints using a preset three-dimensional reconstruction model, to expand the vertices of the first human body mesh feature map to obtain a second human body mesh feature map, and to obtain a first three-dimensional human body feature model based on the second human body mesh feature map. The three-dimensional reconstruction model includes an input layer, which is used to obtain the first human body mesh feature map based on the human body joint points; The input layer includes at least one first graph convolutional unit, a first data dimension transformation layer, a fully connected layer, and a second data dimension transformation layer connected in series. At least one first graph convolutional unit is used to increase the feature dimension of the human body joints and output a feature map. The fully connected layer is used to augment the input first feature map with joints. The first data dimension conversion layer and the second data dimension conversion layer are used to perform dimension conversion on the input second feature map. The second data dimension conversion layer outputs the first human body mesh feature map.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images