Image-based step-by-step generation method and device for human body reconstruction

Abstract

The present invention discloses an image-based, step-by-step generative method and device for human body reconstruction. The method comprises: extracting image features of a given human body image, decoding to generate heat maps in the x, y, and z directions, reseparating them after concatenated encoding, and then regressing the coordinates of human joint points through heat map integration; passing the heat map distribution of human joint points through a transformer encoder to perform information exchange between human joint points and human mesh model vertices, upsampling the human mesh model vertices, and batch normalization, thereby gradually generating a final human mesh model; inputting the human mesh model vertex coordinates into a universal SMPL model regressor, and outputting the human joint point coordinates corresponding to the human model as constraints for human mesh model reconstruction. Based on human three-dimensional posture estimation, the present invention introduces an attention mechanism to optimize the heat map distribution in different directional dimensions, and adopts a coarse-to-fine approach to gradually generate a human mesh model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for human body reconstruction based on image step-by-step generation. Background Art

[0002] Existing deep learning-based human body reconstruction technologies are mainly divided into two categories: the first category is three-dimensional human body reconstruction based on parametric models, which estimates the human body model parameters through neural networks and uses the parametric model to directly generate a three-dimensional human body mesh model; the second category of three-dimensional human body reconstruction does not use parametric models to generate human body mesh models, but directly regresses the coordinates of the vertices of the human body three-dimensional model based on image feature information.

[0003] There are two common approaches to directly regressing the coordinates of vertices in a 3D human model based on image features: First, a neural network is used to estimate the 3D human pose and the heatmap distribution of the 3D human mesh model along the x, y, and z axes, and then the heatmap integral is used to regress the coordinates of the human joints and the vertices of the human mesh model. Second, a human model template is used to gradually regress the 3D mesh coordinates of the human model through a transformer and gradual dimensionality reduction.

[0004] Among the above-mentioned existing technologies, the first major disadvantage is that the heat map estimation of each directional dimension is relatively independent, and the correlation between the heat map information of different directional dimensions is lost; the second major disadvantage is that it requires the use of a human body model template, and the model calculation amount is large and the training time is long. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide an image-based step-by-step generative human body reconstruction method and device. The present invention introduces an attention mechanism to learn the relationship between the distribution of heat maps in different directional dimensions, and optimizes the three-dimensional human joint point estimation and human body mesh model reconstruction through the interaction of heat map information in different directional dimensions; at the same time, based on the human joint point heat map information, a coarse-to-fine method is used to gradually regress the vertex coordinates of the human body mesh model, which is independent of the human body model template and reduces the model calculation amount and training time.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In one aspect, the present invention provides an image-based step-by-step human body reconstruction method, comprising the following steps:

[0008] Extract the image features of a given human body image, decode the image features to generate a heat map in the three dimensions of x, y, and z, then concatenate the heat map distributions in the three dimensions of x, y, and z, and separate the heat map distributions in the three dimensions of x, y, and z after encoding, and then regress the coordinates of the human joint points through heat map integration;

[0009] The heat map distribution of human joints is passed through the transformer encoder to perform information interaction between human joints and human mesh model vertices, upsample the human mesh model vertices, and perform batch normalization, and then gradually generate the final human mesh model;

[0010] The coordinates of the human joint points are input into the human mesh model. After passing through the general SMPL model regressor, the coordinates of the human joint points corresponding to the human model are output as the reconstruction constraints of the human mesh model.

[0011] As a preferred technical solution, the image features of a given human body image are extracted as follows:

[0012] Given a human image, crop the human image based on the marked or detected border, remove the background effect, retain the pure human image, and adjust the size of the human image to match the subsequent neural network processing;

[0013] The processed image is input into the encoder for feature extraction to obtain the human image feature F P , F P The feature dimension is c×h×w;

[0014] For the obtained human image features, first use the inverse convolution operation in the x and y axis directions The feature map F p The dimension is increased to c′×8h×8w. Then, in the x-axis direction, the y-axis dimension is averaged by avg. y , and then use one-dimensional convolution Get the heat map distribution P in the x-axis direction H，x ∈J×8h; Similarly, in the y-axis direction, the x-axis dimension is averaged avg x , and then use one-dimensional convolution Get the heat map distribution P in the y-axis direction H，y ∈J×8w. ;

[0015]

[0016]

[0017] In the z-axis direction, perform the averaging operation avg on the x and y axis dimensions x，y , using one-dimensional convolution f p and feature deformation operations Convert the feature dimension to c′×D, and finally use one-dimensional convolution Get the heat map distribution P in the z-axis direction H，z ∈J×D.

[0018]

[0019] As a preferred technical solution, the heat map distributions of the three dimensions of x, y, and z are connected, and after encoding by the transformer encoder, the heat map distributions of the three dimensions of x, y, and z are separated, specifically:

[0020] Heat map distribution P in the three dimensions of x, y, and z H，x , P H，y , P H，z ∈R J×64 Fuse in the last dimension to obtain the fused feature P H =[P H，x , P H，y , P H，z ]∈R J×192 , the fused feature contains the heat map distribution information of three directional dimensions, and then the fused feature is used as the input of the transformer encoder;

[0021] The fused feature P H Input to the N-layer attention module for heat map information interaction between different directions and different joints; each layer of attention module performs four operations in sequence: multi-head attention, residual connection and regularization, feedforward network processing, residual connection and regularization, and finally outputs the heat map distribution P′ H ∈R J×192 ;

[0022] The output heat map distribution P′ H Pass through independent feature fully connected layers fc x 、fc y 、fc z Perform feature mapping and then distribute it with the original heat map P H，x 、P H，y 、P H，z Add and re-separate into the heat map distribution P′ in the three dimensions of x, y, and z H，x , P′ H，y , P′ H，z ;

[0023] P′ (H，x) =P H，x +fc x (P′ H )

[0024] P′ (H，y) =P H，y +fc y (P′ H )

[0025] P′ (H，z) =P H，z +fcz (P′ H )

[0026] For the heat map distribution of the three dimensions of each joint point, use soft-argmax to regress the coordinate points to obtain P′ C，x , P′ C，y , P′ C，z ∈R J×1 , and then perform the connection operation to obtain the final human joint point coordinates P′ C =[P′ C，x , P′ C，y , P′ C，z ]∈R J×3 ;.

[0027] As a preferred technical solution, the soft-argmax is defined as follows:

[0028]

[0029] As a preferred technical solution, the human body joint point and human body mesh model vertex information interaction is specifically as follows:

[0030] Given heatmap feature F H =[P′ H，x , P′ H，y , P′ H，z ]∈R J×192 , obtain feature F through position embedding embed =F H +PE, position embedded PE is defined as follows: pos is the position, i is the feature dimension subscript, d model is the feature dimension;

[0031] After position embedding, feature F embed Input Transformer encoder, Transformer encoder contains N blocks, each block contains a multi-head attention module and a feedforward neural network; in each block, F embed First, the normalized attention weights are calculated by the multi-head attention module and then the features are transformed by the feedforward neural network. Finally, after N blocks, the output That is the vertex heat map distribution of the human body model with V vertices.

[0032] As a preferred technical solution, the human body model vertex upsampling is specifically as follows:

[0033] The output after Transformer encoder processing Input to the 1×1 convolution layer for upsampling operation, the output is That is, after the 1×1 convolution operation, the number of vertices of the existing model is doubled, thereby achieving model refinement.

[0034] As a preferred technical solution, the human body joint point coordinates are input into the human body mesh model, and after passing through the general SMPL model regressor, the human body joint point coordinates corresponding to the human body model are output to complete the human body reconstruction, specifically:

[0035]

[0036] Among them, SMPL.J_regressor∈24×6890 is the joint regression matrix of the SMPL model, M′ c ∈6890×3 is the human body mesh model estimated by the above method, The coordinates of the human joint points obtained by regression.

[0037] On the other hand, the present invention provides an image-based step-by-step generative human body reconstruction system, which is applied to the image-based step-by-step generative human body reconstruction method, and includes a human body posture estimation module, a human body network generation module, and a human body joint point regression module;

[0038] The human posture estimation module is used to extract image features of a given human image, decode the image features to generate a heat map in the three dimensions of x, y, and z, then concatenate the heat map distributions in the three dimensions of x, y, and z, separate the heat map distributions in the three dimensions of x, y, and z after encoding, and then regress the coordinates of the human joint points through heat map integration;

[0039] The heat map distribution of human joints is passed through the transformer encoder to perform information interaction between human joints and human mesh model vertices, upsample the human mesh model vertices, and perform batch normalization, and then gradually generate the final human mesh model;

[0040] The human joint point regression module is used to input the human joint point coordinates into the human mesh model, and after passing through the general SMPL model regressor, output the human joint point coordinates corresponding to the human body model as the human mesh model reconstruction constraints.

[0041] Another aspect of the present invention provides an electronic device, comprising:

[0042] at least one processor; and,

[0043] a memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the image-based step-by-step generative human body reconstruction method.

[0045] In another aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the image-based step-by-step generation method for human body reconstruction.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] 1. This invention introduces an attention mechanism, enabling human pose estimation and morphology reconstruction to better utilize the correlation information of heatmap distributions in different directional dimensions, thereby achieving fine-tuning of heatmap distributions in different directional dimensions. Existing solutions directly use heatmaps in a single directional dimension to estimate human pose and perform body reconstruction, losing the interaction of information in different directional dimensions.

[0048] 2. The present invention adopts a coarse-to-fine step-by-step generative human body reconstruction scheme, which reduces the dependence on the human body model template and reduces the time required for training. The existing method relies on the human body model template, has a large amount of calculation, and takes a long time to train.

[0049] 3. The coarse-to-fine step-by-step human body reconstruction scheme adopted by the present invention can reduce the video memory requirements required for training compared to the direct generation scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 4 is a flow chart of a method for human body reconstruction based on image-based step-by-step generation according to an embodiment of the present invention;

[0052] Figure 2 1 is a schematic diagram of a process for estimating a human body posture according to an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of heat map information interaction between different directions and dimensions and different joint points according to an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of human body network imaging according to an embodiment of the present invention;

[0055] Figure 5 A schematic structural diagram of an image-based step-by-step human body reconstruction system provided by an embodiment of the present invention;

[0056] Figure 6 It is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments and drawings in this application. It should be understood that the drawings are only for illustrative purposes and cannot be understood as limiting this patent. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0058] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0059] like Figure 1 As shown, the human body reconstruction method based on the image of the present embodiment mainly includes the following three steps: (1) human body posture estimation step; (2) human body mesh generation step; (3) human body joint point regression step; on the one hand, the present application introduces an attention mechanism to learn the relationship between the heat map distributions of different directional dimensions, and optimizes the three-dimensional human joint point estimation and human body mesh model reconstruction through the interaction of heat map information of different directional dimensions; on the other hand, based on the human joint point heat map information, the vertex coordinates of the human body mesh model are gradually regressed in a coarse-to-fine manner, which does not rely on the human body model template and reduces the model calculation amount and training time.

[0060] The following describes the image-based step-by-step generative human body reconstruction method in detail according to the specific workflow:

[0061] (1) Human body posture estimation step;

[0062] like Figure 2Figure 2 shows the detailed process of human pose estimation. This module is implemented using an encoder, decoder, and Transformer encoder. Given a human image I as input, the encoder extracts features from I to obtain image features F. These features are decoded by the decoder to generate a heat map distribution in the x, y, and z directions. These heat maps are then concatenated and fed into an N-layer Transformer encoder. The output is then separated into the x, y, and z heat map distributions, which are then regressed using heat map integration to obtain the coordinates of the human joints.

[0063] Furthermore, the specific process of human posture estimation is as follows:

[0064] (1.1) Human body image feature extraction;

[0065] (1.1.1) Given a human image I, crop the image based on the annotated or detected bounding boxes to remove background influence, retain the pure human image, and resize the image to match the subsequent neural network processing.

[0066] (1.1.2) Input the processed image into the encoder for feature extraction to obtain the human image feature F P . F P The feature dimension is c×h×w.

[0067] For the obtained human image features, first use the inverse convolution operation in the x and y axis directions The feature map F p The dimension is increased to c′×8h×8w. Then, in the x-axis direction, the y-axis dimension is averaged by avg y , and then use one-dimensional convolution Get the heat map distribution P in the x-axis direction H，x ∈J×8h; Similarly, in the y-axis direction, the x-axis dimension is averaged avg x , and then use one-dimensional convolution Get the heat map distribution P in the y-axis direction H，y ∈J×8w.

[0068]

[0069]

[0070] In the z-axis direction, perform the averaging operation avg on the x and y axis dimensions x，y , using one-dimensional convolution f p and feature deformation operations Convert the feature dimension to c′×D, and finally use one-dimensional convolution Get the heat map distribution P in the z-axis direction H，z∈J×D.

[0071]

[0072] In the specific implementation stage of this embodiment, 8h=8w=D=64.

[0073] Furthermore, when performing feature extraction, the residual network ResNet50 can be used. ResNet50 has two basic blocks, namely Conv Block and Identity Block. The input and output dimensions of Conv Block are different, so they cannot be connected in series continuously. Its function is to change the dimension of the network; the input dimension and output dimension of Identity Block are the same, and they can be connected in series to deepen the network. This embodiment uses the residual network ResNet50 to extract human body features very well; of course, this application is not limited to the residual network ResNet50 when performing feature extraction. Other residual networks that can implement the technical solution of this application are applicable to this application and will not be repeated here.

[0074] (1.2) Generate heatmaps in x, y, and z directions;

[0075] (1.2.1) The extracted human image features F are input to the decoder, and the multi-dimensional heat map distribution of the three dimensions of x, y, and z is output. In this embodiment, the 64-dimensional heat map distribution P of J joint points is output. H，x , P H，y , P H，z ∈R J×64 .

[0076] (1.3) Exchange of heat map information in different directions and dimensions;

[0077] (1.3.1) Heat map distribution P in the three dimensions of x, y, and z H，x , P H，y , P H，z ∈R J×64 Concatenate and fuse in the last dimension to obtain the fused feature P H =[P H，x , P H，y , P H，z ]∈R J×192 , this feature will contain the heat map distribution information of three directional dimensions, and then the fused feature will be used as the input of the transformer encoder.

[0078] (1.3.2) Fusion feature P H Input to the N-layer attention module (such as Figure 3) to interact with heat map information in different directions and dimensions and between different joints. Among them, each layer of attention module sequentially performs four operations: multi-head attention, residual connection and regularization, feedforward network processing, residual connection and regularization, and finally outputs the heat map distribution P′ H ∈R J×192 .

[0079] (1.3.3) The output heat map distribution P′ H Pass through independent feature fully connected layers fc x 、fc y 、fc z Perform feature mapping and then distribute it with the original heat map P H，x 、P H，y 、P H，z By adding them together, we can re-separate them into the heat map distribution P′ in the three dimensions of x, y, and z H，x , P′ H，y , P′ H，z ;

[0080] P′ (H，x) =P H，x +fc x (P′ H )

[0081] P′ (H，y) =P H，y +fc y (P′ H )

[0082] P′ (H，z) =P H，z +fc z (P′ H )

[0083] (1.3.4) For the heat map distribution of the three dimensions of each joint point, use soft-argmax to regress the coordinate points to obtain P′ C，x , P′ C，y , P′ C，z ∈R J×1 , and then perform the connection operation to obtain the final human joint point coordinates P′ C =[P′ C，x , P′ C，y , P′ C，z ]∈R J×3 .

[0084] Furthermore, in this embodiment, soft-argmax is defined as follows:

[0085] (1.3.5) The human pose estimation module uses joint loss and skeleton loss as training indicators, which are defined as follows: Among them, P C is the real data, P′ C Predict data for the network.

[0086] (2) Human body mesh model generation steps;

[0087] like Figure 4 As shown, for the intermediate output P′ given in step (1) H =[P′ H，x , P′ H，y , P′ H，z ]∈R J×192 , P′ H After multiple steps of the following: ① the heat map features are input into the Transformer encoder for information interaction; ② the heat map features are input into the 1×1 convolutional network for upsampling of mesh vertices to increase the number of vertices in the mesh model; ③ the heat map features are batch normalized; the final human body mesh model is gradually generated.

[0088] The specific process of the above human body mesh model is as follows:

[0089] Here we assume that the heat map feature input of a single layer operation is F H ∈B×V×192, where B is the batch number, V is the sum of the current joint points and the number of model vertices, and 192 is the current feature dimension.

[0090] (2.1) Interaction of joint point and model vertex information;

[0091] (2.1.1) Given the heat map feature F H =[P′ H，x , P′ H，y , P′ H，z ]∈R J×192 , obtain feature F through position embedding embed =F H +PE, position embedded PE is defined as follows: pos is the position, i is the feature dimension subscript, d model is the feature dimension (set to 256).

[0092] After position embedding, feature F embed Input Transformer encoder, Transformer encoder contains N blocks, each block contains a multi-head attention module and a feedforward neural network (see Figure 3 ). In each block, F embedFirst, the normalized attention weights are calculated by the multi-head attention module and then the features are transformed by the feedforward neural network. Finally, after N blocks, the output That is the vertex heat map distribution of the human body model with V vertices.

[0093] (2.2) Model vertex upsampling;

[0094] (2.2.1) The output after the Transformer encoder is processed Input to the 1×1 convolution layer for upsampling operation, the output is That is, after the 1×1 convolution operation, the number of vertices of the existing model is doubled, thereby achieving model refinement.

[0095] (2.3) Batch regularization;

[0096] (2.3.1) Vertex heat map after sampling Perform batch normalization.

[0097] In summary, after the above steps (2.1)-(2.3), the number of vertices of the human body model doubles, and the model construction process from coarse to fine is realized. Finally, similar to the human body posture estimation module, the output heat map distribution F′ H Re-separate the heat map distribution F′ into three dimensions: x, y, and z H，x , F′ H，y , F′ H，z , use soft-argmax to regress the coordinate points. Among them, vertex loss is introduced as the optimization target. L vertex =||V′ C -V C ||1,V C is the real data, V′ C Predict data for the network.

[0098] (3) Human joint point regression step;

[0099] Taking the human body mesh model V′ C As input, after passing through the general SMPL model regressor, the coordinates of the human joint points corresponding to the human body model can be output Joint point loss is then introduced to constrain the generation of human body models.

[0100] Specifically:

[0101]

[0102] Among them, SMPL.J_regressor∈24×6890 is the joint regression matrix of the SMPL model, M′ c∈6890×3 is the human body mesh model estimated by the above method, The coordinates of the human joint points obtained by regression.

[0103] Further,

[0104] The technical solution described in this invention provides more accurate human pose estimation and body reconstruction results. By introducing an attention mechanism to process heatmap distributions in different dimensional directions, it can more effectively utilize heatmap information in different dimensional directions to guide human pose estimation and body reconstruction. Furthermore, model training time is shortened. Compared to methods that rely on human model templates, this invention adopts a coarse-to-fine generation approach, which requires less computation than methods that directly optimize based on human model templates, thereby shortening model training time.

[0105] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0106] Based on the same concept as the image-based step-by-step generative human body reconstruction method in the above-mentioned embodiment, the present invention also provides an image-based step-by-step generative human body reconstruction method system, which can be used to implement the above-mentioned image-based step-by-step generative human body reconstruction method. For ease of explanation, the structural diagram of the embodiment of the image-based step-by-step generative human body reconstruction method system only shows the parts relevant to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0107] like Figure 6 As shown, in another embodiment of the present application, an image-based step-by-step generation method system 100 for human body reconstruction is provided, comprising a human body posture estimation module 101 , a human body network generation module 102 , and a human body joint point regression module 103 .

[0108] Furthermore, the human posture estimation module 101 is used to extract image features of a given human image, decode the image features to generate a heat map in the three dimensions of x, y, and z, concatenate the heat map distributions in the three dimensions of x, y, and z, and separate the heat map distributions in the three dimensions of x, y, and z after encoding, and then regress the coordinates of the human joint points through heat map integration;

[0109] Furthermore, the human body network generation module 102 is used to pass the human joint point heat map distribution through the transformer encoder to perform human joint point and human body mesh model vertex information interaction, human body mesh model vertex upsampling and batch normalization, and then gradually generate the final human body mesh model;

[0110] Furthermore, the human joint point regression module 103 is used to input the human joint point coordinates into the human mesh model, and after passing through the general SMPL model regressor, output the human joint point coordinates corresponding to the human model as the human mesh model reconstruction constraints.

[0111] The three modules of this system are trained in two stages. In the first stage, the parameters of the human body posture estimation module are trained. Given an input data set, the human body joint coordinates P′ are obtained through the above human body posture estimation process. C , and joint point loss and skeleton loss are used as training indicators, which are defined as follows: L joint =||P′ C -P C ||1, Among them, P C is the real data, P′ C The data is predicted by the network. In this stage, the data flow does not pass through the human body mesh generation module and the human body joint point regression module. In the second stage, the human body mesh generation module and the human body joint point regression module are trained. At this time, the parameters of the human body posture estimation module are fixed, and the x, y, and z direction heat map distribution estimated by the human body posture estimation module is input into the human body mesh generation module to estimate the human body mesh model M′. C , M′ C Then the joint coordinates after regression are output through the joint regression module In this stage, vertex loss and joint point regression loss are used as training indicators, which are defined as follows: L vertex =||M′ C -M C ||1 and

[0112] It should be noted that the image-based step-by-step generation method system of the present invention corresponds one-to-one to the image-based step-by-step generation method of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the image-based step-by-step generation method of the human body reconstruction method are applicable to the embodiment of the image-based step-by-step generation method of the human body reconstruction method. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0113] In addition, in the implementation of the image-based step-by-step generation method system for human body reconstruction in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the image-based step-by-step generation method system for human body reconstruction is divided into different program modules to complete all or part of the functions described above.

[0114] like Figure 6 As shown, in one embodiment, an electronic device for implementing an image-based step-by-step generation method for human body reconstruction is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an image-based step-by-step generation program 203 for human body reconstruction.

[0115] The first memory 202 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 may include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can be used not only to store application software installed on the electronic device 200 and various types of data, such as the code of the image-based step-by-step generation human body reconstruction program 203, but also to temporarily store data that has been output or is about to be output.

[0116] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules (such as federated learning defense programs, etc.) stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0117] Figure 6 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 6 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0118] The image-based step-by-step human body reconstruction program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When executed in the first processor 201, the following can be achieved:

[0119] Extract the image features of a given human body image, decode the image features to generate a heat map in the three dimensions of x, y, and z, concatenate the heat map distributions in the three dimensions of x, y, and z, and separate the heat map distributions in the three dimensions of x, y, and z after encoding, and then regress the coordinates of the human joint points through heat map integration;

[0120] The heat map distribution is processed through the interaction of joint point and model vertex information, upsampling of human body model vertices and batch normalization to gradually generate the final human body mesh model;

[0121] The coordinates of the human joint points are input into the human mesh model. After passing through the general SMPL model regressor, the coordinates of the human joint points corresponding to the human model are output to complete the human body reconstruction.

[0122] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A step-by-step generative human body reconstruction method based on images, characterized in that: The steps include: Extract the image features of a given human body image, decode the image features to generate a heat map in the three dimensions of x, y, and z, then concatenate the heat map distributions in the three dimensions of x, y, and z, and separate the heat map distributions in the three dimensions of x, y, and z after encoding, and then regress the coordinates of the human joint points through heat map integration; The heat map distribution of human joints is passed through the transformer encoder to perform information interaction between human joints and human mesh model vertices, upsample the human mesh model vertices, and perform batch normalization, and then gradually generate the final human mesh model; The human joint coordinates are input into the human mesh model. After passing through the general SMPL model regressor, the human joint coordinates corresponding to the human model are output as the reconstruction constraints of the human mesh model. The interaction of the human body joint points and the human body mesh model vertex information is specifically as follows: Given heatmap feature F H =[p′ H,x ,p′ H,y ,P′ H,z ]∈R J×192 , obtain feature F through position embedding embed =F H +PE, position embedded PE is defined as follows: pos is the position, i is the feature dimension subscript, d model is the feature dimension; After position embedding, feature F embed Input Transformer encoder, Transformer encoder contains N blocks, each block contains a multi-head attention module and a feedforward neural network; in each block, F embed First, the normalized attention weights are calculated by the multi-head attention module and then the features are transformed by the feedforward neural network. Finally, after N blocks, the output That is, the vertex heat map distribution of the human body model with V vertices; The human body model vertex upsampling is specifically as follows: The output after Transformer encoder processing Input to the 1×1 convolution layer for upsampling operation, the output is Where B represents the batch number, that is, after the 1×1 convolution operation, the number of vertices of the existing model is doubled, thereby achieving model refinement.

2. The image-based step-by-step human body reconstruction method according to claim 1, characterized in that: The image features of the given human body image are extracted as follows: Given a human image, crop the human image based on the marked or detected border, remove the background effect, retain the pure human image, and adjust the size of the human image to match the subsequent neural network processing; The processed image is input into the encoder for feature extraction to obtain the human image feature F P , F P The feature dimension is c×h×w; For the obtained human image features, first use the inverse convolution operation in the x and y axis directions The feature map F p The dimension is increased to c′×8h×8w, and then the averaging operation avg is performed on the y-axis dimension in the x-axis direction. y , and then use one-dimensional convolution Get the heat map distribution P in the x-axis direction H,x ∈J×8h; Similarly, in the y-axis direction, the x-axis dimension is averaged avg x , and then use one-dimensional convolution Get the heat map distribution P in the y-axis direction H,y ∈J×8w; In the z-axis direction, perform the averaging operation avg on the x and y axis dimensions x,y , using one-dimensional convolution f p and feature deformation operations Convert the feature dimension to c′×D, and finally use one-dimensional convolution Get the heat map distribution P in the z-axis direction H,z ∈J×D; 3. The image-based step-by-step human body reconstruction method according to claim 2, wherein: The heat map distributions of the three dimensions of x, y, and z are then connected and separated after being encoded by the transformer encoder, specifically: Heat map distribution P in the three dimensions of x, y, and z H,x ,P H,y ,P H,z ∈R J×64 Fuse in the last dimension to obtain the fused feature P H =[P H,x ,P H,y ,P H,z ]∈R J×192 , the fused feature contains the heat map distribution information of three directional dimensions, and then the fused feature is used as the input of the transformer encoder; The fused feature P H Input to the N-layer attention module for heat map information interaction between different directions and different joints; each layer of attention module performs four operations in sequence: multi-head attention, residual connection and regularization, feedforward network processing, residual connection and regularization, and finally outputs the heat map distribution P′ H ∈R J×192 ; The output heat map distribution P′ H Pass through independent feature fully connected layers fc x 、fc y 、fc z Perform feature mapping and then distribute it with the original heat map P H,x 、P H,y 、P H,z Add and re-separate into the heat map distribution P′ in the three dimensions of x, y, and z H,x ,P′ H,y ,P′ H,z ; P′ (H,x) =P H,x +fc x (P′ H ) P′ (H,y) =P H,y +fc y (P′ H ) P ′(H,z) =P H,z +fc z (P′ H ) For the heat map distribution of the three dimensions of each joint point, use soft-argmax to regress the coordinate points to obtain P′ C,x ,P′ C,y ,P′ C,z ∈R J×1 , and then perform the connection operation to obtain the final human joint point coordinates P′ C =[P′ C,x ,P′ C,y ,P′ C,z ]∈R J×3 .

4. The image-based step-by-step generative human body reconstruction method according to claim 3, characterized in that: The soft-argmax is defined as follows:

5. The image-based step-by-step generative human body reconstruction method according to claim 1, characterized in that: The human body joint point coordinates are input into the human body mesh model, and after passing through the general SMPL model regressor, the human body joint point coordinates corresponding to the human body model are output to complete the human body reconstruction, specifically: Among them, SMPL.J_regressor∈24×6890 is the joint regression matrix of the SMPL model, M′ c ∈6890×3 is the human body mesh model estimated by the above method, The coordinates of the human joint points obtained by regression.

6. An image-based step-by-step generative human body reconstruction system, characterized in that: The image-based step-by-step generative human body reconstruction method applied to any one of claims 1-5, characterized by comprising a human body posture estimation module, a human body network generation module, and a human body joint point regression module; The human posture estimation module is used to extract image features of a given human image, decode the image features to generate a heat map in the three dimensions of x, y, and z, then concatenate the heat map distributions in the three dimensions of x, y, and z, separate the heat map distributions in the three dimensions of x, y, and z after encoding, and then regress the coordinates of the human joint points through heat map integration; The heat map distribution of human joints is passed through the transformer encoder to perform information interaction between human joints and human mesh model vertices, upsample the human mesh model vertices, and perform batch normalization, and then gradually generate the final human mesh model; The human joint point regression module is used to input the human joint point coordinates into the human mesh model, and after passing through the general SMPL model regressor, output the human joint point coordinates corresponding to the human body model as the human mesh model reconstruction constraints.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the image-based step-by-step generation method for human body reconstruction as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the image-based step-by-step generation method for human body reconstruction according to any one of claims 1 to 5 is implemented.

Images