An end-to-end human 3D model reconstruction method based on frequency domain

Abstract

The application discloses a kind of end-to-end human three-dimensional model reconstruction methods based on frequency domain, comprising the following steps: step 1: to single given color image is down-sampling;Step 2: feature extraction is carried out to original drawing and down-sampling image;Step 3: high-resolution features extracted in original drawing are used to obtain unobstructed human UV chart;Step 4: using the global features and UV space features extracted in down-sampling image, feature extraction is carried out in frequency domain, and offset UV chart representing clothes details is obtained;Step 5: superimpose unobstructed human UV chart and offset UV chart, sampling is carried out, and the final reconstructed human body is obtained.The application uses human prior information to ensure the rationality of reconstruction result, and uses offset UV chart above corresponding human surface to describe clothes details, and simultaneously uses high-frequency and low-frequency features of frequency domain to make clothes reconstruction result more accurate.

Description

Technical Field

[0001] This invention relates to a three-dimensional model reconstruction method, and more particularly to an end-to-end human three-dimensional model reconstruction method based on the frequency domain. Background Technology

[0002] Digitizing the human body from a single color image is a significant and poorly solved problem in computer vision and graphics. It can serve various applications in other vision domains, such as realistic body modeling and virtual try-on. Previously, much work focused on fitting parametric models to obtain usable reconstructed human bodies. With the development of deep learning technology, thanks to the significant advantages of deep networks in parametric fitting tasks, human body reconstruction based on parametric models has achieved increasingly better results. However, due to the smooth representation characteristic of parameter space, details on the human body, such as clothing wrinkles and deformations caused by movement, cannot be represented.

[0003] Besides directly representing the human body using parametric models, there are also non-parametric methods that can effectively model the human body, such as voxels, implicit functions, or direct regression prediction of the human body's vertices. The reconstruction results of these methods can reveal some details, but these details are not explicitly combined with prior knowledge of the human body shape, and cannot infer poses that are not present in the dataset. Therefore, the performance of the final method depends on the comprehensiveness of the dataset. Currently, for the task of predicting the human body with clothing, some methods combine the above two representations, using non-parametric relative displacement to represent details on the basis of predicting the parametric human body, thus modeling the human body with clothing. However, due to the complexity of clothing types, this method usually requires predefining specific clothing types during training, making it difficult to generalize to scenarios with different clothing types. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide an end-to-end human three-dimensional model reconstruction method based on the frequency domain, which addresses the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, this invention discloses an end-to-end human 3D model reconstruction method based on the frequency domain, comprising the following steps:

[0006] Step 1: Downsample the original image (i.e., a single given color image) to obtain the downsampled image. Specific methods include:

[0007] Step 1-1: For all input images, first scale them to 512×512 pixels to obtain the original image.

[0008] Steps 1-2 involve downsampling the original 512×512 pixel image to obtain a 256×256 pixel image, which is the downsampled image.

[0009] Step 2: Extract features from both the original image and the downsampled image to obtain high-resolution features, global features, and UV spatial features, including the following steps:

[0010] Step 2-1: Use the ResNet-50 structure to extract features from the original image to obtain high-scoring features. Specific methods include:

[0011] Using the ResNet-50 architecture, the original image with a pixel size of 512×512 in the RGB (red, green, blue) three channels is first subjected to convolution operations with a kernel size of 7×7, normalization operations, and ReLU activation operations. Then, the pooling layer of the ResNet-50 architecture is used to reduce the dimensionality of the data after the above operations, resulting in features with a dimension of 256×256×64. After passing through four bottleneck layer modules, the intermediate features of the four bottleneck layer modules are finally fused to obtain high-resolution features with a dimension of 256×256×128.

[0012] Step 2-2: Using the encoder-decoder architecture, feature extraction is performed on the downsampled image to obtain global features and UV space features. Specific methods include:

[0013] An encoder-decoder architecture is used to extract features from the downsampled image. The encoder uses a ResNet-50 structure to perform 7×7 convolution, normalization, and ReLU activation operations on the downsampled image with a pixel size of 256×256 in the three input channels, resulting in features of dimension 256×256×64. Then, the features are passed through four bottleneck layers, which fuse the intermediate features of the modules to obtain global features of dimension 256×256×128. The decoder distinguishes between the front and back of the human body in the image and decodes the points belonging to the front of the human body to obtain UV space features.

[0014] In the four-layer bottleneck layer module, the number of bottleneck layer modules in each layer is 3, 4, 6 and 4 respectively.

[0015] Step 3: Use the high-resolution features extracted from the original image to obtain an unoccluded human UV map. That is, directly decode the high-resolution features obtained in Step 2-1 to obtain an unoccluded human UV map.

[0016] The aforementioned direct decoding involves using a normalized exponential function for decoding.

[0017] Step 4: Using the global and UV spatial features extracted from the downsampled image, feature extraction is performed in the frequency domain to obtain an offset UV map representing clothing details. This includes the following steps:

[0018] Step 4-1: Perform a Fast Fourier Transform on the global features obtained in Step 2-2 to convert them to the frequency domain and obtain the global features in the frequency domain.

[0019] Step 4-2: Use low-frequency filtering in the frequency domain to remove redundant information and obtain the low-frequency characteristics of the human body.

[0020] Step 4-3: Use the inverse Fourier fast transform to convert the low-frequency features after low-frequency filtering in step 4-2 back to the spatial domain, and obtain the features converted back to the spatial domain. These features are the features that highlight the details of the human body and clothing.

[0021] Step 4-4: Using the features converted back to the spatial domain in Step 4-3 and the UV spatial features in Step 2-2, decode and generate the offset UV map.

[0022] Step 5: Overlay the unoccluded human UV map and the offset UV map, sample, and reconstruct the final human 3D model, completing the end-to-end human 3D model reconstruction based on the frequency domain. This specifically includes:

[0023] By overlaying the unobstructed human body UV map and UV offset map to obtain a new UV map, sampling is performed to obtain the final reconstructed 3D human body model.

[0024] Beneficial effects:

[0025] 1) A novel end-to-end dual human reconstruction network is proposed, which can simultaneously reconstruct unoccluded human bodies and clothing offsets, and the two reconstruction tasks can promote each other.

[0026] 2) Using non-parametric offset UV maps to represent positional offsets on unoccluded human bodies allows for more flexible representation of clothing on clothed human bodies.

[0027] 3) Use frequency domain features to reconstruct clothing on the human body, and divide clothing into low-frequency features and high-frequency features for separate prediction. Low-frequency features represent the overall shape of clothing, and high-frequency features represent the fold details of clothing. At the same time, the frequency domain features are further refined in the spatial domain. Attached Figure Description

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0029] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.

[0030] Figure 2 This is a schematic diagram of the neural network used in this invention.

[0031] Figure 3 This is a schematic diagram of the human body UV map and a 3D model of the human body.

[0032] Figure 4 This is a schematic diagram illustrating the result of implementing the present invention. Detailed Implementation

[0033] This invention provides an end-to-end human 3D model reconstruction method based on the frequency domain, which improves upon the shortcomings of existing human reconstruction methods in representing clothing details. Its core lies in obtaining information about the unobstructed human body and clothing details through feature extraction from a single color image. Furthermore, UV offset is employed in the representation of clothing details, i.e., projecting the 3D model onto a 2D space, representing the offset of the clothed human body relative to the unobstructed human body. This utilizes prior information about the human body, and simultaneously uses both high-frequency and low-frequency features to ensure the accuracy of the offset. Figure 1 As shown, the specific steps include:

[0034] Step 1: Downsample a single given color image;

[0035] Step 2: Extract features from the original image and the downsampled image;

[0036] Step 3: Use the high-resolution features extracted from the original image to obtain an unoccluded human UV map, where the UV map refers to the image obtained by projecting a three-dimensional human body into a two-dimensional space.

[0037] Step 4: Using the global features and UV spatial features extracted from the downsampled image, perform feature extraction in the frequency domain to obtain an offset UV map representing clothing details;

[0038] Step 5: Overlay the unobstructed human UV map and the offset UV map, and sample to obtain the final reconstructed human body.

[0039] Further, step 1 includes downsampling a single given color image, comprising the following steps:

[0040] Step 1-1: For all input images, first scale them down to 512×512 to obtain the original images;

[0041] Steps 1-2 involve downsampling the original 512×512 image. For each 2×2 pixel grid in the original image, the average of four pixels is taken as the value of a single new pixel, resulting in a 256×256 image.

[0042] Furthermore, step 2 includes an image feature extraction step, which includes the following steps:

[0043] Step 2-1: Use the ResNet-50 structure to extract features from the original image. The ResNet-50 structure (reference: Deepestial learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.) first performs 7×7 convolutions, normalization, and ReLU (Rectified Luminaire) activation on the input 512×512×3 image. Then, it enters a pooling layer to reduce the dimensionality of the data to obtain 256×256×64 features. Next, it passes through four bottleneck layers with 3, 4, 6, and 4 layers respectively, and finally, the intermediate features of the modules are fused to obtain high-resolution features of 256×256×128.

[0044] Step 2-2: Feature extraction is performed on the downsampled image. An encoder-decoder architecture is used, where the encoder, based on a ResNet-50 structure, performs 7×7 convolution, normalization, and ReLU activation on the input 256×256×3 image to obtain 256×256×64 features. These features are then passed through four bottleneck layers of 3, 4, 6, and 4 layers respectively, and finally, the intermediate features of these layers are fused to obtain 256×256×128 global features. The decoder first obtains the human body contour using an additional input human body mask, distinguishes between the front and back of the body, and decodes the points belonging to the front of the body using a normalized exponential function to obtain UV spatial features.

[0045] Furthermore, the feature processing steps in step 3 specifically include:

[0046] Step 3-1: For the high-scoring features obtained in Step 2-1, the normalized exponential function is directly used for decoding to obtain the unoccluded human UV map.

[0047] Furthermore, the feature processing steps in step 4 specifically include:

[0048] Step 4-1: For the global features obtained in Step 2-2, perform a Fast Fourier Transform to convert them to the frequency domain. The low-frequency regions in the frequency domain mainly correspond to the approximate shape of the clothing surface and the folds and wrinkles of the clothing. By learning the low-frequency components of the clothing in the frequency domain, this invention can reconstruct the shape and details of the clothing.

[0049] Step 4-2: In the frequency domain, a low-pass Gaussian filter with a window size of 256 and a variance of 20 is used to perform a weighted average of the pixel values ​​of the entire image. The value of each pixel is obtained by weighting its own value and the values ​​of other pixels in the neighborhood, removing redundant information and obtaining the low-frequency features of the human body.

[0050] Step 4-3: Using inverse Fourier fast transform according to formulas (1) and (2), the frequency domain features after low-frequency filtering are transformed to the spatial domain, where... The connection represents the feature, ξ represents the inverse fast Fourier transform, and φ represents a 1×1 convolution. ψ represents a 3×3 convolution, and convolution ψ represents a 5×5 convolution.

[0051]

[0052] Step 4-4: For the low-frequency features in Step 4-3 and the UV spatial features in Step 2-2, perform feature concatenation between the two, and use the new features obtained to decode and generate an offset UV map using a normalized exponential function.

[0053] Further, in step 5, following step 3-1 where an unobstructed human UV map was obtained, and step 4-4 where a UV shift map was obtained, the following steps are performed:

[0054] Step 5-1: In the two-dimensional UV space, according to formula (3), the coordinates of the unoccluded human body UV map and the UV offset map are superimposed to obtain a new UV map. Then, according to formula (4), the 6890 vertices of the human body model are sampled on the UV map to obtain the final reconstructed human body. Where U pred_clot Represents the UV coordinates of a human body with clothing on, U pred_nake U represents the UV coordinates of the unobstructed human body. pred_disp Represents the offset UV coordinates; u i and v i V represents the UV coordinates of a vertex in the UV graph. i X represents the vertex coordinates, where X is the vertex UV coordinate (u... i ,v i The mapping from 1000 vertices to 3D spatial coordinates, where N is the 6890 vertices of the 3D human body model.

[0055] U pred_clot =U pred_nake +U pred_disp …(3)

[0056] V i =X(u i ,v i )i=1,2,...,N…(4)

[0057] Example 1:

[0058] This invention discloses an end-to-end human 3D model reconstruction method based on the frequency domain. Its core lies in obtaining information about the unobstructed human body and clothing details through feature extraction from a single color image. Furthermore, UV offset is used in the representation of clothing details, utilizing prior information about the human body, and both high-frequency and low-frequency features are used to ensure the accuracy of the offset. Figure 2 As shown, the specific steps include:

[0059] Step 1: Downsample a single given color image;

[0060] Step 2: Extract features from the original image and the downsampled image;

[0061] Step 3: Use the high-resolution features extracted from the original image to obtain an unoccluded human UV map;

[0062] Step 4: Using the global features and UV spatial features extracted from the downsampled image, perform feature extraction in the frequency domain to obtain an offset UV map representing clothing details;

[0063] Step 5: Overlay the unobstructed human UV map and the offset UV map, perform sampling, and obtain the final reconstructed human body.

[0064] Step 1 involves downsampling a single given color image, including the following steps:

[0065] Step 1-1: For the input image, first scale it to 512×512 to obtain the original image;

[0066] Steps 1-2 involve downsampling the original 512×512 image. For each 2×2 pixel grid in the original image, the average of four pixels is taken as the value of a single new pixel, resulting in a 256×256 image.

[0067] Step 2 includes an image feature extraction step, which includes the following steps:

[0068] Step 2-1: Use the trained ResNet-50 structure to extract features from the original image. The ResNet-50 structure first performs 7×7 convolution, normalization, and ReLU activation on the input 512×512×3 image, and then enters the pooling layer to reduce the dimensionality of the data to obtain 256×256×64 features; then it goes through four bottleneck layer modules with 3, 4, 6, and 4 layers respectively, and finally the intermediate features of the modules are fused to obtain high-resolution features of 256×256×128.

[0069] Step 2-2: Feature extraction is performed on the downsampled image. An encoder-decoder architecture is used, where the encoder, with a pre-trained ResNet-50 structure, performs 7×7 convolution, normalization, and ReLU activation on the input 256×256×3 image to obtain 256×256×64 features. These features are then passed through four bottleneck layers of 3, 4, 6, and 4 layers respectively, and finally, the intermediate features of these layers are fused to obtain 256×256×128 global features. The decoder first accepts a human body mask as input. The human body mask is an image representing the outline of the human body in the original image; points contained in the mask are considered to belong to the front of the human body. The decoder uses this as a basis to distinguish between the front and back of the human body, and decodes the global features of the points belonging to the front of the human body using a normalized exponential function to obtain UV spatial features.

[0070] The feature processing steps in step 3 specifically include:

[0071] Step 3-1: For the high-scoring features obtained in Step 2-1, use the normalized exponential function to decode and obtain an unoccluded human UV map, such as... Figure 3 As shown in the top left, the bottom left is the 3D model of the unobstructed human body obtained by sampling the UV map of the unobstructed human body.

[0072] The feature processing steps in step 4 specifically include:

[0073] Step 4-1: Perform a Fast Fourier Transform on the global features obtained in Step 2-2 to convert them to the frequency domain.

[0074] Step 4-2: In the frequency domain, a low-pass Gaussian filter with a window size of 256 and a variance of 20 is used to perform a weighted average of the pixel values ​​of the entire image. The value of each pixel is obtained by weighting its own value and the values ​​of other pixels in the neighborhood, removing redundant information and obtaining the low-frequency features of the human body.

[0075] Step 4-3: Apply the inverse Fourier fast transform to the low-frequency features in step 4-2 according to the above formulas (1) and (2) to transform them into the spatial domain;

[0076] Step 4-4: For the low-frequency features converted to the spatial domain in Step 4-3 and the UV spatial features in Step 2-2, perform feature concatenation between the two, and use the normalized exponential function to decode the new features to generate an offset UV map.

[0077] In step 5, after obtaining the unobstructed human UV map in step 3-1 and the UV shift map in step 4-4, the following steps are performed:

[0078] Step 5-1: In the two-dimensional UV space, according to the above formula (3), superimpose the coordinates of the unobstructed human UV map and the UV offset map to obtain a new clothing human UV map, as follows: Figure 3 As shown in the upper right, according to the above formula (4), 6890 vertices of the human body model are sampled on the UV map to obtain the final reconstructed human body, as shown in the figure. Figure 3 As shown in the bottom right corner.

[0079] like Figure 4 The results achieved according to the present invention are shown in the figure, where the left side is the input image, the middle side is the predicted human body, and the right side is the real human body.

[0080] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding an end-to-end human 3D model reconstruction method based on the frequency domain, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0081] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0082] This invention provides a concept and method for end-to-end human 3D model reconstruction based on the frequency domain. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A frequency-domain-based end-to-end human three-dimensional model reconstruction method, characterized in that, The method comprises the following steps: Step 1: downsampling the original image, i.e. a single given color image, to obtain a downsampled image; Step 2: extracting features from the original image and the downsampled image respectively to obtain high-resolution features, global features and UV space features; Step 3: obtaining an unoccluded human body UV map from the high-resolution features extracted from the original image; Step 4: extracting features in the frequency domain using the global features and the UV space features extracted from the downsampled image to obtain an offset UV map representing clothing details; Step 5: superimposing the unoccluded human body UV map and the offset UV map, sampling, and reconstructing a final human body three-dimensional model to complete the end-to-end human body three-dimensional model reconstruction based on the frequency domain; In step 2, the feature extraction from the original image and the downsampled image respectively comprises the following steps: Step 2-1: using a ResNet-50 structure to extract features from the original image to obtain high-resolution features; Step 2-2: using an encoder-decoder architecture to extract features from the downsampled image to obtain global features and UV space features; In step 4, the feature extraction in the frequency domain comprises the following steps: Step 4-1: performing a fast Fourier transform on the global features obtained in step 2-2 to convert them to the frequency domain to obtain global features in the frequency domain; Step 4-2: using low-frequency filtering in the frequency domain to remove redundant information to obtain low-frequency features of the human body part; Step 4-3: using inverse fast Fourier transform to convert the low-frequency features filtered in step 4-2 back to the spatial domain to obtain features converted back to the spatial domain, which are the features highlighting the clothing detail information of the human body; Step 4-4: using the features converted back to the spatial domain in step 4-3 and the UV space features in step 2-2 to decode and generate an offset UV map.

2. The frequency-domain based end-to-end human three-dimensional model reconstruction method of claim 1, wherein, In step 1, the downsampling of the original image, i.e. a single given color image, comprises the following steps: Step 1-1: scaling the size of all input pictures to 512x512 pixels as the original image, i.e. the original image; Step 1-2: downsampling the original image of 512x512 pixels to obtain an image of 256x256 pixels, i.e. the downsampled image.

3. The method of claim 2, wherein, In step 2-1, the use of the ResNet-50 structure to extract features from the original image comprises the following steps: Using the ResNet-50 structure, first performing a convolution operation with a kernel size of 7x7, a normalization operation and a ReLU activation operation on the original image with an RGB three-channel pixel size of 512x512, then entering the pooling layer of the ResNet-50 structure to reduce the dimension of the result data after the above operations to obtain features with a dimension of 256x256x64; then passing through four bottleneck layer modules, and finally fusing the intermediate features of the four bottleneck layer modules to obtain high-resolution features with a dimension of 256x256x128.

4. The method of claim 3, wherein, In step 2-2, the feature extraction from the downsampled image comprises the following steps: The down-sampled image is subjected to feature extraction using an encoder-decoder architecture, wherein the encoder adopts a ResNet-50 structure, performs a 7*7 convolution operation, a normalization operation and a ReLU activation operation on the input down-sampled image with three channels and a pixel size of 256*256 to obtain a feature with a dimension of 256*256*64, and then passes through four bottleneck layer modules, and finally the four bottleneck layer modules fuse the intermediate features to obtain a global feature with a dimension of 256*256*128; the decoder distinguishes the front and back of the human body in the image and decodes the points belonging to the front of the human body to obtain a UV space feature.

5. The method of claim 4, wherein, In the four bottleneck layer modules described in steps 2-1 and 2-2, the number of bottleneck layer modules in each layer is 3, 4, 6 and 4, respectively.

6. The method of claim 5, wherein, In step 3, the high-score feature extracted from the original image is used to obtain an unoccluded human body UV map, that is, the high-score feature obtained in step 2-1 is directly decoded to obtain an unoccluded human body UV map.

7. The method of claim 6, wherein, In step 5, the final human body three-dimensional model is reconstructed, specifically including: After superimposing the unoccluded human body UV map and the UV offset map to obtain a new UV map, sampling is performed to obtain the final reconstructed human body three-dimensional model.

8. The method of claim 7, wherein, In step 3, the decoding is directly performed, that is, the decoding is performed using a normalization exponential function.

Images