A lightweight 3D virtual clone generation method suitable for terminal devices

Abstract

This invention discloses a lightweight 3D virtual avatar generation method suitable for terminal devices. It utilizes a mobile phone camera to capture the user's image information in real time, preprocesses the image by cropping it to 256×256 pixels and standardizing it, extracts low-level and high-level features from the image using a dual-branch network, and fuses these features to characterize the image at different granularities. A regression module outputs the body shape parameters of the SMPL human model to reconstruct the 3D human model. Finally, a 3D rendering tool is used to obtain a 2D image of the 3D model from the camera's perspective K, which is then output to the user's screen. This invention allows users to quickly create their own virtual avatar in the metaverse using embedded devices such as smartphones.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a lightweight method for generating 3D virtual clones. Background Technology

[0002] With the arrival of the metaverse era, more and more companies and research institutions are beginning to explore the use of technologies such as VR and AR to try to build bridges between the virtual and real worlds. Virtual humans are an important component of the metaverse. Also known as digital humans, virtual humans refer to virtual human figures simulated on computers that resemble real people.

[0003] Traditional methods of creating 3D virtual avatars involve a significant amount of manual work. First, 3D modeling software is used to create a model of the person. Then, to make the avatar move, the 3D model needs to be bound to a skeletal system, and motion capture technology is used to drive human movement. With the development of deep learning and 3D reconstruction technologies, many methods, such as PiFu and Human Nerf, can accurately recreate 3D models of people using a small number of 2D images. However, these methods rely on powerful GPU resources and are limited by the complexity of the models, preventing virtual human technology from becoming widely adopted in people's lives. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a lightweight 3D virtual avatar generation method suitable for terminal devices. It utilizes a mobile phone camera to capture real-time image information of the current user, preprocesses the image by cropping it to 256×256 pixels and standardizing it, extracts low-level and high-level features of the image using a dual-branch network, and performs feature fusion to characterize the image at different granularities. A regression module outputs the body shape parameters of the SMPL human model to reconstruct the 3D human model. Finally, a 3D rendering tool is used to obtain the 3D model from the camera's perspective. The invention generates a two-dimensional image and outputs it to the user's screen. This invention allows users to quickly create their own virtual avatar in the metaverse using embedded devices such as smartphones.

[0005] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0006] Step 1: Capture the user's full-body RGB image using a camera;

[0007] Step 2: Preprocess the image obtained in Step 1, including: (1) adjusting the resolution of the image to A×A, and (2) performing normalization processing;

[0008] Step 3: Low-level feature extraction: Downsample the preprocessed image to extract its low-level features. The downsampling part contains three convolutional layers. The first layer is a Conv2D convolutional layer, and the other two layers are depth-separable convolutional layers DSConv. The kernel size of each convolutional layer is 3×3, and the stride is 2.

[0009] Step 4: High-level feature extraction: Based on the low-level features, a high-level feature extractor is used to further obtain the global contextual information of the image. The advanced feature extractor consists of three bottleneck residual modules and one pyramid pooling layer. The bottleneck residual modules are used to further extract deeper information, and the pyramid pooling layer is used to aggregate contextual information from different regions.

[0010] Step 5: Feature Fusion: Perform bilinear interpolation on the high-level features to obtain feature vectors with the same dimensions as the low-level features; add the low-level features... With advanced features Obtain the fused feature vector :

[0011] ;

[0012] A bottleneck layer is used to further process the fused vector, followed by average pooling, to obtain the final 256-dimensional image feature vector. ;

[0013] Step 6: Initialize the SMPL human model using the standard human body parameters provided by SMPL, which include body shape parameters. Posture parameters and camera position parameters ;

[0014] Step 7: Regress SMPL parameters: Transfer the feature vector respectively with The data is then spliced ​​together, followed by a three-step regression to obtain the final result. as well as :

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] in, and All are linear fully connected networks. This represents a vector concatenation operation. and They are respectively and Intermediate vectors during the processing;

[0022] Step 8: Based on parameters The 6890 vertices of the SMPL 3D model were recovered:

[0023] ;

[0024] Step 9: Render the 3D model using the Sim3DR rendering tool to obtain the 3D model from the camera's perspective. The two-dimensional image is generated and output to the user's screen:

[0025] .

[0026] Preferably, .

[0027] The beneficial effects of this invention are as follows:

[0028] The network parameter count of this invention is only 3.231M, while the network parameter count of the HMR algorithm based on ResNet50 is 26.978M. Compared to the HMR algorithm, this invention significantly reduces the number of network parameters, thus requiring only 67.59ms to infer an image on a CPU. This invention maintains good human body reconstruction performance while being lightweight. Attached Figure Description

[0029] Figure 1 This is a model diagram of the method of the present invention.

[0030] Figure 2 This is a rendering of the human body reconstruction according to an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] The present invention is based on the following principle: the dual-branch network encodes images at different resolutions, which can reduce the amount of computation while maintaining accuracy: (1) the deep network uses small resolution images as input and uses a deeper network to extract global context information; (2) the shallow network uses large resolution images as input to capture spatial detail information and edge information of the input image; (3) the final image feature is represented by the fusion of the results of the two branches, which helps the model to understand human images more effectively.

[0033] like Figure 1 As shown, a lightweight 3D virtual avatar generation method suitable for terminal devices includes the following steps:

[0034] Step 1: Use the phone's camera to capture a full-body RGB image of the user.

[0035] Step 2: Input image preprocessing, including: (1) adjusting the resolution of the image to 256×256, and (2) standardizing the cropped image.

[0036] Step 3: Low-level feature extraction: Downsampling is performed on the preprocessed image to extract low-level features at high resolution. The downsampling part contains three convolutional layers. The first layer is a standard convolutional layer (Conv2D), and the other two layers are depth-separable convolutional layers (DSConv). The kernel size of each convolutional layer is 3×3, and the stride is 2.

[0037] Step 4: High-level feature extraction: Based on the low-level features, further obtain the global context information of the image. The advanced feature extractor consists of three linear bottleneck residual modules and a pyramid pooling layer. The bottleneck layer is used to further extract deeper information, and the pyramid pooling layer is used to aggregate contextual information from different regions.

[0038] Step 5: Feature Fusion: First, to ensure consistent vector dimensions, bilinear interpolation is performed on the high-level features to obtain feature vectors with the same dimensions as the low-level features; second, the low-level features are added together. With advanced features Obtain the fused feature vector :

[0039] ;

[0040] Finally, a bottleneck layer is used to further process the fused vector, followed by average pooling, to obtain the final 256-dimensional image feature vector. .

[0041] Step 6: Initialize the SMPL human model using the standard human body parameters provided by SMPL, which include body shape parameters. Posture parameters and camera position parameters .

[0042] Step 7: Regress SMPL parameters: Transfer the feature vector respectively with The data is then spliced ​​together, followed by a three-step regression to obtain the final result. as well as :

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] in, and All are linear fully connected networks. This represents a vector concatenation operation. and They are respectively and Intermediate vectors in the processing process

[0050] Step 8: Based on parameters The 6890 vertices of the SMPL 3D model were recovered:

[0051] .

[0052] Step 9: Render the 3D model using the Sim3DR rendering tool to obtain the 3D model from the camera's perspective. The two-dimensional image is generated and output to the user's screen.

[0053] . Specific implementation examples:

[0055] 1. Capture the user's full-body RGB image using the phone's camera.

[0056] 2. Input image preprocessing, including: (1) adjusting the resolution of the image to 256×256, and (2) standardizing the cropped image.

[0057] 3. Low-level feature extraction: The preprocessed image is downsampled to extract low-level features at high resolution. The downsampling part contains three convolutional layers. The first layer is a standard convolutional layer (Conv2D), and the other two layers are depth-separable convolutional layers (DSConv). The kernel size of each convolutional layer is 3×3, and the stride is 2.

[0058] 4. Advanced Feature Extraction: Building upon low-level features, further global contextual information of the image is obtained. The advanced feature extractor consists of three linear bottleneck residual modules and a pyramid pooling layer. The bottleneck layer is used to further extract deeper information, and the pyramid pooling layer is used to aggregate contextual information from different regions.

[0059] 5. Feature Fusion: First, to ensure consistent vector dimensions, bilinear interpolation is performed on high-level features to obtain feature vectors with the same dimensions as low-level features; second, the low-level features are added together. With advanced features Obtain the fused feature vector :

[0060] ;

[0061] Finally, a linear bottleneck layer is used to further process the fused vectors, followed by average pooling, to obtain the final 256-dimensional image feature vector. .

[0062] 6. Initialize the SMPL human model using the standard human body parameters provided by SMPL, which include body shape parameters. Posture parameters and camera position parameters .

[0063] 7. Regress SMPL parameters: Transform the feature vector respectively with The data is then spliced ​​together, followed by a three-step regression to obtain the final result. as well as :

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] ;

[0069] .

[0070] 8. Based on parameters The 6890 vertices of the SMPL 3D model were recovered:

[0071] .

[0072] Step 9: Render the 3D model using the Sim3DR rendering tool to obtain the 3D model from the camera's perspective. The two-dimensional image is generated and output to the user's screen:

[0073] .

[0074] like Figure 2 A rendering of a reconstructed human body.

Claims

1. A lightweight 3D virtual avatar generation method suitable for terminal devices, characterized in that, Includes the following steps: Step 1: Capture a full-body RGB image of the user using the user's smartphone camera; Step 2: Preprocess the image obtained in Step 1, including: (1) adjusting the resolution of the image to 256×256, and (2) performing normalization processing; Step 3: Low-level feature extraction: Downsample the preprocessed image to extract its low-level features. The downsampling part contains three convolutional layers. The first layer is a Conv2D convolutional layer, and the other two layers are depth-separable convolutional layers DSConv. The kernel size of each convolutional layer is 3×3, and the stride is 2. Step 4: High-level feature extraction: Based on the low-level features, a high-level feature extractor is used to further obtain the global contextual information of the image. The advanced feature extractor consists of three bottleneck residual modules and one pyramid pooling layer. The bottleneck residual modules are used to further extract deeper information, and the pyramid pooling layer is used to aggregate contextual information from different regions. Step 5: Feature Fusion: Perform bilinear interpolation on the high-level features to obtain feature vectors with the same dimensions as the low-level features; add the low-level features... With advanced features Obtain the fused feature vector : ； A bottleneck layer is used to further process the fused vector, followed by average pooling, to obtain the final 256-dimensional image feature vector. ; Step 6: Initialize the SMPL human model using the standard human body parameters provided by SMPL, which include body shape parameters. Posture parameters and camera position parameters ; Step 7: Regress SMPL parameters: Transfer the feature vector respectively with The data is then spliced ​​together, followed by a three-step regression to obtain the final result. as well as : ； ； ； ； ； ； in, and All are linear fully connected networks. This represents a vector concatenation operation. and They are respectively and Intermediate vectors during the processing; Step 8: Based on parameters The 6890 vertices of the SMPL 3D model were recovered: ； Step 9: Render the 3D model using the Sim3DR rendering tool to obtain the 3D model from the camera's perspective. The image is a two-dimensional image and output to the user's smartphone screen: 。

Images