Hand new view angle image generation method combining MANO model and 3DGS
By combining the MANO model and 3DGS method, the hand shape parameters are extracted and the initialization point cloud is collected and the Gaussian ellipsoid model is optimized, which solves the problem of slow and low quality of new perspective images in the hand, and achieves high-quality and fast rendering New perspective images on the hand to meet the real-time needs of virtual reality applications.
Patent Information
- Application Number
- CN202410925863.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-11
AI Technical Summary
The existing new hand perspective image generation method has problems such as slow rendering speed, low image quality, missing hand details and incomplete shapes, which cannot meet the real-time needs of virtual reality applications.
Combining the MANO model and 3DGS method, the hand shape and pose parameters are extracted through convolutional neural network, a three-dimensional hand grid is established, and the point cloud is collected and initialized, and the Gaussian ellipsoid model is optimized by using the 3DGS method to achieve high-quality new perspective hand Image rendering.
The quality and rendering speed of new perspective images on the hand are improved, meeting the needs of interactive real-time applications, and the generated images are closer to the real situation, and the PSNR and LPIPS indicators have also been improved.
Smart Images

Figure CN119068142A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for generating new-view images according to multi-view images of a hand, and in particular to a method for generating new-view images of a hand by combining a MANO model and 3DGS, and belongs to the field of computer vision. Background Art
[0002] New perspective image generation refers to generating new perspective images of an object based on the information provided by multiple perspective images of a known object. New perspective image generation has a wide range of applications in the field of modern virtual reality technology. One of the more typical applications is multi-perspective image generation of hands. Because gestures are the basic human-computer interaction means of modern virtual reality applications, achieving realistic hand image rendering from any perspective helps to enhance the immersion of virtual reality applications. However, due to the complex joint structure and motion deformation of the hand, it is very difficult to render realistic hand images. At present, some methods have been proposed for the problem of new perspective image generation of hands, but these methods have problems such as blurred hand details and slow rendering speed.
[0003] The MANO (hand Model with Articulated and Non-rigid defOrmations) model is an articulated, parameterized, hand posture estimation model proposed by Romero J et al. in 2017 (Romero J, Tzionas D, Black M J. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, 2017, 36(6): 1-17.). The model is able to calculate a three-dimensional hand mesh model consisting of 778 vertices and 1538 faces based on the hand morphological parameters and posture parameters predicted from the hand image. Based on the mesh model, a new perspective image of the hand with skin texture can be rendered through texture mapping. The problem is that the hand image rendered in this way will be stretched or deformed at the surface and edge, and the realism is affected. For this reason, some studies have introduced neural radiation fields into this field, and used the hand mesh generated by the MANO model as the prior knowledge for sampling the neural radiation field, so as to speed up the modeling of the neural radiation field. The hand images rendered by this type of method are of higher quality overall, but the rendering speed is slow and cannot meet the needs of real-time applications; in addition, there is also the problem of missing high-frequency details of the image.
[0004] 3DGS (3D Gaussian Splatting) is a high-fidelity scene fast reconstruction and rendering method proposed by Kerbl B et al. (Kerbl B, Kopanas G, Leimkühler T, et al. 3D Gaussian Splatting for Real-time Radiance Field Rendering. ACM Transactions on Graphics, 2023, 42(4): 1-14.). This method models the 3D scene as a series of 3D Gaussian ellipsoids based on the known multi-view images of the 3D scene, and then quickly renders the 3D scene model represented by Gaussian into an image through highly parallelized differentiable rendering technology. This provides a new idea for solving the problems of slow rendering speed and lack of high-frequency details in the existing new-view image generation of hands. However, when this method is directly applied to the generation of new-view images of hands, the quality of the generated images is not high, and the peak signal-to-noise ratio (PSNR) index is relatively low; in addition, there is an incomplete hand shape phenomenon in the generated new-view hand images. Therefore, the generation of high-quality new-view images of hands remains a challenge. Summary of the invention
[0005] Aiming at the problem that the existing new perspective image generation speed of the hand is slow and the image quality is not high, the present invention proposes a new perspective image generation method of the hand combining the MANO model and 3DGS.
[0006] A method for generating a hand new perspective image by combining the MANO model and 3DGS comprises the following steps:
[0007] Step 1: Create a 3D mesh of the hand based on the MANO model.
[0008] To this end, we first use multi-view images of the hand to train a convolutional neural network, and use the trained convolutional neural network to extract hand shape parameters from the multi-view images of the hand. and attitude parameters The hand shape parameters and attitude parameters Input the MANO model and calculate the 3D mesh of the hand.
[0009] Step 2: Collect the hand initialization point cloud under the guidance of the hand 3D mesh.
[0010] The acquisition process of the hand initialization point cloud includes three parts: grid-guided surface point cloud sampling, grid-guided internal point cloud sampling, and noise regularization point cloud sampling.
[0011] In order to make the results of surface point cloud sampling reflect both the size of the structure and the local geometric features, the grid-guided surface point cloud sampling adopts a sampling method that combines surface area weighting and feature curvature weighting.
[0012] The grid-guided internal point cloud sampling is related to the multi-view images of the hand. Given a view and its hand image, first start from the viewpoint corresponding to the view, project a ray pointing to the image pixel, and then calculate the intersection of the projected ray and the 3D mesh model of the hand. When the projected ray and the 3D mesh model of the hand have two and only two intersections, and the distance between the two intersections is greater than a preset threshold, a sampling point is selected between the two intersections of the projected ray.
[0013] Noise regularized point cloud sampling is to randomly select a certain number of sampling points according to a uniform probability distribution within the bounding box of the 3D mesh model of the hand.
[0014] The final hand initialization point cloud is the union of the above three sampling results.
[0015] Step 3: Use the 3DGS method to build a hand 3DGS model for the hand multi-view images.
[0016] The 3DGS method is a technology and method for modeling three-dimensional scenes based on multi-view images. The modeling result is a 3DGS model represented by a series of Gaussian ellipsoids with color, shape and position information.
[0017] To establish the 3DGS model of the hand, firstly, the initial 3DGS model of the hand is created according to the 3DGS method using the hand initialization point cloud obtained in step 2 as the initial position of the Gaussian ellipsoid; then the 3DGS model of the hand is optimized using the known multi-view images of the hand.
[0018] The optimization process of the hand 3DGS model is: render the 3DGS model at a known viewing angle to obtain the corresponding rendered image, calculate the loss between the rendered image and the real image at this viewing angle, update the color, shape, position and other parameters of the Gaussian ellipsoid in the 3DGS model according to the loss, and obtain a new 3DGS model; repeat this process until the loss between the rendered image and the real image reaches the expected target.
[0019] Step 4: Generate hand images at any new perspective based on the optimized hand 3DGS model rendering.
[0020] Beneficial Effects
[0021] Compared with the prior art and methods, the present invention has the following advantages and characteristics:
[0022] (1) The present invention uses the MANO model to estimate the three-dimensional mesh of the hand from the hand image, and uses the three-dimensional mesh to guide the initialization of point cloud sampling, which can better utilize the three-dimensional posture information of the hand and help improve the quality of the new perspective image of the hand.
[0023] (2) In order to maximize the expression of hand features by the hand initialization point cloud, the present invention performs sampling of the initialization point cloud from three aspects: grid-guided surface point cloud sampling, grid-guided internal point cloud sampling, and noise regularization sampling. The initialization point cloud obtained by these samplings is more suitable for the complex geometric structure of the hand, making the generated hand image closer to the real situation, thereby obtaining a high-quality new perspective image of the hand.
[0024] (3) The present invention utilizes the highly parallelized differentiable rendering technology of 3DGS to achieve fast rendering of new perspective images of the hand, which can meet the needs of interactive real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 The processing flow of the method of the present invention;
[0026] Figure 2 Figure 1. Convolutional neural network architecture for extracting hand shape and posture parameters.
[0027] Figure 3 3D mesh of the hand for some gestures;
[0028] Figure 4 Initialization point cloud sampling of some gestures, where the first row is the hand images of different gestures, the second row is the 3D mesh model of the hand, the third row is the surface point cloud sampling result, the fourth row is the internal point cloud sampling result, the fifth row is the noise regularization point cloud sampling result, and the sixth row is the final merged hand initialization point cloud;
[0029] Figure 5 The method of the present invention generates new perspective images for different gestures, wherein the odd-numbered rows are newly generated images, and the even-numbered rows are corresponding real images;
[0030] Figure 6 Comparison of the hand new perspective images generated by the method of the present invention and the 3DGS method, wherein the first column is the hand new perspective image generated by the 3DGS method, the second column is the hand new perspective image generated by the method of the present invention, and the third column is the real image;
[0031] Figure 7Qualitative comparison results of the method of the present invention with the SelfRecon method, the HumanNeRF method, and the HandAvatar method, wherein the first row is the result of the SelfRecon method, the second row is the result of the HumanNeRF method, the third row is the result of the HandAvatar method, and the fourth row is the result of the method of the present invention. DETAILED DESCRIPTION
[0032] The specific implementation of the present invention is described below in conjunction with the accompanying drawings and examples. It is worth noting that the examples are only used to explain the present invention and do not constitute any limitation to the present invention.
[0033] The goal of the present invention is to generate a hand image at any new perspective based on a series of given multi-view images of the hand. The image referred to here is a common RGB image. The method of the present invention first uses a convolutional neural network to obtain hand shape parameters and posture parameters from the hand image, and then uses these parameters to calculate the three-dimensional hand mesh according to the MANO model, and then uses the three-dimensional mesh to guide the initialization of the hand point cloud, and then uses the 3DGS method to establish a hand 3DGS model based on the initialized point cloud and the given multi-view images of the hand, and finally renders the hand 3DGS model to generate a high-quality hand image at any new perspective.
[0034] Figure 1 The processing flow of the method for generating a hand new perspective image combining the MANO model and 3DGS is given in the present invention, and its main steps are as follows:
[0035] Step 1: Create a 3D hand mesh based on the MANO model.
[0036] Step 1 contains sub-steps 1.1 and 1.2.
[0037] Step 1.1: Use a convolutional neural network to extract hand shape parameters from a series of multi-view hand images and attitude parameters
[0038] The convolutional neural network structure used in this embodiment to extract hand shape and posture parameters is as follows: Figure 2 As shown in Figure 2, the backbone network of the network is a pre-trained ResNet-50, and necessary convolutional layers, maximum pooling layers, and fully connected layers are added at both ends of the backbone network to connect the input and output data of the entire grid.
[0039] This embodiment uses the public hand multi-view image dataset InterHand2.6M (Moon, G., Yu, SI., Wen, H., Shiratori, T., Lee, KM InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image, ECCV 2020.) as the implementation test object. The dataset includes 554 gestures, each of which has 139 images from different perspectives, and the pixel resolution of each image is 512×334.
[0040] In this embodiment, 21 images of each hand gesture are randomly selected as training data, and the remaining 118 images of each hand gesture are used as test data. The training data of each hand gesture is used to train the neural network used in step 1.1, and then the hand shape parameters are extracted from the hand multi-view images through the trained network. and attitude parameters
[0041] Step 1.2: Set the hand shape parameters and attitude parameters Input the MANO model and calculate the 3D mesh of the hand.
[0042] The specific calculation process of the hand 3D mesh is consistent with the calculation process of the MANO model described in the background art, and will not be repeated here. The finally calculated 3D hand mesh consists of 778 vertices and 1538 triangles.
[0043] The three-dimensional hand mesh of the partial gesture obtained in step 1.2 of this embodiment is as follows: Figure 3 As shown, the 1st, 3rd and 5th rows are hand images, and the 2nd, 4th and 6th rows are the 3D hand meshes corresponding to the previous row.
[0044] Step 2: Collect the hand initialization point cloud under the guidance of the hand 3D mesh.
[0045] The purpose of this step is to collect the initialization point cloud required for hand 3DGS modeling. The collection process includes grid-guided surface point cloud sampling, grid-guided internal point cloud sampling, and noise regularization point cloud sampling. The final sampling result is the union of these three parts of the sampling results.
[0046] The specific processes of grid-guided surface point cloud sampling, grid-guided internal point cloud sampling, and noise regularized point cloud sampling are shown in steps 2.1, 2.2, and 2.3, respectively.
[0047] Step 2.1: Mesh-guided surface point cloud sampling.
[0048] The purpose of step 2.1 is to collect point clouds on the surface of the hand 3D mesh. In order to make the results of surface point cloud sampling reflect both the size of the structure and the local geometric features, the present invention adopts a sampling method that combines surface area weighting and feature curvature weighting. To this end, first, for each triangle face i of the hand 3D mesh, the area A of the triangle face i is calculated in turn. i , characteristic curvature T i , comprehensive weight w i And the sampling probability p i , and then according to the sampling probability p i Randomly sample n within triangle i i point.
[0049] The area A of triangle i i The calculation method is:
[0050]
[0051] Where v1, v2, and v3 are vector representations of the three edges that make up triangle patch i.
[0052] The characteristic curvature T of triangle patch i i is the local curvature S of triangle i i and Gaussian curvature G i The weighted average of .
[0053] The local curvature S of triangle patch i i It is defined as the average value of the difference between the normal vectors of triangle i and its adjacent triangles, and is calculated as follows:
[0054]
[0055] Where N i is the normal vector of triangle i, N ij is the normal vector of triangle j adjacent to triangle i, and k is the number of triangles adjacent to triangle i. Here, two triangles are adjacent means that they have common adjacent edges.
[0056] Gaussian curvature G of triangle patch i i It is defined as the average value of the Gaussian curvature of each vertex of triangle patch i, that is
[0057]
[0058] Where K ip is the Gaussian curvature of vertex p of triangle patch i, and the calculation method is defined as:
[0059]
[0060] where θ ipj is the internal angle of vertex p of triangle i in the adjacent triangle j, A ij is the area of triangle j adjacent to triangle i, and k is the number of triangles adjacent to triangle i.
[0061] After obtaining the local curvature S of triangle patch i i and Gaussian curvature G i Then, the characteristic curvature T of triangle patch i i The calculation method is:
[0062] T i =βS i +(1-β)G i ,
[0063] Wherein β is a weighting coefficient, which can be adjusted according to actual needs. In this embodiment, β=0.8.
[0064] The comprehensive weight w of triangle patch i i From area A i and characteristic curvature T i The comprehensive calculation method is:
[0065] w i =A i ×(1+αT i ),
[0066] Wherein α is an adjustable parameter. In this embodiment, α=0.8.
[0067] The sampling probability p of triangle patch i i Defined as its comprehensive weight w i The ratio relative to the combined weight of all triangles, that is:
[0068]
[0069] where w i and w j is the comprehensive weight of the triangles, and H is the number of triangles in the 3D mesh of the hand.
[0070] The number of sampling points n in triangle patch i i Defined as:
[0071]
[0072] Where N is the total number of sampling points of the surface cloud set according to actual needs, p i is the sampling probability of triangle patch i. In this embodiment, N=800.
[0073] Step 2.2: Mesh-guided interior point cloud sampling.
[0074] The purpose of step 2.2 is to collect point clouds inside the 3D mesh model of the hand. This collection process is related to the multi-view images of the hand. Given a view and its hand image, first start from the viewpoint corresponding to the view, project a ray pointing to the image pixel, and then calculate the intersection of the projected ray and the 3D mesh model of the hand. When the projected ray has only two intersections with the 3D mesh model of the hand, and the distance between the two intersections is greater than the preset threshold δ, a sampling point is selected between the two intersections of the projected ray.
[0075] The intersection points of the projected light and the 3D mesh model of the hand are P near and P far , P near Indicates the point close to the viewpoint, P far Represents a point far from the viewpoint, when ||P far -P near When ||>δ, the sampling point P on the projection ray is calculated according to the following formula, that is
[0076] P=P near +λ×(P far -P near ),
[0077] Wherein λ is a coefficient between 0 and 1, which is used to control the position of the sampling point. λ can be set according to actual needs. In this embodiment, δ=0.05, λ=0.2.
[0078] The data set used in this embodiment contains 554 gestures, each of which contains 139 images from different perspectives. After the internal point cloud sampling is performed according to the above method, the total number of sampling points will be too many. In this regard, this embodiment only selects 1% of the sampling points as the final internal point cloud sampling result.
[0079] Step 2.3: Noise regularized point cloud sampling.
[0080] The purpose of step 2.3 noise regularization point cloud sampling is to introduce uncertainty and reduce the probability of falling into local optimum in the subsequent processing process. To this end, first obtain the bounding box of the 3D mesh model of the hand, and then randomly select a certain number of sampling points in the bounding box according to uniform probability distribution. In this embodiment, 100 sampling points are randomly selected in this step.
[0081] The initialization point cloud sampling results of some gestures obtained in step 2 of this embodiment are as follows: Figure 4As shown, the first row is the hand images with different gestures, the second row is the 3D mesh model of the hand, the third row is the surface point cloud sampling result, the fourth row is the internal point cloud sampling result, the fifth row is the noise regularized point cloud sampling result, and the sixth row is the hand initialization point cloud that is finally merged together.
[0082] Step 3: Use the 3DGS method to build a hand 3DGS model for the hand multi-view images.
[0083] As described in the background technology, the 3DGS method is a technology and method for modeling a three-dimensional scene based on multi-view images, and its modeling result is a three-dimensional scene model represented by a series of Gaussian ellipsoids with color, shape and position information. For the convenience of description, the present invention simply refers to the modeling result of 3DGS as a 3DGS model.
[0084] The process of establishing a hand 3DGS model for a hand multi-view image using the 3DGS method in the present invention includes steps 3.1 and 3.2.
[0085] Step 3.1: Use the hand initialization point cloud obtained in step 2 as the initial position of the Gaussian ellipsoid to create an initial hand 3DGS model.
[0086] The 3DGS model created in step 3.1 is composed of a series of Gaussian ellipsoids. These ellipsoids respectively use the positions of each point in the initialization point cloud obtained in step 2 as the positions of their center points, and the colors and shapes of the ellipsoids are initialized according to the 3DGS method described in the background art.
[0087] Step 3.2: Optimize the hand 3DGS model using known multi-view images of the hand.
[0088] Step 3.2 The basic process of optimizing the 3DGS model is: render the 3DGS model at a known viewing angle to obtain the corresponding rendered image, calculate the loss between the rendered image and the real image at this viewing angle, and update the color, shape, position and other parameters of the Gaussian ellipsoid in the 3DGS model according to the loss to obtain a new 3DGS model; repeat this process until the loss between the rendered image and the real image reaches the expected target.
[0089] The loss of rendered image and real image in the above optimization process It is calculated according to the following formulas (1) to (4), namely:
[0090]
[0091]
[0092] Among them, λ1, λ2, λ3 are adjustable weight coefficients, M is the number of pixels in the image, and y iis the pixel value of the real image, is the pixel value of the rendered image, μ yi and are the local means of the corresponding pixels of the rendered image and the real image in a 11×11 window, σ yi and are the local variances of the corresponding pixels of the rendered image and the real image in a 11×11 window, is the corresponding covariance, C1 and C2 are small constants added to avoid the denominator being zero. In this embodiment, λ1=0.7, λ2=0.2, λ3=0.1, C1=1e-4, C2=9*C1.
[0093] In the above optimization process, the rendering of the 3DGS model and the updating of the 3DGS model are consistent with the 3DGS method described in the background technology, and will not be repeated here. However, unlike the 3DGS method, the present invention adopts a new method to calculate the loss between the rendered image and the real image, namely the above formulas (1) to (4). In addition, the present invention uses the hand initialization point cloud to create the initial 3DGS model, which is also different from the original 3DGS method.
[0094] Step 4: Generate hand images at any new perspective by rendering based on the optimized hand 3DGS model.
[0095] The rendering of the 3DGS model in step 4 is consistent with the method of rendering the 3DGS model involved in step 3.2, that is, the highly parallelized differentiable rendering in the 3DGS method described in the background art, in which the new perspective is the input information and needs to be specified in advance.
[0096] In step 4, this embodiment selects the perspective of a portion of the hand image as the test data as the new perspective, and obtains a new perspective image of a portion of the gesture, such as Figure 5 As shown; the odd-numbered rows are newly generated images, and the even-numbered rows are the corresponding real images. The generated images are very close to the real images.
[0097] To further illustrate the beneficial effects of the present invention, this embodiment uses the test data in the InterHand2.6M data set to conduct a comparative test between the present invention and several related methods in the field. The comparative test results and related situations are as follows.
[0098] Benchmark method. The benchmark methods compared with the present invention include: 1) the original 3DGS method, which uses feature detection and feature matching to construct an initial point cloud; 2) SelfRecon method (Jiang B, Hong Y, BaoH, et al. Selfrecon: Self Reconstruction Your Digital Avatar from MonocularVideo. Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. IEEE, 2022: 5605-5615.); 3) HumanNeRF (Weng CY, Curless B, Srinivasan PP, et al. HumanNerf: Free-viewpoint Rendering of Moving People from MonocularVideo. Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. IEEE, 2022: 16210-16220.); 4) HandAvatar (Chen X, Wang B, Shum HY. Handavatar: Free-pose Hand Animation and Rendering from MonocularVideo. Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. IEEE, 2023:8683-8693.).
[0099] Evaluation indicators. The evaluation indicators used in the comparative test are: 1) PSNR indicator (Sheikh HR, Sabir MF, Bovik A CA Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing, 2006, 15 (11): 3440-3451.); 2) LPIPS indicator (R. Zhang, P. Isola, AAE fros, E. Shechtman and O. Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 586-595.). PSNR is defined based on mean square error and can measure the error between the real image and the generated image. The higher the PSNR value, the smaller the error and the higher the quality of the generated image. LPIPS measures image similarity by calculating the distance between image features. The lower the LPIPS value, the better the image quality generated.
[0100] Experimental results. Figure 6 The figure shows some new perspective images of hands generated by the method of the present invention and the 3DGS method, as well as the corresponding real images; the first column is the new perspective images of hands generated by the 3DGS method, the second column is the new perspective images of hands generated by the method of the present invention, and the third column is the real image. Figure 6 It can be seen that the new perspective image of the hand generated by the method of the present invention has better effect, the skin texture details at the finger joints are clearer, and is closer to the actual situation.
[0101] Table 1 shows the comparison results of the method of the present invention and the 3DGS method in terms of evaluation indicators, from which it can be seen that the method of the present invention improves the PSNR value and reduces the LPIPS value.
[0102] Table 1 Comparison results of evaluation indicators between the method of the present invention and the 3DGS method
[0103]
[0104] Figure 7The qualitative comparison results of the method of the present invention with the SelfRecon, HumanNeRF, and HandAvatar methods are shown in Figure 1, where the first row is the result of the SelfRecon method, the second row is the result of the HumanNeRF method, the third row is the result of the HandAvatar method, and the fourth row is the result of the method of the present invention. Figure 7 From the comparison results shown, it can be seen that the method of the present invention is more suitable for the special form of highly articulated hands, can effectively deal with the self-occlusion problem of the hand structure, and the generated hand form is more delicate and realistic. However, the hand forms obtained by other methods are not realistic enough and have problems such as unclear form caused by occlusion between fingers.
[0105] Table 2 Comparison results of evaluation indicators of the proposed method with SelfRecon, HumanNeRF, and HandAvatar methods
[0106]
[0107]
[0108] Table 2 shows the quantitative comparison results of the method of the present invention with the SelfRecon, HumanNeRF, and HandAvatar methods in terms of evaluation indicators. It can be seen that the method of the present invention is superior to other methods in terms of both PSNR and LPIPS indicators, further demonstrating the beneficial effects of the method of the present invention.
[0109] The above steps and embodiments illustrate the entire process of a method for generating a hand new perspective image combining the MANO model and 3DGS described in the present invention.
[0110] It should be understood that this embodiment is only a specific example of the implementation of the present invention and should not be a limitation of the protection scope of the present invention. Without departing from the spirit and scope of the present invention, any equivalent modification or change to the above content should be included in the scope of protection claimed by the present invention.
Claims
1. A method for generating a new perspective image of a hand by combining the MANO model and 3DGS, characterized in that: The following steps are involved: Step 1: Create a 3D hand mesh based on the MANO model; To this end, we first use multi-view images of the hand to train a convolutional neural network, and use the trained convolutional neural network to extract hand shape parameters from the multi-view images of the hand. and attitude parameters The hand shape parameters and attitude parameters Input the MANO model and calculate the 3D mesh of the hand; Step 2: Collect the hand initialization point cloud under the guidance of the hand 3D grid; The acquisition process of the hand initialization point cloud includes three parts: grid-guided surface point cloud sampling, grid-guided internal point cloud sampling, and noise regularization point cloud sampling; In order to make the results of surface point cloud sampling reflect both the size of the structure and the local geometric features, the grid-guided surface point cloud sampling adopts a sampling method that combines surface area weighting and feature curvature weighting; The grid-guided internal point cloud sampling is related to the multi-view images of the hand. Given a view and its hand image, first start from the viewpoint corresponding to the view, project a ray pointing to the image pixel, and then calculate the intersection of the projected ray and the 3D grid model of the hand. When the projected ray and the 3D grid model of the hand have only two intersections, and the distance between the two intersections is greater than a preset threshold, a sampling point is selected between the two intersections of the projected ray. The noise regularized point cloud sampling is to randomly select a certain number of sampling points according to uniform probability distribution within the bounding box of the 3D mesh model of the hand; The final hand initialization point cloud is the union of the above three sampling results; Step 3: Use the 3DGS method to build a hand 3DGS model for the hand multi-view images; The 3DGS method is a technology and method for modeling three-dimensional scenes based on multi-view images. The modeling result is a 3DGS model represented by a series of Gaussian ellipsoids with color, shape and position information. To establish the 3DGS model of the hand, firstly, the initial 3DGS model of the hand is created using the hand initialization point cloud obtained in step 2 as the initial position of the Gaussian ellipsoid according to the 3DGS method; then, the 3DGS model of the hand is optimized using the known multi-view images of the hand; The optimization process of the hand 3DGS model is as follows: render the 3DGS model at a known viewing angle to obtain the corresponding rendered image, calculate the loss between the rendered image and the real image at the viewing angle, update the color, shape, position and other parameters of the Gaussian ellipsoid in the 3DGS model according to the loss, and obtain a new 3DGS model; repeat this process until the loss between the rendered image and the real image reaches the expected target; Step 4: Generate hand images at any new perspective by rendering based on the optimized hand 3DGS model.
2. The method for generating a hand new perspective image combining the MANO model and 3DGS according to claim 1, characterized in that: The grid-guided surface point cloud sampling in step 2 adopts a sampling method that combines surface area weighting and characteristic curvature weighting. The specific sampling process is: first, for each triangle face i of the hand 3D grid, the area A of triangle face i is calculated in turn. i , characteristic curvature T i , comprehensive weight w i And the sampling probability p i , and then according to the sampling probability p i Randomly sample n within triangle i i point; The area A of triangle i i The calculation method is: Where v1, v2, and v3 are vector representations of the three edges that make up triangle patch i; The characteristic curvature T of triangle patch i i is the local curvature S of triangle i i and Gaussian curvature G i The weighted average of The local curvature S of triangle patch i i It is defined as the average value of the difference between the normal vectors of triangle i and its adjacent triangles, and is calculated as follows: Where N i is the normal vector of triangle i, N ij is the normal vector of triangle j adjacent to triangle i, k is the number of triangles adjacent to triangle i; here, two triangles are adjacent means that they have common adjacent edges; Gaussian curvature G of triangle patch i i It is defined as the average value of the Gaussian curvature of each vertex of triangle patch i, that is Where K ip is the Gaussian curvature of vertex p of triangle patch i, and the calculation method is defined as: where θ ipj is the internal angle of vertex p of triangle i in the adjacent triangle j, A ij is the area of the triangles adjacent to triangle i, k is the number of triangles adjacent to triangle i; After obtaining the local curvature S of triangle patch i i and Gaussian curvature G i Then, the characteristic curvature T of triangle patch i i The calculation method is: T i =βS i +(1-β)G i , Where β is the weighting coefficient, which can be adjusted according to actual needs; The comprehensive weight w of triangle patch i i From area A i and characteristic curvature T i The comprehensive calculation method is: w i =A i ×(1+αT i ), Where θ is an adjustable parameter; The sampling probability p of triangle patch i i Defined as its comprehensive weight w i The ratio relative to the combined weight of all triangles, that is: where w i and w j is the comprehensive weight of the triangles, H is the number of triangles in the 3D mesh of the hand; The number of sampling points n in triangle patch i i Defined as: Where N is the total number of sampling points of the surface cloud set according to actual needs, p i is the sampling probability of triangle i.
3. The method for generating a hand new perspective image combining the MANO model and 3DGS according to claim 1, characterized in that: The method involved in step 2 for selecting a sampling point between two intersection points of the projected rays is: let P near represents the intersection point close to the viewpoint, P far Represents the intersection point far from the viewpoint, sampling point P = P near +λ×(P far -P near ), where λ is a coefficient between 0 and 1, which can be set according to actual needs and is used to control the position of the sampling point.
4. The method for generating a hand new perspective image combining the MANO model and 3DGS according to claim 1, characterized in that: Step 3: In the process of optimizing the hand 3DGS model, the loss between the rendered image and the real image is calculated according to the following formulas (1) to (4): Right now: Among them, λ1, λ2, λ3 are adjustable weight coefficients, M is the number of pixels in the image, and y i is the pixel value of the real image, is the pixel value of the rendered image, μ yi and are the local means of the corresponding pixels of the rendered image and the real image in a 11×11 window, σ yi and are the local variances of the corresponding pixels of the rendered image and the real image in a 11×11 window, is the corresponding covariance, C1 and C2 are small constants added to avoid the denominator being zero. In this embodiment, λ1=0.7, λ2=0.2, λ3=0.1, C1=1e-4, C2=9*C1.
Citation Information
Patent Citations
Multi-view reconstruction method based on grid neural rendering and hexagonal constraint
CN116612251A
Method and system for unbounded scene reconstruction and new view angle synthesis based on 3DGS
CN118135122A
Cited By
Gaussian image rendering method and device
CN119850805A
Information processing device and information processing method
WO2026154573A1
