A hand new perspective image generation method combining a MANO model and a 3D GS

Abstract

The application relates to a method for generating a new view image according to a hand multi-view image, and belongs to the field of computer vision, and aims to solve the problems of slow generation speed and low image quality of an existing hand new view image. In the case of a known hand multi-view image, the method is characterized in that: firstly, a convolutional neural network is used to obtain hand shape parameters and posture parameters from the hand image; then, the parameters are used to calculate a hand three-dimensional grid according to a MANO model; the three-dimensional grid is used to guide hand point cloud initialization; a 3DGS method is used to establish a hand 3DGS model according to the initialized point cloud and the given hand multi-view image; and finally, a high-quality hand image under an arbitrary new view is rendered and generated according to the hand 3DGS model. The method not only makes the generated hand image closer to the real situation through the point cloud initialization guided by the three-dimensional grid, but also realizes the rapid generation of the new view image by using the parallel rendering technology of the 3DGS.

Description

TECHNICAL FIELD

[0001] The application relates to a method for generating a new view image of a hand from multi-view images of the hand, in particular to a method for generating a new view image of a hand by combining a MANO model and a 3D GS, and belongs to the field of computer vision. BACKGROUND

[0002] New view image generation refers to generating a new view image of an object according to the information provided by multiple view images of the object. New view image generation has a wide range of applications in the field of modern virtual reality technology. One of the typical applications is hand multi-view image generation. Because hand gestures are the basic human-computer interaction means in modern virtual reality applications, rendering a realistic hand image in any view helps to improve the immersion of virtual reality applications. However, due to the complex joint structure and motion deformation of the hand, it is difficult to render a realistic hand image. At present, some methods for generating a new view image of a hand have been proposed, but these methods have problems such as blurred hand details and slow rendering speed.

[0003] The MANO (hand Model with Articulated and Non-rigid deformations) model is a jointed, parameterized, hand pose estimation model proposed by Romero J et al. in 2017 (Romero J, Tzionas D, Black M J. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, 2017, 36(6): 1-17.). The model can calculate a three-dimensional hand mesh model composed of 778 vertices and 1538 faces according to the hand shape parameters and pose parameters predicted from the hand image. On the basis of the mesh model, a new view image of the hand with skin texture can be rendered by texture mapping. The problem is that the hand image rendered in this way will appear stretched or deformed at the curved surface and edge, affecting the realism. Therefore, some studies have introduced neural radiance fields into this field, using the hand mesh generated by the MANO model as prior knowledge for neural radiance field sampling to speed up the modeling of neural radiance fields. The hand image rendered by this method has higher overall quality, but the rendering speed is slow and cannot meet the real-time application requirements; in addition, there is a problem of missing high-frequency details in the image.

[0004] 3DGS (3D Gaussian Splatting) is a high-fidelity scene fast reconstruction and rendering method proposed by Kerbl B et al. (Kerbl B, Kopanas G, Leimkühler T, et al. 3d Gaussian Splatting for Real-time Radiance Field Rendering. ACM Transactions on Graphics, 2023, 42(4): 1-14.). According to the known multi-view images of the three-dimensional scene, the method models the three-dimensional scene as a series of three-dimensional Gaussian ellipsoids, and then quickly renders the three-dimensional scene model represented by Gaussian into an image through highly parallelized differentiable rendering technology. This provides a new idea for solving the problem of slow rendering speed and lack of high-frequency details in existing hand novel view image generation. However, when this method is directly applied to hand novel view image generation, the quality of the generated image is not high, and the peak signal-to-noise ratio (PSNR) index is relatively low. In addition, there is a phenomenon of incomplete hand shape in the generated novel view hand image. Therefore, high-quality hand novel view image generation is still a challenge. SUMMARY

[0005] In view of the slow speed and low image quality of existing hand novel view image generation, the present application proposes a hand novel view image generation method combining MANO model and 3DGS.

[0006] A hand novel view image generation method combining MANO model and 3DGS includes the following steps:

[0007] Step 1: Establishing a hand three-dimensional grid based on the MANO model.

[0008] To this end, first train a convolutional neural network using multi-view hand images, and use the trained convolutional neural network to extract hand shape parameters and pose parameters from the multi-view hand images. Input the hand shape parameters and pose parameters into the MANO model to calculate the hand three-dimensional grid.

[0009] Step 2: Collecting hand initialization point cloud under the guidance of hand three-dimensional grid.

[0010] The collection process of hand initialization point cloud includes three parts: surface point cloud sampling guided by the grid, internal point cloud sampling guided by the grid, and noise regularization point cloud sampling.

[0011] In order to make the result of surface point cloud sampling reflect the structure size and represent the local geometric features, the grid-guided surface point cloud sampling adopts a sampling mode combining surface area weighting and feature curvature weighting.

[0012] The grid-guided internal point cloud sampling is related to the hand multi-view image. In the case of a given view and its hand image, first, a light ray is projected from the viewpoint corresponding to the view to the image pixel point, and then the intersection of the projected light ray and the hand three-dimensional grid model is calculated. When the projected light ray has and only has two intersection points with the hand three-dimensional grid model, and the distance between the two intersection points is greater than a pre-set threshold, a sampling point is selected between the two intersection points of the projected light ray.

[0013] The noise regularization point cloud sampling is to randomly select a certain number of sampling points in the bounding box of the hand three-dimensional grid model according to a uniform probability distribution.

[0014] The final hand initialization point cloud is the union of the sampling results of the above three parts.

[0015] Step 3: A hand 3DGS model is established for the hand multi-view image by using the 3DGS method.

[0016] The 3DGS method is a technology and method for modeling a three-dimensional scene according to a multi-view image, and the modeling result is a 3DGS model represented by a series of Gaussian ellipsoids with color, shape and position information.

[0017] In order to establish the hand 3DGS model, first, the hand initialization point cloud obtained in Step 2 is used as the initial position of the Gaussian ellipsoid according to the 3DGS method to create an initial hand 3DGS model; and then the hand 3DGS model is optimized by using the known hand multi-view image.

[0018] The optimization process of the hand 3DGS model is: the 3DGS model is rendered under the known view to obtain the corresponding rendered image, the loss of the rendered image and the real image under the view is calculated, the color, shape and position parameters of the Gaussian ellipsoid in the 3DGS model are updated according to the loss, and a new 3DGS model is obtained; the process is repeated until the loss of the rendered image and the real image reaches the expected target.

[0019] Step 4: The hand image under any new view is generated by rendering according to the optimized hand 3DGS model.

[0020] Advantages

[0021] Compared with the prior art and method, the present application has the following advantages and characteristics:

[0022] (1) The present application uses the MANO model to estimate the hand three-dimensional grid from the hand image, and uses the three-dimensional grid to guide the initialization point cloud sampling, which can better utilize the hand three-dimensional posture information and help improve the quality of the hand new view image.

[0023] (2) In order to maximize the hand initialization point cloud to express the hand features, the present application samples the initialization point cloud from three aspects: grid-guided surface point cloud sampling, grid-guided internal point cloud sampling and noise regularization sampling. The initialization point cloud obtained by these samplings is more suitable for the complex geometry of the hand, so that the generated hand image is closer to the real situation, thereby obtaining a high-quality hand new view image.

[0024] (3) The present application uses the 3DGS highly parallelizable differentiable rendering technology to realize the fast rendering of the hand new view image, which can meet the needs of interactive real-time applications. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The processing flow of the method of the present application;

[0026] Figure 2 Convolutional neural network structure diagram for extracting hand shape and posture parameters;

[0027] Figure 3 Part of the hand gesture three-dimensional grid;

[0028] Figure 4 Part of the gesture initialization point cloud sampling, wherein the first row is the hand image of different gestures, the second row is the hand three-dimensional grid model, the third row is the surface point cloud sampling result, the fourth row is the internal point cloud sampling result, the fifth row is the noise regularization point cloud sampling result, and the sixth row is the final hand initialization point cloud combined together;

[0029] Figure 5 The new view image generated by the method of the present application for different gestures, wherein the odd rows are newly generated images, and the even rows are corresponding real images;

[0030] Figure 6 The hand new view image generated by the method of the present application compared with the 3DGS method, wherein the first column is the hand new view image generated by the 3DGS method, the second column is the hand new view image generated by the method of the present application, and the third column is the real image;

[0031] Figure 7The qualitative comparison results of the method of the present application and the SelfRcon method, the HumanNeRF method and the HandAvatar method, wherein the first row is the result of the SelfRecon method, the second row is the result of the HumanNeRF method, the third row is the result of the HandAvatar method, and the fourth row is the result of the method of the present application. DETAILED DESCRIPTION

[0032] The specific embodiments of the present application are described below in conjunction with the accompanying drawings and examples. It should be pointed out that the examples are only used to explain the present application and do not constitute any limitation on the present application.

[0033] The object of the present application is to generate hand images in any new view according to a series of given multi-view images of the hand. The images referred to here are usually RGB images. The method of the present application first uses a convolutional neural network to obtain hand shape parameters and pose parameters from the hand images, then uses these parameters to calculate the hand three-dimensional mesh according to the MANO model, initializes the hand point cloud by the three-dimensional mesh, then uses the 3DGS method to establish the hand 3DGS model according to the initialized point cloud and the given multi-view images of the hand, and finally renders high-quality hand images in any new view according to the hand 3DGS model.

[0034] Figure 1 The processing flow of the method of generating new view images of the hand combining the MANO model and 3DGS according to the present application is given, and the main steps are as follows:

[0035] Step 1: Establishing a hand three-dimensional mesh based on the MANO model.

[0036] Step 1 includes sub-steps 1.1 and 1.2.

[0037] Step 1.1: Using a convolutional neural network to extract hand shape parameters and pose parameters

[0038] The convolutional neural network structure used to extract hand shape and pose parameters in this embodiment is shown in Figure 2 The backbone network of this network is a pre-trained ResNet-50, and necessary convolutional layers, max-pooling layers and fully connected layers are added at both ends of the backbone network to connect the input and output data of the entire network.

[0039] The present embodiment uses the public hand multi-view image dataset InterHand2.6M (Moon, G., Yu, SI., Wen, H., Shiratori, T., Lee, K. M. InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image, ECCV 2020.) as the implementation test object. The dataset includes a total of 554 gestures, each with 139 images of different views, and each image has a pixel resolution of 512x334.

[0040] The present embodiment randomly selects 21 view images for each gesture as training data, and the remaining 118 view images as test data. The neural network used in step 1.1 is trained using the training data for each gesture, and then the trained network is used to extract hand shape parameters and pose parameters

[0041] Step 1.2: Input the hand shape parameters and pose parameters into the MANO model to calculate the hand three-dimensional mesh.

[0042] The specific calculation process of the hand three-dimensional mesh is consistent with the calculation process of the MANO model described in the background art, which will not be repeated here. The final calculated hand three-dimensional mesh is composed of 778 vertices and 1538 triangular patches.

[0043] The hand three-dimensional mesh of some gestures obtained in step 1.2 of the present embodiment is shown in Figure 3 , where the first, third, and fifth rows are hand images, and the second, fourth, and sixth rows are hand three-dimensional meshes corresponding to the previous row.

[0044] Step 2: Collect hand initialization point cloud under the guidance of hand three-dimensional mesh.

[0045] The purpose of this step is to collect the initialization point cloud required for hand 3DGS modeling. The collection process includes mesh-guided surface point cloud sampling, mesh-guided internal point cloud sampling, and noise-regularized point cloud sampling. The final sampling result is the union of the three sampling results.

[0046] The specific processes of mesh-guided surface point cloud sampling, mesh-guided internal point cloud sampling, and noise-regularized point cloud sampling are shown in steps 2.1, 2.2, and 2.3, respectively.

[0047] Step 2.1: Mesh-guided surface point cloud sampling.

[0048] The purpose of step 2.1 is to collect point cloud on the surface of the hand three-dimensional mesh. In order to make the surface point cloud sampling result reflect both the structure size and the local geometric features, the application adopts a sampling method combining surface area weighting and feature curvature weighting. For this purpose, first, for each triangular patch i of the hand three-dimensional mesh, the area A i , the feature curvature T i , the comprehensive weight w i and the sampling probability p i of the triangular patch i are calculated in turn, and then n i points are randomly sampled in the triangular patch i according to the sampling probability p i .

[0049] The calculation method of the area A i of the triangular patch i is as follows:

[0050]

[0051] Where v1, v2 and v3 are the vector representations of the three edges constituting the triangular patch i.

[0052] The feature curvature T i of the triangular patch i is the weighted average of the local curvature S i and the Gaussian curvature G i of the triangular patch i.

[0053] The local curvature S i of the triangular patch i is defined as the average of the normal vector differences between the triangular patch i and its adjacent triangular patches, and the calculation method is as follows:

[0054]

[0055] Where N i is the normal vector of the triangular patch i, N ij is the normal vector of the triangular patch j adjacent to the triangular patch i, and k is the number of triangular patches adjacent to the triangular patch i. Here, two triangular patches are adjacent if they have a common adjacent edge.

[0056] The Gaussian curvature G i of the triangular patch i is defined as the average of the Gaussian curvatures of the vertices of the triangular patch i, that is,

[0057]

[0058] Where K ip is the Gaussian curvature of the vertex p of the triangular patch i, and the calculation method is defined as:

[0059]

[0060] where θ ipj is the inner angle size of the vertex p of the triangular facet i in the adjacent triangular facet j, A ij is the area of the triangular facet j adjacent to the triangular facet i, and k is the number of triangular facets adjacent to the triangular facet i.

[0061] After obtaining the local curvature S i and the Gaussian curvature G i of the triangular facet i, the characteristic curvature T i of the triangular facet i is calculated as follows:

[0062] T i = βS i + (1-β)G i ,

[0063] where β is a weighting coefficient, which can be adjusted according to actual needs. In the embodiment, β = 0.8.

[0064] The comprehensive weight w i of the triangular facet i is calculated by comprehensively calculating the area A i and the characteristic curvature T i , and the calculation method is as follows:

[0065] w i = A i × (1+αT i ),

[0066] where α is an adjustable parameter. In the embodiment, α = 0.8.

[0067] The sampling probability p i of the triangular facet i is defined as the ratio of the comprehensive weight w i of the triangular facet i to the comprehensive weight of all triangular facets, that is:

[0068]

[0069] where w i and w j are the comprehensive weights of the triangular facets, and H is the number of triangular facets of the hand three-dimensional mesh.

[0070] The number n i of sampling points in the triangular facet i is defined as:

[0071]

[0072] where N is the total number of surface point clouds set according to actual needs, and p i is the sampling probability of the triangular facet i. In the embodiment, N = 800.

[0073] Step 2.2: Grid-guided internal point cloud sampling.

[0074] The purpose of step 2.2 is to collect point cloud in the interior of the hand three-dimensional mesh model. The collection process is related to the hand multi-view images. Given a view angle and its hand image, first, a light ray is projected from the viewpoint corresponding to the view angle to the image pixel point, and then the intersection of the projected light ray and the hand three-dimensional mesh model is calculated. When the projected light ray has and only has two intersection points with the hand three-dimensional mesh model, and the distance between the two intersection points is greater than the pre-set threshold value δ, a sampling point is selected between the two intersection points of the projected light ray.

[0075] For the intersection points of the projected light ray and the hand three-dimensional mesh model, P near and P far , P near represents the point far from the viewpoint, and P far represents the point close to the viewpoint, when ||P far -P near ||>δ, the sampling point P on the projected light ray is calculated according to the following formula, that is,

[0076] P=P near +λ×(P far -P near ),

[0077] Where λ is a coefficient between 0 and 1, which is used to control the position of the sampling point. λ can be set according to actual needs. In this embodiment, δ=0.05, λ=0.2.

[0078] This embodiment uses a data set containing 554 gestures, each gesture containing 139 images of different view angles. After internal point cloud sampling according to the above method, the total number of sampling points will be too much. To this end, this embodiment only selects 1% of the sampling points as the final internal point cloud sampling result.

[0079] Step 2.3: Noise-regularized point cloud sampling.

[0080] The purpose of step 2.3 noise-regularized point cloud sampling is to introduce uncertainty and reduce the probability of falling into local optimum in the subsequent processing process. For this purpose, first, the bounding box of the hand three-dimensional mesh model is obtained, and then a certain number of sampling points are randomly selected in the bounding box according to the uniform probability distribution. This embodiment randomly selects 100 sampling points in this step.

[0081] The initial point cloud sampling results of some gestures obtained in step 2 of this embodiment are as follows: Figure 4The first column is the hand image of different gestures, the second column is the hand three-dimensional mesh model, the third column is the surface point cloud sampling result, the fourth column is the internal point cloud sampling result, the fifth column is the noise regularized point cloud sampling result, and the sixth column is the final hand initialization point cloud combined together.

[0082] Step 3: Establish a hand 3DGS model for the hand multi-view image by using the 3DGS method.

[0083] As described in the background, the 3DGS method is a technology and method for modeling a three-dimensional scene according to multi-view images, and the modeling result is a three-dimensional scene model represented by a series of Gaussian ellipsoids with color, shape and position information. For convenience of description, the modeling result of 3DGS is simply referred to as 3DGS model.

[0084] The process of establishing a hand 3DGS model for the hand multi-view image by using the 3DGS method according to the present application comprises steps 3.1 and 3.2.

[0085] Step 3.1: Take the hand initialization point cloud obtained in step 2 as the initial position of the Gaussian ellipsoid to create an initial hand 3DGS model.

[0086] The 3DGS model created in step 3.1 is composed of a series of Gaussian ellipsoids. The center points of these ellipsoids are respectively located at the positions of the points in the initialization point cloud obtained in step 2, and the colors and shapes of the ellipsoids are initialized according to the 3DGS method described in the background.

[0087] Step 3.2: Optimize the hand 3DGS model using known hand multi-view images.

[0088] The basic process of step 3.2 to optimize the 3DGS model is as follows: render the 3DGS model under a known view to obtain a corresponding rendered image, calculate the loss between the rendered image and the real image under the view, update the parameters such as color, shape and position of the Gaussian ellipsoid in the 3DGS model according to the loss, and obtain a new 3DGS model; repeat this process until the loss between the rendered image and the real image reaches the expected target.

[0089] In the above optimization process, the loss between the rendered image and the real image is calculated according to the following formulas (1)-(4), i.e.:

[0090]

[0091]

[0092] where λ1, λ2, λ3 are adjustable weight coefficients, M is the number of pixels of the image, y iis the pixel value of the real image, is the pixel value of the rendered image, μ yi and are the local mean of the corresponding pixels of the rendered image and the real image in a 11x11 window, respectively, σ yi and are the local variance of the corresponding pixels of the rendered image and the real image in a 11x11 window, respectively, is the corresponding covariance, C1 and C2 are small constants added to avoid the denominator being zero. In this embodiment, λ1 = 0.7, λ2 = 0.2, λ3 = 0.1, C1 = 1e-4, C2 = 9 * C1.

[0093] In the above optimization process, the rendering of the 3DGS model and the updating of the 3DGS model are consistent with the 3DGS method described in the background art, which will not be repeated here. However, unlike the 3DGS method, the present application adopts a new way to calculate the loss between the rendered image and the real image, i.e. the above formulas (1) to (4). In addition, the present application adopts the hand initialization point cloud to create an initial 3DGS model, which is also different from the original 3DGS method.

[0094] Step 4: render the hand image under any new view angle according to the optimized hand 3DGS model.

[0095] The rendering of the 3DGS model in step 4 is consistent with the method of rendering the 3DGS model involved in step 3.2, i.e. the highly parallelizable differentiable rendering in the 3DGS method described in the background art, where the new view angle is input information that needs to be specified in advance.

[0096] In this embodiment, the view angle of part of the hand images selected as test data in step 4 is selected as the new view angle, and the new view angle images of part of the gestures are obtained, as shown in Figure 5 where the odd rows are newly generated images, and the even rows are corresponding real images. The generated images are very close to the real images.

[0097] To further illustrate the beneficial effects of the present application, this embodiment uses the test data in the InterHand2.6M dataset to compare and test the present application method with several related methods in the field. The comparison test results and related circumstances are as follows.

[0098] Benchmark methods. The benchmark methods for comparison test with the present application include: 1) the original 3DGS method, which uses feature detection and feature matching to construct the initialization point cloud; 2) the SelfRecon method (Jiang B, Hong Y, Bao H, et al. Selfrecon: Self Reconstruction Your Digital Avatar from Monocular Video. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2022: 5605-5615.); 3) the HumanNeRF (Weng C Y, Curless B, Srinivasan PP, et al. HumanNerf: Free-viewpoint Rendering of Moving People from Monocular Video. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2022: 16210-16220.); 4) the HandAvatar (Chen X, Wang B, Shum HY. Handavatar: Free-pose Hand Animation and Rendering from Monocular Video. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2023: 8683-8693.).

[0099] Evaluation index. The evaluation index used in the comparative test is: 1) PSNR index (Sheikh H R, Sabir M F, Bovik A C. A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing, 2006, 15(11): 3440-3451.); 2) LPIPS index (R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 586-595.). PSNR is defined based on mean square error, which can measure the error between the real image and the generated image. The higher the PSNR value, the smaller the error, and the higher the quality of the generated image. LPIPS measures the similarity between images by calculating the distance between image features. The lower the LPIPS value, the better the quality of the generated image.

[0100] Experimental results. Figure 6 Part of the hand new view images generated by the method of the present application and the 3DGS method are shown, as well as the corresponding real images; the first column is the hand new view image generated by the 3DGS method, the second column is the hand new view image generated by the method of the present application, and the third column is the real image. From Figure 6 It can be seen that the hand new view image generated by the method of the present application has better effect, the skin texture details at the finger joints are clearer, and is closer to the real situation.

[0101] Table 1 is the comparative results of the evaluation index of the method of the present application and the 3DGS method, from which it can be seen that the method of the present application improves the PSNR value and reduces the LPIPS value.

[0102] Table 1 Comparative results of evaluation index of the method of the present application and the 3DGS method

[0103]

[0104] Figure 7For the qualitative comparison results of the method of the present application and SelfRecon, HumanNeRF and HandAvatar methods, the first row is the result of the SelfRecon method, the second row is the result of the HumanNeRF method, the third row is the result of the HandAvatar method, and the fourth row is the result of the method of the present application. From Figure 7 As can be seen from the comparison results shown in the table, it can be seen that the method of the present application is more suitable for the special morphology of high jointization of the hand, can effectively cope with the self-occlusion problem of the hand structure, and the generated hand morphology is more fine and real, while the hand morphology obtained by other methods is not real enough, and there are problems such as unclear morphology caused by inter-finger occlusion.

[0105] Table 2 Comparison results of evaluation indexes of the method of the present application and SelfRecon, HumanNeRF and HandAvatar methods

[0106]

[0107]

[0108] Table 2 is the quantitative comparison results of the method of the present application and SelfRecon, HumanNeRF and HandAvatar methods in the evaluation indexes, from which it can be seen that the method of the present application is superior to other methods in the PSNR index and the LPIPS index, further indicating the beneficial effects of the method of the present application.

[0109] The above steps and examples illustrate the entire process of the hand new view image generation method combining the MANO model and the 3DGS according to the present application.

[0110] It should be understood that the above embodiments are only specific examples of the implementation of the present application, and should not be regarded as a limitation on the protection scope of the present application. Equivalent modifications or changes to the above content without departing from the spirit and scope of the present application should be included in the scope of protection claimed by the present application.

Claims

1. A method for generating novel hand perspective images combining MANO models and 3DGS, characterized in that, Includes the following steps: Step 1: Create a 3D mesh for the hand based on the MANO model; Therefore, a convolutional neural network is first trained using multi-view images of the hand, and then the trained convolutional neural network is used to extract hand shape parameters from the multi-view images of the hand. and attitude parameters Hand shape parameters and attitude parameters Input the MANO model to calculate the 3D mesh of the hand; Step 2: Acquire initial point cloud data of the hand under the guidance of the 3D hand mesh; The acquisition process of the hand initialization point cloud includes three parts: surface point cloud sampling guided by the grid, internal point cloud sampling guided by the grid, and noise-normalized point cloud sampling; In order to ensure that the results of surface point cloud sampling can reflect both the size of the structure and the local geometric features, the mesh-guided surface point cloud sampling adopts a sampling method that combines surface area weighting and feature curvature weighting. The internal point cloud sampling of the mesh guide is related to the multi-view image of the hand. Given a viewpoint and its hand image, a ray is first projected from the viewpoint corresponding to that viewpoint, pointing to the image pixel. Then, the intersection of the projected ray and the 3D mesh model of the hand is calculated. Only when the projected ray and the 3D mesh model of the hand have exactly two intersections, and the distance between the two intersections is greater than a preset threshold, is a sampling point selected between the two intersections of the projected ray. Noise-regularized point cloud sampling involves randomly selecting a certain number of sampling points within the bounding box of the 3D mesh model of the hand, according to a uniform probability distribution. The final hand initialization point cloud is the union of the above three sampling results; Step 3: Use the 3DGS method to create a 3DGS model of the hand from the multi-view images; 3DGS is a technique and method for modeling three-dimensional scenes based on multi-view images. The modeling result is a 3DGS model represented by a series of Gaussian ellipsoids with color, shape and position information. To establish a 3DGS model of the hand, the initial hand 3DGS model is first created by using the hand initial point cloud obtained in step 2 as the initial position of the Gaussian ellipsoid according to the 3DGS method; then the hand 3DGS model is optimized using known multi-view images of the hand. The optimization process of the hand 3DGS model is as follows: render the 3DGS model under a known viewpoint to obtain the corresponding rendered image, calculate the loss between the rendered image and the real image under that viewpoint, update the parameters such as the color, shape and position of the Gaussian ellipsoid in the 3DGS model according to the loss, and obtain a new 3DGS model; repeat this process until the loss between the rendered image and the real image reaches the expected target. Step 4: Render and generate hand images from any new perspective based on the optimized 3DGS hand model.

2. The method for generating a new perspective image of a hand combining the MANO model and 3DGS according to claim 1, characterized in that, In step 2, the surface point cloud sampling guided by the mesh adopts a sampling method that combines surface area weighting and feature curvature weighting. The specific sampling process is as follows: First, for each triangular facet i of the hand 3D mesh, the area A of triangular facet i is calculated sequentially. i Characteristic curvature T i Comprehensive weight w i and sampling probability p i Then, based on the sampling probability p i Randomly sample n within triangular facet i i point; The area A of triangular facet i i The calculation method is as follows: Where v1, v2, and v3 are the vector representations of the three sides that make up the triangular facet i; The characteristic curvature T of triangular patch i i The local curvature S of triangular patch i i and Gaussian curvature G i The weighted average; Local curvature S of triangular patch i i Defined as the average difference between the normal vectors of triangular facet i and its neighboring triangular facets, the calculation method is as follows: Where N i N is the normal vector of triangle i. ij is the normal vector of triangle j adjacent to triangle i, and k is the number of triangles adjacent to triangle i; here, two triangles are adjacent if they share a common adjacent edge. Gaussian curvature G of triangular patch i i Defined as the average Gaussian curvature of each vertex of triangle i, i.e. Where K ip Let p be the Gaussian curvature of vertex p of triangle i, and the calculation method is defined as follows: Where θ ipj A is the size of the interior angle of vertex p of triangle i in the adjacent triangle j. ij is the area of ​​triangular facet j adjacent to triangular facet i, and k is the number of triangular facets adjacent to triangular facet i. The local curvature S of triangular facet i is obtained. i and Gaussian curvature G i Afterwards, the characteristic curvature T of triangular patch i i The calculation method is as follows: T i =βS i +(1-β)G i , Where β is the weighting coefficient, which can be adjusted according to actual needs; The overall weight w of triangle i i From area A i and characteristic curvature T i The comprehensive calculation shows that the calculation method is as follows: w i =A i ×(1+αT i ), Where θ is an adjustable parameter; The sampling probability p of triangle i i Defined as its comprehensive weight w i The proportion relative to the overall weight of all triangular faces, i.e.: Where w i and w j It is the overall weight of the triangular facets, and H is the number of triangular facets in the 3D mesh of the hand; The number of sampling points n within triangle i i Defined as: Where N is the total number of surface point cloud samples set according to actual needs, p i It is the sampling probability of triangle i.

3. The method for generating a new perspective image of a hand combining the MANO model and 3DGS according to claim 1, characterized in that, The method for selecting a sampling point between the two intersections of the projected light rays involved in step 2 is: Let P near P represents the intersection point closest to the viewpoint. far The sampling point P represents the intersection point farthest from the viewpoint. near +λ×(P far -P near ), where λ is a coefficient between 0 and 1, which can be set according to actual needs to control the position of the sampling point.

4. The method for generating a new perspective image of a hand combining the MANO model and 3DGS according to claim 1, characterized in that, Step 3: In the process of optimizing the 3DGS model of the hand, calculate the loss between the rendered image and the real image according to the following formulas (1) to (4). Right now: Where λ1, λ2, and λ3 are adjustable weight coefficients, M is the number of pixels in the image, and y i These are the pixel values ​​of the actual image. It is the pixel value of the rendered image, μ yi and σ represents the local mean of corresponding pixels in the rendered image and the real image within an 11×11 window, respectively. yi and These are the local variances of corresponding pixels in the rendered image and the real image within an 11×11 window, respectively. The corresponding covariances are C1 and C2, which are small constants added to avoid the denominator being zero. λ1 = 0.7, λ2 = 0.2, λ3 = 0.1, C1 = 1e-4, C2 = 9*C1.

Images