A method for saliency of fuzzy texture features of 3D models based on conditional generative adversarial

Abstract

The present invention discloses a method for saliency of fuzzy texture features of a three-dimensional model based on conditional generative adversarial network, comprising the following steps: 1) generating the shape and texture of a three-dimensional geometric model to obtain the point cloud features of the three-dimensional geometric model; 2) extracting the fuzzy boundary point cloud; 3) saliency of the fuzzy boundary features based on a conditional generative adversarial network. The present invention is based on multi-scale voxelized fuzzy boundary selection, generates textures through a conditional generative adversarial network for mapping, embeds the extracted voxel blocks into a three-dimensional geometric model, realizes the global texture fidelity of the geometric model through multi-scale voxel segmentation, changes the global texture optimization into local texture optimization, and reduces the amount of calculation.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional geometric model reconstruction, and in particular to a method for saliency of fuzzy texture features of three-dimensional models based on conditional generative adversarial methods. Background Art

[0002] The existing 3D geometric model reconstruction technology usually involves the step of model realism. The reconstructed 3D geometric model only has shape information. Therefore, after the 3D geometric shape reconstruction, it is usually necessary to supplement the texture information on the basis of the geometric shape model to make the entire 3D geometric model have a more realistic visual effect.

[0003] In 2019, Wang et al. from the University of the Chinese Academy of Sciences proposed a re-identification method with supervised texture generation (Wang J, Zhong Y, Li Y, et al. Re-Identification Supervised Texture Generation [C] / / 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.), which generates human body texture under the supervision of human re-identification. The synthetic image is rendered by the texture extracted from the input, and the similarity between the rendered image and the input image is maximized by using the re-identification network as a perceptual metric. It is possible to generate textures from a single image and have higher quality than textures generated by other available methods. The disadvantage is that the boundary information of the generated model is not processed, and there is blur at the boundary of the generated texture.

[0004] In 2021, Deng Youming and others from Shenzhen Huili Technology Co., Ltd. proposed a texture generation method, device, equipment and storage medium (application publication number: CN112950739A). By designing a device to obtain the first view texture of the target object and input it into the pre-trained texture generation network, the second view texture is obtained, which reduces the view acquisition requirements of the target object, reduces the cost of texture generation, and improves the accuracy and authenticity of texture generation. The disadvantage is that it uses multi-perspective views for texture generation, and the generation process of boundary texture loses some features, and the texture blurring problem cannot be solved.

[0005] In 2021, Peng Guihui and others from China Coal (Xi'an) Aerial Survey and Remote Sensing Research Institute Co., Ltd. proposed a 3D model texture generation method, device, equipment and storage medium (application publication number: CN113223149A), which selects optional occluded model faces from each model face of multiple models outside the first model, and determines the candidate image of the target model face. According to the type of the target model face, the best image is selected from each candidate image, and the texture of the target model face is generated based on the best image, thereby improving the authenticity of the 3D model texture generation. The disadvantage is that by generating the texture based on the best image, the generated texture only retains the original features.

[0006] In 2021, Li Jituo and others from Zhejiang University proposed a texture fusion method for real-time 3D reconstruction of RGB-D cameras (authorization announcement number: CN110827397B). Through the texture fusion of real-time 3D reconstruction of RGB-D cameras, the confidence weights in the adaptive weight field of the real-time frame are compared with the latest confidence weights of the reference point cloud, and the operation is selected from the three operations of replacement, fusion and retention to update the texture result, so as to realize texture fusion applied to 3D reconstruction. High-quality data is extracted, which can effectively reduce the blur of texture fusion and achieve clear texture reconstruction results. It is embedded in the RGB-D reconstruction framework at a lower computational cost, thereby significantly improving the accuracy of texture reconstruction. The disadvantage is that the texture is updated by replacing and fusing fixed operations, and the results are limited by the clarity of the input data, and the blurry boundary texture is not fully handled.

[0007] In summary, the existing 3D reconstruction methods have a certain fuzziness in the generated texture boundaries, which makes the realism of the 3D reconstruction have certain defects. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the prior art and provide a method for salient fuzzy texture features of a three-dimensional model based on conditional generative adversarial technology, which can effectively improve the blurriness of texture boundaries after three-dimensional reconstruction.

[0009] The object of the present invention is achieved by: a method for saliency of fuzzy texture features of a three-dimensional model based on conditional generative adversarial, comprising the following steps:

[0010] Step 1) generating the shape and texture of the three-dimensional geometric model and obtaining the point cloud features of the three-dimensional geometric model;

[0011] Step 2) extracting fuzzy boundary point cloud;

[0012] Step 3) Feature saliency of fuzzy boundaries based on conditional generative adversarial networks.

[0013] As a further limitation of the present invention, the step 1) specifically includes: using HMR to use an iterative three-dimensional regression module to generate shape, posture and translation parameters for SMPL, and reconstructing the shape of the human body from the input image through SMPL; constructing a Unet network to generate texture mapping to the reconstructed model mesh, and obtaining point cloud features of the three-dimensional geometric model.

[0014] As a further limitation of the present invention, the step 2) specifically comprises:

[0015] Step 2.1) Region growing segmentation based on pixel values; according to the basic features of the point cloud obtained in step 1), the point cloud is sorted according to the size of the RGB pixel value; the point cloud with the lowest RGB pixel value is selected as the initial seed point cloud, and the surrounding adjacent point clouds are compared with the seed point cloud to determine whether the color distance between the adjacent point cloud and the seed point cloud is small enough; the point clouds with the same or similar properties as the seed point cloud in the neighborhood around the seed point cloud are merged into the area where the seed pixel is located, and the new point cloud continues to grow around as a seed until it includes all the point clouds that meet the conditions, and the growth area is obtained; the growth area is further divided, and the point cloud at the fuzzy boundary of the texture is defined as the fuzzy boundary area;

[0016] The color distance is:

[0017]

[0018] Where D(X1,X2) is the color distance between X1 and X2, where R1 G1 B1 and R2 G2 B2 represent the pixel values ​​of point X1 and point X2 in the RGB color domain respectively;

[0019] Step 2.2) Select the fuzzy boundary based on multi-scale voxelization; Construct a cube containing all point clouds of the three-dimensional geometric model established in step 1), divide the entire point cloud space into voxels one by one based on multi-scale voxelization representation, and judge whether the cube represented by the voxel contains the fuzzy boundary area obtained in step 2.1), define the divided cube that can completely contain the fuzzy boundary area obtained in step 2.1) as a valid node, otherwise it is an invalid node, and further divide the valid nodes until there are no valid nodes; construct a data set T with the minimum valid node that can contain a fuzzy boundary area, where T∈(A1,A2,…,A n ).

[0020] As a further limitation of the present invention, the step 3) specifically comprises:

[0021] Step 3.1) Generate texture based on conditional generative adversarial network;

[0022] The texture mapping of the existing voxel block is input as a set of discriminant data into the discriminator model to discriminate the texture output of the generator, and the distinct texture boundary label is added to the discriminator, and batch norm processing is used; the initial noise data is converted into a two-dimensional vector as the input of the generative model, and filled and activated through the transposed convolution layer, and the BN layer and ReLu layer are added to the output of each transposed convolution layer, the pixels of the output image are normalized, and the texture is generated to perform texture mapping on the voxel block, and the output is the generated voxel block, which is input into the discriminant model, and the calculation is repeated so that the discriminant model reaches Nash equilibrium, where the loss function of the entire model is:

[0023]

[0024] Where G represents the generator, D represents the discriminator, z represents the noise variable input to the generator, y represents the distinct texture boundary label, D(x) represents the output of the discriminator, and G(z) represents the output when the noise variable is input to the generator. When G is fixed, D is solved. Fix D and solve for G, Denotes the optimal discriminator obtained, G * Denotes the optimal generator obtained; Finally, a set of voxel blocks T2 with realistic texture boundaries is obtained, where T2∈(A'1, A'2, ..., A' n );

[0025] Step 3.2) embeds the extracted voxel blocks into a three-dimensional geometric model; traverse the three-dimensional geometric model obtained in step 1) and embed the points A1, A2, ..., A in the voxel block data set T segmented in step 2.2) n Replaced with points A'1, A'2, ..., A' in the voxel block set T2 obtained in step 3.1) n , and finally realize a realistic three-dimensional geometric model.

[0026] The present invention adopts the above technical scheme, and has the following beneficial effects compared with the prior art: first, HMR is used to use an iterative three-dimensional regression module to generate shape, posture and translation parameters for SMPL, and the shape of the human body is reconstructed for the input image through SMPL; secondly, a Unet network is constructed to generate texture mapping to the reconstructed model grid; then, the spatial topological information of the point cloud is obtained, and the local boundaries are selected using multi-scale voxels based on the RGB channels; finally, the texture is generated by a conditional generative adversarial network for mapping, and the extracted voxel blocks are embedded in the three-dimensional geometric model. High-resolution texture generation is achieved by combining local discrimination with overall discrimination, which can reduce the calculation time of texture generation, improve the realism of texture boundaries, and achieve unsupervised fuzzy boundary feature saliency. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flowchart of the present invention. DETAILED DESCRIPTION

[0028] like Figure 1 A method for saliency of fuzzy texture features of a three-dimensional model based on conditional generative adversarial is shown, comprising the following steps:

[0029] Step 1) generating the shape and texture of the three-dimensional geometric model and obtaining the point cloud features of the three-dimensional geometric model;

[0030] First, the image is input to HMR (Human Mesh Recovery) and the SMPL (Skinned Multi-Person Linear Model) parameters are predicted. HMR uses an iterative 3D regression module to generate shape, pose and translation parameters for SMPL. The 3D mesh m estimated by the input image is represented as: m = M(β,θ,γ). Secondly, U-Net is used to generate textures, and Opendr is used to render tensors. The generated textures are mapped to the 3D mesh. The UV correspondence provided by SMPL and the rendering function R(m,t) of Opendr are used to assign pixels to the surface of the 3D geometric model. At the same time, linear interpolation is used to fill the gaps, and finally the point cloud features of the 3D geometric model are obtained.

[0031] Step 2) extracting fuzzy boundary point cloud;

[0032] Step 2.1) Region growing segmentation based on pixel values; according to the basic features of the point cloud obtained in step 1), the point cloud is sorted according to the size of the RGB pixel value; the point cloud with the lowest RGB pixel value is selected as the initial seed point cloud, and the surrounding adjacent point clouds are compared with the seed point cloud to determine whether the color distance between the adjacent point cloud and the seed point cloud is small enough; the point clouds with the same or similar properties as the seed point cloud in the neighborhood around the seed point cloud are merged into the area where the seed pixel is located, and the new point cloud continues to grow around as a seed until it includes all the point clouds that meet the conditions, and the growth area is obtained; the growth area is further divided, and the point cloud at the fuzzy boundary of the texture is defined as the fuzzy boundary area;

[0033] The color distance is:

[0034]

[0035] Where D(X1,X2) is the color distance between X1 and X2, where R1 G1 B1 and R2 G2 B2 represent the pixel values ​​of point X1 and point X2 in the RGB color domain respectively;

[0036] Step 2.2) Select the fuzzy boundary based on multi-scale voxelization; Construct a cube containing all point clouds of the three-dimensional geometric model established in step 1), divide the entire point cloud space into voxels one by one based on multi-scale voxelization representation, and judge whether the cube represented by the voxel contains the fuzzy boundary area obtained in step 2.1), define the divided cube that can completely contain the fuzzy boundary area obtained in step 2.1) as a valid node, otherwise it is an invalid node, and further divide the valid nodes until there are no valid nodes; construct a data set T with the minimum valid node that can contain a fuzzy boundary area, where T∈(A1,A2,…,A n ).

[0037] Step 3) Feature saliency of fuzzy boundaries based on conditional generative adversarial network;

[0038] Step 3.1) Generate texture based on conditional generative adversarial network;

[0039] The texture mapping of the existing voxel block is input as a set of discriminant data into the discriminator model to discriminate the texture output of the generator, and the distinct texture boundary label is added to the discriminator, and batch norm processing is used; the initial noise data is converted into a two-dimensional vector as the input of the generative model, and filled and activated through the transposed convolution layer, and the BN layer and ReLu layer are added to the output of each transposed convolution layer, the pixels of the output image are normalized, and the texture is generated to perform texture mapping on the voxel block, and the output is the generated voxel block, and input into the discriminant model, keep the generative model unchanged, train the discriminant model to reach the maximum, keep the discriminant model unchanged, train the generative model to reach the minimum, and repeat the calculation to make the discriminant model reach Nash equilibrium, where the loss function of the entire model is:

[0040]

[0041] Where G represents the generator, D represents the discriminator, z represents the noise variable input to the generator, y represents the distinct texture boundary label, D(x) represents the output of the discriminator, and G(z) represents the output when the noise variable is input to the generator. When G is fixed, D is solved. Fix D and solve for G, Denotes the optimal discriminator obtained, G * Denotes the optimal generator obtained; Finally, a set of voxel blocks T2 with realistic texture boundaries is obtained, where T2∈(A'1, A'2, ..., A' n );

[0042] Step 3.2) embeds the extracted voxel blocks into a three-dimensional geometric model; traverse the three-dimensional geometric model obtained in step 1) and embed the points A1, A2, ..., A in the voxel block data set T segmented in step 2.2) nReplaced with points A'1, A'2, ..., A' in the voxel block set T2 obtained in step 3.1) n , and finally realize a realistic three-dimensional geometric model.

[0043] The present invention adopts HMR to use an iterative three-dimensional regression module to generate shape, posture and translation parameters for SMPL, and reconstructs the shape of the human body from an input image through SMPL; constructs a Unet network to generate texture mapping to the reconstructed model grid; then, obtains the spatial topological information of the point cloud, uses multi-scale voxels based on the RGB channel to select the local boundary, and finally, generates texture mapping through a conditional generative adversarial network, and uses a local generation discriminant method to achieve high-resolution texture generation, which can reduce the calculation time of texture generation, improve the realism of texture boundaries, and achieve unsupervised high-quality texture generation.

[0044] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A method for saliency of fuzzy texture features of three-dimensional models based on conditional generative adversarial, characterized in that: The following steps are involved: Step 1) generating the shape and texture of the three-dimensional geometric model and obtaining the point cloud features of the three-dimensional geometric model; Step 2) extracting fuzzy boundary point cloud; The step 2) specifically includes: Step 2.1) Region growing segmentation based on pixel values; according to the basic features of the point cloud obtained in step 1), the point cloud is sorted according to the size of the RGB pixel value; the point cloud with the lowest RGB pixel value is selected as the initial seed point cloud, and the surrounding adjacent point clouds are compared with the initial seed point cloud to determine whether the color distance between the adjacent point clouds and the initial seed point cloud is small enough; the point clouds with the same or similar properties as the seed point cloud in the neighborhood around the seed point cloud are merged into the area where the seed pixel is located, and the new point cloud continues to grow around as a seed until it includes all the point clouds that meet the conditions, and the growth area is obtained; the growth area is further divided, and the point cloud at the fuzzy boundary of the texture is defined as the fuzzy boundary area; The color distance is: Where D(X1,X2) is the color distance between X1 and X2, where R1 G1 B1 and R2 G2 B2 represent the pixel values ​​of point X1 and point X2 in the RGB color domain respectively; Step 2.2) Select the fuzzy boundary based on multi-scale voxelization; Construct a cube containing all point clouds of the three-dimensional geometric model established in step 1), divide the entire point cloud space into voxels one by one based on multi-scale voxelization representation, and judge whether the cube represented by the voxel contains the fuzzy boundary area obtained in step 2.1), define the divided cube that can completely contain the fuzzy boundary area obtained in step 2.1) as a valid node, otherwise it is an invalid node, and further divide the valid nodes until there are no valid nodes; construct a data set T with the minimum valid node that can contain a fuzzy boundary area, where T∈(A1,A2,…,A n ); A1, A2, …, A n is a point in the voxel block dataset T; Step 3) Feature saliency of fuzzy boundaries based on conditional generative adversarial network; The step 3) specifically includes: Step 3.1) Texture data generation based on conditional generative adversarial network; The texture mapping of the existing voxel block is input as a set of discriminant data into the discriminator model to discriminate the texture output of the generator, and the distinct texture boundary label is added to the discriminator, and batch norm processing is used; the initial noise data is converted into a two-dimensional vector as the input of the generative model, and filled and activated through the transposed convolution layer, and the BN layer and ReLu layer are added to the output of each transposed convolution layer. The pixels of the output image are normalized to generate textures, and the voxel blocks are texture mapped to obtain the output as the generated voxel block, which is input into the discriminant model, and the calculation is repeated so that the discriminant model reaches Nash equilibrium, where the loss function of the entire model is: Where G represents the generator, D represents the discriminator, z represents the noise variable input to the generator, y represents the distinct texture boundary label, D(x) represents the output of the discriminator, and G(z) = G(x, y) represents the output when the noise variable is input to the generator; solve D with G fixed, Fix D and solve for G, Denotes the optimal discriminator obtained, G * Denotes the optimal generator obtained; Finally, a set of voxel blocks T2 with realistic texture boundaries is obtained, where T2∈(A'1, A'2, ..., A' n );E x～Pd(x) [.] represents the variance of the true data distribution of the discriminator’s texture output, E z～Pz(z) [.] represents the variance of the output when the generator inputs the noise variable; x~Pd(x) represents the real data distribution of the texture output of the discriminator, and z~Pz(z) represents the distribution of the output when the generator inputs the noise variable; Step 3.2) embeds the extracted voxel blocks into a three-dimensional geometric model; traverse the three-dimensional geometric model obtained in step 1) and embed the points A1, A2, ..., A in the voxel block data set T segmented in step 2.2) n Replaced with points A'1, A'2, ..., A' in the voxel block set T2 obtained in step 3.1) n , and finally realize a realistic three-dimensional geometric model.

2. The method for saliency of fuzzy texture features of three-dimensional models based on conditional generative adversarial methods according to claim 1, characterized in that: The step 1) specifically includes: using the HMR iterative 3D regression module to generate shape, posture and translation parameters for SMPL, and reconstructing the shape of the human body from the input image through SMPL; constructing a Unet network to generate texture mapping to the reconstructed model mesh, and obtaining point cloud features of the 3D geometric model.

Images