Method and system for reconstructing high-precision complete shape and texture of object in scene

Abstract

The invention provides a method and a system for reconstructing a high-precision complete shape and texture of an object in a scene. The method comprises the following steps: acquiring a visual three-dimensional grid G, optimizing the visual three-dimensional grid G, performing visual modeling of a three-dimensional scene space, and acquiring a three-dimensional grid optimization visual model; through decomposition type nerve implicit reconstruction of the generation diffusion model prior, the potential generation diffusion model prior is used for filling object missing information of an object invisible area or an occluded area, and combined scene reconstruction is carried out; optimizing a visibility model through a three-dimensional grid, dynamically adjusting the prior loss weight of a potential generation diffusion model, forming a generation prior consistent with the input in a visible region, and performing object shape enhancement; and after the object shape enhancement is completed, three-dimensional grid representation is exported from the implicit field, object texture enhancement is carried out, and a high-precision complete object 3D model with complete texture of each object is precisely reconstructed.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent data processing for three-dimensional modeling of scene image objects, and more specifically, to a system and method for high-precision complete shape and texture reconstruction of objects in a scene. Background Art

[0002] 3D scene reconstruction based on multi-view images is an important research topic in the field of computer vision. Its goal is to restore the complete scene geometry and appearance from multi-view images. In recent years, the development of neural implicit representation has greatly promoted the progress of new view rendering and 3D geometry reconstruction; however, existing methods usually model the entire scene as a whole, which limits its application in fine-grained scene decomposition tasks;

[0003] Decomposed 3D reconstruction aims to decompose the implicit 3D representation of a scene into independent objects, thereby supporting a wider range of downstream applications, such as intelligent AI and robotics tasks; however, current decomposed reconstruction methods often have difficulty in simultaneously reconstructing complete geometric shapes and accurate appearances when dealing with sparse viewpoint inputs or severely occluded areas; although some methods attempt to alleviate the problem of sparse inputs through semantic or geometric regularization, their performance in non-observable areas is still limited due to the lack of effective supplementary information to deal with these under-constrained areas and other issues that remain to be resolved; therefore, it is necessary to propose a system and method for high-precision complete shape and texture reconstruction of objects in a scene to at least partially address the problems existing in the prior art. Summary of the invention

[0004] A series of simplified concepts are introduced in the content of the invention, which will be further described in detail in the specific implementation method section; the content of the invention of the present invention does not mean to attempt to limit the key features and essential technical features of the technical solution claimed for protection, nor does it mean to attempt to determine the scope of protection of the technical solution claimed for protection.

[0005] To at least partially solve the above problems, the present invention provides a method for reconstructing the complete shape and texture of an object in a scene with high precision, comprising:

[0006] S10, obtaining a visibility three-dimensional grid G, optimizing the visibility three-dimensional grid G, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model;

[0007] S20, by generating a decomposed neural implicit reconstruction of the diffusion model prior, using the potential generative diffusion model prior to fill in the missing information of the object in the invisible area or the occluded area, and perform combined scene reconstruction;

[0008] S30, optimizes the visibility model through the three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a priori that is consistent with the input in the visible area, and performs object shape enhancement;

[0009] S40, after completing the object shape enhancement, derives a three-dimensional mesh representation from the implicit field, performs object texture enhancement, and precisely reconstructs a high-precision complete object 3D model with complete texture for each object.

[0010] Preferably, S10 includes:

[0011] S101, obtaining a visible three-dimensional grid G, and initializing the visible three-dimensional grid G ​​to zero;

[0012] S102, optimizing the visibility three-dimensional grid G ​​by using the cumulative transmittance T calculated when rendering the input viewing angle, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model.

[0013] Preferably, S20 includes:

[0014] S201, creating an implicit field for each object in the three-dimensional scene space to express it;

[0015] S202, during the optimization process, the implicit field of each object is used to generate an RGB image, a depth image, a normal vector image, and an object mask image under a specified input viewing angle through volume rendering technology, and the loss is calculated by comparing the implicit field with the corresponding true value of the input image, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

[0016] Preferably, S30 includes:

[0017] S301, based on the initial combined scene reconstruction, randomly sample N viewpoints for each object, generate corresponding N normal vector maps and visibility maps using the implicit field and visibility 3D grid G ​​of the object using volume rendering technology, and obtain a rendered normal vector map;

[0018] S302, inputting the rendered normal vector image and the text description of the object into a potential generative diffusion model to obtain a model predicted image loss;

[0019] S303, optimizing the visibility model according to the three-dimensional grid, adjusting the model prediction image loss, and reconstructing the complete 3D geometric shape structure of the object.

[0020] Preferably, S40 includes:

[0021] S401, after completing the object shape enhancement, using a small number of scene images and mask annotations of the objects therein, deriving a 3D mesh representation from the implicit field of each object, and creating a small neural network M to predict the color of each mesh vertex;

[0022] S402, randomly sampling N viewpoints for each object, and generating corresponding N RGB images and N visibility images by using a small neural network M and a visibility 3D grid G ​​using differentiable rendering;

[0023] S403, input the generated N RGB images together with the text description of the object into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and use the three-dimensional grid optimization visibility model to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and obtain the optimized small neural network M; after the appearance optimization stage, use the optimized small neural network M to export the texture image of each object, combine it with the original three-dimensional grid representation, and obtain a high-precision complete object 3D model with complete texture, and accurately reconstruct a high-precision complete object 3D model with complete texture for each object.

[0024] The present invention provides a high-precision complete shape and texture reconstruction system for objects in a scene, comprising:

[0025] A visibility model construction optimization subsystem is constructed to obtain a visibility three-dimensional grid G, optimize the visibility three-dimensional grid G, perform visibility modeling of the three-dimensional scene space, and obtain a three-dimensional grid optimized visibility model;

[0026] The prior decomposition and combination scene reconstruction subsystem performs combined scene reconstruction by generating a decomposition-based neural implicit reconstruction of the diffusion model prior and using the potential generative diffusion model prior to fill in the missing information of objects in invisible or occluded areas;

[0027] The dynamic prior object shape enhancement subsystem optimizes the visibility model through a three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a prior that is consistent with the input in the visible area, and performs object shape enhancement;

[0028] The texture-enhanced 3D reconstruction subsystem, after completing the object shape enhancement, derives the 3D mesh representation from the implicit field, performs object texture enhancement, and accurately reconstructs a high-precision complete object 3D model with complete texture for each object.

[0029] Preferably, the visibility model construction optimization subsystem includes:

[0030] A visibility three-dimensional grid module obtains a visibility three-dimensional grid G ​​and initializes the visibility three-dimensional grid G ​​to zero;

[0031] The cumulative transmittance visibility module optimizes the visibility three-dimensional grid G ​​by calculating the cumulative transmittance T when rendering the input view, performs visibility modeling of the three-dimensional scene space, and obtains a three-dimensional grid optimized visibility model.

[0032] Preferably, the prior decomposition and combination scene reconstruction subsystem includes:

[0033] The object independent implicit field expression module creates an independent implicit field for each object in the three-dimensional scene space for expression;

[0034] The volume rendering comparison scene reconstruction module, during the optimization process, uses volume rendering technology to utilize the implicit field of each object to generate the RGB image, depth image, normal vector image, and object mask image under the specified input perspective, and compares them with the corresponding true value of the input image to calculate the loss, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

[0035] Preferably, the dynamic prior object shape enhancement subsystem includes:

[0036] The post-rendering normal vector generation module randomly samples N viewpoints for each object based on the initial combined scene reconstruction, and generates N corresponding normal vector maps and visibility maps using the volume rendering technology through the implicit field and visibility 3D grid G ​​of the object to obtain the post-rendering normal vector map;

[0037] The image loss model prediction module inputs the rendered normal vector map and the text description of the object into the potential generative diffusion model to obtain the model predicted image loss;

[0038] The loss-adjusted shape reconstruction module optimizes the visibility model according to the three-dimensional grid, adjusts the model prediction image loss, and reconstructs the complete 3D geometric shape structure of the object.

[0039] Preferably, the texture enhanced 3D reconstruction subsystem comprises:

[0040] The mesh vertex color prediction module, after completing the object shape enhancement, uses a small number of scene images and mask annotations of the objects in them to derive a 3D mesh representation from the implicit field of each object and create a small neural network M to predict the color of each mesh vertex;

[0041] The differentiable rendering generation module randomly samples N viewpoints for each object, and uses a small neural network M and a visibility 3D grid G ​​to generate the corresponding N RGB images and N visibility images using differentiable rendering;

[0042] The color texture prediction 3D generation module inputs the generated N RGB images together with the text description of the object into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and uses the three-dimensional grid optimization visibility model to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and obtain the optimized small neural network M; after the appearance optimization stage, the optimized small neural network M is used to export the texture image of each object, combined with the original three-dimensional grid representation, to obtain a high-precision complete object 3D model with complete texture, and accurately reconstruct a high-precision complete object 3D model with complete texture for each object.

[0043] The beneficial effects of the above technical solution include:

[0044] The present invention provides a method and system for reconstructing the complete shape and texture of an object in a scene with high precision. The method obtains a visible three-dimensional grid G, optimizes the visible three-dimensional grid G, performs visibility modeling of a three-dimensional scene space, and obtains a three-dimensional grid optimized visibility model. The method generates a diffusion model prior through decomposition-type neural implicit reconstruction, and uses a potential generative diffusion model prior to fill in the missing information of the object in the invisible area or the occluded area, so as to perform combined scene reconstruction. The method optimizes the visibility model through the three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a generative prior consistent with the input in the visible area, and performs object shape enhancement. After completing the object shape enhancement, the three-dimensional grid is derived from the implicit field. It indicates that object texture enhancement is performed to accurately reconstruct a high-precision complete object 3D model with complete texture for each object; the present invention is based on the decomposition-based neural implicit reconstruction of the generative diffusion model prior, and uses the potential generative diffusion model prior to fill in the missing information of the invisible or occluded area; designs visibility modeling for the implicit neural radiation field; through visibility modeling, dynamically adjusts the loss weight of the potential generative diffusion model prior to solve the problem of inconsistency between the generative prior and the input in the visible area; under the condition of including a small number of scene images and mask annotations of the objects therein, accurately reconstructs a high-precision complete object 3D model with complete texture for each object, significantly improving the processing accuracy; it has important technical significance and significant effects.

[0045] The present invention provides a method and system for high-precision complete shape and texture reconstruction of objects in a scene. Other advantages, objectives and features of the present invention will be reflected in part through the following description, and in part will be understood by technicians in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0047] Figure 1This is a diagram of an embodiment of a method and system for reconstructing the complete shape and texture of an object in a scene with high precision according to the present invention.

[0048] Figure 2 This is a diagram of an embodiment of a method for reconstructing the complete shape and texture of an object in a scene with high precision according to the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments so that those skilled in the art can implement the invention with reference to the description. As shown in the drawings, the present invention provides a method for reconstructing the complete shape and texture of an object in a scene with high precision, including:

[0050] S10, obtaining a visibility three-dimensional grid G, optimizing the visibility three-dimensional grid G, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model;

[0051] S20, by generating a decomposed neural implicit reconstruction of the diffusion model prior, using the potential generative diffusion model prior to fill in the missing information of the object in the invisible area or the occluded area, and perform combined scene reconstruction;

[0052] S30, optimizes the visibility model through the three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a priori that is consistent with the input in the visible area, and performs object shape enhancement;

[0053] S40, after completing the object shape enhancement, derives a three-dimensional mesh representation from the implicit field, performs object texture enhancement, and precisely reconstructs a high-precision complete object 3D model with complete texture for each object.

[0054] The principle and effect of the above technical solution are as follows: the present invention provides a method for reconstructing the complete shape and texture of an object in a scene with high precision, including: obtaining a visible three-dimensional grid G, optimizing the visible three-dimensional grid G, performing visibility modeling of a three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model; performing a combined scene reconstruction by using a decomposed neural implicit reconstruction of a priori generative diffusion model, and using a priori potential generative diffusion model to fill in the missing information of an object in an invisible area or an obscured area of ​​the object; optimizing the visibility model through a three-dimensional grid, dynamically adjusting the loss weight of a priori potential generative diffusion model, forming a priori consistent with the input in a visible area, and performing object shape enhancement; after completing the object shape enhancement, deriving a three-dimensional grid representation from an implicit field, performing object texture enhancement, and accurately reconstructing a high-precision complete object 3D model with complete texture for each object; calculating a loss function, the formula is as follows:

[0055] In the decomposed scene reconstruction stage, the total loss function is:

[0056] L recon=L C +L N +L D +L S

[0057] In the object shape enhancement stage, the total loss function is:

[0058]

[0059] In the object texture enhancement stage, the total loss function is:

[0060]

[0061] The present invention is based on decomposition-based neural implicit reconstruction of generative diffusion model priors, and uses potential generative diffusion model priors to fill in missing information in invisible or occluded areas; designs visibility modeling for implicit neural radiation fields; through visibility modeling, dynamically adjusts the loss weights of potential generative diffusion model priors to solve the problem of inconsistency between generative priors and inputs in visible areas; under the condition of including a small number of scene images and mask annotations of objects therein, accurately reconstructs a high-precision complete object 3D model with complete texture for each object, significantly improving processing accuracy; and has important technical significance and significant effects.

[0062] In one embodiment, S10 includes:

[0063] S101, obtaining a visible three-dimensional grid G, and initializing the visible three-dimensional grid G ​​to zero;

[0064] S102, optimizing the visibility three-dimensional grid G ​​by using the cumulative transmittance T calculated when rendering the input viewing angle, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model.

[0065] The principle and effect of the above technical solution are: obtaining a visible three-dimensional grid G, initializing the visible three-dimensional grid G ​​to zero; optimizing the visible three-dimensional grid G ​​by calculating the cumulative transmittance T when rendering the input perspective, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model; obtaining the visible three-dimensional grid G ​​includes: using multiple input perspective observation points to have higher visibility, and in the volume rendering process of the neural radiation field, setting the cumulative transmittance T of the three-dimensional point p to represent the probability that the light reaches the three-dimensional point p without intersecting with other particles; a higher cumulative transmittance T indicates that the three-dimensional point p has a greater probability of visibility in the input perspective; obtaining the visible three-dimensional grid G.

[0066] In one embodiment, S20 includes:

[0067] S201, creating an implicit field for each object in the three-dimensional scene space to express it;

[0068] S202, during the optimization process, the implicit field of each object is used to generate an RGB image, a depth image, a normal vector image, and an object mask image under a specified input viewing angle through volume rendering technology, and the loss is calculated by comparing the implicit field with the corresponding true value of the input image, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

[0069] The principle and effect of the above technical solution are: an implicit field is created separately for each object in the three-dimensional scene space to express it; during the optimization process, the implicit field of each object is used to generate the RGB image, depth image, normal vector image and object mask image under the specified input perspective through volume rendering technology, and the loss is calculated by comparing with the corresponding true value of the input image, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

[0070] In one embodiment, S30 includes:

[0071] S301, based on the initial combined scene reconstruction, randomly sample N viewpoints for each object, generate corresponding N normal vector maps and visibility maps using the implicit field and visibility 3D grid G ​​of the object using volume rendering technology, and obtain a rendered normal vector map;

[0072] S302, inputting the rendered normal vector image and the text description of the object into a potential generative diffusion model to obtain a model predicted image loss;

[0073] S303, optimizing the visibility model according to the three-dimensional grid, adjusting the model prediction image loss, and reconstructing the complete 3D geometric shape structure of the object.

[0074] The principle and effect of the above technical solution are as follows: according to the initial combined scene reconstruction, N viewpoints are randomly sampled for each object, and the corresponding N normal vector maps and visibility maps are generated by volume rendering technology through the implicit field and visibility three-dimensional grid G ​​of the object to obtain the rendered normal vector map; the rendered normal vector map and the text description of the object are input into the potential generative diffusion model to obtain the model prediction image loss; the visibility model is optimized according to the three-dimensional grid, the model prediction image loss is adjusted, and the complete 3D geometric shape structure of the object is reconstructed; the visibility model is optimized according to the three-dimensional grid, the model prediction image loss is adjusted, and the complete 3D geometric shape structure of the object is reconstructed, including: optimizing the visibility model according to the three-dimensional grid, reducing the loss weight of the area with greater visibility; and increasing the loss weight of the area with less visibility; finally, the adjusted image loss is used to optimize the implicit field of the object and reconstruct the complete 3D geometric shape structure of the object.

[0075] In one embodiment, S40 includes:

[0076] S401, after completing the object shape enhancement, using a small number of scene images and mask annotations of the objects therein, deriving a 3D mesh representation from the implicit field of each object, and creating a small neural network M to predict the color of each mesh vertex;

[0077] S402, randomly sampling N viewpoints for each object, and generating corresponding N RGB images and N visibility images by using a small neural network M and a visibility 3D grid G ​​using differentiable rendering;

[0078] S403, input the generated N RGB images together with the text description of the object into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and use the three-dimensional grid optimization visibility model to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and obtain the optimized small neural network M; after the appearance optimization stage, use the optimized small neural network M to export the texture image of each object, combine it with the original three-dimensional grid representation, and obtain a high-precision complete object 3D model with complete texture, and accurately reconstruct a high-precision complete object 3D model with complete texture for each object.

[0079] The principle and effect of the above technical solution are as follows: after completing the object shape enhancement, a three-dimensional mesh representation is derived from the implicit field of each object using a mask annotation including a small number of scene images and the objects therein, and a small neural network M is created to predict the color of each mesh vertex; N viewpoints are randomly sampled for each object, and the corresponding N RGB images and N visibility images are generated using the small neural network M and the visibility three-dimensional mesh G using differentiable rendering; the generated N RGB images are input into the latent generative diffusion model together with the text description of the object to obtain the image color texture loss predicted by the model; and the three-dimensional mesh optimized visibility model is used to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and the optimized small neural network M is obtained; after the appearance optimization stage is completed, the texture image of each object is derived using the optimized small neural network M, and combined with the original three-dimensional mesh representation, a high-precision complete object 3D model with complete texture is obtained, and a high-precision complete object 3D model with complete texture of each object is accurately reconstructed; the processing accuracy is significantly improved; it has important technical significance and significant effects.

[0080] The present invention provides a high-precision complete shape and texture reconstruction system for objects in a scene, comprising:

[0081] A visibility model construction optimization subsystem is constructed to obtain a visibility three-dimensional grid G, optimize the visibility three-dimensional grid G, perform visibility modeling of the three-dimensional scene space, and obtain a three-dimensional grid optimized visibility model;

[0082] The prior decomposition and combination scene reconstruction subsystem performs combined scene reconstruction by generating a decomposition-based neural implicit reconstruction of the diffusion model prior and using the potential generative diffusion model prior to fill in the missing information of objects in invisible or occluded areas;

[0083] The dynamic prior object shape enhancement subsystem optimizes the visibility model through a three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a prior that is consistent with the input in the visible area, and performs object shape enhancement;

[0084] The texture-enhanced 3D reconstruction subsystem, after completing the object shape enhancement, derives the 3D mesh representation from the implicit field, performs object texture enhancement, and accurately reconstructs a high-precision complete object 3D model with complete texture for each object.

[0085] The principle and effect of the above technical solution are as follows: the present invention provides a high-precision complete shape and texture reconstruction system for objects in a scene, including: a visibility model construction optimization subsystem, which obtains a visibility three-dimensional grid G, optimizes the visibility three-dimensional grid G, performs visibility modeling of a three-dimensional scene space, and obtains a three-dimensional grid optimized visibility model; a priori decomposition and combination scene reconstruction subsystem, which generates a priori neural implicit reconstruction of the diffusion model, and uses the potential generation diffusion model prior to fill in the missing information of the object in the invisible area or the occluded area of ​​the object, and performs combined scene reconstruction; a dynamic priori object shape enhancement subsystem, which optimizes the visibility model through the three-dimensional grid, dynamically adjusts the loss weight of the potential generation diffusion model prior, forms a generation prior consistent with the input in the visible area, and performs object shape enhancement; a texture enhancement 3D reconstruction subsystem, after completing the object shape enhancement, derives a three-dimensional grid representation from the implicit field, performs object texture enhancement, and accurately reconstructs a high-precision complete object 3D model with complete texture for each object; calculates the loss function, and the formula is as follows:

[0086] In the decomposed scene reconstruction stage, the total loss function is:

[0087] L recon =L C +L N +L D +L S

[0088] In the object shape enhancement stage, the total loss function is:

[0089]

[0090] In the object texture enhancement stage, the total loss function is:

[0091]

[0092] The present invention is based on decomposition-based neural implicit reconstruction of generative diffusion model priors, and uses potential generative diffusion model priors to fill in missing information in invisible or occluded areas; designs visibility modeling for implicit neural radiation fields; through visibility modeling, dynamically adjusts the loss weights of potential generative diffusion model priors to solve the problem of inconsistency between generative priors and inputs in visible areas; under the condition of including a small number of scene images and mask annotations of objects therein, accurately reconstructs a high-precision complete object 3D model with complete texture for each object, significantly improving processing accuracy; and has important technical significance and significant effects.

[0093] In one embodiment, the visibility model construction optimization subsystem includes:

[0094] A visibility three-dimensional grid module obtains a visibility three-dimensional grid G ​​and initializes the visibility three-dimensional grid G ​​to zero;

[0095] The cumulative transmittance visibility module optimizes the visibility three-dimensional grid G ​​by calculating the cumulative transmittance T when rendering the input view, performs visibility modeling of the three-dimensional scene space, and obtains a three-dimensional grid optimized visibility model.

[0096] The principle and effect of the above technical solution are: a visibility model construction optimization subsystem, including: a visibility three-dimensional grid module, obtaining a visibility three-dimensional grid G, and initializing the visibility three-dimensional grid G ​​to zero; a cumulative transmittance visibility module, optimizing the visibility three-dimensional grid G ​​by calculating the cumulative transmittance T when rendering the input perspective, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model; obtaining the visibility three-dimensional grid G ​​includes: using multiple input perspective observation points to have higher visibility, and in the volume rendering process of the neural radiation field, setting the cumulative transmittance T of the three-dimensional point p to represent the probability that the light reaches the three-dimensional point p without intersecting with other particles; a higher cumulative transmittance T indicates that the three-dimensional point p has a greater probability of visibility in the input perspective; obtaining the visibility three-dimensional grid G.

[0097] In one embodiment, the a priori decomposition and combination scene reconstruction subsystem includes:

[0098] The object independent implicit field expression module creates an independent implicit field for each object in the three-dimensional scene space for expression;

[0099] The volume rendering comparison scene reconstruction module, during the optimization process, uses volume rendering technology to utilize the implicit field of each object to generate the RGB image, depth image, normal vector image, and object mask image under the specified input perspective, and compares them with the corresponding true value of the input image to calculate the loss, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

[0100] The principle and effect of the above technical solution are: a priori decomposition and combination scene reconstruction subsystem, including: an object separate implicit field expression module, which creates an implicit field for each object in the three-dimensional scene space to express it; a volume rendering comparison scene reconstruction module, which, during the optimization process, uses volume rendering technology to use the implicit field of each object to generate an RGB image, depth map, normal vector map and object mask map under a specified input perspective, and compares them with the corresponding true value of the input image to calculate the loss, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

[0101] In one embodiment, the dynamic prior object shape enhancement subsystem includes:

[0102] The post-rendering normal vector generation module randomly samples N viewpoints for each object based on the initial combined scene reconstruction, and generates N corresponding normal vector maps and visibility maps using the volume rendering technology through the implicit field and visibility 3D grid G ​​of the object to obtain the post-rendering normal vector map;

[0103] The image loss model prediction module inputs the rendered normal vector map and the text description of the object into the potential generative diffusion model to obtain the model predicted image loss;

[0104] The loss-adjusted shape reconstruction module optimizes the visibility model according to the three-dimensional grid, adjusts the model prediction image loss, and reconstructs the complete 3D geometric shape structure of the object.

[0105] The principle and effect of the above technical solution are as follows: a dynamic priori object shape enhancement subsystem includes: a post-rendering normal vector generation module, which randomly samples N viewing angles for each object according to the initial combined scene reconstruction, and generates corresponding N normal vector maps and visibility maps through the implicit field and visibility three-dimensional grid G ​​of the object using volume rendering technology to obtain the rendered normal vector map; a picture loss model prediction module, which inputs the rendered normal vector map and the text description of the object into the potential generative diffusion model to obtain the model predicted picture loss; a loss adjustment shape reconstruction module, which optimizes the visibility model according to the three-dimensional grid, adjusts the model predicted picture loss, and reconstructs the complete 3D geometric shape structure of the object; optimizes the visibility model according to the three-dimensional grid, adjusts the model predicted picture loss, and reconstructs the complete 3D geometric shape structure of the object, including: optimizing the visibility model according to the three-dimensional grid, reducing the loss weight of the area with greater visibility; and increasing the loss weight of the area with less visibility; finally, using the adjusted picture loss to optimize the implicit field of the object and reconstruct the complete 3D geometric shape structure of the object.

[0106] In one embodiment, a texture enhanced 3D reconstruction subsystem includes:

[0107] The mesh vertex color prediction module, after completing the object shape enhancement, uses a small number of scene images and mask annotations of the objects in them to derive a 3D mesh representation from the implicit field of each object and create a small neural network M to predict the color of each mesh vertex;

[0108] The differentiable rendering generation module randomly samples N viewpoints for each object, and uses a small neural network M and a visibility 3D grid G ​​to generate the corresponding N RGB images and N visibility images using differentiable rendering;

[0109] The color texture prediction 3D generation module inputs the generated N RGB images together with the text description of the object into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and uses the three-dimensional grid optimization visibility model to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and obtain the optimized small neural network M; after the appearance optimization stage, the optimized small neural network M is used to export the texture image of each object, combined with the original three-dimensional grid representation, to obtain a high-precision complete object 3D model with complete texture, and accurately reconstruct a high-precision complete object 3D model with complete texture for each object.

[0110] The principle and effect of the above technical solution are as follows: the texture enhanced 3D reconstruction subsystem includes: a mesh vertex color prediction module, which, after completing the object shape enhancement, uses a mask annotation including a small number of scene images and the objects therein to derive a 3D mesh representation from the implicit field of each object, and creates a small neural network M to predict the color of each mesh vertex; a differentiable rendering generation module, which randomly samples N viewpoints for each object, and generates corresponding N RGB images and N visibility images through the small neural network M and the visibility 3D mesh G using differentiable rendering; a color texture prediction 3D generation module, which generates the N RGB images generated The image and the text description of the object are input into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and the three-dimensional grid optimized visibility model is used to adjust the image color texture loss of N RGB images to optimize the small neural network M, and the optimized small neural network M is obtained; after the appearance optimization stage, the texture image of each object is exported using the optimized small neural network M, and combined with the original three-dimensional grid representation, a high-precision complete object 3D model with complete texture is obtained, and a high-precision complete object 3D model with complete texture of each object is accurately reconstructed; the processing accuracy is significantly improved; it has important technical significance and significant effects.

[0111] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for reconstructing the complete shape and texture of an object in a scene with high precision, characterized in that: include: S10, obtaining a visibility three-dimensional grid G, optimizing the visibility three-dimensional grid G, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model; S20, by generating a decomposed neural implicit reconstruction of the diffusion model prior, using the potential generative diffusion model prior to fill in the missing information of the object in the invisible area or the occluded area, and perform combined scene reconstruction; S30, optimizes the visibility model through the three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a priori that is consistent with the input in the visible area, and performs object shape enhancement; S40, after completing the object shape enhancement, derives a three-dimensional mesh representation from the implicit field, performs object texture enhancement, and precisely reconstructs a high-precision complete object 3D model with complete texture for each object.

2. A method for reconstructing the complete shape and texture of an object in a scene with high precision according to claim 1, characterized in that: S10 includes: S101, obtaining a visible three-dimensional grid G, and initializing the visible three-dimensional grid G ​​to zero; S102, optimizing the visibility three-dimensional grid G ​​by using the cumulative transmittance T calculated when rendering the input viewing angle, performing visibility modeling of the three-dimensional scene space, and obtaining a three-dimensional grid optimized visibility model.

3. The method for reconstructing the complete shape and texture of an object in a scene with high precision according to claim 1, characterized in that: The S20 includes: S201, creating an implicit field for each object in the three-dimensional scene space to express it; S202, during the optimization process, the implicit field of each object is used to generate an RGB image, a depth image, a normal vector image, and an object mask image under a specified input viewing angle through volume rendering technology, and the loss is calculated by comparing the implicit field with the corresponding true value of the input image, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

4. The method for reconstructing the complete shape and texture of an object in a scene with high precision according to claim 1, characterized in that: S30 includes: S301, based on the initial combined scene reconstruction, randomly sample N viewpoints for each object, generate corresponding N normal vector maps and visibility maps using the implicit field and visibility 3D grid G ​​of the object using volume rendering technology, and obtain a rendered normal vector map; S302, inputting the rendered normal vector image and the text description of the object into a potential generative diffusion model to obtain a model predicted image loss; S303, optimizing the visibility model according to the three-dimensional grid, adjusting the model prediction image loss, and reconstructing the complete 3D geometric shape structure of the object.

5. The method for reconstructing the complete shape and texture of an object in a scene with high precision according to claim 1, characterized in that: S40 includes: S401, after completing the object shape enhancement, using a small number of scene images and mask annotations of the objects therein, deriving a 3D mesh representation from the implicit field of each object, and creating a small neural network M to predict the color of each mesh vertex; S402, randomly sampling N viewpoints for each object, and generating corresponding N RGB images and N visibility images by using a small neural network M and a visibility 3D grid G ​​using differentiable rendering; S403, input the generated N RGB images together with the text description of the object into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and use the three-dimensional grid optimization visibility model to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and obtain the optimized small neural network M; after the appearance optimization stage, use the optimized small neural network M to export the texture image of each object, combine it with the original three-dimensional grid representation, and obtain a high-precision complete object 3D model with complete texture, and accurately reconstruct a high-precision complete object 3D model with complete texture for each object.

6. A high-precision complete shape and texture reconstruction system for objects in a scene, characterized in that: include: A visibility model construction optimization subsystem is constructed to obtain a visibility three-dimensional grid G, optimize the visibility three-dimensional grid G, perform visibility modeling of the three-dimensional scene space, and obtain a three-dimensional grid optimized visibility model; The prior decomposition and combination scene reconstruction subsystem performs combined scene reconstruction by generating a decomposition-based neural implicit reconstruction of the diffusion model prior and using the potential generative diffusion model prior to fill in the missing information of objects in invisible or occluded areas; The dynamic prior object shape enhancement subsystem optimizes the visibility model through a three-dimensional grid, dynamically adjusts the loss weight of the potential generative diffusion model prior, forms a prior that is consistent with the input in the visible area, and performs object shape enhancement; The texture-enhanced 3D reconstruction subsystem, after completing the object shape enhancement, derives the 3D mesh representation from the implicit field, performs object texture enhancement, and accurately reconstructs a high-precision complete object 3D model with complete texture for each object.

7. A high-precision complete shape and texture reconstruction system for objects in a scene according to claim 6, characterized in that: The visibility model construction optimization subsystem includes: A visibility three-dimensional grid module obtains a visibility three-dimensional grid G ​​and initializes the visibility three-dimensional grid G ​​to zero; The cumulative transmittance visibility module optimizes the visibility three-dimensional grid G ​​by calculating the cumulative transmittance T when rendering the input view, performs visibility modeling of the three-dimensional scene space, and obtains a three-dimensional grid optimized visibility model.

8. The high-precision complete shape and texture reconstruction system for objects in a scene according to claim 6, characterized in that: A priori decomposition and combination scene reconstruction subsystem includes: The object independent implicit field expression module creates an independent implicit field for each object in the three-dimensional scene space for expression; The volume rendering comparison scene reconstruction module, during the optimization process, uses volume rendering technology to utilize the implicit field of each object to generate the RGB image, depth image, normal vector image, and object mask image under the specified input perspective, and compares them with the corresponding true value of the input image to calculate the loss, thereby optimizing the implicit field of the object and performing combined scene reconstruction.

9. The high-precision complete shape and texture reconstruction system for objects in a scene according to claim 6, characterized in that: Dynamic prior object shape enhancement subsystem, including: The post-rendering normal vector generation module randomly samples N viewpoints for each object based on the initial combined scene reconstruction, and generates N corresponding normal vector maps and visibility maps using the volume rendering technology through the implicit field and visibility 3D grid G ​​of the object to obtain the post-rendering normal vector map; The image loss model prediction module inputs the rendered normal vector map and the text description of the object into the potential generative diffusion model to obtain the model predicted image loss; The loss-adjusted shape reconstruction module optimizes the visibility model according to the three-dimensional grid, adjusts the model prediction image loss, and reconstructs the complete 3D geometric shape structure of the object.

10. The high-precision complete shape and texture reconstruction system for objects in a scene according to claim 6, characterized in that: Texture-enhanced 3D reconstruction subsystem, including: The mesh vertex color prediction module, after completing the object shape enhancement, uses a small number of scene images and mask annotations of the objects in them to derive a 3D mesh representation from the implicit field of each object and create a small neural network M to predict the color of each mesh vertex; The differentiable rendering generation module randomly samples N viewpoints for each object, and uses a small neural network M and a visibility 3D grid G ​​to generate the corresponding N RGB images and N visibility images using differentiable rendering; The color texture prediction 3D generation module inputs the generated N RGB images together with the text description of the object into the potential generative diffusion model to obtain the color texture loss of the image predicted by the model; and uses the three-dimensional grid optimization visibility model to adjust the image color texture loss of the N RGB images to optimize the small neural network M, and obtain the optimized small neural network M; after the appearance optimization stage, the optimized small neural network M is used to export the texture image of each object, combined with the original three-dimensional grid representation, to obtain a high-precision complete object 3D model with complete texture, and accurately reconstruct a high-precision complete object 3D model with complete texture for each object.

Images