Semantic-guided zero-shot generalization 3D object reconstruction method

Abstract

The application relates to a zero-shot generalization three-dimensional object reconstruction method based on semantic guidance, and belongs to the technical fields of computer vision and image processing. The method comprises the following steps: the superiority of a fractional distillation sampling strategy in the three-dimensional reconstruction process of a single image is utilized; prompt words are designed to guide a multi-modal large language model to generate descriptions of an image from coarse granularity to fine granularity and guide the generation of a three-dimensional result; and a multi-modal data alignment strategy is adopted to realize the alignment of semantic and visual modes and integrate semantic information into the generated three-dimensional structure. The application can solve the problem of generating a three-dimensional object structure from a single image in a zero-shot generalization manner, and experiments on a real data set prove the superiority of the application.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to a semantically guided zero-shot generalization method for 3D object reconstruction. Background Technology

[0002] In recent years, with the rapid development of computing devices and capabilities, significant progress has been made in the field of AI-generated content. Generative models have achieved great success in natural language processing and image generation. Recent developments, such as ChatGPT, Wenxin Yiyan, Kimi, and Doubao, have revolutionized many academic and industrial fields. In the 3D field, with the continuous increase in the amount of 3D data and breakthroughs in other generative technologies, 3D generation technology has also made great strides.

[0003] Driven by the application of intelligent robots, the demands for environmental perception technology in this field are increasing. Two-dimensional visual images alone are insufficient to meet the perception needs in complex environments; therefore, 3D reconstruction technology is particularly crucial in enhancing the environmental perception capabilities of intelligent agents. By constructing accurate 3D models, systems can better understand and perceive their surroundings. 3D vision technology can significantly improve autonomous decision-making, spatial perception, and interaction capabilities, enabling systems to react more intelligently in complex scenarios. Especially in fields requiring high-precision perception, such as autonomous driving and medical diagnosis, the value of 3D technology is becoming increasingly prominent, laying a solid foundation for the future development of intelligence and automation. By generating realistic 3D models, machines can gain a deeper understanding of spatial structures and the complex relationships between objects, providing strong support for the decision-making capabilities of automated systems.

[0004] However, traditional 3D reconstruction methods typically require high hardware costs or complex algorithms to extract the 3D information of objects. These methods rely on precise image matching and geometric calculations, and are easily affected by factors such as lighting variations, occlusion, and noise. Currently, deep learning-based methods significantly reduce the need for precise matching of input data, while multimodal large models, based on their rich prior knowledge, also provide guidance for generating the 3D structure of unknown parts. Therefore, deep learning-based methods and multimodal large models based on their rich prior knowledge have become the main approaches for 3D reconstruction. However, existing methods cannot yet solve the problem of generating 3D object structures from a single image with zero-shot generalization. Summary of the Invention

[0005] To address the shortcomings of the aforementioned technologies, the present invention aims to provide a semantically guided zero-shot generalization method for 3D object reconstruction. This invention can utilize semantic information of different granularities to guide the 3D generation process of the model, completing the task of 3D reconstruction of a single image. Extensive experimental testing on commonly used 3D reconstruction datasets demonstrates the superiority of this technology.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A semantically guided zero-shot generalization 3D object reconstruction method establishes cross-modal information fusion between semantics and 3D vision to generate the optimal 3D object structure based on semantic guidance, including the following steps:

[0008] Preprocessing steps: Separate the target object from the background in the image to obtain the target object mask image, and extract the depth information and surface normal information of the target object.

[0009] Model construction steps: A 3D implicit structure sub-model is constructed based on two-stage fractional distillation sampling from coarse to fine; a multimodal large language model is introduced to generate coarse and fine-grained descriptions of the image and calculate fine-grained similarity; 2D and 3D rendering network models are constructed; simultaneously, the CLIP network is used for coarse-grained semantic alignment to optimize the generated new perspective; a semantic feature-guided multimodal 3D implicit representation network is constructed through a visual text alignment strategy; a total loss function is constructed for backpropagation during iterative learning to call back the network parameters of each part of the multimodal 3D reconstruction network.

[0010] Iterative processing steps: Using a single image as input, the preprocessing results are used to iteratively solve the constructed multimodal 3D reconstruction network, and finally output the reconstructed image Mesh.

[0011] The two-stage fractional distillation sampling based on coarse-to-fine rendering includes: constructing a 3D implicit structure sub-model based on NeRF and DMTet networks, initializing an implicit neural network representation of the 3D scene, sampling spatial points and predicting their color and density, and combining volumetric rendering to obtain a multi-view image (Novel View). By guiding the quality of the generated multi-view image, the generated realistic image is gradually optimized. The rendering methods from 3D structure to multi-view are different in the two stages. The first stage uses a NeRF network for neural radiation fields, while the second stage uses a DMTet network for refinement. The parameters are updated through backpropagation, thereby updating the generated 3D structure and obtaining the multi-view (Novel View) of the image rendering result. At the same time, the 3D reconstruction result of the first stage is used as the initialization input for the second stage to accelerate the generation speed of the refinement process.

[0012] Constructing a 3D implicit structure submodel and using it to generate multi-view images includes the following steps:

[0013] NeRF takes the position (x, y, z) and view direction (θ, φ) of each point in 3D space as input, and uses a neural network to predict the color (r, g, b) and volume density (σ) of that point in a specific direction; the formula is expressed as:

[0014] fΘ(x,y,z,θ,φ)→(r,g,b,σ)

[0015] The neural network f is a multilayer perceptron (MLP) that implicitly represents the entire 3D scene.

[0016] During initialization, NeRF samples a large number of points within a three-dimensional space. The neural network accumulates and calculates the points on the light rays to obtain the color and density of each sampled point, thereby generating the image pixel value C(r).

[0017]

[0018] Where T(t) is transmittance, representing the probability that light is not absorbed before a depth t; σ(t) is volume density; and c(t) is color value.

[0019] The multimodal large language model specifically includes the following steps:

[0020] Set prompt words to be used by the LLaVA multimodal large language model to generate specified descriptions of images, resulting in coarse-grained descriptions (Text). c and fine-grained description of Text inv ;

[0021] For the generated large segments of fine-grained, multi-perspective, invariant text descriptions (Text) inv Segment the sentences and compare them with coarse-grained Text features. c The images are then concatenated, and similarity calculations are performed between the concatenated images and both the input and generated new perspective images. The Softmax method is used to normalize the similarity. Since the generated multi-view invariant features are of variable length, the two final concatenated features (Text...) are controlled. inv Components after statement splitting and Text c The sum of similarities of multi-view invariant features is equal, and the first norm is used to control for this.

[0022] After constraint normalization, the variable-length multi-view invariant feature description and image similarity have the same similarity value under different viewpoints; the loss function is calculated by the following formula:

[0023]

[0024] The Softmax method is used to normalize the similarity, and the Sim operation is used to calculate the similarity between the image and the multi-view invariant text, where r is the viewpoint of the input image, and I...r For the input front view, G θ (v i ) is from v i View rendered from a perspective, Text inv This generates fine-grained, multi-perspective, invariant text information.

[0025] The two-dimensional structure is rendered using a standard diffusion model:

[0026] Add Gaussian noise and coarse-grained text information to the 2D image rendering result of the multi-view Novel view. c Together, they are used as input to the standard diffusion model Stable Diffusion to learn from the rendered 2D image result; the output of this step is the loss function L. 2D ;

[0027] The loss function can be calculated using the following formula:

[0028]

[0029] ∈,∈ φ φ and θ are the process parameters for adding and predicting noise, respectively. The value of parameter θ is updated based on the solution results. For the coarse-grained stage, θ is the parameter value of the NeRF's MLPs; for the fine-grained stage, it is the parameter value of SDF, triangle deformation, and color field. φ φ represents the added noise, the predicted noise, and the prior parameters of the 2D spread model, respectively, and z represents the image with added noise. t Let w(t) be the result of adding noise to the image in t steps, w(t) be the weight corresponding to the prediction in step t, and e be the result of text encoding.

[0030] The three-dimensional structure is rendered using a multi-view diffusion model:

[0031] Input the input image of the Novel view and the translation and rotation matrix (R, T) of the new view into the Zero-1-to-3 multi-view diffusion model to obtain the corresponding new view image. Use 3D rendering to obtain the rendering result of this stage. Use the new view image obtained by the diffusion model to constrain the rendering result and update the 3D model.

[0032]

[0033] Where (R,T) is the camera pose passed to the multi-view generation, z t The noise feature map is obtained by adding random Gaussian noise with a time step of t to the latent space feature map of I.

[0034] The new perspective generated by using the CLIP network for coarse-grained semantic alignment optimization has a loss function calculated using the following formula:

[0035] L clip =|Adapter(E m (I r ))-E t (Text c )|

[0036] Text c For coarse-grained textual descriptions of images, the Adapter consists of two fully connected linear layers and a ReLU activation function. m and E t The CLIP is divided into image encoders and text encoders.

[0037] The construction of semantic features is guided by a visual text alignment strategy, including: strong constraint calculation of L for the two-dimensional rendering structure of the viewpoints corresponding to the input image. pre L d L n ;

[0038] L pre The L2 norm loss is used for the preprocessed RGB and mask images of the original viewpoint image and the input image obtained from rendering;

[0039]

[0040] ⊙Element-wise multiplication between corresponding elements of the two matrices, where M is the foreground mask obtained by integrating the volume density of rays along each pixel;

[0041] L d The Pearson negative correlation loss is applied to the original viewpoint depth image and the input depth map obtained from rendering.

[0042]

[0043] Where cov(.) represents the covariance, and σ(.) represents the standard deviation. r The depth estimation map of the original view rendering result, where d is the preprocessed depth map;

[0044] L n To estimate the normal vector for each point using the finite difference of depth, a 2D normal map n is rendered from the normal vector, and the following loss is applied:

[0045] L n =‖n-τ(g(n,k))‖

[0046] Where τ(.) is the stop gradient operation, g(.) is the Gaussian blur, and the size of the convolution kernel k is set to 9×9.

[0047] The total loss function for model training is:

[0048] L total =λ1L 2D +λ2L 3D +λ3L clip +λ4L inv +λ5L pre +λ6L d +λ7L n

[0049] Where λ1-λ7 are the weight coefficients of the corresponding loss function.

[0050] The present invention has the following beneficial effects and advantages:

[0051] 1. This invention does not require a large amount of data to train the model; it only requires a single image to train the model and generate a three-dimensional structure.

[0052] 2. This invention introduces a large model prior to guide the generation of 3D structures. It utilizes the rich prior knowledge and image understanding capabilities of the large model to provide guidance for 3D reconstruction.

[0053] 3. This invention solves the problems of 3D reconstruction without images, paired 3D structures, and multi-view images. Attached Figure Description

[0054] Figure 1 This is a flowchart of the method of the present invention;

[0055] Figure 2 This is the overall structure diagram of the network model invented;

[0056] Figure 3 This is a rendering of the three-dimensional reconstruction according to the method of this invention. Detailed Implementation

[0057] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The method steps are described below with reference to the accompanying drawings.

[0059] Please see Figure 1 and Figure 2 This invention provides a semantically guided zero-shot generalization method for 3D object reconstruction. It includes volume rendering and multi-granularity semantic guidance, specifically comprising the following steps:

[0060] S1: Preprocessing the input image: Separate the target object mask from the background in the image, and estimate the object's depth information and surface normal information using existing methods. It is necessary to separate the foreground target object from the background and obtain the image's depth information and surface normal information based on the separated target image.

[0061] S2: Constructing a multimodal 3D reconstruction network: (S21) Constructing a 3D implicit structure sub-model based on two-stage fractional distillation sampling from coarse to fine; (S22) Introducing a multimodal large language model to generate coarse and fine-grained descriptions of images and calculating fine-grained similarity; (S23) Constructing 2D and 3D optimized network models; (S24) Simultaneously using the CLIP network for coarse-grained semantic alignment to optimize the generated new perspective; (S25) Constructing a semantic feature-guided 3D implicit representation of the multimodal 3D reconstruction network through a visual text alignment strategy; (S26) Constructing a total model loss function for backpropagation during model training to call back the network parameters of each part of the multimodal 3D reconstruction network.

[0062] S21: The multimodal 3D reconstruction network employs a two-stage fractional distillation sampling (SDS)-guided generation strategy. The two stages differ only in the rendering method from the 3D structure to multiple viewpoints. The first stage uses a Neural Radiation Field (NeRF) network, while the second stage uses a DMTet network for refinement. Backpropagation updates parameters, thereby updating the generated 3D structure and obtaining the multi-view novel view of the image rendering result. Simultaneously, the 3D reconstruction result from the first stage serves as the initial input for the second stage, accelerating the generation speed of the refinement process.

[0063] S21-1: The first stage constructs a 3D implicit structural representation: An implicit neural network is initialized to represent the 3D scene, spatial points are sampled and their color and density are predicted, and volumetric rendering is combined to obtain multi-view images. The quality of the generated multi-view images is then guided to progressively optimize the generated realistic images. Specifically, this includes the following steps:

[0064] S21-1a: NeRF takes the position (x, y, z) and view direction (θ, φ) of each point in 3D space as input, and uses a neural network to predict: (r, g, h): representing the color of the point in a specific direction. Volume density (σ): representing the scene occupancy of the point, usually related to the absorption or scattering ability of light. The formula is expressed as:

[0065] fΘ(x,y,z,θ,φ)→(r,g,b,σ)

[0066] The neural network f is typically a multilayer perceptron (MLP) that implicitly represents the entire 3D scene.

[0067] S21-1b: During initialization, NeRF samples a large number of points within a three-dimensional space. These sampled points are then used for volumetric rendering via a neural network, calculating the color and density of each point to obtain the final image.

[0068] S21-1c: In the coarse-grained generation stage, random initialization is used to initialize the NeRF neural network weight parameters. The initial volume density σ is small, indicating that most areas in the space are empty. Color values ​​(r, g, b) are randomly distributed, and the initial color predictions are meaningless. This means that in the early stages of training, the network's representation of the scene is completely chaotic, gradually obtaining a reasonable scene representation through optimization. In the fine-grained generation stage, the training results from the coarse-grained stage are used as the initialization for the fine-grained rendering network.

[0069] S21-1d: By accumulating calculations on points along the light rays, NeRF generates image pixel values ​​using a volumetric rendering formula, where:

[0070]

[0071] T(t) is transmittance, representing the probability that light is not absorbed before a depth t; σ(t) is volume density; c(t) is color value.

[0072] S21-1e: The training objective is to optimize the loss function so that the multi-view images rendered by the network are closer to real objects. The loss function is a constraint on semantics and vision under two-dimensional and three-dimensional conditions.

[0073] S21-2: The second stage uses the DMTet network for refinement. The results obtained in the NeRF stage are used to initialize the DMTet, and the same rendering method as NeRF is used. The loss function is used to constrain and refine the local structure of the 3D structure.

[0074] S22: Constructing a multimodal semantic guidance module:

[0075] Using a multimodal large language model, extract coarse-grained semantic features (Text) from images. c Based on the prior knowledge of the semantic model, generate fine-grained representations (Text). inv This guides the generation of three-dimensional structures. Specifically, it includes the following steps:

[0076] S22-1: Multimodal Semantic Guidance Module. This module designs prompts to guide the LLaVA multimodal large language model in generating specified descriptions of images, divided into coarse-grained descriptions (Text). c and fine-grained description of Text inv Among them, Text c An example is A photo of a red drum set with a drum, cymbals, and a stand; Text inv An example is The multiviewinvariant features of the object in the image are a drum set with a red drumshell, a snare drum, a bass drum, and a cymbal. The drum set is set against a black background. The material of the object is not specified in the caption.

[0077] S22-2: For the generated large segments of fine-grained, multi-perspective, invariant text descriptions... inv Segment the sentences and compare them with coarse-grained Text features. c The images are then stitched together, and then similarity is calculated between the stitched images and the input image mask of the target object, as well as the generated new multi-view Novel view. The output of this step is the loss function L. inv .

[0078] The Softmax method is used for similarity normalization. Since the generated multi-view invariant features are of variable length, the last two concatenated features (Text...) are controlled... inv Components after statement splitting and Text c The similarity sum of multi-view invariant features is controlled by using a norm and a normalized norm to ensure that the normalized, variable-length multi-view invariant features describe the similarity to the image, having the same similarity value across different viewpoints. The loss function can be calculated using the following formula:

[0079]

[0080] The Softmax method normalizes the similarity, while Sim calculates the similarity between an image and multi-view invariant text, where r is the viewpoint of the input image, and I... r For the input front view, G θ (v i ) is from v i View rendered from a perspective, Text inv This generates fine-grained, multi-perspective, invariant text information.

[0081] S23: Two-dimensional and three-dimensional reverse optimization guidance;

[0082] S23-1: The two-dimensional structure is guided by the standard diffusion model as follows:

[0083] Add Gaussian noise and coarse-grained text information to the 2D image rendering result of the multi-view novel. c Together, these conditions are used as inputs to the standard diffusion model, Stable Diffusion, to guide the rendering of the 2D image. The output of this step is the loss function L. 2D .

[0084] The loss function can be calculated using the following formula:

[0085]

[0086] ∈,∈ φ φ and θ are the process parameters for adding and predicting noise, respectively. The value of parameter θ is updated based on the solution results. For the coarse-grained stage, θ is the parameter value of the NeRF MLPs; for the fine-grained stage, it is the parameter value of SDF, triangle deformation, and color field. φ represents the prior parameters of the added noise, predicted noise, and 2D spread model, respectively, and z represents the image with added noise. t Let w(t) be the result of adding noise to the image in t steps, w(t) be the weight corresponding to the prediction in step t, and e be the result of text encoding.

[0087] S23-2: The three-dimensional structure is constrained by a multi-view diffusion model as follows:

[0088] The input image of the Novel view and the translation and rotation matrix (R, T) of the new view are input into the Zero-1-to-3 multi-view diffusion model to obtain the corresponding new view image. The rendering result of this stage is obtained by 3D rendering. The rendering result is constrained by the new view image obtained by the diffusion model, and the 3D model is updated. The constraint can be calculated by the following formula:

[0089]

[0090] Where (R,T) is the camera pose passed to the multi-view generation, i.e., a view-dependent diffusion model. t The noise feature map is obtained by adding random Gaussian noise with a time step of t to the latent space feature map of I.

[0091] S24: Simultaneously, the CLIP network is used for coarse-grained semantic text processing. c Alignment is performed to optimize the generated new viewpoint; this step outputs the loss function L. clip The loss function can be calculated using the following formula:

[0092] L clip =|Adapter(E m (I r ))-E t (Text c )|

[0093] Text c For coarse-grained textual descriptions of images, the Adapter consists of two fully connected linear layers and a ReLU activation function. m and E t These represent CLIP's image encoder and text encoder, respectively.

[0094] S25: Perform strong constraint calculations on the 2D rendering results from the input viewpoint. pre L d L n r represents the viewpoint corresponding to the input image.

[0095] S25-1:L pre The L2 norm loss is applied to the preprocessed RGB values ​​of the original viewpoint image and the input image, as well as the mask image. The loss function can be calculated using the following formula:

[0096]

[0097] The element-wise multiplication of corresponding elements between the two matrices, where M is the foreground mask obtained by integrating the volume density of rays along each pixel.

[0098] S25-2:L d The Pearson negative correlation loss is applied to the rendered original viewpoint depth image and the input depth map. The loss function can be calculated using the following formula:

[0099]

[0100] Where cov(.) represents the covariance, and σ(.) represents the standard deviation. r The depth estimation map of the original view rendering result, where d is the preprocessed depth map.

[0101] S25-3:L n To estimate the normal vector (normal) for each point using the finite difference of depth, a 2D normal map (n) is rendered from the normal vector, and a loss is applied to it:

[0102] L n =‖n-τ(g(n,k))‖

[0103] Where τ(.) represents stopping the gradient operation, g(.) represents Gaussian blur, and k is the convolution kernel, with a kernel size of 9×9. S26: The total loss function for model training is:

[0104] L total =λ1L 2D +λ2L 3D +λ3L clip +λ4L inv +λ5L pre +λ6L d +λ7L n

[0105] L pre L is the L2 norm loss for the original viewpoint image and the preprocessed RGB and mask images of the input image obtained from rendering. d The Pearson negative correlation loss is used to calculate the original viewpoint depth image and the input depth map for rendering. n To estimate the normal vector at each point using the finite difference of depth, a 2D normal map n is rendered from the normal vector. λ represents the weighting coefficients of the corresponding loss function.

[0106] In summary, the semantic prior is incorporated into the training process of step S2, and constraints are imposed on both the 2D diffusion results and the 3D rendering results. The generation of the 3D structure is ultimately constrained through backpropagation. The coarse-grained rendering stage and the fine-grained rendering stage follow the same process, differing only in the rendering method: NeRF is used in the coarse-grained rendering stage, while the DMTet rendering method is used in the fine-grained rendering stage.

[0107] S3: Iterative processing step: Using a single image as input, the preprocessing results are used to iteratively solve the multimodal 3D reconstruction network constructed in step S2, and finally output the reconstructed image Mesh.

[0108] like Figure 3 The image shows the reconstruction results on three datasets. The multi-view images are visualizations obtained through 3D rendering.

[0109] Finally, it should be noted that the above description is a preferred embodiment of the present invention. It should be pointed out that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should be considered within the scope of protection of the present invention.

Claims

1. A semantic-guided zero-shot generalization 3D object reconstruction method, characterized in that, The cross-modal information fusion between semantics and three-dimensional vision is established to generate an optimal three-dimensional object structure based on semantic guidance, including the following steps: A preprocessing step: separating the target object from the background in the image to obtain a target object mask image mask, and extracting the depth information depth and object surface normal information normal of the target object; A model construction step: constructing a three-dimensional implicit structure submodel based on a two-stage fractional distillation sampling from coarse to fine; introducing a multi-modal large language model to generate coarse and fine-grained descriptions of the image and calculating the fine-grained similarity; constructing a rendering network model of two-dimensional structure and three-dimensional structure; simultaneously performing coarse-grained semantic alignment using a CLIP network to optimize the generated new perspective; realizing a multi-modal three-dimensional reconstruction network 3D implicit representation guided by semantic features through a visual text alignment strategy; and constructing a model total loss function for backward propagation and callback of network parameters of each part of the multi-modal three-dimensional reconstruction network in the iterative learning process; The two-stage fractional distillation sampling from coarse to fine includes: constructing a three-dimensional implicit structure submodel based on a NeRF network and a DMTet network, initializing an implicit neural network to represent a three-dimensional scene, sampling a space point and predicting its color and density, combining volume rendering to obtain a multi-view image Novel View, and gradually optimizing the generated realistic image by guiding the quality of the generated multi-view image; the rendering mode of the three-dimensional structure to the multi-view is different in the two stages, the first stage uses a neural radiation field NeRF network, and the second stage uses a DMTet network to realize refinement; the parameters are updated through back propagation, thereby updating the generated three-dimensional structure to obtain a multi-view Novel view of the image rendering result; and the three-dimensional reconstruction result of the first stage is used as the initialization input of the second stage to accelerate the generation speed of the refinement process; The semantic feature guidance is achieved by a visual text alignment strategy, including: performing strong constraint calculation L pre 、 d 、 n ; L pre The L2 norm loss of the rendered original view image and the pre-processed RGB and mask images of the input image is obtained. The element-wise multiplication between the corresponding elements of the two matrices, M is the foreground mask obtained by integrating the body density along the ray of each pixel; L d Pearson negative correlation loss between the rendered original view depth image and the input depth image; where cov(.) denotes the covariance and σ(.) denotes the standard deviation; d r a depth estimation map of the original view rendering result, d is a pre-processed depth map; L n To estimate the normal vector of each point using the limited difference of depth, render a two-dimensional normal map n from the normal vectors and impose the following loss: L n =‖n-τ(g(n,k))‖ Where τ(.) is a stop gradient operation, g(.) is a Gaussian blur, and the size of the convolution kernel k is set to 9x9; An iterative processing step: using a single image as input, using the preprocessing result to iteratively solve the constructed multi-modal three-dimensional reconstruction network, and finally outputting a reconstructed image Mesh.

2. The semantic guidance based zero-shot generative 3D object reconstruction method of claim 1, wherein, Constructing a three-dimensional implicit structure submodel to generate a multi-view image includes the following steps: NeRF takes the position (x, y, z) of each point in a three-dimensional space and the viewing direction (θ, φ) as input, and predicts the color (r, g, b) and volume density (σ) of the point in a specific direction through a neural network; the formula is represented as: fΘ(x,y,z,θ,φ)→(r,g,b,σ) Where the neural network f is a multi-layer perceptron (MLP), which implicitly represents the entire three-dimensional scene; During initialization, NeRF samples a large number of points in a three-dimensional space, performs cumulative calculation on the points on the light ray through the neural network, obtains the color and density of each sampling point, and thereby generates an image pixel value C(r); Where T(t) is the transmittance, representing the probability of light not being absorbed before depth t; σ(t) is the volume density; c(t) is the color value.

3. The method of claim 1, wherein, The multi-modal large language model specifically comprises the following steps: A prompt word is set for the LLaVA multi-modal large language model to generate a specified description of an image to obtain a coarse-grained description Text c and a fine-grained description Text inv ; For the generated large section of fine-grained multi-view invariant text description Text inv The sentence segmentation is performed, and the coarse-grained features Text c are spliced, and similarity calculation is performed with the input image and the generated new view image respectively; wherein the similarity is normalized by using the Softmax method; since the generated multi-view invariant features are of indefinite length, the two features (Text inv The components after the sentence segmentation and Text c ) and a norm are used to control the sum of the similarities of the multi-view invariant features to be equal.

4. The method of claim 3, wherein, The similarity between the constraint normalized variable-length multi-view invariant feature description and the image has the same similarity value under different views; the loss function is calculated by the following formula: wherein the Softmax method is used to normalize the similarity, the Sim operation is used to calculate the similarity between the image and the multi-view invariant text, r is the view of the input image, I r is the input front view, G θ (v i ) is the view rendered from the v i view, Text inv is the generated fine-grained multi-view invariant text information.

5. The method of claim 1, wherein, Where the two-dimensional structure is rendered by a standard diffusion model: In the two-dimensional image rendering result of the multi-view Novel view, Gaussian noise noise and coarse-grained text information Test are added c together as the input of the standard diffusion model Stable Diffusion, and the two-dimensional image rendering result is learned; the output of this step is a loss function L 2D ; The calculation of the loss function can be performed by the following formula: ∈,∈ φ ,φ,θ are the process parameters of adding noise and predicting noise, and the value of parameter θ is updated according to the solving result; for the coarse-grained stage, θ is the parameter value of the MLP of NeRF, and for the fine-grained stage, θ is the parameter value of SDF, triangle deformation and color field; ∈,∈ φ ,φ are the prior parameters of the added noise, the predicted noise and the 2D spreading model respectively, z is the image added with noise, z t is the result of the image after adding noise for t steps, w(t) is the weight corresponding to the tth step of prediction, and e is the result of text encoding.

6. The method of claim 1, wherein, Where the three-dimensional structure is rendered by a multi-view diffusion model: The input image of the multi-view Novelview and the translation rotation matrix (R, T) of the new view are input into the multi-view diffusion model Zero-1-to-3 to obtain the corresponding new view image, the rendering result of this stage is obtained in a three-dimensional rendering manner, the new view image obtained by the diffusion model is used to constrain the rendering result, and the three-dimensional model is updated; where (R, T) is the camera pose passed to multi-view generation, z t is the noisy feature map obtained by adding random Gaussian noise with time step t in the latent space feature map of I.

7. The method of claim 1, wherein, The new view generated by the coarse-grained semantic alignment optimization using the CLIP network, the loss function is calculated by the following formula: L clip = |Adapter(E m (I r ))- E t (Text c )| Text c For the coarse-grained text description of the image, Adapter is composed of two linear fully connected layers and a ReLU activation function; E m and E t The image encoder and text encoder of CLIP are represented by two linear fully connected layers and a ReLU activation function.

8. The semantic-guided zero-shot generative 3D object reconstruction method according to any one of claims 1-7, characterized in that, The total loss function of model training is: L total = λ1L 2D + λ2L 3D + λ3L clip + λ4L inv + λ5L pre + λ6L d + λ7L n Where λ1-λ7 are weight coefficients corresponding to the loss function.

Images