A method and apparatus for monocular object 3D reconstruction based on depth estimation

Abstract

This specification discloses a method and apparatus for monocular object 3D reconstruction based on depth estimation, comprising: acquiring a multi-view image to be processed; segmenting the multi-view image to be processed based on an image segmentation model to obtain multi-view scene cues and multi-view object images; obtaining consistency features based on the multi-view object images through a preset large visual model combination, and obtaining deconstruction features based on the multi-view scene cues through a preset Transformer model; fusing and reconstructing the consistency features and the deconstruction features through a diffusion Unet model to obtain a super-resolution hidden code; performing multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden code to obtain depth information; and performing 3D reconstruction based on the depth information to generate a monocular target image.

Description

Technical Field

[0001] This specification relates to the fields of computer technology and image recognition technology, and in particular to a method and apparatus for monocular object 3D reconstruction based on depth estimation. Background Technology

[0002] Multi-view object reconstruction, as an important research direction in the field of computer vision, provides technical means for the three-dimensional reconstruction of target objects.

[0003] Traditional multi-view stereo (MVS) methods reconstruct 3D objects through modular processes such as feature matching, depth prediction, and fusion. However, recent breakthroughs in Neural View Synthesis (NVS), particularly the introduction of Neural Radiance Fields (NeRF), have significantly propelled this field forward. NeRF models geometry and viewpoint-dependent appearance using multilayer perceptrons and optimizes rendering through volumetric rendering, resulting in superior rendering quality. Despite advancements in computer vision technology, multi-view 3D object reconstruction still faces numerous challenges. The lack of sophisticated depth estimation and mesh generation strategies during reconstruction leads to the inability to capture minute details when dealing with complex scenes or object surfaces, resulting in low resolution and insufficient detail. Overemphasis on rendering effects can distort 3D reconstruction results, leading to geometric inconsistencies. Furthermore, the failure to consider multi-view consistency representation results in poor accuracy of multi-view information integration, degrading the overall quality of 3D reconstruction.

[0004] Based on this, this specification provides a method and apparatus for monocular object 3D reconstruction based on depth estimation. Summary of the Invention

[0005] This specification provides a method and apparatus for monocular object 3D reconstruction based on depth estimation, which addresses the following technical problems: In the prior art, the lack of refined depth estimation and mesh generation strategies during reconstruction results in the inability to capture minute details when processing complex scenes or object surfaces, leading to low resolution and insufficient detail depiction; due to an overemphasis on rendering effects, the 3D reconstruction results are distorted, and the reconstruction results have defects in geometric consistency; at the same time, due to the failure to consider the problem of multi-view consistency representation, the accuracy of multi-view information integration is poor, resulting in a decline in the overall quality of 3D reconstruction.

[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0007] This specification provides an embodiment of a monocular object 3D reconstruction method based on depth estimation, including:

[0008] Acquire multi-view images to be processed;

[0009] Based on the image segmentation model, the multi-view image to be processed is segmented to obtain multi-view scene clues and multi-view object images.

[0010] Based on the multi-view object images, consistency features are obtained through a preset combination of large visual models, and deconstruction features are obtained through a preset Transformer model based on the multi-view scene cues.

[0011] The consistency feature and the deconstruction feature are fused and reconstructed using the diffusion Unet model to obtain the super-resolution hidden code;

[0012] Multi-level depth prediction is performed based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information;

[0013] Based on the depth information, a three-dimensional reconstruction is performed to generate a monocular target image.

[0014] This specification also provides an embodiment of a monocular object 3D reconstruction device based on depth estimation, comprising:

[0015] The acquisition module acquires multi-view images to be processed;

[0016] The segmentation module, based on an image segmentation model, segments the multi-view image to be processed to obtain multi-view scene clues and multi-view object images;

[0017] The intermediate processing module obtains consistency features based on the multi-view object images through a preset combination of large visual models, and obtains deconstruction features based on the multi-view scene cues through a preset Transformer model.

[0018] In the prediction module, the consistency features and the deconstruction features are fused and reconstructed using a diffusion Unet model to obtain a super-resolution hidden code.

[0019] The depth estimation module performs multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information;

[0020] The 3D reconstruction module performs 3D reconstruction based on the depth information to generate a monocular target image.

[0021] The monocular object 3D reconstruction method based on depth estimation provided in this specification acquires a multi-view image to be processed; segments the multi-view image based on an image segmentation model to obtain multi-view scene cues and multi-view object images; obtains consistency features based on the multi-view object images through a preset large visual model combination, and obtains deconstruction features based on the multi-view scene cues through a preset Transformer model; fuses and reconstructs the consistency features and the deconstruction features through a diffusion Unet model to obtain a super-resolution hidden code; performs multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden code to obtain depth information; and performs 3D reconstruction based on the depth information to generate a monocular target image. This method can achieve high-precision and separable 3D reconstruction, significantly improving the accuracy and robustness of the reconstruction. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A schematic diagram of the system architecture of a monocular object 3D reconstruction method based on depth estimation provided in the embodiments of this specification;

[0024] Figure 2 A flowchart illustrating a monocular object 3D reconstruction method based on depth estimation provided in the embodiments of this specification;

[0025] Figure 3 A framework diagram of a monocular object 3D reconstruction method based on depth estimation provided in the embodiments of this specification;

[0026] Figure 4 This is a schematic diagram of a monocular object 3D reconstruction device based on depth estimation, provided as an embodiment of this specification. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0028] In monocular object depth estimation, Lee predicts object meshes and renders depth maps based on Mesh-RCNN. Old-net, on the other hand, predicts depth directly from monocular RGB images and designs novel SDDR and NGPH, validating the importance of using shape priors to better handle intra-class variations.

[0029] Multi-view 3D reconstruction methods include point cloud, mesh, voxel, and implicit representation methods, each with its own advantages. Point cloud-based reconstruction methods, such as Lin and Wei, significantly improve reconstruction accuracy through dense point cloud generation and multi-view aggregation. Regarding mesh representation, Pixel2Mesh++ generates multiple views through deformation. Figure 3 D-grids and FOUND optimization improve model accuracy by optimizing surface deformation. Voxel representations, such as Multi-view 3D Reconstruction with Transformers, utilize Transformers technology to improve voxel representation efficiency, while 3D-C2FT enhances reconstruction detail through hierarchical optimization. Implicit representation methods, such as FvOR, combine shape and pose optimization to achieve robust reconstruction under limited viewpoint conditions, while GaussianObject reduces the required input viewpoints through Gaussian initialization.

[0030] Existing monocular object 3D reconstruction methods generally suffer from problems such as low resolution, geometric inconsistency, and poor integration of multiple views in multi-view reconstruction.

[0031] Based on this, the embodiments of this specification provide a monocular object 3D reconstruction method based on depth estimation. By accurately segmenting target object elements in complex scenes, the method ensures the independence and clear representation of each object; by combining scene cues extracted from a large model with object structural features, it ensures the consistency of representation under multiple perspectives, providing support for depth estimation; by using a diffusion model for depth estimation, it recovers the absolute depth at the scene level, ensuring the accurate position and size of objects in 3D space; finally, it transforms the depth information into a 3D mesh to achieve complete landscape reconstruction and improve the detail performance of monocular object reconstruction.

[0032] Figure 1 This diagram illustrates the system architecture of a monocular object 3D reconstruction method based on depth estimation, as provided in the embodiments of this specification. Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0033] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various client applications can be installed on terminal devices 101, 102, and 103, such as dedicated programs for monocular object 3D reconstruction based on depth estimation.

[0034] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various dedicated or general-purpose electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module.

[0035] Server 105 can be a server that provides various services, such as a backend server that provides services to client applications installed on terminal devices 101, 102, and 103. For example, the server can perform monocular object 3D reconstruction based on depth estimation so that the results of the monocular object 3D reconstruction based on depth estimation can be displayed on terminal device servers 101, 102, and 103.

[0036] Server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module.

[0037] Figure 2 This is a flowchart illustrating a monocular object 3D reconstruction method based on depth estimation, provided in an embodiment of this specification. From a programming perspective, the execution entity of the process can be a program hosted on an application server or application terminal. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, the method includes:

[0038] Step S201: Obtain the multi-view image to be processed.

[0039] In the embodiments of this specification, the multi-view image to be processed includes at least three views, and the multi-view image to be processed is an image containing a target object. The target object is the object that needs to be reconstructed in three dimensions. It can be understood that the multi-view image to be processed includes both the target image and the background image. It can be understood that in the multi-view image to be processed, all images other than the target image are background images.

[0040] In the embodiments of this specification, the acquisition of multi-view images can be achieved by taking pictures from different angles using a camera.

[0041] Step S203: Based on the image segmentation model, segment the multi-view image to be processed to obtain multi-view scene clues and multi-view object images.

[0042] In the embodiments of this specification, the multi-view scene clues are object-level tokens, and the multi-view scene clues include the features, position, and detection box information of each element;

[0043] The multi-view scene cues are obtained based on the SAM2 model, and the formula for the multi-view scene cues is:

[0044] τ=S(I)

[0045] in,

[0046] τ represents the multi-view scene clues;

[0047] S represents the SAM2 model;

[0048] I represents the input multi-view image to be processed.

[0049] An object-level token is a token that represents a specific object or salient region in an image. In the embodiments of this specification, an object-level token is a data unit of feature information of a target image. Specifically, it includes the features, position, and detection box information of each element in the target image. Further, the detection box information includes the position, size, and semantics of the detection box.

[0050] The object-level tokens of multi-view scene cues improve the reconstruction accuracy of the model and ensure the capture of minute details in the scene, ultimately making the 3D reconstruction results more accurate and reliable.

[0051] The SAM2 model is a video and image object segmentation model. The specific structure of the model is existing technology and will not be described in detail here.

[0052] Step S205: Based on the multi-view object images, obtain consistency features through a preset visual large model combination, and based on the multi-view scene cues, obtain deconstruction features through a preset Transformer model.

[0053] In the embodiments of this specification, the preset visual large model combination includes three visual large models, which are connected in parallel. The preset visual large model includes: a visual large model for semantic feature extraction, a visual large model for texture feature extraction, and a visual large model for structural feature extraction.

[0054] The consistency features include: semantic consistency features, attentional consistency features, and descriptive consistency features.

[0055] In the embodiments described in this specification, the formula for the consistency feature is:

[0056]

[0057] in,

[0058] This represents the consistency feature;

[0059] f represents the consistency feature extraction function;

[0060] M represents the preset combination of large visual models;

[0061] I′ represents the multi-view object image;

[0062] The preset visual large model combination consists of DINOV2, Stable Diffusion, and BLIP in parallel. DINOV2 is used to extract semantic features from the multi-view object image to obtain the semantic consistency feature; Stable Diffusion is used to enhance the texture features in the multi-view object image to obtain the attention consistency feature; and BLIP is used to extract structural features from the multi-view object image to obtain the description consistency feature.

[0063] In the embodiments of this specification, semantic consistency features refer to the semantic similarity or consistency of image content captured from different viewpoints. In this embodiment, "semantics" mainly refers to the content in the image, including but not limited to information about objects, scenes, textures, and structures.

[0064] Attention consistency features refer to attention-related features extracted from an image using a specific visual model, ensuring the consistency of these features across different viewpoints. In this embodiment, attention consistency features are primarily achieved through the image's texture features.

[0065] The description of consistency features refers to the extracted structural information, specifically the spatial layout and shape of objects.

[0066] In the embodiments of this specification, consistency features are extracted by using multi-view object images as input to a preset large visual model and then extracting consistency features through a consistency feature extraction function. Further, the consistency feature extraction function is a function that extracts corresponding features within the preset large visual model; that is, the function or module for consistency feature extraction within the preset large visual model serves as the consistency feature extraction function, or it can be described as a function within the preset large visual model used to implement consistency feature extraction.

[0067] By acquiring consistent features, we can ensure the stability and consistency of features across multiple perspectives. Integrating the outputs of different models enhances the robustness of features and effectively reduces inconsistencies caused by changes in perspective.

[0068] Deconstruction features refer to features with specific geometric meanings extracted from images or 3D models, such as normals, contours, and corner points. These features are crucial for understanding the geometry and structure of an object.

[0069] In the embodiments of this specification, the deconstruction features include: normal deconstruction features, contour deconstruction features, and corner deconstruction features;

[0070] The formula for the deconstruction feature is:

[0071] Ψ = g(T, τ)

[0072] in,

[0073] Ψ represents the deconstruction feature;

[0074] g represents the attention mechanism function;

[0075] T represents the preset Transformer model;

[0076] τ represents the multi-perspective scene clues.

[0077] In the embodiments of this specification, the attention mechanism function is used to extract deconstructed features by taking multi-view scene cues as input to a preset Transformer model and using a self-attention function.

[0078] In the embodiments described in this specification, the preset Transformer model includes a self-attention feedback layer, residual connections and layer normalization, a feedforward neural network, residual connections and layer normalization.

[0079] In the embodiments of this specification, the normal information in the normal deconstruction feature reflects the direction of a point on the surface of an object, which is very important for the lighting calculation and understanding of the surface properties of the object.

[0080] In contour deconstruction features, contour information defines the boundaries of an object, which is very useful for understanding the shape and boundaries of an object.

[0081] In corner deconstruction features, corner points are special points on the surface of an object, usually appearing on the edges or corners of the object, and are an important part of the object's geometric structure.

[0082] By deconstructing features, more refined and accurate feature information can be extracted, thereby achieving more precise 3D reconstruction.

[0083] Step S207: The consistency feature and the deconstruction feature are fused and reconstructed using the diffusion Unet model to obtain the super-resolution hidden code.

[0084] After obtaining consistency features and deconstruction features, they are further used as input to the diffusion Unet model for effective fusion, thereby improving the precision and accuracy of the encoding.

[0085] In the embodiments of this specification, the consistency features and the deconstruction features are fused and reconstructed using a diffusion Unet model to obtain super-resolution hidden codes, specifically including:

[0086] Using the consistency feature and the deconstruction feature as inputs, the three-layer implicit coding is obtained by inputting them into the diffusion Unet model;

[0087] The three-layer implicit coding is fused using an attention mechanism to obtain the super-resolution implicit coding;

[0088] The formula for the super-resolution hidden coding is:

[0089]

[0090] in,

[0091] r indicates super-resolution hidden coding;

[0092] u represents the diffusion Unet model;

[0093] This represents the consistency feature;

[0094] Ψ represents the deconstruction feature.

[0095] In the embodiments described in this specification, the diffusion Unet model can be the SDXL model.

[0096] Step S209: Perform multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information.

[0097] In the embodiments of this specification, the step of performing multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information specifically includes:

[0098] The super-resolution hidden coding layers and the multi-view scene cues are used as inputs to the depth decoder. The depth decoder uses adaptive weight values ​​to determine the contribution value of each layer of the super-resolution hidden coding to obtain the depth information.

[0099] The formula for the depth information is:

[0100]

[0101] in,

[0102] D represents the depth information;

[0103] h represents the depth decoder;

[0104] α i This represents the fusion weights corresponding to the implicit coding of each layer;

[0105] r i Indicates the implicit coding of each layer;

[0106] n represents the number of implicit coding layers;

[0107] β represents the fusion weight of the multi-view scene clues;

[0108] τ represents the multi-perspective scene clues.

[0109] In the embodiments of this specification, the depth decoder is an absolute depth recovery decoder, and the depth decoder is composed of: a 1*1 convolutional layer, a 3*3 deconvolutional layer, a 3*3 convolutional layer, and a Sigmoid activation function.

[0110] Continuing from the previous example, in the embodiments of this specification, the number of hidden coding layers n is 3.

[0111] This embodiment integrates multi-view scene cues and super-resolution hidden coding, which can significantly improve the accuracy of depth prediction and make full use of feature information from different sources, thereby ensuring the comprehensive capture of global and local details of the scene, and thus providing accuracy and reliability for subsequent 3D reconstruction steps.

[0112] Step S211: Perform three-dimensional reconstruction based on the depth information to generate a monocular target image.

[0113] In the embodiments described in this specification, 3D reconstruction is performed using a differentiable Marching Cube. The differentiable Marching Cube algorithm transforms depth information into 3D geometry, generating a monocular target image.

[0114] Based on depth information, 3D reconstruction is performed using the differentiable Marching Cube algorithm. On the one hand, the differentiability property ensures that the gradient propagates effectively throughout the reconstruction process, making the geometric transformation smoother and avoiding distortion or discontinuity problems that may occur in traditional methods. On the other hand, by combining the continuity of depth information with surface features, the algorithm accurately captures the minute changes and details of objects to ensure that high-precision geometric shapes can be generated, providing a more realistic visual experience.

[0115] In the embodiments described in this specification, the formula for three-dimensional reconstruction is:

[0116]

[0117] in,

[0118] O represents the generated three-dimensional geometry, which in this specification is specifically a monocular target image;

[0119] MC stands for Differentiable Marching Cube algorithm;

[0120] Represents the depth gradient;

[0121] k represents the surface normal vector;

[0122] Ω represents the integration field;

[0123] ∈ represents the curvature adjustment factor;

[0124] k(D) represents the curvature function;

[0125] λ represents the regularization parameter;

[0126] dA represents the integral;

[0127] j represents the number of shape functions, j = 1 to m;

[0128] m represents the number of accumulated shape functions;

[0129] p j Represents the weights of the shape function;

[0130] q j Represents the shape function.

[0131] To further understand the depth estimation-based monocular object 3D reconstruction method provided in this embodiment, the framework of the depth estimation-based monocular object 3D reconstruction method will be explained below.

[0132] Figure 3 This is a framework diagram of a monocular object 3D reconstruction method based on depth estimation, provided for embodiments of this specification. Figure 3 As shown, in the multi-view scene cue and object segmentation module, the multi-view scene image (i.e., the multi-view image to be processed) is segmented into scene elements by the SAM2 model to generate multi-view object images and scene cues; in the consistency constraint-based super-resolution implicit coding prediction module, the multi-view object images are then combined by a preset visual large model to generate consistency features; in the object deconstruction-based super-resolution implicit coding prediction module, scene cues are input into the object deconstruction Transformer to generate deconstruction features; the deconstruction features and consistency features are jointly input into the diffusion model Unet to generate super-resolution implicit coding; finally, in the multi-level depth prediction and 3D reconstruction module based on scene cues, depth prediction is performed based on super-resolution implicit coding and scene cues, and depth information is obtained through a depth decoder; finally, a differentiable Marching Cube (Neural Marching Cube) is used to perform 3D reconstruction of the depth to generate the reconstructed object.

[0133] The monocular object 3D reconstruction method based on depth estimation provided in the embodiments of this specification utilizes the prior knowledge of a large visual model to perform multi-view scene cue acquisition and object segmentation, super-resolution implicit coding prediction, multi-level depth prediction, and 3D reconstruction. By integrating the powerful representation capabilities of the large model, this invention achieves high-precision and separable 3D reconstruction, significantly improving the accuracy and robustness of the reconstruction.

[0134] The above describes in detail a monocular object 3D reconstruction method based on depth estimation. Correspondingly, this specification also provides a monocular object 3D reconstruction device based on depth estimation, such as... Figure 4 As shown. Figure 4 This is a schematic diagram of a monocular object 3D reconstruction device based on depth estimation, provided as an embodiment of this specification. The monocular object 3D reconstruction device includes:

[0135] Module 401 acquires the multi-view images to be processed;

[0136] The segmentation module 403, based on the image segmentation model, segments the multi-view image to be processed to obtain multi-view scene clues and multi-view object images;

[0137] The intermediate processing module 405 obtains consistency features based on the multi-view object images by combining preset visual large models, and obtains deconstruction features based on the multi-view scene cues by using a preset Transformer model.

[0138] Prediction module 407: The consistency feature and the deconstruction feature are fused and reconstructed using a diffusion Unet model to obtain a super-resolution hidden code;

[0139] The depth estimation module 409 performs multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information.

[0140] The 3D reconstruction module 411 performs 3D reconstruction based on the depth information to generate a monocular target image.

[0141] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0142] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0143] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0144] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0147] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A monocular object 3D reconstruction method based on depth estimation, characterized in that, The monocular object 3D reconstruction method includes: Acquire multi-view images to be processed; Based on an image segmentation model, the multi-view image to be processed is segmented to obtain multi-view scene cues and multi-view object images. The multi-view scene cues are object-level tokens, and each element's features, location, and bounding box information is included. The multi-view scene cues are obtained based on the SAM2 model, and the formula for the multi-view scene cues is: ;in, This indicates the multi-perspective scene clues; This refers to the SAM2 model; This indicates that the input is the multi-view image to be processed; Based on the multi-view object images, consistency features are obtained through a pre-defined large visual model combination, and deconstruction features are obtained through a pre-defined Transformer model based on the multi-view scene cues. The consistency features include: semantic consistency features, attention consistency features, and descriptive consistency features. The acquisition of these consistency features is as follows: ;in, The consistency feature is represented by f; the consistency feature extraction function is represented by M; and the preset visual large model combination is represented by M. The multi-view object image is represented by the preset large visual model combination, which consists of DINOV2, Stable Diffusion, and BLIP. DINOV2 is used to extract semantic features from the multi-view object image to obtain semantic consistency features; Stable Diffusion is used to enhance texture features in the multi-view object image to obtain attention consistency features; BLIP is used to extract structural features from the multi-view object image to obtain descriptive consistency features; the deconstruction features include normal deconstruction features, contour deconstruction features, and corner deconstruction features; the formula for the deconstruction features is: ;in, This represents the deconstruction feature; The attention mechanism function is represented by T; T represents the preset Transformer model. This indicates the multi-perspective scene clues; The consistency feature and the deconstruction feature are fused and reconstructed using the diffusion Unet model to obtain the super-resolution hidden code; Multi-level depth prediction is performed based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information. Specifically, this includes: using the hidden codes of each layer of the super-resolution hidden coding and the multi-view scene cues as input to the depth decoder; the depth decoder uses adaptive weight values ​​to determine the contribution value of each layer of the super-resolution hidden coding to obtain the depth information; the formula for the depth information is: Where D represents the depth information; h represents the depth decoder; This represents the fusion weights corresponding to the implicit coding of each layer; This represents the implicit coding at each layer; n represents the number of implicit coding layers. This represents the fusion weight of the multi-view scene clues; This indicates the multi-perspective scene clues; Based on the depth information, a 3D reconstruction is performed to generate a monocular target image. The formula for the 3D reconstruction is: Where O represents the generated 3D geometry, specifically a monocular target image; MC represents the differentiable Marching Cube algorithm; The depth gradient is represented by k; the surface normal vector is represented by k. Represent the integration field; denoted by , where k(D) represents the curvature adjustment factor; k(D) represents the curvature function. represents the regularization parameter; dA represents the integral; j represents the number of shape functions, j=1~m; m represents the number of accumulated shape functions; Represents the weights of the shape function; Represents the shape function.

2. The monocular object 3D reconstruction method as described in claim 1, characterized in that, The preset visual large model combination includes three visual large models connected in parallel. The preset visual large models include: a visual large model for semantic feature extraction, a visual large model for texture feature extraction, and a visual large model for structural feature extraction.

3. The monocular object 3D reconstruction method as described in claim 1, characterized in that, The preset Transformer model includes a self-attention feedback layer, residual connections and layer normalization, a feedforward neural network, residual connections and layer normalization.

4. The monocular object 3D reconstruction method as described in claim 1, characterized in that, The consistency features and the deconstruction features are fused and reconstructed using the diffusion Unet model to obtain the super-resolution hidden code, specifically including: Using the consistency feature and the deconstruction feature as inputs, the three-layer implicit coding is obtained by inputting them into the diffusion Unet model; The three-layer implicit coding is fused using an attention mechanism to obtain the super-resolution implicit coding; The formula for the super-resolution hidden coding is: in, This indicates super-resolution hidden coding; This represents the diffusion Unet model; This represents the consistency feature; This represents the deconstruction feature.

5. The monocular object 3D reconstruction method as described in claim 1, characterized in that, The depth decoder consists of, in sequence: a 1*1 convolutional layer, a 3*3 deconvolutional layer, a 3*3 convolutional layer, and a Sigmoid activation function.

6. A monocular object 3D reconstruction device based on depth estimation, characterized in that, The monocular object 3D reconstruction device includes: The acquisition module acquires multi-view images to be processed; The segmentation module, based on an image segmentation model, segments the multi-view image to be processed, obtaining multi-view scene cues and multi-view object images. The multi-view scene cues are object-level tokens, including the features, location, and bounding box information of each element. The multi-view scene cues are obtained based on the SAM2 model, and the formula for the multi-view scene cues is: ;in, This indicates the multi-perspective scene clues; This refers to the SAM2 model; This indicates that the input is the multi-view image to be processed; The intermediate processing module, based on the multi-view object images, obtains consistency features through a preset large visual model combination, and obtains deconstruction features through a preset Transformer model based on the multi-view scene cues. The consistency features include: semantic consistency features, attention consistency features, and descriptive consistency features; the acquisition of the consistency features is as follows: ;in, The consistency feature is represented by f; the consistency feature extraction function is represented by M; and the preset visual large model combination is represented by M. The multi-view object image is represented by the preset large visual model combination, which consists of DINOV2, Stable Diffusion, and BLIP. DINOV2 is used to extract semantic features from the multi-view object image to obtain semantic consistency features; Stable Diffusion is used to enhance texture features in the multi-view object image to obtain attention consistency features; BLIP is used to extract structural features from the multi-view object image to obtain descriptive consistency features; the deconstruction features include normal deconstruction features, contour deconstruction features, and corner deconstruction features; the formula for the deconstruction features is: ;in, This represents the deconstruction feature; The attention mechanism function is represented by T; T represents the preset Transformer model. This indicates the multi-perspective scene clues; In the prediction module, the consistency features and the deconstruction features are fused and reconstructed using a diffusion Unet model to obtain a super-resolution hidden code. The depth estimation module performs multi-level depth prediction based on the multi-view scene cues and the super-resolution hidden coding to obtain depth information. Specifically, it includes: using the hidden codes of each layer of the super-resolution hidden coding and the multi-view scene cues as input to the depth decoder; the depth decoder uses adaptive weight values ​​to determine the contribution values ​​of each layer of the super-resolution hidden coding to obtain the depth information; the formula for the depth information is: Where D represents the depth information; h represents the depth decoder; This represents the fusion weights corresponding to the implicit coding of each layer; This represents the implicit coding at each layer; n represents the number of implicit coding layers. This represents the fusion weight of the multi-view scene clues; This indicates the multi-perspective scene clues; The 3D reconstruction module performs 3D reconstruction based on the depth information to generate a monocular target image. The formula for the 3D reconstruction is: Where O represents the generated 3D geometry, specifically a monocular target image; MC represents the differentiable Marching Cube algorithm; The depth gradient is represented by k; the surface normal vector is represented by k. Represent the integration field; denoted by , where k(D) represents the curvature adjustment factor; k(D) represents the curvature function. represents the regularization parameter; dA represents the integral; j represents the number of shape functions, j=1~m; m represents the number of accumulated shape functions; Represents the weights of the shape function; Represents the shape function.

Images