Depth estimation-based monocular object three-dimensional reconstruction method and device

Abstract

The embodiment of the invention discloses a monocular object three-dimensional reconstruction method and device based on depth estimation. The monocular object three-dimensional reconstruction method comprises the steps that a to-be-processed multi-view image is acquired; based on an image segmentation model, segmenting the to-be-processed multi-view image to obtain a multi-view scene clue and a multi-view object image; based on the multi-view object image, obtaining a consistency feature through a preset visual large model combination, and based on the multi-view scene clue, obtaining a deconstruction feature through a preset Transform model; carrying out fusion reconstruction on the consistency feature and the deconstruction feature through a diffusion Unet model to obtain a super-division hidden code; performing multi-level depth prediction based on the multi-view scene clues and the super-division implicit coding to obtain depth information; and performing three-dimensional reconstruction based on the depth information to generate a monocular target image.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology and image recognition technology, and in particular to a method and device for three-dimensional reconstruction of a monocular object based on depth estimation. Background Art

[0002] Multi-view object reconstruction, as an important research direction in the field of computer vision, provides a technical means for the three-dimensional reconstruction of target objects.

[0003] Traditional multi-view stereo methods (MVS) achieve 3D object reconstruction through modular processes such as feature matching, depth prediction and fusion. However, the breakthroughs in Neural View Synthesis (NVS) in recent years, especially the introduction of Neural Radiance Fields (NeRF), have greatly promoted the development of this field. NeRF models geometric shapes and view-dependent appearances through multi-layer perceptrons, and optimizes them through volume rendering, resulting in excellent rendering quality. With the development of computer vision technology, multi-view object 3D reconstruction still faces many problems. The lack of sophisticated depth estimation and mesh generation strategies in the reconstruction process leads to the inability to capture tiny details when dealing with complex scenes or object surfaces, resulting in low resolution and insufficient detail depiction; due to excessive emphasis on rendering effects, the 3D reconstruction results are distorted, and the reconstruction results have defects in geometric consistency; at the same time, due to the lack of consideration of the problem of multi-view consistency representation, the integration of multi-view information is poor in accuracy, resulting in a decrease in the overall quality of 3D reconstruction.

[0004] Based on this, this specification provides a method and device for three-dimensional reconstruction of a monocular object based on depth estimation. Summary of the invention

[0005] The embodiments of the present specification provide a method and device for three-dimensional reconstruction of monocular objects based on depth estimation, which are used to solve the following technical problems: In the prior art, there is a lack of sophisticated depth estimation and mesh generation strategies in the reconstruction process, which results in the inability to capture tiny details when processing complex scenes or object surfaces, and thus leads to low resolution and insufficient detail portrayal; due to excessive emphasis on rendering effects, the three-dimensional reconstruction results are distorted, and the reconstruction results have defects in geometric consistency; at the same time, due to the failure to consider the problem of multi-view consistency representation, the accuracy of the integration of multi-view information is poor, resulting in a decrease in the overall quality of the three-dimensional reconstruction.

[0006] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0007] The embodiment of this specification provides a method for three-dimensional reconstruction of a monocular object based on depth estimation, including:

[0008] Acquire multi-view images to be processed;

[0009] Based on the image segmentation model, the multi-view image to be processed is segmented to obtain multi-view scene clues and multi-view object images;

[0010] Based on the multi-view object images, consistent features are obtained by combining a preset large visual model, and based on the multi-view scene clues, deconstructed features are obtained by using a preset Transformer model;

[0011] The consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code;

[0012] Perform multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information;

[0013] Three-dimensional reconstruction is performed based on the depth information to generate a monocular target image.

[0014] The embodiment of the present specification also provides a monocular object three-dimensional reconstruction device based on depth estimation, comprising:

[0015] An acquisition module, for acquiring multi-view images to be processed;

[0016] A segmentation module, based on an image segmentation model, segments the multi-view image to be processed to obtain multi-view scene clues and multi-view object images;

[0017] An intermediate processing module, based on the multi-view object image, obtains consistency features through a preset large visual model combination, and based on the multi-view scene clues, obtains deconstruction features through a preset Transformer model;

[0018] In the prediction module, the consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code;

[0019] A depth estimation module performs multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information;

[0020] The three-dimensional reconstruction module performs three-dimensional reconstruction based on the depth information to generate a monocular target image.

[0021] The embodiment of the present specification provides a method for three-dimensional reconstruction of a monocular object based on depth estimation, which obtains a multi-view image to be processed; based on an image segmentation model, the multi-view image to be processed is segmented to obtain multi-view scene clues and multi-view object images; based on the multi-view object image, consistency features are obtained through a preset large visual model combination, and based on the multi-view scene clues, deconstruction features are obtained through a preset Transformer model; the consistency features and the deconstruction features are fused and reconstructed through a diffusion Unet model to obtain super-resolution latent codes; multi-level depth prediction is performed based on the multi-view scene clues and the super-resolution latent codes to obtain depth information; three-dimensional reconstruction is performed based on the depth information to generate a monocular target image, which can achieve high-precision and separable three-dimensional reconstruction, and significantly improve the accuracy and robustness of reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 A schematic diagram of the system architecture of a method for three-dimensional reconstruction of a monocular object based on depth estimation provided in an embodiment of this specification;

[0024] Figure 2 A schematic diagram of a flow chart of a method for three-dimensional reconstruction of a monocular object based on depth estimation provided in an embodiment of this specification;

[0025] Figure 3 A framework diagram of a method for 3D reconstruction of a monocular object based on depth estimation provided in an embodiment of this specification;

[0026] Figure 4 A schematic diagram of a monocular object 3D reconstruction device based on depth estimation provided in an embodiment of this specification. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0028] In terms of monocular object depth estimation, Lee predicted object meshes and rendered depth maps based on Mesh-RCNN, while Old-net directly predicted depth from monocular RGB images and designed novel SDDR and NGPH, verifying the importance of using shape priors to better cope with intra-class changes.

[0029] Multi-view 3D reconstruction methods include point cloud, mesh, voxel and implicit representation, each with its own advantages. Point cloud-based reconstruction methods such as Lin and Wei significantly improve reconstruction accuracy by generating dense point clouds and aggregating multiple views. In terms of mesh representation, Pixel2Mesh++ generates multi-view images by deformation. Figure 3 D mesh, FOUND optimizes surface deformation to improve model accuracy. Voxel representation such as Multi-view 3D Reconstruction with Transformers uses Transformers technology to improve voxel representation efficiency, while 3D-C2FT improves reconstruction details through hierarchical optimization. Implicit representation methods such as FvOR combine shape and posture optimization to achieve robust reconstruction under conditions of few viewing angles, and GaussianObject reduces the required input viewing angles through Gaussian initialization.

[0030] Existing 3D reconstruction methods for monocular objects generally have problems such as low resolution, geometric inconsistency, and poor multi-view integration in multi-view reconstruction.

[0031] Based on this, the embodiments of this specification provide a method for three-dimensional reconstruction of monocular objects based on depth estimation, which ensures the independence and clear expression of each object by accurately segmenting the target object elements in complex scenes; combines the scene clues and object structural features extracted by the large model to ensure the consistency of representation under multiple perspectives and provide support for depth estimation; uses the diffusion model to perform depth estimation and restore the absolute depth at the scene level to ensure the accurate position and size of the object in the three-dimensional space; finally, converts the depth information into a three-dimensional mesh to achieve complete landscape reconstruction and improve the detail performance of monocular object reconstruction.

[0032] Figure 1 A schematic diagram of the system architecture of a method for 3D reconstruction of a monocular object based on depth estimation provided in an embodiment of this specification. Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0033] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various client applications may be installed on the terminal devices 101, 102, 103. For example, a dedicated program such as a method for performing a monocular 3D reconstruction of a depth-estimation-based object may be installed.

[0034] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various dedicated or general electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or they may be implemented as a single software or software module.

[0035] The server 105 may be a server that provides various services, such as a backend server that provides services for client applications installed on the terminal devices 101, 102, and 103. For example, the server may perform a three-dimensional reconstruction of a monocular object based on depth estimation, so as to display the three-dimensional reconstruction result of the monocular object based on depth estimation on the terminal device servers 101, 102, and 103.

[0036] The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 105 is software, it may be implemented as multiple software or software modules (e.g., multiple software or software modules for providing distributed services), or as a single software or software module.

[0037] Figure 2 A flowchart of a method for 3D reconstruction of a monocular object based on depth estimation provided in an embodiment of this specification. From a program perspective, the execution subject of the process can be a program installed on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the method includes:

[0038] Step S201: Acquire multi-view images to be processed.

[0039] In the embodiment of the present specification, the multi-view image to be processed includes at least three view angles, and the multi-view image to be processed is an image containing a target object. The target object is an object that needs to be three-dimensionally reconstructed. It can be understood that the multi-view image to be processed includes the target image and the background image. It can be understood that in the multi-view image to be processed, images other than the target image are all background images.

[0040] In the embodiment of the present specification, the multi-view images may be obtained by shooting from different angles using a camera.

[0041] Step S203: Based on the image segmentation model, the multi-view image to be processed is segmented to obtain multi-view scene clues and multi-view object images.

[0042] In the embodiment of this specification, the multi-view scene clue is an object-level token, and the multi-view scene clue includes the characteristics, position and detection frame information of each element;

[0043] The multi-view scene clues are obtained based on the SAM2 model, and the formula of the multi-view scene clues is:

[0044] τ=S(I)

[0045] in,

[0046] τ represents the multi-view scene clues;

[0047] S represents the SAM2 model;

[0048] I represents the input multi-view image to be processed.

[0049] An object-level token refers to a token that represents a specific object or a significant area in an image. In the embodiments of this specification, an object-level token is a data unit of feature information of a target image. Specifically, it includes features, positions, and detection box information of each element in the target image. Furthermore, the detection box information includes: the position, size, and semantics of the detection box.

[0050] Object-level tokens of multi-view scene cues improve the reconstruction accuracy of the model and ensure the capture of tiny details in the scene, ultimately making the 3D reconstruction results more accurate and reliable.

[0051] The SAM2 model is a video and image object segmentation model. The specific structure of the model is prior art and will not be described in detail here.

[0052] Step S205: Based on the multi-view object image, consistent features are obtained through a preset large visual model combination, and based on the multi-view scene clues, deconstruction features are obtained through a preset Transformer model.

[0053] In the embodiment of this specification, the preset visual large model combination includes three visual large models, the three visual large models are connected in parallel, and the preset visual large models include: a visual large model for semantic feature extraction, a visual large model for texture feature extraction, and a visual large model for structural feature extraction;

[0054] The consistency features include: semantic consistency features, attention consistency features and description consistency features.

[0055] In the embodiments of this specification, the formula of the consistency feature is:

[0056]

[0057] in,

[0058] Indicates said consistency feature;

[0059] f represents the consistency feature extraction function;

[0060] M represents the preset large visual model combination;

[0061] I′ represents the multi-view object image;

[0062] The preset large visual model combination consists of: DINOV2, Stable Diffusion and BLIP in parallel, wherein the DINOV2 is used to extract semantic features in the multi-view object image and obtain the semantic consistency feature; the Stable Diffusion is used to enhance the texture features in the multi-view object image and obtain the attention consistency feature; the BLIP is used to extract structural features in the multi-view object image and obtain the description consistency feature.

[0063] In the embodiments of this specification, the semantic consistency feature refers to the semantic similarity or consistency of the image content captured at different viewing angles. In this embodiment, "semantics" mainly refers to the content in the image, including but not limited to information about objects, scenes, textures, and structures.

[0064] The attention consistency feature refers to the attention-related features in the image extracted by a specific visual model, ensuring the consistency of these features under different viewing angles. In this embodiment, the attention consistency feature is mainly realized by the texture feature of the image.

[0065] The description of consistent features refers to the extracted structural information, specifically, mainly the spatial layout, shape and other information of the object.

[0066] In the embodiments of this specification, the consistency feature is extracted by using the multi-view object image as the input of the preset visual macro model through the consistency feature extraction function. Further, the consistency feature extraction function is a function for extracting corresponding features in the preset visual macro model, that is, the function or module for extracting consistency features in the preset visual macro model is used as the consistency feature extraction function, or the function for realizing consistency feature extraction in the preset visual macro model.

[0067] By obtaining consistent features, we can ensure the stability and consistency of features under multiple perspectives. By integrating the outputs of different models, we can enhance the robustness of features and effectively reduce the inconsistency caused by perspective changes.

[0068] Deconstructed features refer to features with specific geometric meanings extracted from images or 3D models, such as normals, contours, and corners. These features are crucial for understanding the geometric shape and structure of objects.

[0069] In the embodiments of this specification, the deconstruction features include: normal deconstruction features, contour deconstruction features and corner deconstruction features;

[0070] The formula for the deconstructed feature is:

[0071] Ψ=g(T,τ)

[0072] in,

[0073] Ψ represents the deconstruction feature;

[0074] g represents the attention mechanism function;

[0075] T represents the preset Transformer model;

[0076] τ represents the multi-view scene cue.

[0077] In the embodiments of the present specification, the attention mechanism function is used to use multi-view scene clues as the input of a preset Transformer model, and to extract deconstruction features through a self-attention function.

[0078] In the embodiment of the present specification, the preset Transformer model includes a layer of self-attention feedback, residual connection and layer normalization, a feedforward neural network, residual connection and layer normalization.

[0079] In the embodiments of this specification, in the normal deconstruction feature, the normal information reflects the direction of a point on the surface of an object, which is very important for the lighting calculation and understanding of the surface properties of the object.

[0080] In the contour deconstruction feature, the contour information defines the boundary of the object and is very useful for understanding the shape and boundary of the object.

[0081] In the corner deconstruction feature, the corner point is a special point on the surface of an object, which usually appears on the edge or corner of the object and is an important part of the geometric structure of the object.

[0082] By deconstructing features, more detailed and accurate feature information can be extracted, thereby achieving more accurate three-dimensional reconstruction.

[0083] Step S207: The consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code.

[0084] After obtaining the consistency features and deconstruction features, they are further used as input and input into the diffusion Unet model for effective fusion to improve the fineness and accuracy of the encoding.

[0085] In the embodiment of this specification, the consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution implicit coding, which specifically includes:

[0086] Taking the consistency feature and the deconstruction feature as input, inputting them into the diffusion Unet model to obtain a three-layer latent code;

[0087] The three layers of hidden codes are fused through an attention mechanism to obtain the super-resolution hidden code;

[0088] The formula of the super-resolution implicit coding is:

[0089]

[0090] in,

[0091] r represents super-resolution hidden coding;

[0092] u represents the diffusion Unet model;

[0093] Indicates said consistency feature;

[0094] Ψ represents the deconstruction feature.

[0095] In the embodiment of this specification, the diffusion Unet model may be a SDXL model.

[0096] Step S209: performing multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information.

[0097] In the embodiment of this specification, the multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain the depth information specifically includes:

[0098] Taking the hidden codes of each layer of the super-resolution hidden code and the multi-view scene clues as inputs of a depth decoder, and the depth decoder using an adaptive weight value to determine a contribution value of each layer of the hidden code of the super-resolution hidden code to obtain the depth information;

[0099] The formula for the depth information is:

[0100]

[0101] in,

[0102] D represents the depth information;

[0103] h represents the depth decoder;

[0104] α i Represents the fusion weights corresponding to the hidden codes of each layer;

[0105] r i Represents the hidden code of each layer;

[0106] n represents the number of hidden coding layers;

[0107] β represents the fusion weight of the multi-view scene cues;

[0108] τ represents the multi-view scene cue.

[0109] In the embodiment of the present specification, the depth decoder is an absolute depth recovery decoder, and the composition of the depth decoder includes: a 1*1 convolution layer, a 3*3 deconvolution layer, a 3*3 convolution layer and a Sigmoid activation function.

[0110] Continuing with the previous example, in the embodiment of this specification, the number of implicit coding layers n is 3.

[0111] This embodiment integrates multi-view scene cues and super-resolution latent coding, which can significantly improve the accuracy of depth prediction and make full use of feature information from different sources to ensure comprehensive capture of the global and local details of the scene, thereby providing accuracy and reliability for subsequent three-dimensional reconstruction.

[0112] Step S211: performing three-dimensional reconstruction based on the depth information to generate a monocular target image.

[0113] In the embodiments of this specification, the three-dimensional reconstruction is performed using a differentiable Marching Cube. The depth information is converted into a three-dimensional geometric shape through a differentiable Marching Cube algorithm to generate a monocular target image.

[0114] Based on depth information, three-dimensional reconstruction is performed using the differentiable Marching Cube algorithm. On the one hand, the differentiable characteristics are used to ensure that gradients are effectively propagated throughout the entire reconstruction process, making geometric transformations smoother and avoiding distortion or discontinuity problems that may occur in traditional methods. On the other hand, by combining the continuity of depth information with surface features, the subtle changes and details of objects can be accurately captured to ensure that high-precision geometric shapes can be generated, providing a more realistic visual experience.

[0115] In the embodiment of this specification, the formula for three-dimensional reconstruction is:

[0116]

[0117] in,

[0118] O represents the generated three-dimensional geometric shape, which in this specification is specifically a monocular target image;

[0119] MC stands for the differentiable Marching Cube algorithm;

[0120] represents the depth gradient;

[0121] k represents the surface normal vector;

[0122] Ω represents the integration domain;

[0123] ∈ represents the curvature adjustment factor;

[0124] k(D) represents the curvature function;

[0125] λ represents the regularization parameter;

[0126] dA represents integration;

[0127] j represents the number of shape functions, j = 1 to m;

[0128] m represents the number of accumulated shape functions;

[0129] p j represents the weight of the shape function;

[0130] q j Represents the shape function.

[0131] In order to further understand the monocular object 3D reconstruction method based on depth estimation provided by the present illustrative embodiment, the framework of the monocular object 3D reconstruction method based on depth estimation will be described below.

[0132] Figure 3 This is a framework diagram of a method for 3D reconstruction of a monocular object based on depth estimation provided in an embodiment of this specification. Figure 3 As shown in the figure, in the multi-view scene clue and object segmentation module, the multi-view scene image (i.e., the multi-view image to be processed) is segmented into scene elements by the SAM2 model to generate a multi-view object image and scene clues; in the super-resolution latent coding prediction module based on consistency constraints, the multi-view object image is then combined with a preset large visual model to generate a consistent feature; in the super-resolution latent coding prediction module based on object deconstruction, the scene clue is input into the object deconstruction Transformer to generate a deconstruction feature; the deconstruction feature and the consistency feature are jointly input into the diffusion model Unet to generate a super-resolution latent code; finally, in the multi-level depth prediction and 3D reconstruction module based on scene clues, depth prediction is performed based on the super-resolution latent code and scene clues, and depth information is obtained through a depth decoder; finally, the differentiable Marching Cube (Neural Marching Cube) performs 3D reconstruction of the depth to generate a reconstructed object.

[0133] The monocular object 3D reconstruction method based on depth estimation provided in the embodiments of this specification uses the prior knowledge of the large visual model to obtain multi-view scene clues and object segmentation, super-resolution latent coding prediction, multi-level depth prediction and 3D reconstruction. By integrating the powerful representation ability of the large model, the invention achieves high-precision and separable 3D reconstruction, significantly improving the accuracy and robustness of reconstruction.

[0134] The above content describes in detail a method for 3D reconstruction of a monocular object based on depth estimation. Correspondingly, this specification also provides a device for 3D reconstruction of a monocular object based on depth estimation, such as Figure 4 shown. Figure 4 A schematic diagram of a monocular object 3D reconstruction device based on depth estimation provided in an embodiment of this specification, the monocular object 3D reconstruction device includes:

[0135] An acquisition module 401 is used to acquire a multi-view image to be processed;

[0136] The segmentation module 403 segments the multi-view image to be processed based on the image segmentation model to obtain multi-view scene clues and multi-view object images;

[0137] The intermediate processing module 405 obtains consistency features based on the multi-view object images through a preset large visual model combination, and obtains deconstruction features based on the multi-view scene clues through a preset Transformer model;

[0138] Prediction module 407, the consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code;

[0139] A depth estimation module 409 performs multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information;

[0140] The three-dimensional reconstruction module 411 performs three-dimensional reconstruction based on the depth information to generate a monocular target image.

[0141] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0143] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0144] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0145] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0146] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0147] The above is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for three-dimensional reconstruction of a monocular object based on depth estimation, characterized in that: The monocular object three-dimensional reconstruction method comprises: Acquire multi-view images to be processed; Based on the image segmentation model, the multi-view image to be processed is segmented to obtain multi-view scene clues and multi-view object images; Based on the multi-view object images, consistent features are obtained by combining a preset large visual model, and based on the multi-view scene clues, deconstructed features are obtained by using a preset Transformer model; The consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code; Perform multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information; Three-dimensional reconstruction is performed based on the depth information to generate a monocular target image.

2. The monocular object 3D reconstruction method according to claim 1, characterized in that: The multi-view scene clues are object-level tokens, and the multi-view scene clues include the features, position and detection frame information of each element; The multi-view scene clues are obtained based on the SAM2 model, and the formula of the multi-view scene clues is: τ=S(I) in, τ represents the multi-view scene clues; S represents the SAM2 model; I represents the input multi-view image to be processed.

3. The monocular object 3D reconstruction method according to claim 1, characterized in that: The preset visual large model combination includes three visual large models, the three visual large models are connected in parallel, and the preset visual large models include: a visual large model for semantic feature extraction, a visual large model for texture feature extraction, and a visual large model for structural feature extraction; The consistency features include: semantic consistency features, attention consistency features and description consistency features.

4. The monocular object 3D reconstruction method according to claim 3, characterized in that: The acquisition of the consistency feature is as follows: in, Indicates said consistency feature; f represents the consistency feature extraction function; M represents the preset large visual model combination; I′ represents the multi-view object image; The preset large visual model combination consists of: DINOV2, Stable Diffusion and BLIP in parallel, wherein the DINOV2 is used to extract semantic features in the multi-view object image and obtain the semantic consistency features; the Stable Diffusion is used to enhance the texture features in the multi-view object image and obtain the attention consistency features; the BLIP is used to extract structural features in the multi-view object image and obtain the description consistency features.

5. The monocular object 3D reconstruction method according to claim 1, characterized in that: The deconstruction features include: normal deconstruction features, contour deconstruction features and corner deconstruction features; The formula for the deconstructed feature is: Ψ=g(T,τ) in, Ψ represents the deconstruction feature; g represents the attention mechanism function; T represents the preset Transformer model; τ represents the multi-view scene cue.

6. The monocular object 3D reconstruction method according to claim 1, characterized in that: The preset Transformer model includes a layer of self-attention feedback, residual connection and layer normalization, a feedforward neural network, residual connection and layer normalization.

7. The monocular object 3D reconstruction method according to claim 1, characterized in that: The consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code, which specifically includes: Taking the consistency feature and the deconstruction feature as input, inputting them into the diffusion Unet model to obtain a three-layer latent code; The three layers of hidden codes are fused through an attention mechanism to obtain the super-resolution hidden code; The formula of the super-resolution implicit coding is: in, r represents super-resolution hidden coding; u represents the diffusion Unet model; Indicates said consistency feature; Ψ represents the deconstruction feature.

8. The monocular object 3D reconstruction method according to claim 1, characterized in that: The performing multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information specifically includes: Taking the hidden codes of each layer of the super-resolution hidden code and the multi-view scene clues as inputs of a depth decoder, and the depth decoder using an adaptive weight value to determine a contribution value of each layer of the hidden code of the super-resolution hidden code to obtain the depth information; The formula for the depth information is: in, D represents the depth information; h represents the depth decoder; α i Represents the fusion weights corresponding to the hidden codes of each layer; r i Represents the hidden code of each layer; n represents the number of hidden coding layers; β represents the fusion weight of the multi-view scene cues; τ represents the multi-view scene cue.

9. The monocular object 3D reconstruction method according to claim 8, characterized in that: The deep decoder is composed of: a 1*1 convolutional layer, a 3*3 deconvolutional layer, a 3*3 convolutional layer and a Sigmoid activation function.

10. A monocular object 3D reconstruction device based on depth estimation, characterized in that: The monocular object three-dimensional reconstruction device comprises: An acquisition module, for acquiring multi-view images to be processed; A segmentation module, based on an image segmentation model, segments the multi-view image to be processed to obtain multi-view scene clues and multi-view object images; An intermediate processing module, based on the multi-view object images, obtains consistency features through a preset large visual model combination, and based on the multi-view scene clues, obtains deconstruction features through a preset Transformer model; In the prediction module, the consistency feature and the deconstruction feature are fused and reconstructed through a diffusion Unet model to obtain a super-resolution latent code; A depth estimation module performs multi-level depth prediction based on the multi-view scene clues and the super-resolution implicit coding to obtain depth information; The three-dimensional reconstruction module performs three-dimensional reconstruction based on the depth information to generate a monocular target image.

Images