Three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction

Abstract

The application discloses a three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction, interactive extraction of semantic segmentation masks and target images of target objects based on a pre-trained SAM segmentation model, introduction of SfM algorithm to calculate camera pose information and sparse structure information of multi-view scene graphs, input of the target images, the segmentation masks and the camera pose information into a grid encoder to train to obtain a rough polygonal grid and texture features, reconstruction of a rough polygonal grid and a texture graph of the target object, and output of a three-dimensional reconstruction result of the target object after fine-grained rough polygonal grid and texture graph. The application constructs an interactive segmentation method through a pre-trained large model, avoids invalid occupation of a background area. An implicit surface reconstruction algorithm based on a neural radiation field is used to construct fine-grained grid and texture information of the object to be reconstructed, so that three-dimensional reconstruction of the target object can be quickly and accurately completed in a complex environment, and the application has good value and promotion prospect.

Description

Technical Field

[0001] This invention relates to a method for reconstructing three-dimensional objects based on interactive segmentation priors and implicit surface reconstruction, belonging to the fields of computer vision and three-dimensional computer graphics. Background Technology

[0002] 3D object reconstruction aims to recover the 3D structure of objects and the environment in a scene by extracting depth information from 2D images. Traditional reconstruction methods typically rely on techniques such as stereo vision, beam normals, and multi-view geometry to infer point clouds in 3D space by matching images or videos from multiple perspectives. However, these methods often struggle to guarantee reconstruction accuracy and computational efficiency when faced with complex scenes, especially those with background interference, object occlusion, or low-texture areas. Therefore, improving the accuracy, speed, and robustness of 3D reconstruction in practical applications remains a pressing issue.

[0003] Neural Radiation Field (NeRF) technology has made significant progress in the field of 3D object representation and reconstruction in recent years. NeRF uses deep learning to represent 3D objects through implicit functions, accurately capturing complex geometric structures and lighting effects, especially excelling in novel view composition and high-quality rendering tasks. However, the 3D object representation learned by NeRF is implicit, typically requiring traditional methods such as volume rendering to query depth information from the implicit field and then convert it into structural information. Therefore, NeRF technology cannot directly learn accurate mesh models. NeRF technology also lacks editable texture information, making its output object models incompatible with traditional graphics rendering pipelines. Furthermore, NeRF technology requires substantial computational resources for object reconstruction, so performance remains an unresolved issue when handling large-scale data.

[0004] Furthermore, in 3D object reconstruction, background interference and occlusion by irrelevant objects are key factors affecting reconstruction quality and efficiency. Background regions and irrelevant objects consume significant computational resources, leading to inefficient and inaccurate calculations during view synthesis. Especially in complex scenes, partial occlusion or weakly textured areas can prevent the reconstruction of the target object from fully recovering its structural information. Therefore, effectively removing background interference and handling occlusion issues to focus on the accurate reconstruction of specific target objects has become an important research direction in the field of 3D object reconstruction. Although some studies in recent years have attempted to combine deep learning techniques for target segmentation and background removal to improve reconstruction efficiency and accuracy, some methods rely on complex post-processing steps, making them difficult to implement in real-time applications. Others still have low segmentation accuracy, especially when object boundaries are blurred or severely occluded, often resulting in structural errors. Additionally, past segmentation methods require training and fine-tuning for the specific scene before deployment. This leads to poor generalization performance and introduces unnecessary training overhead, making them unsuitable for direct deployment in new environments. Therefore, addressing these two technical needs, there is an urgent need to design a high-performance, highly generalizable, and efficient 3D object reconstruction method. Summary of the Invention

[0005] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] To address the problems and shortcomings of existing technologies, this invention aims to provide a 3D object reconstruction method based on interactive segmentation priors and implicit surface reconstruction. Utilizing deep learning, large models, and implicit surface reconstruction techniques, an interactive segmentation algorithm is constructed using a pre-trained model. Prior segmentation knowledge is extracted from the object to be reconstructed in complex scenes, and the implicit surface reconstruction algorithm is used to construct the fine mesh and texture information of the object. This method focuses on the reconstruction of a specific object within a scene, and is particularly suitable for reconstructing objects in scenes with complex backgrounds and occlusions. The interactive segmentation algorithm removes interference from the background and occlusions, providing accurate 3D object assets for subsequent scene understanding, object recognition, and virtual reality applications. This addresses the problems mentioned in the background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] This invention discloses a method for reconstructing three-dimensional objects based on interactive segmentation priors and implicit surface reconstruction, comprising the following steps:

[0009] Step 1, in response to acquiring multi-view scene images to be processed;

[0010] Step 2: Interactively extract the segmentation mask and target image of the target object based on the pre-trained SAM segmentation model;

[0011] Step 3: Introduce the SfM algorithm to calculate the camera pose information and sparse structure information of the multi-view scene graph;

[0012] Step 4: Input the target image, segmentation mask, and camera pose information into the mesh encoder to train and obtain a coarse 3D mesh and texture features;

[0013] Step 5: Load the trained model weights and reconstruct a coarse 3D mesh and texture map of the target object;

[0014] Step 6: Refine the coarse 3D mesh and texture map to output the 3D reconstruction result of the target object.

[0015] Preferably, step 2 includes the following steps:

[0016] Step 2.1: Load the SAM segmentation model and its model weights, and input the multi-view scene image;

[0017] Step 2.2: Obtain the target object region and generate a segmented region based on the prompt information of the target object region;

[0018] Step 2.3: Analyze the segmented region and the semantic segmentation results of the multi-view scene image prediction;

[0019] Step 2.4: Generate a binarized segmentation mask from the semantic segmentation results through multi-layer feature extraction and layer-by-layer aggregation;

[0020] Step 2.5: Combine the binarized segmentation mask and the multi-view scene image to output the target image from multiple perspectives.

[0021] Preferably, step 4 includes the following steps:

[0022] Step 4.1: Construct a 3D mesh and initialize it as a coarse mesh structure;

[0023] Step 4.2: Input the segmentation mask, target image, and camera pose information into the mesh encoder to extract coarse geometric and texture features;

[0024] Step 4.3: Optimize the 3D mesh by combining the rough geometric and texture features.

[0025] Preferably, step 4.1 further includes the following steps:

[0026] Step 4.1.1: Divide the three-dimensional mesh into voxel meshes of uniform size and shape using a regular mesh;

[0027] Step 4.1.2: Construct the position of the target object on the three-dimensional mesh using the ray stepping method to generate a coarse mesh structure;

[0028] Step 4.1.3: Optimize the surface of the three-dimensional mesh using ray tracing.

[0029] Preferably, step 4.2 further includes the following steps:

[0030] Step 4.2.1: Input the segmentation mask, target image and camera pose information into the neural network to extract coarse geometric features;

[0031] Step 4.2.2: Combine the rough geometric features to obtain rough texture features by implicitly modeling the texture mapping of the object surface;

[0032] Step 4.2.3: The rough texture features are decomposed into specular reflection components and diffuse reflection components.

[0033] Preferably, the neural network in step 4.2.1 is a combination of three fully connected layers, which learns and captures structural knowledge of the three-dimensional mesh constructed in step 4.1 and stores it in the coarse geometric features.

[0034] Preferably, in step 4.3, the optimization of the three-dimensional mesh first involves adjusting the resolution of the three-dimensional mesh by analyzing local density optimization to enhance the expression of complex textures; then, combined with illumination consistency optimization, the three-dimensional mesh can accurately express the details of the target object.

[0035] Preferably, in step 6, the coarse 3D mesh and texture map are processed using a visual method. Figure 1 Consistency optimization generates refined texture maps, ensuring a high degree of matching between textures and geometric meshes.

[0036] As a second aspect of this application, the present invention also discloses an electronic device, comprising:

[0037] At least one processor, and a memory communicatively connected to said at least one processor;

[0038] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the steps of the above-described three-dimensional object reconstruction method based on interactive segmentation priors and implicit surface reconstruction.

[0039] As a third aspect of this application, the present invention also discloses a computer storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention provides a 3D object reconstruction method based on interactive segmentation priors and implicit surface reconstruction. It can automatically generate high-precision 3D reconstruction results, improving reconstruction speed and providing excellent real-time performance. It can quickly and accurately reconstruct target objects in complex environments. The interactive segmentation of the SAM segmentation model in this invention allows for direct segmentation in various scenes through user interaction prompts, avoiding fine-tuning and training in new scenes, ensuring its generalization performance, and preventing the ineffective use of computing resources by background regions, thus efficiently handling complex scenes. Implicit surface reconstruction addresses the problem of previous NeRF-based methods failing to generate high-precision display models and texture maps, directly generating a coarse mesh structure of the object to be reconstructed. Simultaneously, texture information such as specular reflection and diffuse reflection is learned through neural networks and explicitly fine-tuned during subsequent learning. This makes the 3D reconstruction more operable and editable, supporting subsequent editing and optimization, and meeting the high requirements of 3D models in practical applications. This invention utilizes deep learning technology to automatically generate training datasets. By combining segmentation masks, camera pose information, and input images, it constructs high-quality 3D reconstruction models, avoiding the high costs of manual data collection and annotation, significantly reducing data production time and costs, while also possessing strong scene scalability. Furthermore, it avoids the problem of texture information loss in existing methods by directly generating accurate texture maps and optimizing lighting effects during the 3D reconstruction process. This results in reconstruction results with not only good geometric structure but also high-quality texture information. The generated texture maps and 3D meshes can be directly applied to virtual reality, augmented reality, and other fields, demonstrating significant practical value and promising prospects for wider application. Attached Figure Description

[0042] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:

[0043] Figure 1 This is a flowchart showing the steps of the three-dimensional object reconstruction method in an embodiment of the present invention.

[0044] Figure 2 This is a flowchart illustrating the execution of the three-dimensional object reconstruction method in an embodiment of the present invention.

[0045] Figure 3 This is a flowchart illustrating the steps of SAM-based interactive segmentation prior in an embodiment of the present invention.

[0046] Figure 4 In this embodiment of the invention, the result after selecting the target based on SAM interactive segmentation prior is shown in (a) as the input image and (b) as the target object region determined by clicking the mouse.

[0047] Figure 5 The following is a diagram showing the effect of generating a mask based on SAM interactive segmentation prior in this embodiment of the invention: (a) is the input image, (b) is the target object region determined by mouse click, and (c) is the generated mask image.

[0048] Figure 6 This is a flowchart illustrating the steps for calculating camera pose information based on SfM in an embodiment of the present invention.

[0049] Figure 7 This is a flowchart illustrating the generation process of reconstructing the mesh and texture of the target object in an embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Detailed Implementation

[0051] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0052] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0053] This invention discloses a 3D object reconstruction method based on interactive segmentation priors and implicit surface reconstruction. An interactive segmentation program is constructed using a large segmentation model (SAM) that requires no training. The segmentation mask containing the reconstructed object is obtained through user mouse click prompts. This invention also introduces an implicit surface reconstruction method. An implicit representation is learned through a neural network, and a coarse mesh model and texture information of the generated object are displayed during the learning process. This is continuously optimized during learning to obtain a high-precision polygonal mesh and editable texture map of the target object, facilitating downstream applications. The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0054] Reference Figure 1 and Figure 2 As shown, it includes the following steps:

[0055] Step 1, in response to acquiring multi-view scene images to be processed;

[0056] Step 2: Interactively extract the segmentation mask and target image of the target object based on the pre-trained SAM segmentation model;

[0057] Step 3: Introduce the SfM algorithm to calculate the camera pose information and sparse structure information of the multi-view scene graph;

[0058] Step 4: Input the target image, segmentation mask, and camera pose information into the mesh encoder to train and obtain a coarse polygon mesh and texture features;

[0059] Step 5: Load the trained model weights and reconstruct a rough polygonal mesh and texture map of the target object;

[0060] Step 6: Refine the coarse polygonal mesh and texture map to output the 3D reconstruction result of the target object.

[0061] First, we use an industrial camera to acquire RGB images of the scene from multiple perspectives as input multi-view scene images. These multi-view scene images should cover the main perspectives of the target object, and each scene image should have a corresponding timestamp and resolution information to provide data support for subsequent segmentation and reconstruction steps. Then, we use an interactive segmentation method to generate a semantic segmentation mask for the object to be reconstructed. The main goal is to remove background information from the multi-view scene images, avoiding the generalization and fine-tuning required by traditional learning-based segmentation methods in different scenes.

[0062] Specifically, in step 2, the SAM segmentation model is used to complete the semantic segmentation mask. Since the SAM segmentation model has already been trained on massive amounts of data, interactive segmentation can be performed without further training. Semantic segmentation is performed on the output scene graph through user interaction, such as mouse clicks or region selection to determine the region of interest of the target object, automatically generating a semantic segmentation mask. The segmentation mask accurately marks the boundaries of the target object while removing background and irrelevant regions. Figures 3 to 5 As shown, it includes the following steps:

[0063] Step 2.1: Load the SAM segmentation model and its model weights, and input multi-view scene images;

[0064] Step 2.2: Obtain the target object region and generate segmented regions based on the prompt information of the target object region;

[0065] Step 2.3: Analyze the segmented regions and predict semantic segmentation results for multi-view scene images;

[0066] Step 2.4: Generate a binarized segmentation mask by extracting features from multiple layers and aggregating them layer by layer from the semantic segmentation results;

[0067] Step 2.5: Combine the binarized segmentation mask and the multi-view scene image to output the multi-view target image.

[0068] Specifically, multi-view RGB scene images to be processed are acquired. These images should cover the main viewpoints of the target object to ensure the integrity of the segmentation results. For example... Figure 4 Image (a) shows the input RGB scene image, including the target object and its background scene. The background scene needs to be removed through segmentation. A pre-trained SAM segmentation model is used, with SAM segmentation model weights loaded. The SAM segmentation model is a general segmentation model based on the Transformer structure, supporting various user interaction methods (such as clicking, bounding box selection, etc.) and requiring no repeated training and fine-tuning. In this invention, we use mouse clicks to interact with the SAM segmentation model. The user clicks on key locations of interest on the target object, and the SAM segmentation model generates prompt information based on the click location and automatically generates segmentation regions based on the prompt information, such as... Figure 4 As shown in (b). If the user needs to click on multiple key points of the target object, the SAM segmentation model can generate more accurate segmentation regions based on the click information, and then output the corresponding segmentation mask representation based on these segmentation regions.

[0069] M = SAM(I,C,P)

[0070] Where I represents the input image, and C is the set of key points clicked by the user, represented as C = {c1, c2, ..., c...} n}, where P is the prior parameter of the model and M is the output segmentation mask. The SAM segmentation model predicts the semantic regions related to the target and generates segmentation results by analyzing the distribution of click keypoints and image content. The segmentation mask retains only the target object portion, while the background region is removed and represented as black. The mask is generated through multi-layer feature extraction and layer-by-layer aggregation, using binary representation to distinguish the target object from the background region, resulting in a binary segmentation mask representation, as shown below.

[0071]

[0072] Among them, f l For the features of the l-th layer, w l Here, F represents the corresponding weight parameters, and F is the aggregated feature map. By combining the generated binarized segmentation mask with the multi-view scene images, background regions can be removed, retaining only the target object information. For example... Figure 5As shown, the processed target object is clearly visible without background interference. This allows the processed image data to be used for subsequent 3D reconstruction tasks.

[0073] It is important to note that the fundamental purpose of using the segmentation method in this invention is to remove background and clutter occlusions from the scene, allowing subsequent reconstruction methods to be performed on the object to be reconstructed and reducing computational overhead. Previous semantic segmentation-based prior generation schemes required training and fine-tuning within a specified scene. However, this invention differs from previous semantic segmentation schemes, which necessitate an additional, time-consuming and labor-intensive training process for the segmentation network during the object reconstruction step. Therefore, this invention introduces the SAM semantic segmentation method into the reconstruction process, avoiding the cumbersome training steps.

[0074] Since the acquired multi-view scene images lack camera pose information, the SfM method is used to extract the camera's intrinsic and extrinsic parameter matrices, and sparse structure information is calculated using triangulation. By combining structured light technology with multi-view geometry principles, the SfM method is used to generate camera pose information for each input multi-view scene image. Camera pose information includes translation matrix T and rotation matrix R, which are crucial for geometric constraints during 3D scene reconstruction. The input multi-view scene image data must contain rich overlapping regions. The multi-view scene image data should cover different viewpoints of the target object to provide sufficient information for subsequent feature extraction and matching. A set of scale- and rotation-invariant feature points is extracted from the multi-view scene images using the SIFT algorithm and triangulation. The feature point extraction formula is expressed as Keypoint =

[0075] {k1,k2,…,k n}, where each keypoint k includes coordinate position, scale, and orientation information. Then, the nearest neighbor search algorithm is used to determine corresponding feature point pairs in different multi-view scene images, and multi-view matching is performed on the extracted feature point pairs to obtain matching pairs. The set P of matching pairs, i.e., matching points, is denoted as,

[0076] P={(k i ,k j )∣k i ∈Image1,k j ∈Image2};

[0077] Where, k i and k j This represents the matching points in a matching pair, each originating from a scene image from a different viewpoint. Next, the RANSAC algorithm is used to robustly optimize and remove erroneous matches, and the fundamental matrix F is calculated. The formula for calculating the fundamental matrix F is as follows: Where, x i and x jLet F represent the normalized coordinates of the matching points and denote the fundamental matrix satisfying the epipolar constraint relationship between the point pairs. The camera's rotation matrix R and translation matrix T are calculated by decomposing the fundamental matrix F and the intrinsic parameter matrix K. The specific formulas are as follows:

[0078] E=KTFK[R|T]=Decompose(E)[R|T]=\text{Decompose}(E)[R∣T]=

[0079] Decompose(E);

[0080] E=K T FK;

[0081] [R|T] = Decompose(E);

[0082] Here, E is the essential matrix, calculated from the fundamental matrix F and the camera intrinsic matrix K. Industrial cameras typically come with a pre-calibrated intrinsic matrix K, which can be directly used. The camera's pose information, i.e., the rotation matrix R and translation matrix T, can be obtained by decomposing the essential matrix E. Finally, the calculated rotation matrix R and translation matrix T are stored in correspondence with the multi-view scene image data, serving as important inputs for subsequent 3D scene reconstruction. The specific steps are as follows: Figure 6 As shown, the camera positions and their relative relationships in multi-view images are illustrated. This invention achieves accurate camera pose calculation by combining SfM technology with feature point extraction, matching, and robust optimization. It can handle complex multi-view scenes and provide accurate geometric constraints for subsequent 3D scene reconstruction.

[0083] Next, we need to input the target object image and camera pose information obtained in steps 2 and 3 into the mesh encoder for training. Specifically, this involves a mesh and texture generation method based on implicit surface reconstruction. This step combines the target image, camera pose information, mesh initialization, and a deep learning optimization network to generate high-quality geometric meshes and texture maps of the object through the mesh encoder. Furthermore, during network learning, this method prioritizes generating coarse explicit polygonal meshes and extracting texture information (specular and diffuse reflection), and continuously optimizes it in subsequent learning processes, resulting in an accurate and complete polygonal mesh model that is also pipeline-friendly. Specifically, it includes the following steps:

[0084] Step 4.1: Construct a 3D mesh and initialize it to generate a coarse mesh structure;

[0085] Step 4.2: Input the segmentation mask, target image, and camera pose information into the mesh encoder to extract coarse geometric and texture features;

[0086] Step 4.3: Optimize the 3D mesh by combining rough geometric and texture features.

[0087] Furthermore, such as Figure 7 As shown, the input consists of multi-view target images and their corresponding camera pose information (such as rotation matrix R and translation matrix T). These inputs provide geometric and lighting information for subsequent mesh generation and texture mapping. The input multi-view target images contain images of the target object from different perspectives, while the camera pose information ensures that these target object images can be correctly mapped to 3D space. The 3D mesh is constructed and initialized with a coarse mesh structure, including the following steps:

[0088] Step 4.1.1: Divide the 3D mesh into voxel meshes of uniform size and shape using a regular mesh.

[0089] Step 4.1.2: Construct the position of the target object on the 3D mesh using the ray stepping method to generate a coarse mesh structure;

[0090] Step 4.1.3: Optimize the surface of the three-dimensional mesh using ray tracing.

[0091] Specifically, in the mesh initialization phase, a regular mesh is used for spatial partitioning. This means the 3D mesh is divided into voxel meshes of uniform size and shape, with each mesh cell containing a mesh node representing the surface of the target object. Specifically, the position of the target object on the 3D mesh is constructed using ray tracing, directly learning and generating a coarse 3D mesh structure. Then, ray tracing is used to optimize the positions of the surface mesh nodes. Specifically, using the input multi-view target images and camera pose information, rays are projected from multiple perspectives, and 3D points are sampled on the rays to obtain the initial positions of the mesh nodes. The multi-view target images and camera pose information are then input into the mesh encoder to extract and generate coarse geometric and texture features, including the following steps:

[0092] Step 4.2.1: Input the segmentation mask, target image and camera pose information into the neural network to extract coarse geometric features;

[0093] Step 4.2.2: Combine the rough geometric features with the texture mapping of the object surface through implicit modeling to obtain rough texture features;

[0094] Step 4.2.3 decomposes the coarse texture features into specular reflection components and diffuse reflection components.

[0095] Specifically, the mesh encoder inputs segmentation masks, multi-view target images, and camera pose information into a neural network to extract coarse geometric features, supporting the generation of polygonal meshes. This process includes multiple linear layers, activation function layers, and a layer-by-layer feature refinement process. For example... Figure 7As shown, the geometric features extracted after geometric transformation of the segmentation mask, multi-view target images, and camera pose information through multiple convolutional layers (including Linear, SoftPlus, Linear, and ReLU) are represented as follows:

[0096] F g =Net g (I,R,T);

[0097] Among them, F g The generated geometric features are represented by I, where I represents the input multi-view target image, and R and T are the camera rotation and translation matrices, respectively. To enable the neural network to learn effectively, this invention uses the Net network... g The design is a combination of three fully connected MLP layers, and the specific operation formula is as follows:

[0098] MLP=ReLU(SoftPlus(Linear(I,R,T));

[0099] The structural knowledge of the 3D mesh output from step 4.1 can be learned and captured, and stored in the geometric feature F. g This is to facilitate subsequent optimization and use.

[0100] The neural network further learns and generates coarse texture features by combining geometric features generated from multi-view convolutions. It generates high-quality texture maps by jointly learning geometric features and color distributions in multi-view object images, implicitly modeling the texture mapping of the object surface. Specifically, during training, the neural network constrains color consistency across different viewpoints to obtain an optimized texture representation, ensuring that the final generated texture matches the color variations of the real object surface.

[0101] F s =Net s (F g ,I);

[0102] Among them, F s For the generated texture features, F g Let I be the geometric features and I be the input multi-view scene image, enabling the network to generate high-precision object textures. Next, we decompose the obtained coarse texture features into specular and diffuse reflection components. This decomposition maintains the editability of the generative model, allowing it to interact well in downstream tasks.

[0103] The above steps generate coarse geometric and texture features, which are then used to optimize the mesh surface, improving its smoothness and detail representation. This step involves analyzing local density to optimize mesh resolution, reducing surface irregularities and enhancing the representation of complex textures. Illumination consistency optimization is also incorporated to ensure the mesh structure accurately reflects the lighting interactions of real objects, thereby improving geometric accuracy and texture mapping quality. The optimized coarse mesh can more accurately represent object details, making it suitable for subsequent 3D vision tasks, rendering, or physical simulations.

[0104] Next, the performance of the current model is evaluated by calculating the training error and validation set performance metrics (such as PSNR and SSIM). Based on the evaluation results, the network weight parameters are further optimized and saved for subsequent online detection and reconstruction to improve the model's convergence speed and reconstruction quality. A pre-trained mesh encoder and related modules are loaded to reconstruct a coarse 3D mesh and texture map of the target object. A mesh generation module is used to generate a coarse 3D mesh of the target object, and coarse texture features (specular and diffuse reflection information) are calculated to refine the 3D mesh and texture map. Combining the input multi-view image data, the generated coarse 3D mesh, and coarse texture features, the model is then refined using visual... Figure 1 A consistency optimization method generates high-quality texture maps, ensuring a high degree of matching between the texture and the geometric mesh. The optimized and adjusted mesh and texture map are then output. The output 3D mesh contains the object's geometry, while the texture map provides the lighting details of the object's surface. The results can be directly applied to multiple fields such as virtual reality, augmented reality, and game design. Finally, the generated 3D reconstruction model is evaluated. The model's performance is verified by calculating the errors between real data and the reconstruction results (such as positional errors and texture similarity). The results are also visualized for easy manual review.

[0105] To implement the above embodiments, this application also discloses an electronic device. (Refer to...) Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0106] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 8 Each box shown can represent a device or multiple devices as needed.

[0107] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0108] It should be noted that, in some embodiments of this disclosure, the computer storage medium described above can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0109] In some embodiments of this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0110] The aforementioned computer storage medium may be included within the aforementioned electronic device, or it may exist independently and not assembled into the electronic device. The aforementioned computer storage medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to implement a three-dimensional object reconstruction method.

[0111] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings.

[0113] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0114] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for reconstructing three-dimensional objects based on interactive segmentation priors and implicit surface reconstruction, characterized in that, Includes the following steps: Step 1, in response to acquiring multi-view scene images to be processed; Step 2: Interactively extract the segmentation mask and target image of the target object based on the pre-trained SAM segmentation model; Step 3: Introduce the SfM algorithm to calculate the camera pose information and sparse structure information of the multi-view scene graph; Step 4: Input the target image, segmentation mask, and camera pose information into the mesh encoder to train and obtain a coarse 3D mesh and texture features; Step 5: Load the trained model weights and reconstruct a coarse 3D mesh and texture map of the target object; Step 6: Refine the coarse 3D mesh and texture map to output the 3D reconstruction result of the target object.

2. The method for reconstructing a 3D object based on interactive segmentation prior and implicit surface reconstruction according to claim 1, characterized in that, Step 2 includes the following steps: Step 2.1: Load the SAM segmentation model and its model weights, and input the multi-view scene image; Step 2.2: Obtain the target object region and generate a segmented region based on the prompt information of the target object region; Step 2.3: Analyze the segmented region and the semantic segmentation results of the multi-view scene image prediction; Step 2.4: Generate a binarized segmentation mask from the semantic segmentation results through multi-layer feature extraction and layer-by-layer aggregation; Step 2.5: Combine the binarized segmentation mask and the multi-view scene image to output the target image from multiple perspectives.

3. The method for reconstructing a 3D object based on interactive segmentation prior and implicit surface reconstruction according to claim 2, characterized in that, Step 4 includes the following steps: Step 4.1: Construct a 3D mesh and initialize it as a coarse mesh structure; Step 4.2: Input the segmentation mask, target image, and camera pose information into the mesh encoder to extract coarse geometric and texture features; Step 4.3: Optimize the 3D mesh by combining the rough geometric and texture features.

4. The three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction according to claim 3, characterized in that, Step 4.1 also includes the following steps: Step 4.1.1: Divide the three-dimensional mesh into voxel meshes of uniform size and shape using a regular mesh; Step 4.1.2: Construct the position of the target object on the three-dimensional mesh using the ray stepping method to generate a coarse mesh structure; Step 4.1.3: Optimize the surface of the three-dimensional mesh using ray tracing.

5. The three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction according to claim 4, characterized in that, Step 4.2 also includes the following steps: Step 4.2.1: Input the segmentation mask, target image and camera pose information into the neural network to extract coarse geometric features; Step 4.2.2: Combine the coarse geometric features with the texture mapping of the object surface through implicit modeling to obtain coarse texture features; Step 4.2.3: The rough texture features are decomposed into specular reflection components and diffuse reflection components.

6. The three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction according to claim 5, characterized in that: The neural network in step 4.2.1 is configured as a combination of three fully connected layers to learn and capture structural knowledge of the three-dimensional mesh constructed in step 4.1, and store it in the rough geometric features.

7. The three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction according to claim 6, characterized in that: In step 4.3, the optimization of the three-dimensional mesh first involves adjusting the resolution of the three-dimensional mesh by analyzing local density optimization to enhance the expression of complex textures; then, combined with lighting consistency optimization, the three-dimensional mesh can accurately express the details of the target object.

8. The three-dimensional object reconstruction method based on interactive segmentation prior and implicit surface reconstruction according to claim 3, characterized in that: In step 6, the coarse 3D mesh and texture map are optimized using view consistency to generate a refined texture map, ensuring a high degree of matching between the texture and the geometric mesh.

9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 8.

10. A computer storage medium storing a computer program thereon, characterized in that: When the computer program is executed by the processor, it performs the steps as described in any one of claims 1 to 8.

Images