A three-dimensional reconstruction method for transparent mask removal

Abstract

The application discloses a three-dimensional reconstruction method for transparent mask elimination, which firstly adopts a physical-based reflection removal module. By integrating spatial consistency and edge prior, the real scene content can be effectively separated from the glass reflection, and a clear transmission image is generated. At the same time, an explicit point cloud constraint is used to assist in positioning the object surface. In the second stage, a neural volume rendering framework is adopted to jointly optimize the SDF and the radiation field representation to reconstruct the accurate geometry and appearance of the object inside the glass shell. Through the two-stage framework of removing the reflection interference first and then jointly optimizing the radiation field representation, the reconstruction robustness and fineness under complex reflection conditions are significantly enhanced.

Description

Technical Field

[0001] This invention relates to the field of 3D modeling technology applications, and in particular to a 3D reconstruction method for eliminating transparent masks. Background Technology

[0002] While existing 3D reconstruction techniques have made significant progress in general scenes, they still have obvious limitations in environments containing glass enclosures. First, most mainstream methods assume that the imaging process is unaffected by strong reflection and refraction, making it difficult to effectively distinguish between the real surface and specular reflection components of an object in glass scenes. In areas where reflection and transmission are mixed, these algorithms often produce incorrect matching or unstable depth estimates. Second, some methods attempt to use multi-view... Figure 1 While existing methods may rely on prior knowledge or learning, they still struggle to address the multilayered transmission and nonlinear refraction effects inherent in glass materials, leading to issues such as texture artifacts, structural distortions, and geometric discontinuities in the reconstruction results. Furthermore, current reflection elimination or glass detection models are typically separated from the 3D reconstruction process, lacking a joint modeling mechanism, which makes robust component separation and accurate modeling difficult to achieve under complex lighting conditions.

[0003] The closest prior art is the applicant's earlier application, Chinese invention patent application CN 118736114 A, published on October 1, 2024, entitled "A Method for 3D Reconstruction of Semi-transparent Objects Based on Neural Implicit Symbolic Distance Field," which discloses a method for 3D reconstruction of semi-transparent objects based on neural implicit symbolic distance field. This method relates to the fields of computer vision and 3D reconstruction, and can perform 3D reconstruction and synthesis of new perspectives in scenes containing semi-transparent objects. This invention represents objects in a scene as an implicit symbolic distance field model fitted by a multilayer perceptron. Using input RGB data, camera pose, optimized scene physical information (color, voxel density), and the implicit symbolic distance field represented by a neural network, the 3D reconstruction result is finally extracted from the optimized implicit distance field model. This method can not only correctly recover the geometry of semi-transparent objects but also obtain images synthesized from new perspectives. It solves the problems of poor reconstruction accuracy and low quality of new perspective synthesis in existing methods for semi-transparent object scenes.

[0004] Application number 202411311558.1, application date September 2024. Chinese Invention Patent Application No. 20, entitled "Reconstruction Method, Apparatus, Device, and Storage Medium for a 3D Model," discloses a reconstruction method, apparatus, device, and storage medium for a 3D model. The method includes: acquiring multiple images of an object to be reconstructed from multiple perspectives; inputting each image of the object to be reconstructed into a trained de-glare model for de-glare processing to obtain multiple de-glare reconstructed object images; wherein the de-glare model sequentially performs multiple downsampling and multiple upsampling operations on the input image, and fuses the results of each upsampling operation with the input image through channel stitching and transposed convolution to obtain de-glare images; using a pre-trained de-occlusion model, based on the occlusion mask map of the occlusion region in each of the de-glare reconstructed object images, performing de-occlusion processing on the occlusion regions of each of the de-glare reconstructed object images to obtain multiple processed reconstructed object images; and using each of the processed reconstructed object images for 3D reconstruction to obtain a 3D model of the object to be reconstructed.

[0005] In practical applications, target objects are often encased in transparent glass shells for protection and isolation. However, these shells introduce complex optical interference. The specular reflection of glass obstructs direct observation of the internal objects, while refraction and multilayer transmission distort incoming and outgoing light, resulting in images containing real textures, reflection artifacts, and distorted false structures simultaneously, significantly increasing the difficulty of 3D reconstruction. Although existing methods can achieve high-quality reconstructions in generally non-transparent environments, they lack reliable mechanisms to separate reflection and transmission components when dealing with glass scenes, making it difficult to distinguish between real surfaces and optical artifacts caused by glass. This limitation not only leads to texture aliasing and geometric distortion in the reconstruction results but also severely interferes with pose estimation and subsequent analysis, failing to meet the requirements for high-precision 3D understanding.

[0006] Existing technical solution 1: Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning forMulti-View Reconstruction with Reflection Baseline Method

[0007] Main methods: such as Figure 1As shown, firstly, images acquired from multiple perspectives are analyzed. An anomaly detection method identifies which regions are reflective surfaces, thus determining which locations might interfere with the reconstruction. Then, based on the intensity of these reflective regions, color errors in the images are weighted to automatically reduce the impact of reflection errors. Simultaneously, the viewpoint information is combined to create a more reasonable model of the object's true appearance. Finally, all the above information is input into an implicit surface learning network, enabling the system to recover a more accurate and coherent 3D shape of the object even under conditions of strong reflection.

[0008] Implementation process: (1) First, the multi-view images are analyzed. By combining the anomaly detection method with the multi-view information, the reflection score of each pixel is calculated to locate the area affected by reflection in the image.

[0009] (2) Add reflection score to rendering and error calculation, further construct the photometric loss of reflection perception, so that the influence of reflection area is automatically reduced. At the same time, combine the radiation modeling of the view direction to improve the color and appearance consistency in the reflection area.

[0010] (3) The reflection score, reflection perception loss and implicit surface representation are used together to train the model.

[0011] This method proposes a photometric loss function that considers reflection to resolve ambiguity caused by reflection. This loss function weakens the importance of reflective surfaces by reducing the weight of reflective pixels. Its loss function relies on the estimated reflection score to locate these pixels. An anomaly detector is used, which leverages multi-view context to obtain the reflection score. This context is captured by projecting surface points onto a visible light source view using a visibility identification module. In this figure, Source images, Reference image, Geometry Network, Signed distance, Surface point projection, Visibility identification, Reflection score estimation, Radiance Network predicts radiance (radiance color) and surface normal (normal vector) based on position and orientation, Reflection score fraction, and Reflection-aware photometric loss are also considered.

[0012] Existing technical solution 1 reduces reflection interference through reflection scoring, but it still cannot handle the refraction path of light after passing through transparent materials, thus it cannot be used in complex optical path scenarios such as glass containers. Secondly, reflection detection relies on anomaly detection networks, which have limited accuracy. If the reflection area is not accurately located, geometric misjudgment will still occur.

[0013] Existing technical solution 2: Seeing Through the Glass: Neural 3D Reconstruction of Object Inside aTransparent Container Baseline Method

[0014] The main method is as follows: First, the scene is divided into two regions: the outside and the inside of the transparent container. The light behavior in each region is processed separately to simplify the complex light path caused by multiple reflections and refractions. Then, each camera ray is tracked to simulate the reflection and refraction it produces on the glass surface. The colors corresponding to these ray paths are calculated by either a neural network (inner region) or a background model (outer region). Finally, the colors of all sub-paths are accumulated in a physical way, and the network is continuously trained to generate results consistent with the real image, thereby gradually restoring the real three-dimensional shape of the object encased in glass.

[0015] Implementation process: (1) First, the scene is divided into two parts, the inside and the outside, according to the glass interface, so that the originally complex problem of light reflection and refraction can be decomposed and managed.

[0016] (2) Perform reflection and refraction tracking on each camera ray, break down a ray into multiple real possible ray paths, and calculate the color of these paths using a neural network or background model respectively.

[0017] (3) By physically re-accumulating the colors of these light paths and comparing and optimizing them with real images, the model gradually learns to penetrate the glass and recover the real three-dimensional shape of the internal objects.

[0018] Among them, Scene representation is the representation of the scene, Volumetric Integration is the volume integration, Reconstruction is the reconstruction, Target is the target, Accumulation is the accumulation, and Ray-tracing is the ray tracing.

[0019] Although the existing technical solution 2 divides the scene into background and objects and models them separately, this method relies on the known geometry of transparent containers. Glass is usually regarded as an ideal transparent medium, but deviations may still occur when dealing with rough glass or multi-layered materials, which is difficult to meet in practical applications.

[0020] Existing technical solution 3: Looking Through the Glass: Neural Surface Reconstruction AgainstHigh Specular Reflections Benchmark Dataset

[0021] Main Method: The core idea of ​​this method is to enable the neural network to automatically "identify" which pixels are contaminated by strong reflections from glass when reconstructing a 3D surface, thereby preventing the network from treating reflections as real objects. Specifically, the method involves: first, identifying which areas are highlight reflections from multi-view images and estimating their reliability; then, based on these reliability estimates, dynamically reducing the influence of reflective areas during rendering and error calculation to avoid misleading geometric learning; finally, through continuous optimization, making the network focus more on realistic surface texture, enabling the reconstruction of the object's true 3D structure even under large areas of strong highlights caused by glass.

[0022] Implementation process: (1) First, this method utilizes multiple views Figure 1 The reflective regions in the image are detected by consistency and brightness features, and a "reflection confidence score" is assigned to each pixel.

[0023] (2) Incorporate this score into the rendering loss so that the weight of the reflection area is automatically reduced during optimization, thereby reducing the interference of reflection on geometric judgment.

[0024] (3) By jointly learning surface shape, normal and appearance, the network can still focus on the real object in a strong reflection environment, thus obtaining more accurate and stable 3D reconstruction.

[0025] Among them, sampling points and perspective These are input into the target object path and the planar path, respectively. In the object path, implicitly... Generated by a head neural network. Surface modules are... , and As input, render weights are generated. In the planar path, the reflection module... and Generate the normal to the auxiliary plane 3D Position and rendering weights Finally, we use the appearance function. Two appearances can be obtained by drawing the volume.

[0026] The detection of the reflection area in the existing technical solution 3 depends on brightness and multiple views. Figure 1 Inconsistency can occur when the reflection is particularly complex or the quality of multiple views is poor, as the determination of the reflection area may not be stable enough, thus affecting the reconstruction accuracy.

[0027] This invention application is filed to address the shortcomings of existing technologies. Summary of the Invention

[0028] In view of the above-mentioned shortcomings of existing technologies, this invention proposes a 3D reconstruction method for eliminating transparent masks. Without relying on a prior container model, this method enables the 3D reconstruction system to accurately identify and handle the effects of reflection and refraction caused by glass, thereby achieving high-quality 3D reconstruction of glass-encased objects. Simply put, it allows the system to "penetrate" the glass and see the internal objects, while maintaining geometric stability and clear textures under complex lighting conditions.

[0029] To achieve the above-mentioned objectives, the technical solution provided by this invention is as follows: Beneficial effects: There are two main challenges in dealing with objects with glass enclosure interference: (1) The inherent ambiguity introduced by the glass enclosure. The strong specular reflection on the top surface of the glass enclosure can obscure the background, and the enclosed object will be further reflected by the four sides of the enclosure. In addition, such complex reflections and prism occlusion have not been explored in previous reflection removal studies. (2) Facing object texture and external residual artifacts, residual artifacts are treated as anomalous surfaces. Offset reflection layers present in multi-view often affect surface localization. These artifacts are similar to occlusions, but they are mixed with the color of the object itself, resulting in inaccurate color estimation and increasing the difficulty of achieving multi-view. Figure 1 The difficulty of achieving consistency.

[0030] To address these challenges, this invention introduces a two-stage processing flow. For the first challenge, it extends the performance boundaries of existing reflection removal models by performing glass shell removal. The output of this stage provides a relatively clean image, offering more accurate color and geometry information for subsequent tasks. For the second challenge, the method replaces the traditional view-dependent representation with a parameterized reflection direction representation. By introducing a geometrically consistent network, the network's predictions are explicitly constrained to surface points. This design enhances the smoothness of the radiation field and facilitates the learning and interpolation of reflection properties, thereby achieving more realistic color reconstruction.

[0031] Terminology Explanation: SDF: Signed Distance Field MD: Mahalanobis Distance Z-score: Standardized Anomaly Measure Background_RGB: RGB color model for background. (RGB color values ​​for the background) GT: Ground-truth (True Annotation Values / True Reference Data) L1 / L2: L1 / L2 norm(Manhattan norm / Euclidean norm) L1 / L2 norm Adam: Adaptive Moment Estimation Warm-up: Learning rate warm-up strategy 3DGS: 3D Gaussian Splatting NeRF: Neural Radiance Fields U-Net: Encoder-decoder convolutional neural network architecture Transformer: A deep neural network architecture based on self-attention mechanism Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 Here is a framework diagram of the existing technical solution 1; Figure 2 Here is a framework diagram of the existing technical solution 2; Figure 3 Here is a framework diagram of the existing technical solution 3; Figure 4 This is a general framework diagram of the three-dimensional reconstruction method for eliminating transparent masks according to the present invention; Figure 5 This is a flowchart of the overall process of the 3D reconstruction method for eliminating transparent masks according to the present invention; Figure 6 This is a diagram illustrating data collection. Figure 6 'a' represents the interference details caused by the glass dome during object reconstruction. Figure 6 b represents the interference of various reflections and glass edges during the shooting process on the shooting perspective. Figure 6 c is a magnified view showing details such as glass reflection interference. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0034] Figure 4 This is a general framework diagram of the 3D reconstruction method for eliminating transparent masks according to the present invention. Figure 5 This is a flowchart illustrating the overall process of the 3D reconstruction method for eliminating transparent masks according to the present invention. Figure 4 , Figure 5 The 3D reconstruction method for eliminating transparent masks of the present invention, as shown, includes the following steps: Step S1, pretreatment to remove the glass cover; A physics-based reflection removal module is used to separate real scene content from glass reflection by integrating spatial consistency and edge priors, generating a clear transmission image. Step S10: Use a pre-trained feature extraction network to encode the input image, build a glass shell estimation module on the encoded features, estimate the spatial distribution of the glass shell and generate the corresponding spatially variable confidence map. The glass casing estimation module is represented as follows:

[0035] Among them, receiving The confidence map is used as input and learned for prediction, thus providing accurate reflection location cues for subsequent reflection removal; T represents the target image. This indicates that the image is in pixels. The gradient strength; We obtain a 0-1 mask; Step S11: By integrating spatial consistency and edge prior, gradient calculation is performed on the input image to extract edge information, and an initial estimate of the reflection region is constructed based on the gradient response; By using multi-view feature alignment and reflection prior modeling, gradient calculation is performed on the input multi-view image to extract edge features, and glass candidate regions are constructed based on the edge response. Step S12: By calculating the neighborhood difference of the reflection response map in the horizontal and vertical directions and applying a penalty constraint to the difference, the drastic changes in the reflection response values ​​between adjacent pixels are suppressed, so that the reflection area presents a continuous distribution in space. By combining multi-view geometric consistency constraints, the consistency error between pixels is calculated through cross-view image reprojection in order to identify regions that do not meet the multi-view consistency requirement. The edge information is fused with the multi-view consistency information to obtain the spatial distribution map of the glass shell. Step S13: The reflection region estimation map is used as guiding information and channel-stitched with the original image and input into the image restoration network. During the feature extraction process, the reflection region estimation result is used to generate suppression weights, and the texture and structural information of the transmission layer are restored through the encoder-decoder structure to finally generate a clear transmission image. A confidence map is constructed based on local statistical characteristics and multi-view consistency error to characterize the reliability of the glass region estimation; a spatial smoothing constraint is applied to the spatial distribution map to ensure the continuity of the glass shell region.

[0036] Step S2: Use explicit point cloud constraints to assist in locating the object surface; By using the sparse 3D point cloud obtained by motion recovery structure as an explicit geometric prior, the 3D points in the sparse 3D point cloud are regarded as located on the surface of the object. By constraining its function value in the signed distance function (SDF) to be close to zero, the implicit geometric representation is directly supervised, which helps to determine the position of the object surface. Although the image has been preprocessed to remove the glass cover in the initial stage, locating the surface without the glass cover remains challenging due to residual artifacts. These artifacts lead to feature matching errors, triangulation failures, and increased reprojection errors. Therefore, this patent employs a physics-based dereflection network, where the non-reflective portion effectively eliminates inconsistencies to reduce matching errors. Simultaneously, to ensure that feature points primarily correspond to the true geometric surface, this patent method uses an explicit point cloud to impose strong geometric constraints on the SDF. Specifically, by utilizing a sparse 3D point cloud obtained from a motion recovery structure as an explicit geometric prior, the 3D points in the point cloud are considered to be located on the object surface. By constraining their function value in the signed distance function (SDF) to be close to zero, the implicit geometric representation is directly supervised, aiding in determining the location of the object surface.

[0037]

[0038] This is the SDF loss function. Wherein, The number of sampling points. The SDF value predicted by the network. This represents the actual surface SDF value. It is an L1 norm.

[0039] Step S3: Using a neural volumetric rendering framework, jointly optimize the SDF and radiation field representation to reconstruct the accurate geometry and appearance of the object inside the glass shell. Firstly, based on multi-view Figure 1 Consistency and reflection perception are used to locate pixels in the image that are affected by reflection interference, and their color and photometric errors are weighted. Subsequently, in the volume rendering framework, the SDF used to determine the surface position and the radiation field used to model the view-dependent color are predicted simultaneously for each ray sampling point, and joint backpropagation optimization is performed through rendering errors.

[0040] Residual artifacts appear at inconsistent locations across different views, leading to color inconsistencies and often distorting surface color estimates. Specifically, this patented method combines light color rendering with a Gaussian distribution. To accurately locate reflection anomalies, this patented method empirically estimates viewpoint-dependent reflectivity scores based on object surface color differences. Because exposure and shadow variations are significant in real-world scenes, color distributions can shift, making Mahalanobis distance less robust to inconsistent lighting and limited viewpoint sampling. An anomaly metric (i.e., variance) based on Z-scores is employed in the early training phase, providing more robust estimates under uncontrolled lighting conditions, thus enhancing stability on real-world data. The anomaly detector obtains surface classifications to minimize the difference between rendered and true colors, requiring no 3D supervision.

[0041]

[0042] in, Indicates the first A perspective on light The color on it is uncertain. For light, From the current perspective, For perspective indexing, For reference, number of viewpoints From the perspective exist The color on it. It is an L1 norm. Indicates the first Weight of each perspective Indicates light rays The mean color difference. This represents the standard deviation of color difference. For numerically stable terms, This is a hyperparameter.

[0043] like Figure 6 As shown, Figure 6 This is a diagram illustrating data collection. Figure 6 'a' represents the interference details caused by the glass dome during object reconstruction. Figure 6 b represents the interference of various reflections and glass edges during the shooting process on the shooting perspective. Figure 6 c is a magnified view showing details such as glass reflection interference.

[0044] In the specific implementation process (1) Training data in real-world scenarios a) Lighting and Environmental Settings: Two sets of data were collected, one for the removal and the other for the reconstruction of the glass shell. Simulated artificial and natural light conditions in different scenarios (including indoor and outdoor) significantly improved the generalizability of the data. Target objects of varying difficulty (e.g., texture and shape) were also selected. The influence of multiple light sources was avoided during data acquisition. For the removal of the glass shell, precise alignment was difficult to ensure during removal and placement. This problem was even more pronounced in outdoor environments. Therefore, it was challenging to obtain strictly aligned, realistic data for the reconstruction task.

[0045] b) For the glass removal training data, the scenes are completely different from the reconstructed data. To improve the model's generalization ability, this patent further introduces a more challenging scene. It includes over 1000 images from different scenes. A network pre-trained on the glass removal data is used to process the other unlabeled data. For the 3D reconstruction data: it contains 10 different 360° realistic scenes demonstrating the real-world impact of the glass shell on the object.

[0046] c) The data format was enhanced by transforming the scene and using glass enclosures of different sizes and shapes. This data comprehensively demonstrates the challenges of reconstructing common objects with glass enclosures in the real world. Objects were captured in multiple scenes, with the target object placed on an arbitrary plane and enclosed by a glass enclosure.

[0047] (2) Training of a volume rendering-based reflection perception network

[0048] a) Data Sampling and Forward Rendering. This refers to the data sampling, ray construction, and forward rendering process during the training of the reflexive perceptron. The training process uses a "random image + batch ray sampling" approach: each iteration randomly selects a training image and samples several rays from it, constructing their origin, direction, near / far distance, and ground truth pixels (including optional masks). Whether to use a white background is determined based on the configuration. These rays, along with the current annealing ratio, are then fed into the renderer to obtain the predicted color, weight statistics, and intermediate values ​​related to regularization; the annealing coefficient gradually increases with the number of training steps. The entire process remains modular: the data module is only responsible for rays and ground truth (GT), the renderer is responsible for forward inference, and the loss module calculates the target based on the rendering results.

[0049] b) Loss Function. The training objective of this patent consists of multiple weighted losses, including a color reconstruction term for reflection perception, Eikonal regularization, mask consistency term, and isosurface (SDF) regularization.

[0050]

[0051] in, Enclosure is a hyperparameter. For Eikonal regularization loss, specifically, in At each sampling point, the implicit function is constrained. gradient The mean square error is close to 1. Color loss is represented by the loss of color across all light rays. Predict color With true colors Differences by uncertainty weight The total error is weighted. To address the issue of excessive color errors in specular and specular areas in real-world scenes, leading to overfitting of a small number of unreliable pixels in traditional L1 / L2 training, this method introduces a "reflection score" into the color prediction process for each ray, thus resolving the aforementioned color uncertainty. The color error is scaled based on this score to weaken high-reflectivity pixels and highlight low-reflectivity areas, allowing optimization to focus more on pixels that stably reflect the diffuse appearance and geometry. Simultaneously, a lower bound is set for the score to avoid gradient anomalies. To improve surface localization accuracy, this method introduces point cloud zero-isosurface supervision, projecting observation points onto an implicit signed distance function (SDF) field and constraining its predicted values ​​to approach zero using the L1 norm, thus ensuring that the implicit surface is spatially consistent with the actual observation data. Furthermore, Eikonal regularization is used to constrain the SDF gradient length to approach the unit norm to maintain the continuity of the distance field and avoid local distortions. In addition, the aggregated weights obtained from ray volume rendering are subjected to binary cross-entropy supervision with the real foreground mask to achieve mask consistency constraints in the foreground region. Specifically, accumulating spatial sampling points along camera rays yields the corresponding ray weight distribution, which reflects the contribution of different locations to the final image. By introducing a realistic foreground region mask, a binary cross-entropy constraint is applied to the predicted ray weights, resulting in a higher response in the foreground region and effective suppression of the background region, thereby achieving consistency between the rendering result and the real foreground structure.

[0052] c) Optimization Strategy. This module describes the details of parameter updates and training scheduling. All trainable parameters (including the field network (for predicting SDF, density, or feature representation), the variance network (for adjusting the distribution shape of the SDF-to-volume density mapping), the color rendering network (for predicting color based on features and viewpoint), and the external neural radiation field (NeRF) module (for modeling the scene exterior or background region)) are updated uniformly through the Adam optimizer. The initial learning rate is specified by the configuration item. A two-stage strategy is used for the learning rate: linear growth is used during the warm-up phase; after the warm-up, a cosine annealing strategy is used, so that the base learning rate is scaled by this factor. Each parameter update first clears the gradient to zero, calculates the loss, backpropagates, and then updates the parameters by the optimizer. To ensure training reversibility and visualization, the following workflow is executed at a fixed frequency during training: the current model parameters, optimizer state, and iteration steps are periodically saved as checkpoints; image-level verification is periodically triggered (rendering a full-resolution image at the selected camera viewpoint and exporting the rendered image and normal vector map); and mesh extraction is performed at a set frequency to assist in 3D quality assessment. During training, the annealing ratio accepted by the renderer, whether a white background is used, and the masking strategy are all uniformly passed to the verification and mesh extraction stages to ensure that the training and evaluation conditions are consistent.

[0053] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0054] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause an electronic device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0056] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A three-dimensional reconstruction method for eliminating transparent masks, characterized in that, Includes the following steps: Step S1, pretreatment to remove the glass cover; A physics-based reflection removal module is used to separate real scene content from glass reflection by integrating spatial consistency and edge priors, generating a clear transmission image. Step S10: Use a pre-trained feature extraction network to encode the input image, build a glass shell estimation module on the encoded features, estimate the spatial distribution of the glass shell and generate the corresponding spatially variable confidence map. Step S11: By integrating spatial consistency and edge prior, gradient calculation is performed on the input image to extract edge information, and an initial estimate of the reflection region is constructed based on the gradient response; Step S12: By calculating the neighborhood difference of the reflection response map in the horizontal and vertical directions and applying a penalty constraint to the difference, the drastic changes in the reflection response values ​​between adjacent pixels are suppressed, so that the reflection area presents a continuous distribution in space. Step S13: The reflection region estimation map is used as guiding information and channel-stitched with the original image and input into the image restoration network. During the feature extraction process, the reflection region estimation result is used to generate suppression weights, and the texture and structural information of the transmission layer are restored through the encoder-decoder structure to finally generate a clear transmission image. Step S2: Use explicit point cloud constraints to assist in locating the object surface; By using the sparse 3D point cloud obtained by motion recovery structure as an explicit geometric prior, the 3D points in the sparse 3D point cloud are regarded as located on the object surface. By constraining its function value in the signed distance function SDF to be close to zero, the implicit geometric representation is directly supervised, which helps to determine the position of the object surface. Step S3: Using a neural volumetric rendering framework, jointly optimize the SDF and radiation field representation to reconstruct the accurate geometry and appearance of the object inside the glass shell. First, based on multi-view consistency and reflection perception, the pixels in the image affected by reflection interference are located, and their color and photometric errors are weighted. Subsequently, in the volume rendering framework, the SDF used to determine the surface position and the radiation field used to model the view-dependent color are predicted simultaneously for each ray sampling point, and joint backpropagation optimization is performed through rendering errors.

2. The three-dimensional reconstruction method for eliminating transparent masks according to claim 1, characterized in that, In step S10, the glass casing estimation module is represented as follows: Among them, receiving The confidence map is used as input and learned for prediction, thus providing accurate reflection location cues for subsequent reflection removal; T represents the target image. This indicates that the image is in pixels. The gradient strength; We obtain a 0-1 mask.

3. The three-dimensional reconstruction method for eliminating transparent masks according to claim 1, characterized in that, In step S2, specifically, the sparse 3D point cloud obtained by motion reconstruction structure is used as an explicit geometric prior. The 3D points in the point cloud are regarded as located on the object surface, and their function value in the signed distance function SDF is constrained to be close to zero, thereby directly supervising the implicit geometric representation and assisting in determining the position of the object surface. It is the SDF loss function, where, The number of sampling points. The SDF value predicted by the network. This represents the actual surface SDF value. It is an L1 norm.

4. The three-dimensional reconstruction method for eliminating transparent masks according to claim 1, characterized in that, In step S3, specifically, the light color rendering is combined with a Gaussian distribution, and the reflectivity score related to the viewpoint is empirically estimated based on the color difference of the object surface; an anomaly metric variance based on Z-score is used in the early training phase to provide a more robust estimate under uncontrolled lighting conditions, thereby enhancing stability on real-world data; the anomaly detector obtains surface classification to minimize the difference between the rendered color and the real color, without any 3D supervision; represented as: in, Indicates the first A perspective on light Color uncertainty on; For light, From the current perspective, For perspective indexing, For reference, number of viewpoints From the perspective exist The color on; It is an L1 norm. Indicates the first Weight of each perspective Indicates light rays Mean color difference; Indicates the standard deviation of color difference; For numerically stable terms, This is a hyperparameter.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in claims 1 to 4.

Images