3D reconstruction method of concave mirror objects based on single-pixel imaging and implicit representation

Abstract

The present invention discloses a three-dimensional reconstruction method for concave mirror objects based on single-pixel imaging and implicit expression, which mainly solves the limitation problem of traditional reconstruction methods when dealing with multiple reflections of concave mirror objects. It includes: building a phased reconstruction model, which is composed of single-pixel imaging technology based on Fourier spectrum and SDF implicit expression method. In the first stage, single-pixel imaging technology is used to capture the reflected light information, and the point pair relationship is obtained by separating the single reflection and multiple reflection light paths; in the second stage, combined with the SDF implicit expression method, these point pair relationships are used as physical priors to realize the reconstruction of the geometric structure of the object. This method shows excellent adaptability and robustness in the case of complex geometric shapes and reflection interference, and its effectiveness is verified through simulation experiments and actual data acquisition experiments.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and optics, and relates to a concave mirror object reconstruction method based on single-pixel imaging and implicit expression, which is used to accurately process high reflection and multiple reflection phenomena and realize high-quality three-dimensional structure reconstruction of objects with complex geometric shapes. Background Art

[0002] 3D reconstruction of concave mirror objects remains a challenging and critical task in computer vision and optics research. Due to their uniquely complex geometry and extremely high reflectivity, concave mirror objects often induce multiple reflections in practical applications. This phenomenon significantly interferes with and hinders important tasks such as 3D reconstruction and ray tracing.

[0003] Traditional multi-view reconstruction methods deal with this problem mainly based on geometric principles, trying to match the point correspondences in multiple views and then try to estimate the 3D coordinates and depth values ​​of the object. However, the high reflectivity of mirror objects seriously violates the multi-view reconstruction principle. Figure 1 The fundamental principle of consistency makes it difficult for these traditional methods to obtain accurate results during the reconstruction process, resulting in unsatisfactory reconstruction results. Although many researchers have actively explored and tried various different approaches to improve this dilemma, such as using object masks to obtain geometric information to reduce the interference of specular reflections on reconstruction work to a certain extent; focusing on using various image processing techniques to remove reflective components in images; using environmental matting and streak reflectometry to eliminate interference caused by external environmental factors, and using differentiable rendering technology to optimize light path constraints; and introducing advanced technologies such as polarization information and light encoding to further improve the reconstruction effect, unfortunately, when faced with highly complex reflection situations, these methods still have difficulty in effectively processing and coping with them.

[0004] With the rapid development of neural network technology, a series of reconstruction methods based on neural rendering and implicit surface representation have emerged rapidly. Among them, the DVR and IDR methods proposed in 2020 became representative neural surface reconstruction methods at the time. However, they suffer from significant drawbacks: excessive reliance on foreground masks for supervision, which makes their training extremely unstable and difficult to obtain reliable results. NeRF and its variants attempt to construct scenes using volume rendering techniques. However, due to the lack of necessary surface constraints during the modeling process, implicit representation of the scene faces significant difficulties, making it difficult to accurately restore the true form of objects. Recent research works include NeuS, which uses an MLP to convert SDF values ​​into density and employs a sigmoid function for volume rendering; NeRFRen, which trains neural density fields based on the introduction of specular planes; Kopanas et al., which employ Neural Point Catacaustics to improve the rendering quality of specular objects by introducing a transformation field; and Ref-NeRF, which separates diffuse and specular reflections by integrating directional encoding and re-parameterizes the appearance to improve specular rendering. Although these methods have made progress to a certain extent, they generally lack physical prior constraints and can only show good results on synthetic datasets. Once applied to actual complex scenes, especially when dealing with complex light paths, they still seem powerless.

[0005] In the development of imaging detection technology, single-pixel detectors have demonstrated significant advantages in light capture due to their unique operating principles. For example, Pittman et al. successfully implemented two-photon entangled ghost imaging, and Shapiro et al. implemented single-pixel computational ghost imaging. The emergence of these technologies has brought new ideas and methods to the field of imaging detection. Durate et al. proposed a single-pixel imaging technology based on compressed sensing. By projecting a series of unrelated random illumination patterns onto the target object, it successfully obtained undersampled data of the object in the spatial domain. However, the compressed sensing algorithm used in this technology suffers from excessive computational time and is not ideal when imaging complex objects. Summary of the Invention

[0006] To overcome the shortcomings of the aforementioned existing technologies, this paper proposes a concave mirror object reconstruction method based on single-pixel imaging and implicit representation. This method focuses on processing multiple reflection information from concave mirror objects, achieving 3D reconstruction through single-pixel imaging and implicit representation. The method consists of two phases: the first phase uses single-pixel imaging to acquire pixel information, processing the light response coefficient to separate the reflected rays and obtain corresponding relationships; the second phase performs multi-view reconstruction, using point-pair relationships as physical priors and reconstructing the geometric structure using reflection paths. This method provides an innovative solution for high-precision reconstruction, breaking through traditional bottlenecks.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following: a concave mirror object reconstruction method based on single-pixel imaging and implicit expression, comprising the following steps:

[0008] (1) Use Fourier spectrum sampling technology to extract phase information from the object, construct a Fourier basis pattern and project it onto the surface of the concave mirror object to obtain the object to be reconstructed containing the spatial properties of the object surface;

[0009] (2) Build a single-pixel imaging system based on Fourier spectrum, capture the complex light path through structured light projection, use the physical prior provided by the object to be reconstructed, constrain the light path propagation model, and calculate the light intensity;

[0010] (3) Capture the reflected light intensity through a single-pixel detector, separate the light response coefficients (LRC) of single-reflection and multiple-reflection light, and establish the correspondence between camera pixels and LCD screen coordinates;

[0011] (4) Building a geometric reconstruction model based on SDF implicit expression, the geometric reconstruction model includes a surface attribute prediction network and a reflection component rendering network;

[0012] (5) Preprocessing the reflected light information of the concave mirror object, and then using the preprocessed information to train the geometric reconstruction model;

[0013] (6) Use the trained geometric reconstruction model to perform three-dimensional reconstruction of the concave mirror object and generate a high-quality 3D model.

[0014] Furthermore, in step (1), the Fourier basis pattern projected onto the target object is expressed as:

[0015]

[0016] in Indicates the light intensity, Indicates contrast, and yes , The spatial frequency of the direction, is the initial phase; these Fourier-generated base patterns are projected onto the surface of the concave mirror object, and are captured by the camera after being reflected by the concave mirror object. The detector can collect the reflected light intensity expressed as:

[0017]

[0018] in is the projection area of ​​the Fourier basis pattern, Represents the object to be reconstructed, and the Fourier weight coefficient of the mirror object is obtained according to the four-step phase shift algorithm:

[0019] ;

[0020] in is the imaginary unit, and finally the Fourier weight coefficient Do the inverse Fourier transform , and obtain a result proportional to the object image, expressed as:

[0021]

[0022] in Indicates direct proportion.

[0023] Furthermore, the principle of the single-pixel imaging system in step (2) is as follows:

[0024] Assume S1 and S2 are two areas on the LCD screen, It is the point where the camera pixel receives the reflected light. For a single reflection path, the light starts from S1 and reaches the pixel after being reflected by the surface of the mirror object. ; For multiple reflections, the light starts from S2 and reaches the same point after multiple reflections from the mirror object The light intensity captured by the camera pixel is the superposition of the single reflection and multiple reflection light intensities, expressed as:

[0025]

[0026] in and Respectively represent the lighting areas corresponding to single and multiple reflections on the LCD screen, and These two functions represent the difference between the point on the LCD screen and the point on the LCD screen in the case of single reflection and multiple reflection respectively. to camera pixels Light response coefficient LRC; is the light field function, which describes the For the direction of light and phase light field distribution.

[0027] Furthermore, in step (3), the projection of the light response coefficient LRC in the horizontal and vertical directions is analyzed, namely:

[0028]

[0029]

[0030] and is the brightness transfer function in the spatial domain, which describes the and Directional light distribution; is the symbol for the inverse Fourier transform, which is used to convert the representation in the frequency domain into the representation in the spatial domain; and is the brightness transfer function in the frequency domain, which describes the and The frequency characteristics of the light distribution in the direction, where and Respectively expressed in and frequency components in the direction.

[0031] Considering each pixel of the camera as a single-pixel detector, the LCD screen area associated with single and multiple reflections of the mirror object is separated into two peaks in the projected one-dimensional LRC. The double-Gaussian model is then used to obtain the sub-pixels of the two peaks. Subsequently, the geometric optical analysis method is used to extract the peak of the single reflection, thereby establishing a correspondence between each camera pixel and the LCD screen.

[0032] Furthermore, in step (4), the position encoding PE is used to process the sampled light path to capture geometric features and spatial information, and this information is input into the surface attribute prediction network to predict the normal of the object surface. , diffuse reflection Weight, roughness and material properties ; Then the integrated direction encoding IDE converts the predicted surface attribute information into parameters related to lighting and reflection and inputs them into the reflection component rendering network. Finally, the reflection component rendering network calculates the specular reflection Components, which are further processed and ultimately rendered as pixel colors .

[0033] Furthermore, the specific implementation of step (5) includes the following sub-steps:

[0034] (5a) Through ray tracing and light path sampling, the propagation of light in the scene is simulated to complete the reflection classification after the light interacts with the concave surface;

[0035] (5b) The separated light path information and point pair relationship are input into the geometric reconstruction model based on SDF implicit expression, and the pixel color is calculated according to the comprehensive bidirectional reflectance distribution function (BRDF) description method based on the Cook-Torrance model;

[0036] (5c) Train the geometric reconstruction model based on the implicit expression of SDF, optimize the weights of the neural network, and minimize the loss function, including the normal vector loss and the difference between the rendered image and the input image.

[0037] Furthermore, the specific implementation of (5a) is as follows:

[0038] Any pixel on a given image , defining the rays emitted from it ,in It's the camera point. is the direction of the light, and the color value is accumulated along the direction of the ray to get the color representation of the pixel. The color of this pixel is defined as:

[0039]

[0040] in It is The color of the sampling point, It is the first The weight of the sampling points, is the opacity density function, which describes the The scattering and absorption characteristics of Cumulative function of Defined from the camera optical center to The transmittance of a point, which describes the energy attenuation of light passing through the scene; is an integral variable representing the integral from the camera point Along the light direction Arrival location Every point on the path;

[0041] When classifying reflected light, for a single reflection light path, the light starts from the starting point Starting from the concave mirror surface, it reflects once and reaches the LCD screen point , through the reflection direction Reparameterize the viewing direction; is the direction of the reflected light, is the normal vector of the point;

[0042] The multiple reflection process is recursive. For each reflection point, the new reflected light will reflect off the surface, and the new reflection direction will be calculated again. At each reflection, the intensity of the reflected light will be calculated according to the BRDF formula. The reflection direction is predicted based on the surface properties. , the normal vector of the object surface and SDF, calculate a point on the concave mirror surface , at a point on the known incident light In this case, calculate the reflection direction:

[0043]

[0044] in, Indicates the number of times the light is reflected. It is The direction vector of the secondary reflection, It is Normal vector of the secondary reflection point; finally, the light reaches the point on the screen after multiple reflections .

[0045] Furthermore, the specific implementation of (5b) is as follows:

[0046] The calculation formula of specular reflection BRDF is:

[0047]

[0048] in Describes the direction of light when it is reflected from the surface of an object. Reflected to the outgoing direction Reflection characteristics; half-angle vector Indicates the direction of incident light and viewing direction The average value of the unit vector, roughness Determines the concentration of highlights. Indicates the direction of the microsurface normal;

[0049] For the diffuse reflection part, the Lambertian lighting model is used, and its BRDF calculation formula is:

[0050]

[0051] in Describes the reflection characteristics of light when it is diffusely reflected on the surface of an object, albedo Indicates the ability of the object surface to reflect light; for mirror objects, the BRDF function combines the specular reflection and diffuse reflection parts, where the metalness Control the weight of the two. The specific expression is:

[0052] ;

[0053] In the incident direction and emission direction The characteristics of surface reflection light; predicting material properties through surface property prediction network 、 、 , and adjust the direction of the reflected light , thus obtaining the mirror concave surface point , and finally the diffuse component is processed by the rendering equation and the specular component To calculate its true color , expressed as:

[0054]

[0055] The irradiance rate Represents the total amount of light on the area plane, which is affected by the angle between the incident light and the plane normal The cosine value of That is, when the angle between the direct radiation and the normal line is larger, the light is weaker.

[0056] Furthermore, the loss function in (5c) consists of three parts: normal vector loss , rendering loss , Eikonal regularization loss , the loss function is defined as:

[0057]

[0058] in and Control the effects of normal vector loss and regularization loss on training respectively;

[0059] The normal loss optimizes the accuracy of the geometry by minimizing the difference between the surface normal and the Snell normal and constraining the local smoothness of the light depth value. Specifically, during the light calculation process, the predicted normal is used to process the SDF surface points. For each point on the surface , output a three-dimensional vector in the surface attribute prediction network and normalize it to obtain the predicted normal vector , calculate the predicted normal vector and the normal vector of the point pair relationship The loss value :

[0060]

[0061] where it is defined , Represents the depth value of the light along the direction of the reflected light, measured by the distance between the camera projection center and the intersection point of the SDF surface. is the depth value predicted by the surface attribute prediction network; and Represents the distance between vectors; set Only rays that undergo single and double bounces are included;

[0062] Furthermore, the rendering loss is used to minimize the rendering color and true colors The specific formula is:

[0063]

[0064] in is the L1 norm;

[0065] Eikonal regularization loss: In order to regularize the gradient of the surface attribute prediction network and avoid overfitting, the Eikonal regularization term is added To regularize the SDF function , thus ensuring The function gradient modulus is kept at 1, which conforms to the basic characteristics of the distance field.

[0066]

[0067] in is the number of sampling points, Refers to sampling points, Is the network Sampling points predictions, is the gradient of this prediction, and is the L2 norm of the gradient, that is, the modulus of the gradient.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] This paper addresses the limitations of existing 3D reconstruction methods for concave mirror objects when dealing with multiple reflections and complex optical paths, such as the poor performance of traditional multi-view reconstruction methods and the lack of physical prior constraints in neural rendering-based methods. By doing so, a method for reconstructing concave mirror objects based on single-pixel imaging and implicit representation is proposed. Single-pixel imaging technology is used to separate reflected light rays to obtain corresponding relationships, and SDF implicit representation is used in conjunction with physical priors to reconstruct the geometric structure. Its effectiveness has been verified through design simulations and actual experiments. Compared with existing technologies, this method is more adaptable and robust in complex scenarios, breaking through traditional bottlenecks and achieving high-precision 3D reconstruction of concave mirror objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is the overall technical flow chart of the present invention.

[0071] Figure 2 This is a flow chart of point-to-point relationship processing of the present invention.

[0072] Figure 3 The single-pixel imaging method of the present invention is used to collect the optical path diagram of the concave mirror object, wherein (a) is the generation and acquisition of the Fourier basis pattern for projection; (b) indicates that the light emitted by the LCD screen is reflected by the concave mirror object to form single reflection and multiple reflection light, and is received by the camera; (c) indicates that multiple reflections produce a double image phenomenon, which is displayed as a ghost image; (d) indicates that the characteristics of the single and multiple reflection light are separated and analyzed by Fourier transforming the horizontal and vertical projections; and (e) indicates that the pixel correspondence between the LCD screen and the camera imaging plane is established to identify the single and multiple reflection imaging points.

[0073] Figure 4 This is a diagram of the geometric reconstruction model architecture based on SDF implicit expression of the present invention. DETAILED DESCRIPTION

[0074] The present invention will be further described below with reference to the accompanying drawings.

[0075] like Figure 1 As shown, the embodiment of the present invention provides a 3D reconstruction method for concave mirror objects based on single-pixel imaging and implicit expression. The overall implementation process is as follows:

[0076] Step 1: Use Fourier spectrum sampling technology to extract phase information from the object, construct a Fourier basis pattern and project it onto the surface of the concave mirror object to obtain the object to be reconstructed containing the spatial properties of the object surface.

[0077] Fourier spectrum sampling technology is a sampling method based on the Fourier transform principle. It converts the signal from the time domain or spatial domain to the frequency domain and samples the resulting Fourier spectrum to obtain discrete information for subsequent processing and analysis. To obtain the Fourier coefficients of the object image, a Fourier basis pattern is first generated using a computer, the pattern is projected onto the object, and the resulting light signal is detected using a single-pixel detector. The Fourier basis pattern projected onto the target object can be expressed as:

[0078]

[0079] in Indicates the light intensity, Indicates contrast, and yes , The spatial frequency of the direction, As the initial phase, we can take Four values, these Fourier-generated base patterns are projected onto the surface of the concave mirror object, and are captured by the camera after being reflected by the concave mirror object. The detector can collect the reflected light intensity expressed as:

[0080]

[0081] in is the projection area of ​​the Fourier basis pattern, Represents the object to be reconstructed. According to the four-step phase shift algorithm, the Fourier weight coefficient of the mirror object can be obtained:

[0082]

[0083] in is the imaginary unit, and finally the Fourier weight coefficient Do the inverse Fourier transform , and obtain a result proportional to the object image, expressed as:

[0084]

[0085] Step 2: Build a single-pixel imaging system based on Fourier spectrum. The system captures complex light paths through structured light projection and single-pixel detectors, uses the physical priors provided by the object to be reconstructed (such as surface roughness and specular reflection characteristics), constrains the light path propagation model, and calculates the light intensity.

[0086] like Figure 2As shown in the figure, during the imaging process, the stripe pattern emitted by the LCD (Liquid Crystal Display) screen is reflected by the mirror object and reaches the imaging plane through the optical center of the camera. Due to the high reflectivity of the mirror object, one pixel of the camera can correspond to multiple reflected light signals, and there may be single reflection and multiple reflection paths. In order to find the correspondence between the camera pixel and the LCD screen coordinates, it is necessary to construct a coefficient matrix for the camera pixel with the same resolution as the LCD screen. Assume that S1 and S2 are two areas on the LCD screen, and P is the point where the camera pixel receives the reflected light. For a single reflection path, the light starts from S1, reflects off the surface of the mirror object, and reaches the pixel. ; For multiple reflections, the light starts from S2 and reaches the same point after multiple reflections from the mirror object Since the multiple reflection path is longer than the single reflection path, the attenuation and propagation time of the light are different, and the fringe information captured by the camera is relatively blurred, which generates the so-called ghost imaging. Therefore, the light intensity captured by the camera pixel is the superposition of the single reflection and multiple reflection light intensities, which can be expressed as:

[0087]

[0088] in and Respectively represent the lighting areas corresponding to single and multiple reflections on the LCD screen, and These two functions represent the difference between the point on the LCD screen and the point on the LCD screen in the case of single reflection and multiple reflection respectively. to camera pixels Light response coefficient LRC; is the light field function, which describes the For the direction of light and phase light field distribution.

[0089] Step 3: Capture the reflected light intensity through a single-pixel detector, separate the light response coefficient (LRC) of single reflection and multiple reflection light, and establish the correspondence between the camera pixels and the LCD screen coordinates.

[0090] Reference Figure 3 As shown, by performing Fourier transform on the horizontal and vertical projections, the characteristics of single and multiple reflected light can be separated and analyzed. In order to obtain the correspondence between camera pixels and LCD screen coordinates, the horizontal and vertical projections of LRC must be analyzed, namely:

[0091]

[0092]

[0093] and is the brightness transfer function in the spatial domain, which describes the and Directional light distribution; is the symbol for the inverse Fourier transform, which is used to convert the representation in the frequency domain into the representation in the spatial domain; and is the brightness transfer function in the frequency domain, which describes the and The frequency characteristics of the light distribution in the direction, where and Respectively expressed in and frequency components in the direction.

[0094] Each pixel of the camera can be regarded as a single-pixel detector. In the projected one-dimensional LRC, the LCD screen area associated with the single reflection and multiple reflection light of the mirror object is separated into two peaks. Then, the double Gaussian model is used to obtain the sub-pixels of the two peaks. Subsequently, the geometric optical analysis method can be used to extract the peak of the single reflection, thereby establishing the correspondence between each camera pixel and the LCD screen.

[0095] Step 4: Build a geometric reconstruction model based on SDF implicit expression. The model combines the physical prior constraints on the reflected light path. The model consists of a surface property prediction network and a reflection component rendering network.

[0096] Reference Figure 4 ,First, use the ray tracing method to sample the light path ,in It is the starting point of light. is the direction of the light, is a parameter representing the position of the light on the path. When light is emitted from the camera and interacts with a concave object, the reflection is divided into single reflection and multiple reflection, which have been distinguished in the previous processing. In this geometric reconstruction model, the sampled light path is first processed using position encoding (PE) to capture geometric features and spatial information. This information is input into the surface property prediction network (SDF network) to predict the normal of the object surface. , diffuse reflection Weight, roughness and material properties Then the integrated direction encoding (IDE) converts the predicted surface attribute information into parameters related to lighting and reflection and inputs them into the reflection component rendering network. Finally, the reflection component rendering network (multi-layer perceptron) calculates the specular reflection Components, which are further processed and ultimately rendered as pixel colors SDF network: The SDF network is a neural network-based representation method used to learn and predict the SDF value of a point in space. It usually approximates the SDF function through a deep learning model (such as a multi-layer perceptron MLP).

[0097] Step 5: Preprocess the reflected light information of the concave mirror object, and then use the processed information to train the geometric reconstruction model, which specifically includes the following sub-steps:

[0098] 5.1) Through ray tracing and light path sampling, the propagation of light in the scene is simulated to complete the reflection classification after the light interacts with the concave surface. The zero-level set representation of SDF (Signed Distance Field is a mathematical function used to represent the distance from any point in space to the nearest object surface, and distinguishes points inside, outside, or on the surface of the object by symbols. SDF can be explicitly stored in a grid or represented by an implicit function.) is used to generate images, thereby optimizing the parameters of the neural SDF and color field. Specifically, for any pixel point on a given image, , defining the rays emitted from it ,in It's the camera point. is the direction of the light, and the color value is accumulated along the direction of the ray to obtain the color representation of the pixel. The color of this pixel can be defined as:

[0099]

[0100] in It is The color of the sampling point, It is the first The weight of the sampling points is based on the opaque density function proposed in the Neus method. The function combines the gradient characteristics of the SDF surface to ensure the accuracy of the surface normal vector during rendering, while setting the weight peak at the intersection of the light and the object to improve the geometric expression ability. is the opacity density function, which describes the In addition, Cumulative function of Defined from the camera optical center to The transmittance of a point, which describes the energy attenuation of light passing through the scene. is an integral variable representing the integral from the camera point Along the light direction Arrival location This design effectively enhances the adaptability of rendering to complex geometric surfaces.

[0101] When classifying reflected light, for a single reflection light path, the light starts from the starting point Starting from the concave mirror surface, it reflects once and reaches the LCD screen point , through the reflection direction Reparameterize the viewing direction; is the direction of the reflected light, This reparameterization makes the relationship between the viewing direction and the reflection direction more direct, which helps to describe the BRDF (bidirectional reflectance distribution function) characteristics of specular objects.

[0102] The multiple reflection process is recursive. For each reflection point, the new reflected light will reflect off the surface, and the new reflection direction will be calculated again. At each reflection, the intensity of the reflected light will be calculated according to the BRDF formula. The network outputs the reflection direction based on the surface properties prediction. , the normal vector of the object surface and SDF, calculate a point on the concave mirror surface , at a point on the known incident light In this case, calculate the reflection direction:

[0103]

[0104] in, Indicates the number of times the light is reflected. It is The direction vector of the secondary reflection, It is Normal vector of the secondary reflection point; finally, the light reaches the point on the screen after multiple reflections .

[0105] 5.2) Input the separated light path information and point pair relationship into the geometric reconstruction model based on SDF implicit expression, and calculate the pixel color according to the comprehensive BRDF description method based on the Cook-Torrance model. First, BRDF (bidirectional reflectance distribution function) is used to define the effect of the radiant illumination in a given incident direction on the radiance in a given outgoing direction, which consists of specular reflection and diffuse reflection. In order to accurately model specular reflection, the Cook-Torrance model is often used in the BRDF lighting model to represent the specular reflection part. This model uses the normal distribution function , Fresnel equation and geometric functions To comprehensively model the optical reflection behavior of the surface, so as to accurately describe the specular reflection effect of the surface. The calculation formula of the specular reflection BRDF is:

[0106]

[0107] in Describes the direction of light when it is reflected from the surface of an object. Reflected to the outgoing direction Reflection characteristics; half-angle vector Indicates the direction of incident light and viewing direction The average value of the unit vector, roughness Determines the concentration of highlights. Indicates the direction of the microsurface normal.

[0108] For the diffuse reflection part, the Lambertian lighting model is usually used, and its BRDF calculation formula is:

[0109]

[0110] in Describes the reflection characteristics of light when it is diffusely reflected on the surface of an object, albedo Indicates the ability of an object's surface to reflect light. For mirror objects, the BRDF function combines specular reflection and diffuse reflection, where metalness Control the weight of the two. The specific expression is:

[0111]

[0112] In the incident direction and emission direction The characteristics of surface reflection light; predicting material properties through surface property prediction network 、 、 , and adjust the direction of the reflected light , thus obtaining the mirror concave surface point , and finally the diffuse component is processed by the rendering equation and the specular component To calculate its true color , expressed as:

[0113]

[0114] The irradiance rate Represents the total amount of light on the area plane, which is affected by the angle between the incident light and the plane normal The cosine value of That is, when the angle between the direct radiation and the normal line is larger, the light is weaker.

[0115] 5.3) Train the geometric reconstruction model based on the implicit expression of SDF, optimize the weights of the neural network, minimize the normal vector loss and the difference between the rendered image and the input image. The loss function consists of three parts: normal vector loss, rendering loss, and Eikonal regularization loss:

[0116] (1) Normal loss: This optimizes the accuracy of the geometry by minimizing the difference between the surface normal and the Snell normal and constraining the local smoothness of the ray depth value. Specifically, during the ray calculation process, the predicted normal is used to process the SDF surface points. For each point on the surface, , output a three-dimensional vector in the surface attribute prediction network and normalize it to obtain the predicted normal vector , calculate the predicted normal vector and the normal vector of the point pair relationship The loss value :

[0117]

[0118] where it is defined , Represents the depth value of the light along the direction of the reflected light, measured by the distance between the camera projection center and the intersection point of the SDF surface. is the depth value predicted by the surface attribute prediction network; and Represents the distance between vectors; set Only rays that have a single or double bounce are included, while rays with more than two bounces have been discarded in the above process.

[0119] (2) Rendering loss: In order to ensure the visual realism of the rendering results and further optimize the reflection component rendering network, a rendering loss function is designed. , used to minimize rendering colors and true colors The specific formula is:

[0120]

[0121] in It is the L1 norm, which can effectively measure the absolute error between the two and ensure that the generated rendered image is more consistent with the real scene.

[0122] (3) Eikonal regularization loss: In order to regularize the gradient of the surface attribute prediction network and avoid overfitting, it is necessary to add the Eikonal regularization term To regularize the SDF function , thus ensuring The function gradient modulus is kept at 1, which conforms to the basic characteristics of the distance field.

[0123]

[0124] in is the number of sampling points, Refers to sampling points, Is the network Sampling points predictions, is the gradient of this prediction, and is the L2 norm of the gradient, that is, the modulus of the gradient. Through this regularization term, the geometric shape of the SDF is more stable and unnecessary fluctuations are reduced.

[0125] Combining the above constraints, the total loss function of the model is defined as:

[0126]

[0127] in and The effects of normal vector consistency loss and regularization loss on training are controlled separately. By combining geometric consistency, color reconstruction, and gradient regularization, this framework effectively improves the 3D reconstruction quality and rendering performance of specular and concave objects, providing theoretical support and practical assurance for high-precision reconstruction in complex scenes.

[0128] Step 6: Input a set of concave mirror object images with camera pose information, use the optimized geometric reconstruction model to perform three-dimensional reconstruction of the concave mirror object, and generate a high-quality 3D model.

[0129] The entire network framework is assisted by the integrated direction encoding (IDE) module, which further optimizes the processing process after each reflection. The rendering module recursively calculates all reflection points and finally generates a complete pixel color. This recursive calculation method not only accurately describes the behavior of light reflecting off a concave mirror, but also effectively addresses the complexity of multiple reflection paths. Through this improvement, the framework successfully overcomes the limitations of traditional methods in dealing with complex reflection paths, significantly improving the accuracy and detail integrity of the final 3D reconstruction results.

[0130] Next, we will illustrate the effects of the invention through specific simulation experiments:

[0131] The specific hardware equipment for this simulation experiment is as follows: computer running memory is 32GB, processor model is i7-12700, graphics card is RTX3080Ti, and video memory is 12GB;

[0132] The specific software environment for this simulation experiment is: Ubuntu 20.04.5 operating system, Pytorch 1.10.0, CUDA 11.3, cuDNN 8, Python 3.7;

[0133] The dataset used in this simulation experiment consists of eight concave objects with low roughness and strong reflectivity. For each object, we used the Cycles renderer in Blender to uniformly render 70 images of the concave surfaces at a resolution of 800 × 800 pixels around the object. We then used OptiX technology for ray tracing to calculate the point-to-point relationship between each reflection point and the ray.

[0134] In simulation experiment 1, image quality was evaluated using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). PSNR measures pixel-level distortion by calculating the mean squared error (MSE) between the original and reconstructed images. Higher values ​​indicate better quality (typically, PSNR > 30 dB is considered excellent). SSIM assesses image similarity based on brightness, contrast, and structure. The formula combines the local mean, variance, and covariance, and takes a value in the range [-1, 1]. A value closer to 1 indicates greater structural similarity.

[0135] This paper uses the above evaluation metrics to conduct comparative experiments on the following cutting-edge models. The NERO method is from the literature: Liu, Yuan, et al. "Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images." ACM Transactions on Graphics (TOG) 42.4 (2023): 1-22;

[0136] The NeRF method comes from the literature: Mildenhall, Ben, et al. "Nerf: Representing scenesas neural radiance fields for view synthesis." Communications of the ACM 65.1(2021): 99-106;

[0137] The NeuS method comes from the literature: Wang, Peng, et al. "Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction." arXiv preprintarXiv:2106.10689 (2021).

[0138] Table 1 Experimental results of four 3D reconstruction methods

[0139]

[0140] As shown in Table 1, the PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) values ​​of the concave mirror object 3D reconstruction method based on single-pixel imaging and implicit expression are higher than those of other methods.

[0141] Based on the above experimental results, it is proved that the method proposed in the present invention has stronger adaptability and robustness in complex scenes, can break through the traditional bottleneck, and achieve high-precision three-dimensional reconstruction of concave mirror objects.

[0142] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. A 3D reconstruction method for concave mirror objects based on single-pixel imaging and implicit expression, characterized by: The steps include: (1) Use Fourier spectrum sampling technology to extract phase information from the object, construct a Fourier basis pattern and project it onto the surface of the concave mirror object to obtain the object to be reconstructed containing the spatial properties of the object surface; (2) Build a single-pixel imaging system based on Fourier spectrum, capture the complex light path through structured light projection, use the physical prior provided by the object to be reconstructed, constrain the light path propagation model, and calculate the light intensity; (3) Capture the reflected light intensity through a single-pixel detector, separate the light response coefficients (LRC) of single-reflection and multiple-reflection light, and establish the correspondence between camera pixels and LCD screen coordinates; (4) Building a geometric reconstruction model based on SDF implicit expression, the geometric reconstruction model includes a surface attribute prediction network and a reflection component rendering network; (5) Preprocessing the reflected light information of the concave mirror object, and then using the preprocessed information to train the geometric reconstruction model; The specific implementation of step (5) includes the following sub-steps: (5a) Through ray tracing and light path sampling, the propagation of light in the scene is simulated to complete the reflection classification after the light interacts with the concave surface; The specific implementation of (5a) is as follows: Any pixel on a given image , defining the rays emitted from it ,in It's the camera point. is the direction of the light, and the color value is accumulated along the direction of the ray to get the color representation of the pixel. The color of this pixel is defined as: ； in It is The color of the sampling point, It is the first The weight of the sampling points, is the opacity density function, which describes the The scattering and absorption characteristics of Cumulative function of Defined from the camera optical center to The transmittance of a point, which describes the energy attenuation of light passing through the scene; is an integral variable representing the integral from the camera point Along the light direction Arrival location Every point on the path; When classifying reflected light, for a single reflection light path, the light starts from the starting point Starting from the concave mirror surface, it reflects once and reaches the LCD screen point , through the reflection direction Reparameterize the viewing direction; is the direction of the reflected light, is the normal vector of the point; The multiple reflection process is recursive. For each reflection point, the new reflected light will reflect off the surface, and the new reflection direction will be calculated again. At each reflection, the intensity of the reflected light will be calculated according to the BRDF formula. The reflection direction is predicted based on the surface properties. , the normal vector of the object surface and SDF, calculate a point on the concave mirror surface , at a point on the known incident light In this case, calculate the reflection direction: ； in, Indicates the number of times the light is reflected. It is The direction vector of the secondary reflection, It is Normal vector of the secondary reflection point; finally, the light reaches the point on the screen after multiple reflections ; (5b) The separated light path information and point pair relationship are input into the geometric reconstruction model based on SDF implicit expression, and the pixel color is calculated according to the comprehensive bidirectional reflectance distribution function (BRDF) description method based on the Cook-Torrance model; (5c) training the geometric reconstruction model based on the implicit expression of SDF, optimizing the weights of the neural network, and minimizing the loss function, including the normal vector loss and the difference between the rendered image and the input image; (6) Use the trained geometric reconstruction model to perform three-dimensional reconstruction of the concave mirror object and generate a high-quality 3D model.

2. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 1, wherein: In step (1), the Fourier basis pattern projected onto the target object is expressed as: ； in Indicates the light intensity, Indicates contrast, and yes , The spatial frequency of the direction, is the initial phase; these Fourier-generated base patterns are projected onto the surface of the concave mirror object, and are captured by the camera after being reflected by the concave mirror object. The reflected light intensity collected by the detector is expressed as: ； in is the projection area of ​​the Fourier basis pattern, Represents the object to be reconstructed, and the Fourier weight coefficient of the mirror object is obtained according to the four-step phase shift algorithm: ； in is the imaginary unit, and finally the Fourier weight coefficient Do the inverse Fourier transform , and obtain a result proportional to the object image, expressed as: ； in Indicates direct proportion.

3. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 1, wherein: The principle of the single-pixel imaging system in step (2) is as follows: Assume S1 and S2 are two areas on the LCD screen, It is the point where the camera pixel receives the reflected light. For a single reflection path, the light starts from S1 and reaches the pixel after being reflected by the surface of the mirror object. ; For multiple reflections, the light starts from S2 and reaches the same point after multiple reflections from the mirror object The light intensity captured by the camera pixel is the superposition of the single reflection and multiple reflection light intensities, expressed as: ； in and Respectively represent the lighting areas corresponding to single and multiple reflections on the LCD screen, and These two functions represent the difference between the point on the LCD screen and the point on the LCD screen in the case of single reflection and multiple reflection respectively. to camera pixels Light response coefficient LRC; is the light field function, which describes the For the direction of light and phase light field distribution.

4. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 1, wherein: In step (3), the projection of the light response coefficient LRC in the horizontal and vertical directions is analyzed, namely: ； ； and is the brightness transfer function in the spatial domain, which describes the and Directional light distribution; is the symbol for the inverse Fourier transform, which is used to convert the representation in the frequency domain into the representation in the spatial domain; and is the brightness transfer function in the frequency domain, which describes the and The frequency characteristics of the light distribution in the direction, where and Respectively expressed in and Frequency components in the direction; Considering each pixel of the camera as a single-pixel detector, the LCD screen area associated with single and multiple reflections of the mirror object is separated into two peaks in the projected one-dimensional LRC. The double-Gaussian model is then used to obtain the sub-pixels of the two peaks. Subsequently, the geometric optical analysis method is used to extract the peak of the single reflection, thereby establishing a correspondence between each camera pixel and the LCD screen.

5. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 1, wherein: In step (4), the position encoding PE is first used to process the sampled light path to capture geometric features and spatial information, and this information is input into the surface attribute prediction network to predict the normal of the object surface. , diffuse reflection Weight, roughness and material properties ; Then the integrated direction encoding IDE converts the predicted surface attribute information into parameters related to lighting and reflection and inputs them into the reflection component rendering network. Finally, the reflection component rendering network calculates the specular reflection Components, which are further processed and ultimately rendered as pixel colors .

6. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 1, wherein: The specific implementation of (5b) is as follows: The calculation formula of specular reflection BRDF is: ； in Describes the direction of light when it is reflected from the surface of an object. Reflected to the outgoing direction Reflection characteristics; half-angle vector Indicates the direction of incident light and viewing direction The average value of the unit vector, roughness Determines the concentration of highlights. Indicates the direction of the microsurface normal; For the diffuse reflection part, the Lambertian lighting model is used, and its BRDF calculation formula is: ； in Describes the reflection characteristics of light when it is diffusely reflected on the surface of an object, albedo Indicates the ability of the object surface to reflect light; for mirror objects, the BRDF function combines the specular reflection and diffuse reflection parts, where the metalness Control the weight of the two. The specific expression is: ； In the incident direction and emission direction The characteristics of surface reflection light; predicting material properties through surface property prediction network 、 、 , and adjust the direction of the reflected light , thus obtaining the mirror concave surface point , and finally the diffuse component is processed by the rendering equation and the specular component To calculate its true color , expressed as: ； The irradiance rate Represents the total amount of light on the area plane, which is affected by the angle between the incident light and the plane normal The cosine value of That is, when the angle between the direct radiation and the normal line is larger, the light is weaker.

7. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 1, wherein: The loss function in (5c) consists of three parts: normal vector loss , rendering loss , Eikonal regularization loss , the loss function is defined as: ； in and Control the effects of normal vector loss and regularization loss on training respectively; The normal loss optimizes the accuracy of the geometry by minimizing the difference between the surface normal and the Snell normal and constraining the local smoothness of the light depth value. Specifically, during the light calculation process, the predicted normal is used to process the SDF surface points. For each point on the surface , output a three-dimensional vector in the surface attribute prediction network and normalize it to obtain the predicted normal vector , calculate the predicted normal vector and the normal vector of the point pair relationship The loss value : ； where it is defined , Represents the depth value of the light along the direction of the reflected light, measured by the distance between the camera projection center and the intersection point of the SDF surface. is the depth value predicted by the surface attribute prediction network; and Represents the distance between vectors; set Only rays that have single and double bounces are included.

8. The method for 3D reconstruction of concave mirror objects based on single-pixel imaging and implicit representation according to claim 7, wherein: Rendering loss is used to minimize the rendered color and true colors The specific formula is: ； in is the L1 norm; Eikonal regularization loss: In order to regularize the gradient of the surface attribute prediction network and avoid overfitting, the Eikonal regularization term is added To regularize the SDF function , thus ensuring The function gradient modulus is kept at 1, which conforms to the basic characteristics of the distance field. ； in is the number of sampling points, Refers to sampling points, Is the network Sampling points predictions, is the gradient of this prediction, and is the L2 norm of the gradient, that is, the modulus of the gradient.

Images