3D reconstruction method of surround scene based on implicit representation

Abstract

The present invention discloses a method for three-dimensional reconstruction of surround-view scenes based on implicit representation, including: using a laser to illuminate a relay wall surface, forming a secondary diffuse reflection spot on the relay wall surface, and generating hidden object projections at different angles on the opposite wall by illuminating different positions on the relay wall; estimating the three-dimensional coordinates of the secondary light source by a light source estimation method; establishing a neural shadow field model by the physical relationship that the secondary light source emits light in all directions in a hemispherical space and terminates at the observation relay wall, and using the cumulative transmittance of light to characterize the probability of light propagating to the surface of the relay wall; representing the three-dimensional space parameterization of the neural shadow field model as a continuous MLP function, and using a binary shadow label to supervise the optimization of the cumulative transmittance of light, and finally obtaining the three-dimensional structure of the hidden space by depth integration. The method can accurately reconstruct the shape and contour details of the hidden object.

Description

Technical Field

[0001] The present invention belongs to the field of non-line-of-sight imaging, and in particular relates to a three-dimensional reconstruction method of a surround view scene based on implicit representation. Background Art

[0002] Traditional imaging systems can only record scene information within the field of view. Once the imaging target is blocked by an obstruction, the detector cannot image the obscured target. Non-line-of-sight (NLOS) imaging technology aims to recover hidden objects using signals scattered by relay walls. This technology has high application potential in fields such as autonomous driving, medical endoscopy and urban counterterrorism.

[0003] Most existing non-line-of-sight imaging methods are set up in corner scenarios with a single relay wall, but in daily life there are many scenarios with opposing relay walls on both sides, such as tunnels, corridors, and alleys. Additional relay walls can provide more information about hidden objects. This imaging problem is called double-reflection non-line-of-sight imaging. In this setting, one side of the relay wall can be illuminated with a laser, and the measurement light received on the other side is recorded. By adjusting the illumination position, multi-view double-reflection light measurements can be collected, and the three-dimensional structure of the hidden object can be reconstructed from these measurements through an algorithm.

[0004] Existing methods have explored schemes using different detectors, such as transient measurements using single-photon avalanche diodes (SPADs), which can take advantage of multiplexed illumination to measure occluded objects. However, the photon signal received by the SPAD is severely attenuated, especially in large-scale scenes with long flight distances, resulting in poor reconstruction quality. In addition, SPAD detectors are expensive, making it challenging to achieve fast detection at low cost. Reconstructing 3D objects from shadow images captured by RGB / RGB-D cameras is a low-cost and effective solution. However, current probability-based space carving methods are affected by the accuracy of pose calibration and the density of discrete grids. They can reconstruct the approximate outline of hidden objects, but lack the ability to reconstruct detailed structures. Summary of the invention

[0005] The object of the present invention is to provide a method for three-dimensional reconstruction of a surround view scene based on implicit representation.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for three-dimensional reconstruction of a surround scene based on implicit representation, comprising:

[0007] Use laser to illuminate the relay wall surface to form a secondary diffuse reflection spot on the relay wall surface, and generate hidden object projections at different angles on the opposite wall by illuminating different positions on the relay wall;

[0008] If the position of the secondary light source cannot be measured, the three-dimensional coordinates of the secondary light source are estimated by a light source estimation method;

[0009] A neural shadow field model is established through the physical relationship that the secondary light source emits light in all directions in the hemispherical space and terminates at the observation relay wall, and the cumulative transmittance of the light is used to characterize the probability of the light propagating to the surface of the relay wall;

[0010] The three-dimensional space parameterization for building the neural shadow field model is represented as a continuous MLP function, and the optimization of the light cumulative transmittance is supervised using binary shadow labels, and finally the three-dimensional structure of the hidden space is obtained through deep integration.

[0011] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0012] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0013] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: the present method achieves high-precision three-dimensional reconstruction of hidden spaces by using a low-cost laser-visible light camera system, while most existing methods have the advantages of high resolution and applicability to large-scale scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 This is a typical schematic diagram of a non-viewing scene.

[0017] Figure 2 Schematic diagram of the NeSF method.

[0018] Figure 3 Schematic diagram of the light source estimation algorithm.

[0019] Figure 4 Reconstruction result map of complex scenes in simulation environment.

[0020] Figure 5 This is the light source estimation test and reconstruction result diagram in the simulation environment.

[0021] Figure 6 This is the result of light source estimation and reconstruction in a real small scene.

[0022] Figure 7 Reconstruction result map for real indoor scenes.

[0023] Figure 8 A schematic diagram for comparing the methods. DETAILED DESCRIPTION

[0024] In recent years, Neural Radiance Field (NeRF) has been successfully applied to multi-view reconstruction. NeRF adopts differentiable neural rendering technology, uses multi-layer perception (MLP) to map 3D spatial coordinates to scene attributes, and reconstructs new viewpoint images through volume rendering. This continuous scene representation can reconstruct the target at arbitrary resolution. Inspired by this, we propose Neural Shadow Field (NeSF) to achieve detailed 3D reconstruction of objects hidden in double-reflected non-viewing scenes.

[0025] NeSF aims to reconstruct hidden objects using shadows projected from different stereo viewpoints, and characterizes the space to be reconstructed by predicting the volume density of 3D points using MLP. This method uses a secondary light source formed by laser irradiation of the relay wall to emit light in all directions in the hemispherical space, while tracing the light that terminates at the observed relay wall. The cumulative transmittance of these light rays is calculated, and the cumulative transmittance is supervised by the binary segmented shadow image to optimize and finally achieve accurate rendering of the depth map of the scene from different viewpoints. In addition, considering that double-reflection non-line-of-sight scenes usually involve parallel relay walls, we propose a light source estimation algorithm to simplify the imaging system of scenes with two parallel relay walls. By exploiting the geometric similarity between the multi-view shadow contours generated by the motion of the light source, we can track the motion trajectory of the light source and thus locate the secondary light source. This process does not require the participation of an RGB-D camera, and the imaging system only needs one RGB camera for shadow detection. Typical non-line-of-sight scenes such as Figure 1 shown.

[0026] A method for 3D reconstruction of a surround scene based on implicit representation of the present invention comprises the following steps:

[0027] (1) Use laser to illuminate the relay wall surface to form a secondary diffuse reflection spot on the relay wall surface, and generate hidden object projections at different angles on the opposite wall by illuminating different positions on the relay wall;

[0028] (2) If the position of the secondary light source cannot be measured, the three-dimensional coordinates of the secondary light source are estimated by the light source estimation method; if it can be measured, the measured position of the secondary light source is directly used as the basis for constructing the light in the following steps;

[0029] (3) A neural shadow field model is established based on the physical relationship that a secondary light source emits light in all directions in a hemispherical space and terminates at the observation relay wall. The cumulative transmittance of light is used to characterize the probability that light propagates to the surface of the relay wall. The details are as follows:

[0030] The coordinates of the secondary light source are taken as the starting point of the light, and the three-dimensional space position of the projected image pixel is taken as the end point of the light, and several rays are constructed, and the direction of the light is the three-dimensional unit direction vector of the ray;

[0031] Each point in space has independent and continuous transmittance, and the probability of light propagating to a certain place gradually decays when passing through these points in space;

[0032] By calculating the cumulative transmittance from the starting point to the end point of the light, the probability that the light emitted by the secondary light source propagates to the surface of the relay wall can be characterized.

[0033] (4) The three-dimensional space parameterization of the constructed neural shadow field model is represented as a continuous MLP function, and the optimization of the light cumulative transmittance is supervised by using binary shadow labels, and finally the three-dimensional structure of the hidden space is obtained through deep integration.

[0034] The method involves two walls, one is a reflective wall, which is used to reflect laser to generate secondary diffuse reflection light source, and the other wall is opposite to the reflective wall and parallel to it, which is used to receive the projection of hidden objects. The relay wall refers to the reflective wall, the opposite wall refers to the projection wall, and the observation relay wall also refers to the projection wall.

[0035] For double reflection non-line-of-sight imaging, the present invention proposes a 3D reconstruction algorithm NeSF, which parameterizes the 3D space with hidden scenes into a continuous MLP function, thereby achieving fine 3D reconstruction of hidden objects.

[0036] The present invention also proposes a new shadow constraint to optimize the voxel space. We use the cumulative transmittance of light to characterize the probability of light propagating to the surface of the relay wall, and use binary shadow images to supervise the results.

[0037] The present invention also proposes a light source estimation algorithm, which uses the shadow of the hidden scene in the parallel plane scene to perform geometric calibration on the point light source moving on the relay wall.

[0038] The problem to be solved by the present invention is summarized as follows: the unknown detection target is blocked by an obstacle and hidden in the non-viewing space at any position behind the obstacle; the obstacles are flanked by planar Lambertian relay walls. By illuminating the relay wall with a laser, a secondary light source is generated on the surface of the wall to illuminate the non-viewing space, and these secondary light sources will emit light in all directions of the hemisphere. If the light is blocked by the hidden object during propagation and cannot reach the observation wall on the opposite side, a shadow will be generated in the corresponding area of ​​the observation wall. By adjusting the lighting position on the relay wall, we can collect the projection results of each angle of the hidden scene. Our goal is to reconstruct the geometry of the hidden object from the multi-angle shadow image.

[0039] The present invention develops a detection system consisting of a laser galvanometer and an RGB / RGB-D camera, in which the RGB-D camera is used for spatial positioning of light sources and shadows. After obtaining the relative positional relationship between the light source and the shadow, the proposed NeSF can accurately voxelize the non-viewing space. Considering that the double-reflection scenes mostly involve parallel relay walls, we introduce a light source estimation algorithm, which determines the position of the light source from a continuous sequence of shadow images, thereby realizing the coordinate alignment between the light source and the shadow only through the RGB camera without the participation of the RGB-D camera.

[0040] Figure 2 Schematic diagram of the NeSF method. NeSF represents the scene as a 3D field of directional particles that absorb light emitted only from secondary light sources in the relay wall, without considering the influence of high-order reflected light for ease of calculation. Since we cannot directly capture the color characteristics of hidden objects, spatial geometry reconstruction relies only on shadow information. Therefore, NeSF represents the continuous NLOS space as a learnable 3D function , the spatial coordinates Mapped to volume density with MLP :

[0041] ;

[0042] in Represents position encoding. In order to facilitate MLP to better approximate high-frequency functions, the spatial coordinates will be applied with position encoding before being input into MLP:

[0043] .

[0044] Among them, L represents the encoding dimension;

[0045] NeSF is described as the visibility probability of the light source to the observation wall, so we only focus on the light reaching the observation wall. The visibility is expressed using cumulative transmittance, which is expressed as , which means that light propagates through the interval without scattering The probability of propagation distance is and upper limit Secondary light source rays The expected cumulative transmittance of is:

[0046] .

[0047] in represents the position reached by the light source ray when it propagates to a distance t, Indicates the starting point of the light. The unit direction vector representing the light source ray.

[0048] For each shadow image, we construct rays originating from the corresponding light source. By tracing the rays reaching the observation relay wall, we estimate the integral of the cumulative transmittance using stratified sampling of discrete samples:

[0049]

[0050] It represents the cumulative transmittance of light when it reaches the i-th sampling point, is a composite parameter used to simplify the representation. represents the transmittance of the i-th sampling point, Indicates the distance between it and the adjacent sampling points.

[0051] Subsequently, we perform binary segmentation on the shadow image to mark whether a shadow exists. These labels will be used to supervise the optimization of NeSF by constraining the cumulative transmittance. Pixels not covered by shadows are labeled as 1, indicating that light is transmitted without occlusion. When light is blocked by a hidden object, a shadow is generated on the observation wall. In this case, the cumulative transmittance of light is close to zero, so we mark the shadowed pixels as 0. The loss function of cumulative transmittance can be expressed as:

[0052]

[0053] Among them, S represents the set of rays corresponding to the shadows of each training batch.

[0054] Finally, we can compute the object depth from any viewpoint by numerically integrating the rays :

[0055]

[0056] Represents light The depth of transmission, It represents the cumulative transmittance of light when it reaches the i-th sampling point, Represents the distance between the ith sampling point and its adjacent sampling points, and N represents the maximum number of sampling points for the ray sampling.

[0057] The depth map can represent the three-dimensional depth information of the scene at a certain perspective, thereby inferring the structure of three-dimensional objects in space.

[0058] This method focuses on the geometric calibration of continuously moving lighting points. By slightly moving the lighting points of the reflecting wall, a similar shadow image is formed on the projected wall. By matching the key points of similar contour features in the image, the matched key points are associated with the edge points of the object corresponding to the contour features in space. Finally, the collinear relationship between the key points, object edge points and lighting points is established, and the position of the unknown light source is inverted using the spatial back-projection algorithm. Figure 3 This is a schematic diagram of the light source estimation algorithm. The specific process of light source estimation is as follows:

[0059] Step 1: For the first shadow image in the sequence Feature extraction is performed to find feature points with geometric invariance; the sequence refers to the time-series shadow image sequence captured on the opposite wall for the continuously moving secondary light source; The corresponding lighting point position is set as the origin of the 3D relative coordinate system, denoted as Next, for subsequent images The same feature extractor is also used to extract features and compare them with The features of the image are matched to obtain several key point pairs with similar geometric features. At this point, we will calculate and The average displacement of all keypoint pairs , and The corresponding lighting point position is recorded as .

[0060] Step 2: Since the subsequent calculations are performed in three-dimensional space, we transform the Euclidean coordinates of the key points in the two-dimensional image coordinate system into 3D coordinates aligned with the light source:

[0061] ;

[0062] in, Represents the key points in the image The pixel coordinates of yes The three-dimensional mapping in the coordinate system of the light source, and Indicates the width and height of the image.

[0063] In three-dimensional space, the light source point of the reflection wall and the key point of the projection wall can form a projection ray, and each pair of key points and their corresponding light source points can form two projection rays that approximately intersect, such as Figure 3 As shown in (a) in the figure, the intersection of these two rays is a point on the edge of the object in the hidden space, and this edge point corresponds to a point on the shadow contour. and , the corresponding intersection point can be calculated :

[0064]

[0065]

[0066] in, is a scaling factor, and The light source and Illuminating hidden space points The projected key points formed.

[0067] The positions of these intersection points will be recorded in the 3D point set In addition, since each key point Uniquely corresponds to a point , we can at each key point and A mapping relationship is established between them, and the set of these mappings is represented as .

[0068] Step 3: Added image will be used Key point set and a 3D point set The mapping relationship between them is used to estimate the position of the light source. When we add a new image When the key point set Cannot directly To establish association, we first extract its features and compare its key points with the existing image set Each image in the image is matched, and each time a new key point With a known key point Can be matched, and there is a mapping relationship , then a new mapping can be created , and this mapping is unique.

[0069] Step 4: Based on the projection relationship described in step 2, when we know the mapping between key points and certain points in space, we can establish a back-projection relationship, such as Figure 3 As shown in (b) in the figure. Each mapping relationship All of them can form a back-projection line in space. Ideally, these back-projection lines will converge at the same point on the reflective wall, which is the desired light source point. However, there will be some outliers in the calculation, which can be filtered out using algorithms such as RANSAC or DBSCAN. By calculating the average coordinates of all points after filtering, the image can be estimated. The corresponding light source position .

[0070] Step 5: To ensure that the algorithm can be iterated continuously, we need to use the newly calculated light source position Update Point Set Similar to the method in step 2, we use and image sets The key point pairs matched between each image in and their corresponding light source positions are used to establish intersecting projection lines and calculate an intersection point of these projection lines. and merge them into Then, we will image Add to Image Set middle.

[0071] Finally, for the newly added images , we can establish through steps 3 and 4 Key points and point sets The mapping relationship between them is calculated The lighting point position In addition, through step 5, we can update To ensure the continuation of iteration.

[0072] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0073] Example

[0074] The test of this embodiment will be carried out in both simulated and real-world scenes. The dataset required for reconstruction consists of the 3D coordinates of the light source and its corresponding projected wall shadow labels. Each shadow label is a binary image, where 0 represents shadow and 1 represents background.

[0075] This embodiment uses Blender to model a double-reflection non-viewing scene. Several diffuse planes are used as walls and occluders, and a point light source is placed on the reflective wall as the illumination source. In addition, a camera is placed in front of the occluder to capture the shadow on the observation wall. The position of the point light source can be queried and modified by a script. By changing the position of the light source and capturing the shadow image at the same time, we can generate shadow image data paired with the light source under different configurations.

[0076] A real-world test scene was built using PVC boards. The baffle was 25.0 cm wide and 39.0 cm high, with 59.4 cm wide relay walls on both sides. We used a 200 mW galvo scanning continuous wave laser with a wavelength of 532 nm for illumination, and used two Realsense D435i RGB-D cameras, one for obtaining the light source position and the other for capturing shadows. In addition, we tested the same imaging system in a 5m×4m×3m indoor environment, using the parallel walls on both sides as relay walls to achieve detection and reconstruction of complex indoor scenes.

[0077] To obtain shadow labels, we use affine transformation to stretch the part of the projected wall in the shadow image to obtain a rectangular image proportional to the actual wall size. We then perform median filtering and histogram equalization on the image and perform adaptive thresholding to segment the shadow area. However, for the light source estimation method, since this method prioritizes contour details, we only use the affine transformed wall image instead of the binary image to iteratively perform the light source estimation process.

[0078] The reconstruction process of the present invention is built on the NeSF implementation under the PyTorch framework. For each pixel in the binary image, we project a ray with 64 coarse samples and 64 fine samples, and train every 1024 rays as a batch on an Nvidia GeForce RTX 3090 GPU. For each scene, we use 25 binary images and their corresponding light source coordinates for reconstruction. We will use images of different sizes in different test scenes. After the network training is completed, we will use deep rendering to calculate multi-view depth images, and use the correspondence between depth map pixels and 3D points to map multiple depth images to 3D coordinates to reconstruct the point cloud results.

[0079] We obtained various types of simulation data through Blender, including single-object scenes, multi-object scenes, and complex large-scale scenes. Figure 4 The reconstruction results of a complex indoor scene are shown. The reconstruction results show that this method can well reconstruct the geometric shapes of occluded objects, such as Figure 4The vegetation detail reconstruction result on the right side of (b) can effectively restore objects with complex geometric structures and perform accurate spatial positioning.

[0080] We constructed a Figure 1 The non-viewing scene with two parallel walls shown in (c) further verifies the effectiveness of the light source estimation algorithm. 160 consecutive lighting points are set, and the position of the light source in the corresponding shadow image sequence is iteratively calculated. Next, the estimated light source coordinates and the actual light source coordinates are used for training under the same experimental setting. Figure 5 The results in show that the proposed method can robustly estimate the trajectory of the sequence of light source points, and most of the light source positions have only small errors compared to the actual positions. In addition, the scene reconstructed using the estimated lighting points can also clearly reproduce various hidden objects, with only small deviations compared to the results obtained using the real coordinates, which also shows that NeSF can remain robust in accurately recovering occluded objects even in the presence of small coordinate errors.

[0081] The present invention also verifies the effectiveness of the hidden object reconstruction algorithm and the light source estimation algorithm in real-world scenes. Figure 6 The results of light source estimation and point cloud reconstruction in a small-scale multi-target scene are shown. We continuously moved the laser to the secondary light source position of the reflecting wall, and simultaneously shot a video of the wall to obtain continuous light source points, and selected 960 light source points for light source estimation. Figure 6 The results shown in (b) show that the distribution shape of the light source point estimates is almost the same as the distribution shape of the actual coordinates. We then selected 25 coordinate points to reconstruct the hidden scene. Figure 6 The point cloud reconstruction results in (c) show that our algorithm can achieve high-quality 3D reconstruction of non-viewing scenes in real-world scenes.

[0082] In addition, we conducted tests in large indoor scenes and obtained the following results: Figure 7 Our algorithm effectively reconstructs objects of various shapes in the non-viewing space, including laptops, boxes, bottles, and even details such as fingers and hair. In addition, we reconstruct point clouds based on depth rendering. The point cloud in the world coordinate system can accurately measure the scale of objects, such as the height of a person is 1.8 meters, the height of a table is 0.6 meters, the height of a bottle is 0.2 meters, and so on.

[0083] To further evaluate the effectiveness of our method, we tested our method using the partially shadowed images provided in the paper (Henley C, Maeda T, Swedish T, et al. Imaging behind occluders using two-bounce light[C] / / Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16. Springer International Publishing, 2020:573-588.). Figure 8 The results shown in demonstrate that our method exhibits better ability in reconstructing details. Compared to voxel-based probabilistic space warping methods, NeSF provides a continuous volumetric representation of the non-viewing 3D space, rather than discarding many less confident details based on probability, which helps preserve more scene information.

[0084] This paper proposes a method for 3D reconstruction in a double-reflection non-viewing scene, which can accurately reconstruct the shape and contour details of hidden objects. Our method uses laser illumination on a relay wall to generate diffuse reflection points as secondary light sources. By illuminating different positions on the relay wall, hidden object projections at different angles are generated on the opposite wall, and multi-view projections are used to reconstruct the 3D shape of the hidden object. Our method establishes a rendering relationship between the 3D shadow field and the 2D shadow through continuous opacity integrals, and uses binary shadow images to optimize the 3D implicit representation. In addition, we propose a light source estimation algorithm to locate the position of a moving light source in the case of parallel planes to solve the situation where the coordinates of the secondary light source are unknown.

[0085] We test the performance of our method in both simulated and real-world environments, and show that our method can be applied to scenes of various scales and has stable reconstruction capabilities even for complex multi-object scenes.

[0086] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0087] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for three-dimensional reconstruction of a surround scene based on implicit representation, characterized in that: include: Use laser to illuminate the relay wall, forming a secondary diffuse reflection spot on the relay wall, and generate hidden object projections at different angles on the opposite wall by illuminating different positions of the relay wall; If the position of the secondary light source cannot be measured, the three-dimensional coordinates of the secondary light source are estimated by a light source estimation method, wherein the secondary light source is formed by laser irradiation of the relay wall; if it can be measured, the measured position of the secondary light source is directly used as the basis for constructing the light in the following steps; A neural shadow field model is established through the physical relationship that the secondary light source emits light in all directions in the hemispherical space and terminates at the observation relay wall, and the cumulative transmittance of the light is used to characterize the probability of the light propagating to the relay wall. The three-dimensional space parameterization of the neural shadow field model is represented as a continuous MLP function, which is a multi-layer perception function. The binary shadow label is used to supervise the optimization of the cumulative transmittance of light, and the three-dimensional structure of the hidden space is obtained through deep integration.

2. The method for 3D reconstruction of surround view scenes based on implicit representation according to claim 1, characterized in that: The three-dimensional coordinates of the secondary light source are estimated by the light source estimation method, as follows: S1, the first shadow image in the time-sequential shadow image sequence Perform feature extraction to find feature points with geometric invariance; The corresponding lighting point position is set as the origin of the 3D relative coordinate system, denoted as ; For subsequent images The same feature extractor is also used to extract features and compare them with The features of are matched to obtain several key point pairs with similar geometric features; the calculated and The average displacement of all keypoint pairs , and The corresponding lighting point position is recorded as ; S2, convert the Euclidean coordinates of the key points in the two-dimensional image coordinate system into 3D coordinates that are aligned with the light source: ； in, , Represents key points in the image The pixel coordinates of yes The three-dimensional mapping in the coordinate system of the light source, and Indicates the width and height of the image; In three-dimensional space, the light source point of the reflection wall and the key point of the projection wall can form a projection ray. Each pair of key points and their corresponding light source points can form two approximately intersecting projection rays. The intersection of the two rays is a point on the edge of the object in the hidden space. This edge point corresponds to a point on the shadow contour. For each pair of key points and , calculate the corresponding intersection point : ； ； in, is a scaling factor, and The light source and Illuminating hidden space points The key points of the projection formed; The positions of these intersection points will be recorded in the 3D point set ; Since each key point Uniquely corresponds to a point , at each key point and A mapping relationship is established between them, and the set of these mappings is represented as ; S3, the newly added image will be used Key point set and a 3D point set The mapping relationship between them is used to estimate the position of the light source; when a new image is added When the key point set Cannot directly To establish association, we first extract its features and compare its key points with the existing image set Each image in the image is matched, and each time a new key point With a known key point Can be matched, and there is a mapping relationship , a new mapping can be created , and this mapping is unique; S4, according to the projection relationship described in step S2, when the mapping between the key point and some points in the space is known, a back-projection relationship is established. A back-projection line can be formed in space to filter out outliers, and the image can be estimated by calculating the average coordinates of all points after filtering. The corresponding light source position ; S5, use the newly calculated light source position Update Point Set ; Used in and image sets The key point pairs matched between each image in and their corresponding light source positions are used to establish intersecting projection lines and calculate an intersection point of these projection lines. and merge them into Then, the image Add to Image Set middle; For newly added images , established through steps S3 and S4 Key points and point sets The mapping relationship between them is calculated The lighting point position ; Update via S5 To ensure the continuation of iteration.

3. The method for 3D reconstruction of surround view scenes based on implicit representation according to claim 2, characterized in that: In step S4, the RANSAC or DBSCAN algorithm is used to filter out outliers.

4. The method for 3D reconstruction of surround view scenes based on implicit representation according to claim 1, characterized in that: The neural shadow field model is established through the physical relationship that the secondary light source emits light in all directions in the hemispherical space and terminates at the observation relay wall. The cumulative transmittance of the light is used to characterize the probability of the light propagating to the surface of the relay wall, as follows: The coordinates of the secondary light source are taken as the starting point of the light, and the three-dimensional spatial position of the projected image pixel is taken as the end point of the light, and several rays are constructed, and the direction of the light is the three-dimensional unit direction vector of the ray; By calculating the cumulative transmittance from the starting point to the end point of the light, the probability of the light emitted by the secondary light source propagating to the surface of the relay wall is characterized.

5. The method for 3D reconstruction of surround view scenes based on implicit representation according to claim 1, characterized in that: The three-dimensional space parameterization of the neural shadow field model is represented as a continuous MLP function, and the optimization of the light cumulative transmittance is supervised by the binary shadow label, and finally the three-dimensional structure of the hidden space is obtained by deep integration. The specific method is as follows: Neural Shadow Field (NeSF) represents continuous NLOS space as a learnable 3D function , the spatial coordinates Mapped to volume density with MLP : ； in, Represents position encoding; spatial coordinates are positionally encoded before being input into the MLP: ； Among them, L represents the encoding dimension; NeSF is described as the visibility probability of the observation wall to the light source, so only the light reaching the observation wall is concerned; the visibility is expressed using cumulative transmittance, which is expressed as , which means that light propagates through the interval without scattering The probability of propagation distance is and upper limit Secondary light source rays The expected cumulative transmittance of is: ； represents the position reached by the light source ray when it propagates to a distance t, Indicates the starting point of the light. Represents the unit direction vector of the light source ray; For each shadow image, construct the rays originating from the corresponding light source; estimate the integral of the accumulated transmittance by tracing the rays reaching the observation relay wall using stratified sampling of discrete samples: ； It represents the cumulative transmittance of light when it reaches the i-th sampling point, is a composite parameter. represents the transmittance of the i-th sampling point, Indicates the distance between it and the adjacent sampling points; The shadow image is binary segmented to mark whether the shadow exists; these labels will be used to supervise the optimization of NeSF by constraining the cumulative transmittance; pixels not covered by the shadow are marked as 1, indicating that the light is transmitted without occlusion; when the light is blocked by the hidden object, a shadow will be generated on the observation wall, and the pixels of the shadow will be marked as 0; the loss function of the cumulative transmittance is expressed as: ； Among them, S represents the set of rays corresponding to the shadows of each training batch; Finally, the object depth is calculated from any viewpoint by numerically integrating the rays: ； in, Represents light The depth of propagation, Represents the distance between the ith sampling point and its adjacent sampling points, and N represents the maximum number of sampling points for the ray sampling.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images