Method and system for converting 2D (two-dimensional) to 3D (three-dimensional) based on Neus model

Abstract

The invention relates to the technical field of image processing, in particular to a method and system for converting 2D into 3D based on a Neus model.The method comprises the steps that firstly, an image set is analyzed, and camera position information of the image set is predicted; secondly, predicting the 3D structure of the model by using an advanced algorithm in combination with the image and camera parameters; and finally, coloring the generated 3D structure and image data set so as to enhance the sense of reality and fidelity of the model. According to the system, the 2D image can be effectively converted into the 3D structure, and accurate coloring is carried out according to camera parameters, so that richer and more vivid experience is provided for a user. On the basis of the Neus model, the automation process of the Neus reconstruction model is perfected, the problems that real data set parameters of the Neus model are difficult to obtain, the denoising problem of input sparse point clouds in the reconstruction process is solved, the problem that a generated model is a white model and lacks color information is solved, and the problem of mutual conversion of camera data sets is solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology and 3D reconstruction, and in particular to a 2D to 3D conversion method and system based on a Neus model. Background Art

[0002] In today's digital age, converting 2D images into 3D models is a critical requirement in many fields, including game development, architectural design, and virtual reality. Manually converting 2D images into 3D models has several significant drawbacks. First, manual modeling consumes considerable time and effort. Complex scenes or objects require point-by-point and line-by-line modeling, significantly increasing the time required. Second, manual modeling is limited by individual skill and experience, resulting in significant discrepancies between modelers and ensuring model accuracy and realism. Furthermore, subjective judgments are often present in the manual modeling process, leading to model inconsistencies or errors. Finally, for large-scale image datasets or complex scenes, manual modeling often fails to meet real-time and efficiency requirements. In summary, manual modeling from 2D images to 3D models suffers from high time costs, difficulties in ensuring accuracy and consistency, strong subjectivity, and low efficiency. Furthermore, using the Neus model poses challenges, such as the difficulty in obtaining real datasets with camera parameters, the inability to automatically remove noise during reconstruction, which affects boundary judgment, and the resulting model being white and lacking color information.

[0003] In summary, the problems existing in the prior art are:

[0004] Manually converting 2D images into 3D models suffers from time-consuming costs, difficulty ensuring accuracy and consistency, strong subjectivity, and low efficiency. Furthermore, using the Neus model poses challenges such as difficulty obtaining real datasets with camera parameters, the inability to automatically remove noise during reconstruction, which affects boundary judgment, and the resulting model being white and lacking color information. Summary of the Invention

[0005] To address the above technical issues, the present invention provides a 2D-to-3D conversion method and system based on the Neus model, which can avoid the shortcomings of high time cost, difficulty in ensuring accuracy and consistency, strong subjectivity, and low efficiency. It also solves many problems existing in the Neus model.

[0006] On the one hand, the 2D to 3D conversion method based on the Neus model provided by this system includes:

[0007] (1) Dataset creation

[0008] The main function of this module is to support the input of image datasets in various formats, including common PNG and JPG images, as well as video data. The system can automatically preprocess the input data to ensure that the quality of the input data meets the subsequent reconstruction requirements. The video data provides a frame extraction function and realizes conversion with multi-view image datasets through custom parameters.

[0009] (2) Camera parameter recovery

[0010] The module's primary function is to restore camera parameters. Users can directly set parameters through a visual interface to restore camera extrinsics. It can automatically estimate extrinsics (such as rotation matrices and translation vectors) based on the input multi-view images. This feature significantly simplifies the camera calibration process and improves the automation of reconstruction. Intermediate point clouds are also generated during the conversion process after the parameter set is restored.

[0011] (3) Point cloud denoising

[0012] The main function of this module is to realize the denoising function of sparse point clouds. In order to solve the noise points that may be generated in the sparse point clouds with reconstruction parameters, the system provides a variety of point cloud filtering and denoising tools, such as statistical filtering SOR, clustering denoising DBSCAN, improved SOR and improved DBSCAN, etc. Users can choose the appropriate denoising method according to the density and noise conditions of the point cloud to ensure the accuracy of the reconstruction parameters.

[0013] (4) Model reconstruction

[0014] This module primarily implements model reconstruction, the core functionality of the system. Using the neural implicit surface reconstruction method described above, the SDF network is trained from a set of images and parameters. Ultimately, the algorithm extracts a dense model surface and generates a mesh model. Users can customize reconstruction parameters within the system and obtain the final result from the set path.

[0015] (5) Model coloring

[0016] The main function of this module is to realize the model coloring function based on texture mapping, which can project the texture information in the multi-view image onto the surface of the three-dimensional model to generate a realistic three-dimensional model.

[0017] The 2D to 3D conversion system based on the Neus model of the present invention includes the following main steps: first, by analyzing the image set, its camera position information is predicted; second, combining the image and camera parameters, the 3D structure of the model is predicted using advanced algorithms; finally, the generated 3D structure and image data set are colored to enhance the realism and fidelity of the model. This system can effectively convert 2D images into 3D structures and accurately color them according to the camera parameters, thereby providing users with a richer and more realistic experience. Based on the Neus model, the automation process of the Neus reconstruction model is further improved, solving the problems of the difficulty in obtaining the parameters of the real data set of the Neus model, the denoising problem of the sparse point cloud input during the reconstruction process, the generated model being a white model lacking color information, and the mutual conversion problem of camera data sets is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a structural diagram of a 2D to 3D system based on the Neus model provided by the present invention.

[0019] Figure 2 This is a diagram of the data set production interface provided by the implementation of the present invention.

[0020] Figure 3 It is a data set production interface path diagram provided by the implementation of the present invention.

[0021] Figure 4 This is a frame extraction diagram of the data set production interface provided by the implementation of the present invention.

[0022] Figure 5 It is an atlas provided by the implementation of the present invention.

[0023] Figure 6 This is a camera parameter recovery interface diagram provided by the implementation of the present invention.

[0024] Figure 7 It is a camera parameter recovery content diagram provided by the implementation of the present invention.

[0025] Figure 8 It is a camera parameter recovery map provided by the embodiment of the present invention.

[0026] Figure 9 This is a point cloud denoising image provided by the implementation of the present invention.

[0027] Figure 10 This is a point cloud noise removal map provided by the present invention.

[0028] Figure 11 This is the starting noise removal map provided by the implementation of the present invention.

[0029] Figure 12 This is a comparison chart of the sparse point cloud denoising results provided by the implementation of the present invention.

[0030] Figure 13 This is a comparison chart of the sparse point cloud denoising results provided by the implementation of the present invention.

[0031] Figure 14 This is a 3D model reconstruction diagram provided by the implementation of the present invention.

[0032] Figure 15 It is a 3D reconstruction generation parameter map provided by the implementation of the present invention.

[0033] Figure 16 It is the starting reconstruction diagram provided by the implementation of the present invention.

[0034] Figure 17 This is a diagram of the model reconstruction result provided by the implementation of the present invention.

[0035] Figure 18 This is a colored diagram of a 3D model provided by the present invention.

[0036] Figure 19 It is a 3D model coloring parameter map provided by the embodiment of the present invention.

[0037] Figure 20 This is the starting 3D model coloring diagram provided by the implementation of the present invention.

[0038] Figure 21 This is a model coloring result diagram provided by the implementation of the present invention.

[0039] Figure 22 Schematic diagram of an image dataset provided by the present invention.

[0040] Figure 23 Schematic diagram of multi-view images of an image dataset provided by the present invention.

[0041] Figure 24 This is a flowchart of camera parameter recovery for an image dataset provided by the present invention.

[0042] Figure 25 It is a pseudo code diagram of the parameter recovery feature extraction algorithm provided by the implementation of the present invention.

[0043] Figure 26 It is a pseudo code diagram of the incremental reconstruction algorithm provided by the present invention;

[0044] Figure 27 This is a diagram of the overall reconstruction architecture provided by the implementation of the present invention.

[0045] Figure 28 This is an implicit reconstruction network structure diagram provided by the implementation of the present invention.

[0046] Figure 29 This is a NeRF network structure diagram provided by the implementation of the present invention.

[0047] Figure 30 This is a pseudo code diagram of the SOR point cloud denoising algorithm based on multi-scale local density weighting provided by the present invention.

[0048] Figure 31 It is a model coloring flowchart provided by the implementation of the present invention.

[0049] Figure 32 It is a texture mapping flow chart provided by the implementation of the present invention.

[0050] Figure 33 This is a diagram of the reconstruction results of the self-made data set provided by the implementation of the present invention.

[0051] Figure 34 This is a diagram of the reconstruction results of the public dataset provided by the implementation of the present invention.

[0052] Figure 35 This is a comparison chart of point clouds of denoising modules based on the improved SOR algorithm provided by the present invention.

[0053] Figure 36 This is a comparison chart of point clouds of denoising modules based on the improved DBSCAN algorithm provided by the present invention.

[0054] Figure 37 It is a comparison diagram of the reconstruction models of the denoising module ablation experiment provided by the implementation of the present invention.

[0055] Figure 38 This is a histogram of the coloring experiment results of the basketball boy model provided by the embodiment of the present invention.

[0056] Figure 39 This is a diagram of the model coloring experiment results provided by the implementation of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] The application principle of the present invention will be further described below with reference to the accompanying drawings.

[0059] like Figure 1 As shown, the 2D to 3D conversion system based on the Neus model provided in this embodiment includes:

[0060] The dataset creation module provides a video-to-image sequence conversion interface and uses timestamp information to extract continuous video frames to construct a dense image dataset. Camera extrinsic parameter recovery then includes feature extraction, feature matching, geometric constraint optimization, and scene graph construction units. It uses a structure-from-motion approach, including five main stages: feature extraction, feature matching and geometric verification, sparse point cloud generation, structure-from-motion reconstruction, and data post-processing, to recursively obtain the camera extrinsic parameter matrix for each image.

[0061] The three-dimensional surface neural reconstruction module is based on the neural implicit surface reconstruction model, combines the spatial signed distance function with the neural radiation field model, and uses a differentiable volume renderer to optimize the model parameters.

[0062] like Figure 2 The 2D to 3D conversion method based on the Neus model provided in this embodiment includes:

[0063] 102. By analyzing the image set, predict its camera position information.

[0064] 103. Combine the image and camera parameters and use advanced algorithms to predict the 3D structure of the model.

[0065] 104. Colorize the generated 3D structure and image datasets to enhance the realism and fidelity of the model.

[0066] The aforementioned 102 also includes:

[0067] 101. Dataset processing: Helps process videos shot by mobile phones and converts them into image sets: Perform the following operations:

[0068] like Figure 3 As shown, on the left, upload the video from the video storage path, change the image path to the image collection storage path, and change the frame extraction interval (assuming that the mobile phone video has 30 frames per second, a 20-second video will be extracted at an interval of 20 frames to obtain 30 pictures.)

[0069] 1021. First, prepare the image dataset of the present invention, such as Figure 4 、 5 shown.

[0070] 1022. Then click "Image External Parameter Recovery" in the navigation tab to enter the camera parameter recovery interface, such as Figure 6 shown.

[0071] 1023. Enter the image path, parameter storage path, principal point information and focal length information on the parameter recovery interface page, such as Figure 7 shown.

[0072] 1024. Click the "Execute Camera Parameter Recovery" button, as shown in the following example: Figure 8 、9 shown.

[0073] 1025. Find the restored parameters in the parameter storage path and the data set is created.

[0074] 1031. First, according to the previous steps, the present invention has generated a data set. The generated point cloud file is used to determine the reconstruction range. However, since the point cloud file contains noise, the noise will affect the judgment of the boundary, and the present invention needs to remove it. Next, the present invention needs to perform denoising on the sparse point cloud in the data set. Figure 10 Enter the working directory path in the interface shown, select the denoising model, and click the denoising button, as shown Figure 11 As shown, point cloud denoising begins, as shown before Figure 12 As shown, after removal Figure 13 shown.

[0075] 1032. Then the present invention needs to rebuild the model. Click "3D Model Reconstruction" in the navigation tab to enter the model reconstruction interface. Figure 14 shown.

[0076] 1033. Enter the number of iterations, learning rate, working directory path, and model output path on the model reconstruction interface page, such as Figure 15 shown.

[0077] 1034. Click the "Start Reconstruction" button to start the reconstruction. Wait until the prompt below changes to "Reconstruction Completed", then click to extract the model surface. Figure 16 shown.

[0078] 1035. Find the rebuilt model in the corresponding working folder, such as Figure 17 shown.

[0079] 1041. The present invention colors the reconstructed model. The reconstructed model lacks color information. Here, the model is texture mapped and colored by the image set and camera parameters. Enter the navigation button as shown in the following example. Figure 18 shown.

[0080] 1042. Enter the working path and coloring output path in the model coloring interface page, and follow the instructions to set the working directory as follows: Figure 19 shown.

[0081] 1043. Click the "Start Coloring" button to start coloring. Wait until the prompt below changes to "Coloring Completed". Figure 20 shown.

[0082] 1044. Find the colored model in the corresponding output path, such as Figure 21 shown.

[0083] The embodiment of the present invention provides an automated conversion of a 2D image dataset into a 3D model, which can implement the method embodiment provided above. For specific functional implementation, please refer to the description in the method embodiment, which will not be repeated here.

[0084] This paper combines the camera parameter prediction of the SFM algorithm with the reconstruction of the Neus network and a colorization module developed based on PCL, achieving automated conversion of 2D image datasets into 3D models. Through video frame extraction and noise filtering, the automation process of the traditional Neus reconstruction model is further improved. Through projective texture mapping, the problem of the Neus color network only mapping to 2D images and failing to recover 3D color information is solved, thus enabling texture colorization.

[0085] like Figure 22 , Schematic diagram of the image dataset.

[0086] like Figure 23 , Schematic diagram of multi-view images in the image dataset.

[0087] like Figure 24 ,Flowchart of camera parameter recovery for image dataset.

[0088] like Figure 25 ,Pseudo code diagram of parameter recovery feature extraction algorithm.

[0089] like Figure 26 ,Pseudo code diagram of incremental reconstruction algorithm;

[0090] like Figure 27 , rebuild the overall architecture diagram.

[0091] like Figure 28 , implicitly reconstruct the network structure graph.

[0092] like Figure 29 , NeRF network structure diagram.

[0093] like Figure 30 , Pseudo code diagram of SOR point cloud denoising algorithm based on multi-scale local density weighting.

[0094] like Figure 31 , model coloring flowchart.

[0095] like Figure 32 , texture mapping flow chart.

[0096] like Figure 33 , the reconstruction result of the self-made dataset.

[0097] like Figure 34 , reconstruction result diagram of public dataset.

[0098] like Figure 35 , point cloud comparison chart of denoising module based on improved SOR algorithm.

[0099] like Figure 36 , comparison chart of point cloud of denoising module based on improved DBSCAN algorithm.

[0100] like Figure 37 ,Comparison of the reconstructed models of the denoising module ablation experiment.

[0101] like Figure 38 ,Histogram of the coloring experiment results of the basketball boy model.

[0102] like Figure 39 , model coloring experimental results.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 2D to 3D system based on the Neus model, characterized in that: include: The dataset creation module provides a video-to-image sequence conversion interface and uses timestamp information to extract continuous video frames to construct a dense image dataset. Camera extrinsic parameter recovery then includes feature extraction, feature matching, geometric constraint optimization, and scene graph construction units. It uses a structure-from-motion approach, including five main stages: feature extraction, feature matching and geometric verification, sparse point cloud generation, structure-from-motion reconstruction, and data post-processing, to recursively obtain the camera extrinsic parameter matrix for each image. The three-dimensional surface neural reconstruction module is based on the neural implicit surface reconstruction model, combines the spatial signed distance function with the neural radiation field model, and uses a differentiable volume renderer to optimize the model parameters.

2. The 2D to 3D conversion system based on the Neus model according to claim 1, characterized in that: The data set creation module specifically includes: The video frame extraction unit supports the input of image datasets in various formats, including common PNG and JPG images, as well as video data. The system can automatically preprocess the input data to ensure that the quality of the input data meets the requirements of subsequent reconstruction. The video data provides a frame extraction function and can achieve conversion with multi-view image datasets through custom parameters. The camera parameter recovery unit includes five main stages: feature extraction, feature matching and geometry verification, sparse point cloud generation, motion structure reconstruction, and data post-processing; (1) Feature extraction In the feature extraction stage, stable and distinguishing feature points are extracted from multiple input images. These feature points are used for subsequent feature matching and three-dimensional point recovery. First, key point detection is performed. In the feature extraction process, key point detection is required for the input image I. Let I g is the grayscale version of the image, then the key point set K is calculated by the detection algorithm F: K=F(I g ) Among them, F uses scale-invariant feature transform (SIFT), and key point detection is achieved by constructing a Gaussian scale space: L(x,y,σ)=G(x,y,σ)*I(x,y) Where G(x, y, σ) is the Gaussian kernel function: σ is the scale parameter of Gaussian blur, which indicates the degree of blur. The difference of two adjacent blurred images in the same group is generated to generate a Gaussian difference map (Difference of Gaussians, DoG), and the difference of images is calculated at different scales: D(x,y,σ)=L(x,y,kσ)-L(x,y,σ) Where k is a multiple of the scale interval, usually k = 2 1 / s (s is the number of image layers in each scale space), perform extreme value detection on D(x, y, σ) to find potential key points; Then calculate the feature descriptor. Once the key point set K is obtained, the descriptor D needs to be calculated for each key point ki. i , used to uniquely identify the local features of the point; assuming that the gradient direction distribution H(k i ), the descriptor consists of multiple direction histograms: D i ={h1,h2,...,hn} Among them, h i Represents the gradient distribution within the local window. For the SIFT descriptor, a 16×16 pixel local window is usually used and divided into 4×4 sub-regions. The gradient histogram in 8 directions is calculated for each sub-region to form a 4×4×8=128-dimensional descriptor: D i ={d1,d2,...,d 128 } Then, key points are screened and optimized. Since some feature points may be affected by noise or located at the edge of the image and unstable, key points need to be screened. Assuming that the responsiveness of the key point is R(k), it is calculated using the Harris corner detection method: R(k)=det(M)-α·(trace(M)) 2 Among them, M is the second-order derivative matrix, α is an empirical parameter, which filters out feature points with low responsiveness to improve the stability of matching. In addition, in order to avoid the concentration of feature points in a certain area of the image, the non-maximum suppression method is usually used to ensure the uniform distribution of key points on the image. Finally, the selected key point set K and its corresponding descriptor set D are stored and applied for subsequent feature matching and 3D reconstruction. This step ensures the stability of the feature points and improves the accuracy of multi-view geometry calculations. (2) Feature matching The feature matching stage uses the nearest neighbor search (NNS) to perform feature matching and combines the robustness strategy to reduce mismatching. Given two images I1 and I2, their local feature descriptor sets D1 and D2 are extracted respectively. For each d i ∈D1, use nearest neighbor search to find the most similar feature point in D2: Use the distance ratio of the nearest neighbor (NN) and the next nearest neighbor (2-NN) feature points to filter matching points: (3) Sparse point cloud generation In the sparse point cloud generation stage, the camera motion is estimated by matching feature points, and the position of the 3D points is restored by triangulation to construct a sparse point cloud. After the feature matching is completed, the position of the 3D points can be calculated using the multi-view geometric relationship. Assume that the pixel coordinates of the same feature points x1 and x2 in the two images I1 and I2 are: <h2 style=";text-align:left;direction:ltr">x1 = K1<h2 style=";text-align:left;direction:ltr"> -1 <h2 style=";text-align:left;direction:ltr"> u1, x2, K2<h2 style=";text-align:left;direction:ltr"> -1 <h2 style=";text-align:left;direction:ltr"> u2 Where K1 and K2 are the camera’s intrinsic matrix, and u1 and u2 are the matching points in image coordinates. The camera’s projection matrices P1 = K1[R1|t1] and P2 = K2[R2|t2] can be used to establish the triangulation equation: λ1x1=R1X+t1 λ2x2=R2X+t2 Among them, X is the coordinate of the three-dimensional point to be solved, λ1 and λ2 are scale factors; using least squares optimization, the optimal X ; (4) Incremental external parameter recovery The incremental extrinsic parameter recovery stage starts from a small initial model, continuously adds new images, optimizes the existing camera poses and 3D points, and performs error optimization. First, an initial image pair is selected, and the two images with the largest overlapping area are found as the starting frame through feature matching. The essential matrix or homography matrix is calculated to determine their relative motion relationship, and the initial sparse point cloud is calculated through triangulation. Then, new perspectives are gradually added, and images with more common viewpoints with the reconstructed perspective are selected for addition. The camera pose of the new perspective is estimated through the perspective n-point positioning method, and the 3D points contributed by the new perspective are calculated through triangulation and merged into the point cloud. Then, error optimization is performed, and global BA is used to adjust all camera poses and point cloud coordinates to minimize the reprojection error, perform robust optimization, and eliminate mismatched points. The cycle iterates continuously until all images are added, the best perspective is continuously selected, the pose is calculated, the point cloud is added, and the entire system is optimized. (5) Data post-processing The data post-processing stage calculates the bounding box of the point cloud and calculates a scale matrix based on the center and maximum radius of the point cloud; this scale matrix aligns the center of the point cloud to the origin while ensuring that the scale of the point cloud in space is appropriate; by applying this scale matrix, all cameras of different perspectives are normalized so that point clouds from different perspectives can be correctly aligned in the same coordinate system.

3. The 2D to 3D conversion system based on the Neus model according to claim 1, characterized in that: The 3D surface neural reconstruction module includes several core modules: denoising and feature extraction, reconstruction network, mesh extraction, and texture mapping. It uses the Structure from Motion (SFM) technique to recover sparse point clouds from multi-view images as the initial geometric representation. The denoising module then removes point cloud noise and extracts boundary information and feature vectors to enhance geometric details. The reconstruction network combines multi-view images, camera parameters, and point cloud features, using an implicit function (SDF) to learn a continuous geometric representation of the scene, effectively capturing the complex surface details of objects. The mesh extraction module extracts the explicit mesh surface from the SDF and projects the color information of the multi-view images onto the mesh using a texture mapping algorithm to generate a textured 3D model. The specific process of the implicit reconstruction module is as follows: (1) Random light sampling First, the network input is a multi-view image of a small object and camera parameter information, and then it starts sampling random light r(t) = o + td, where t is time, and the time starting point (Near Bound) and time ending point (Far Bound) are t n and t f , d is the viewing direction Perform initial sampling of random light and background sampling to obtain the sampling point p(t); (2) Obtain background color information through NeRF network Given a spatial point x = (x, y, z) and an observation direction d, the NeRF network outputs the color (RGB) and density (indicating the light intensity and material transparency at the point); the radiation field formula is: The light angle d and the sampling point coordinate p are position-encoded and input into the NeRF network to obtain the background color information; (3) Obtain feature vectors and SDF gradients through the SDF network The SDF network is modeled by learning the signed distance from each point in space to the nearest surface. This distance is zero for surface points in the scene, positive for the outside of the scene, and negative for the inside of the scene. The SDF network proposed in this paper is implemented by a multi-layer perceptron (MLP); the network input is spatial coordinates (usually three-dimensional coordinates), and the output is the signed distance value from each point to the surface; the network input is X = (x, y, z) ∈ R 3 , the output is [s, f], where s represents the signed distance (SDF value) and f is a 256-character feature vector used to represent additional attributes such as color, texture, etc.; to facilitate the flow of information and enhance the expressiveness of the network, skip connections are inserted into the 4th layer of the network to reduce the gradient vanishing problem and enable the network to learn more low-level and high-level features during training; further calculate the surface normal, and the SDF network calculates the gradient of each input point through automatic differentiation; through backpropagation, the gradient of each point relative to its SDF output is obtained: This process is used to estimate the surface normal and the direction of the point cloud, thereby providing information for subsequent rendering or further geometric optimization; for each input point x, the normal vector is obtained by calculating the gradient of the SDF output of the point relative to the input coordinates. By calculating the gradient, the SDF network can provide the normal information of each point in three-dimensional space; in order to train the SDF network through the volume rendering function, the probability density function φ is defined here s (f(x) This is called S density, where f(x) is the SDF value and the outer layer is the density distribution: (4) Get the initial sampling color through the color network Define c:R 3 ×S 2 →R 3 is the color value of a point in space from a certain viewing angle. For any pixel, let the camera send a ray {p(t)=o+tv|t≥0} from that point, where o is the camera origin and v is the direction of the ray. The pixel color at that point is defined as: C(o,v)=∫0 +∞ w(t)c(p(t),v)dt In traditional work, the weight function is in the form of: w(t) = T(t)σ(t), where σ(t) is called the volume density and T(t) represents the accumulation of light along the ray. To solve the problem that the SDF is trapped in the local optimum, the weight function is defined as: w(t)=T(t)ρ(t),where T(t)==exp(-∫0 t ρ(u)du) where ρ(t) is the opacity, defined as: The weight function constructed in this way is firstly unbiased, and the SDF is 0 at the optimized convergence point. Secondly, the occlusion relationship of multiple collision points can be determined by matching the SDF value condition. Finally, the color of the light is calculated by discrete points on the discrete sampling light similar to NeRF. (5) Combine the bias network to get the final sampled color The bias network is a single-parameter network that aims to provide a control parameter for each three-dimensional sampling point, which affects the degree of expansion or contraction of the point sampling. In this network, the input is an empty vector and the output of the network is a control variable inv s , which is used for subsequent rendering calculations; the network learns a scalar value inv s , this value is used in subsequent transparency and color calculations; During the rendering process, given the SDF value and gradient of each sampling point, the output of the bias network will be used to adjust the cumulative transparency (Δlpha) and color (Color) of these sampling points; the cumulative distribution function (CDF) is used to describe the probability that a random variable takes a value less than or equal to a certain value. In order to distribute more sampling points in important areas during the rendering process to improve the efficiency of calculation and image quality; using CDF, the importance sampling of space can be performed, that is, more sampling points are generated in areas with larger weights to improve sampling efficiency; the CDF value, the bias network parameter inv s Act on the initial sampling color to get the final sampling color; (6) Compare the final sampled color with the real image color to obtain the color loss The gradient generated by the SDF network is used to obtain the reconstruction loss, calculate the total loss, and backpropagate to update the network parameters; (7) Process the trained SDF using the Marching Cubes algorithm Reconstruct the mesh surface of the scene. The mesh generation method includes the following main steps: Step 1: Obtain the scalar field of the space through the SDF network. Use the trained SDF network to sample each point in the three-dimensional space to obtain the scalar field. Specifically, discretize the space with a certain resolution within the set spatial range (defined by the minimum boundary Bound_min and the maximum boundary Bound_max) to obtain a voxel grid. Step 2: Use the Marching Cubes algorithm to extract the surface mesh from the scalar field. Each cube in the voxel grid is checked one by one to determine whether it contains a surface. When the SDF values of the eight corner points of a cube change sign, the algorithm divides the cube into multiple triangles, thus forming a surface. Step 3: Post-process the generated mesh to ensure its quality and accuracy. The vertices and triangles returned by the Marching Cubes algorithm are based on the coordinate system of the voxel grid. In order to align the generated mesh with the actual three-dimensional space, the vertex coordinates need to be normalized; specifically, the coordinates of each vertex are converted from the voxel grid coordinate system to the actual three-dimensional space coordinate system.

4. The 2D to 3D conversion system based on the Neus model according to claim 1, characterized in that: The denoising module specifically includes: Statistical outlier removal based on multi-scale local density weighting for point cloud denoising. The statistical outlier removal algorithm performs statistical analysis on the neighborhood around each point, calculates the difference between each point and its neighborhood, and determines whether to consider the point as an outlier and remove it based on the difference. If the distance between a point and its neighborhood points is far, the point is considered a noise point and removed from the point cloud. The SOR point cloud denoising algorithm based on multi-scale local density weighting aims to solve the denoising effect of the traditional SOR algorithm on densely packed edge areas when processing complex point cloud data. The traditional SOR algorithm uses a fixed number of neighbors k. i , cannot effectively deal with the problem of uneven density distribution in point clouds. This algorithm dynamically adjusts the number of neighbors k i , realizing multi-scale analysis; for areas with higher density, a smaller k i To preserve details; for areas with lower density, a larger k i To enhance the denoising ability; on this basis, this algorithm introduces the local density p i As a weight factor, dynamically adjust the distance threshold d thresh Specifically, a stricter threshold is used in areas with higher density to retain the detailed features of the point cloud; a looser threshold is used in areas with lower density to avoid mistakenly deleting valid points; this adaptive mechanism significantly improves the adaptability of the algorithm to complex point clouds; in order to better handle the changes in local features of point clouds, this section combines the local density p i and standard deviation σ i , calculate the adaptive distance threshold d thresh ; This threshold can be dynamically adjusted according to the local characteristics of the point cloud, thereby better balancing detail preservation and outlier removal during the denoising process.

5. The 2D to 3D conversion system based on the Neus model according to claim 1, characterized in that: The texture mapping colorization module implements a multi-camera texture mapping process based on PCL, which can effectively map multiple texture images to the surface of the three-dimensional model and handle occlusion and visibility issues; The specific process includes: (1) Data preparation First, the 3D model is loaded from the PLY file, including point cloud data and patch information. The patch information is stored in the TextureMesh object as the input for texture mapping. At the same time, the camera's pose, intrinsic parameters, and corresponding texture image path are loaded. The camera's pose includes the rotation matrix and translation vector, and the intrinsic parameters include the focal length, principal point coordinates, and the width and height of the image. This information is stored in the CameraVector for subsequent projection calculations. (2) Camera coordinate system transformation For each camera, the point cloud data of the 3D model is first transformed into the coordinate system of the current camera. This step is achieved by inverting the camera's pose matrix to generate a transformed point cloud. The transformed point cloud is used for subsequent UV coordinate calculation and visibility judgment. (3) UV coordinate calculation In the camera coordinate system, each patch is projected to determine whether it is within the camera's field of view. For visible patches, the UV coordinates of its three vertices in the texture image are calculated and stored in the Projections point cloud. The UV coordinates are calculated based on the camera's intrinsic parameters and the position of the patch's vertices in the camera coordinate system. If a patch is invisible, its UV coordinates are marked as NaN and marked as invisible in the Visibility vector. (4) Occlusion processing To handle the occlusion problem between patches, a KD tree is used to perform a spatial search on the UV coordinates. For each patch, the radius and center of its circumscribed circle are calculated, and the KD tree is searched for adjacent points within the circle. If a neighboring point is within the triangle of the current patch and its depth value is greater than the depth value of the current patch, the current patch is considered occluded and marked as invisible. This step ensures that each patch is only mapped by the most appropriate camera texture, avoiding texture overlap and occlusion problems. (5) Storage of UV coordinates of visible face For visible patches, store their UV coordinates in Tex_coordinates, update the patch information, and store the index of the visible patch in Tex_coord_indices for subsequent texture mapping and rendering; for invisible patches, store them in Occluded_faces and push them to the next camera for processing. This step ensures that all patches can be texture mapped by at least one camera. (6) Processing of invisible faces After processing all cameras, if there are still patches that are not mapped by any camera texture, virtual UV coordinates (-1, -1) are generated for these patches and stored in Tex_coordinates. These virtual UV coordinates can be replaced with default textures or processed by interpolation methods in subsequent rendering; (7) Save the results Save the texture mapped mesh data as an OBJ file and generate the corresponding MTL material file. The OBJ file contains the geometric information and UV coordinates of the model, and the MTL file contains the path and material properties of the texture image.

Images