Three-dimensional reconstruction method based on improved NeRF
By using the improved NeRF method, combined with COLMAP, neural radiation field and diffusion algorithms, the complexity problems of camera coordinate transformation and 3D reconstruction were solved, efficient and accurate 3D model reconstruction and printing were achieved, and the use of support materials was optimized.
Patent Information
- Application Number
- CN202510878005.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-10-17
AI Technical Summary
In the existing technology, the determination of camera internal and external parameters and camera coordinate conversion add additional cost and complexity to deep learning-based 3D reconstruction, and are difficult for ordinary people to achieve.
An improved NeRF method is adopted, combined with COLMAP, neural radiance field and diffusion algorithms, to achieve camera pose estimation and 3D reconstruction through camera coordinate transformation, multi-view reconstruction and STL file preprocessing. Dense point cloud is generated using SFM and MVS algorithms, combined with NeRF network for 3D reconstruction, and STL file processing is optimized through OpenGL and Assimp libraries.
It achieves stable and efficient 3D reconstruction, reduces the complexity of camera pose estimation, improves the accuracy and printing efficiency of 3D models, and reduces the use of support materials.
Smart Images
Figure CN120807779A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional reconstruction method based on an improved NeRF. BACKGROUND
[0002] Camera coordinate conversion plays a crucial role in the theory of three-dimensional reconstruction of objects based on deep learning. Three-dimensional reconstruction using images first requires obtaining the perspective information of the camera, including determining the camera intrinsic and extrinsic parameters, which will affect the flow of image three-dimensional reconstruction, wherein:
[0003] The camera intrinsic parameter is used to describe the optical and imaging characteristics of the camera itself, that is, how to project a point in the three-dimensional world onto a two-dimensional image through the camera, such as the focal length parameter defining the magnification of the camera lens; the optical center parameter defining the intersection of the optical axis and the image plane in the projection model; the distortion coefficient parameter defining the image distortion caused by the optical characteristics of the lens, etc.
[0004] The camera extrinsic parameter is used to describe the position and direction of the camera in the three-dimensional space, that is, the pose of the camera coordinate system relative to the world coordinate system, such as the rotation matrix coefficient used to describe the orientation of the camera; the translation coefficient used to describe the position of the camera in the three-dimensional space, that is, how to move the origin of the world coordinate system to the origin of the camera coordinate system. Although the camera intrinsic and extrinsic parameters can be well calculated through traditional monocular / dual-camera cameras, for ordinary people, this undoubtedly involves additional cost and complexity. SUMMARY
[0005] The application discloses a three-dimensional reconstruction method based on an improved NeRF, which aims to solve the technical problems raised in the background.
[0006] In order to achieve the above-mentioned purpose, the application adopts the following technical solutions:
[0007] A three-dimensional reconstruction method based on an improved NeRF, comprising camera coordinate conversion, a multi-view reconstruction scheme, a few-view reconstruction scheme and STL file preprocessing, the camera coordinate conversion uses multiple two-dimensional images taken by the camera at different positions in space, estimates the three-dimensional structure of the scene and the motion trajectory of the camera through the relative motion of these images, SFM matches the feature points in the image group, estimates the motion trajectory of the camera and the spatial distribution of three-dimensional points through the relative motion of these feature points, that is, analyzes the features of the two-dimensional image to calculate the pose information of the camera, and the information obtained after the conversion of the world coordinate system and the image coordinate system through the camera intrinsic and extrinsic parameters of the SFM is input into MVS for multi-view aggregation to form a dense point cloud, thereby constructing multi-view three-dimensional information.
[0008] In a preferred scheme, the multi-view reconstruction scheme includes COLMAP, neural radiance field and Nerf explicit output; by combining the SFM and MVS algorithms, pose estimation is introduced to realize dense point cloud fusion for three-dimensional reconstruction, and the derived camera pose estimation information often has good stability, and in the COLMAP, the Pipeline pipeline process includes feature detection, feature matching and pose estimation.
[0009] In a preferred scheme, in the pose estimation, COLMAP selects an image with the largest feature point overlap as the initial selection, calculates the fundamental matrix between two images to perform the internal and external parameter conversion iteration of the camera view, and ends until the image group ends, so as to achieve the effect of camera pose estimation. This process is an iterative optimization process, and the camera internal and external parameters are repeatedly adjusted;
[0010] In feature detection, COLMAP uses the Scale-InvariantFeatureTransform algorithm to identify scale and rotation invariant key points from images, constructs a multi-scale pyramid for each image, detects extreme points at each scale and performs accurate positioning, and finally obtains sub-pixel level key point positions through Taylor expansion method;
[0011] The neural radiance field is a neural network that learns the feature relationship between a set of two-dimensional images to reconstruct a three-dimensional scene / object. The core of the neural radiance field is a small multilayer perceptron combined with spatial coordinates and view directions to output the color and volume density of the corresponding points, and finally construct a three-dimensional reconstruction implicit expression of the object.
[0012] In a preferred scheme, the Nerf explicit output transmits the RGB information of a set of two-dimensional images and the information estimated by COLMAP to the Nerf, and obtains the volume density information and color information of the corresponding three-dimensional scene reconstruction. In the multi-view reconstruction scheme, noise is gradually added to the data samples by a diffusion algorithm, and the original data is returned by learning. The Diffusion process includes a forward noise adding process and a reverse noise removing process, that is, the learning process of mutual noise adding and noise removing between the original image X0 and the noise image X T The Diffusion algorithm can predict pictures of different views under the same scene.
[0013] In a preferred scheme, the STL file preprocessing is a key step in three-dimensional printing and numerical control machining, usually including reading, display, rotation, cutting and repair processes, through preprocessing, the support structure of the model file can be better analyzed, and the subsequent machining process can be designed, in the STL file preprocessing, the light, rendering, rotation and cutting operations of the STL file are realized by using the state machine framework of OpenGL, the optimization processing of STL information is realized by using the Assimp library, the cutting of the STL file is realized by using OpenGL based on the implicit cutting plane algorithm, the algorithm allows the user to set the cutting direction flexibly by moving and rotating the implicit plane, so as to optimize the support structure and reduce the use of support material, after the cutting is completed, the program automatically saves the cut file and automatically repairs the cutting surface to ensure the integrity of the model.
[0014] As can be seen from the above, a three-dimensional reconstruction method based on improved NeRF includes camera coordinate conversion, multi-view reconstruction scheme, few-view reconstruction scheme and STL file preprocessing, the camera coordinate conversion utilizes multiple two-dimensional images taken by the camera at different positions in space, estimates the three-dimensional structure of the scene and the motion trajectory of the camera through the relative motion of these images, SFM matches feature points in the image group, estimates the motion trajectory of the camera and the spatial distribution of three-dimensional points through the relative motion of these feature points, that is, the pose information of the camera is calculated through the analysis of the features of the two-dimensional images, the information obtained after the conversion of the world coordinate system and the image coordinate system through the internal and external parameters of the camera of the SFM is input into MVS for multi-view aggregation to form a dense point cloud, and the three-dimensional information of the multi-view is constructed. The three-dimensional reconstruction method based on improved NeRF provided by the application combines COLMAP and NeRF neural radiation field architecture, inputs a set of RGB two-dimensional static object data without camera pose, removes the background of the picture in detail by using the remove-bg algorithm, estimates the reasonable camera pose in parallel by using COLMAP, initializes the NeRF network architecture together with the RGB image group, and realizes the implicit video stream three-dimensional reconstruction process of the original NeRF. BRIEF DESCRIPTION OF DRAWINGS
[0015] Figure 1 A COLMAP pipeline diagram of the three-dimensional reconstruction method based on improved NeRF provided by the application.
[0016] Figure 2 A NeRF architecture diagram of the three-dimensional reconstruction method based on improved NeRF provided by the application.
[0017] Figure 3 A NeRF explicit output flowchart of the three-dimensional reconstruction method based on improved NeRF provided by the application.
[0018] Figure 4 A verification result example diagram of a three-dimensional reconstruction method based on an improved NeRF proposed in the application.
[0019] Figure 5 A Diffusion directed flow chart of a three-dimensional reconstruction method based on an improved NeRF proposed in the application.
[0020] Figure 6 Zero123++ prediction results of a three-dimensional reconstruction method based on an improved NeRF proposed in the application.
[0021] Figure 7 A few-view diffusion reconstruction flow chart of a three-dimensional reconstruction method based on an improved NeRF proposed in the application.
[0022] Figure 8 An STL pretreatment diagram of a three-dimensional reconstruction method based on an improved NeRF proposed in the application. DETAILED DESCRIPTION
[0023] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, rather than all the embodiments of the application.
[0024] The three-dimensional reconstruction method based on an improved NeRF disclosed in the application is mainly applied to the scene in which it is difficult to obtain current real-world STL model data.
[0025] A three-dimensional reconstruction method based on an improved NeRF, comprising camera coordinate conversion, a multi-view reconstruction scheme, a few-view reconstruction scheme and STL file pretreatment.
[0026] In a preferred embodiment, the specific process of camera coordinate conversion can be represented as follows: a plurality of two-dimensional images are taken by using different positions of a camera in space, the three-dimensional structure of a scene and the motion trajectory of the camera are estimated through the relative motion of the images, SFM matches feature points in the image group, and the motion trajectory of the camera and the spatial distribution of three-dimensional points are estimated through the relative motion of the feature points, that is, the pose information of the camera is calculated by analyzing the features of the two-dimensional images, and the specific camera conversion process is as follows:
[0027] Step one: a point P in the world coordinates w (X w , Y w , Z w ) represents the real coordinates of the three-dimensional world where the object is located, and the world coordinates are converted into camera coordinates P c (Xc Y c Z c ):
[0028]
[0029] Step two: Camera coordinates P c (X c ,Y c ,Z c ) are used to describe the position of the object relative to the camera, which is converted to image plane coordinates (x x ,y y ) by the camera's intrinsic parameters, i.e. focal length f and principal point position (c p ,c p ):
[0030]
[0031] Step three: Finally, the image plane coordinates are converted to the image coordinate system, i.e. the UV plane, by pixel size and distortion correction:
[0032]
[0033] where s x and s y are the pixel size, k1, k2 are the radial distortion coefficients, and p1, p2 are the tangential distortion coefficients.
[0034] SFM provides the pose of the camera and a sparse 3D point cloud, while Multi-View Stereo (MVS) further generates a dense 3D point cloud or mesh;
[0035] The specific process is as follows: through the geometric and photometric consistency between multiple views, the dense three-dimensional structure of the scene is reconstructed, i.e. using the disparity information of the same scene point in multiple images, the three-dimensional coordinates of the point are recovered by triangulation, at the same time, as known from the above, the information obtained by converting the world coordinate system and the image coordinate system through the camera intrinsic and extrinsic parameters of SFM is input into MVS for multi-view aggregation to form a dense point cloud, so as to construct the three-dimensional information of multi-view, the core idea is to calculate the cost function (such as absolute difference, square difference, normalized cross correlation, etc.) for each pair of images, estimate the depth by measuring the similarity between pixels, then select the image pair with the largest overlap area and the best view difference for matching and disparity fusion. Finally, the more accurate depth map is converted into a dense three-dimensional point cloud to achieve three-dimensional reconstruction effect.
[0036] In a preferred embodiment, the multi-view reconstruction scheme includes COLMAP, Neural Radiance Field and Nerf Explicit Output;
[0037] COLMAP multi-view reconstruction process needs picture information as input, including its RGB information and camera pose, however, for users, using a depth camera to collect RGB-D (RGB-Depth) images with camera pose will increase additional overhead, while obtaining RGB information can be easily achieved by mobile phones or digital products, therefore, COLMAP as an open source software algorithm, by combining SFM and MVS algorithms, introduces pose estimation to realize dense point cloud fusion for three-dimensional reconstruction, and the derived camera pose estimation information often has good stability. In COLMAP, the following steps are often needed, and the pipeline process is as shown in the accompanying Figure 1 Pipeline process includes feature detection, feature matching and pose estimation;
[0038] Feature detection: COLMA uses SIFT (Scale-Invariant Feature Transform) algorithm to identify scale and rotation invariant key points from images, constructs a multi-scale pyramid for each image, detects extreme points at each scale and performs accurate positioning, and finally obtains sub-pixel level key point position through Taylor expansion method;
[0039] For SIFT, the detailed process can be as follows: first, a multi-scale Gaussian pyramid is constructed for each image, the image is gradually blurred at different scales, and a series of scale spaces are generated, in each scale space, the algorithm detects extreme points by calculating the difference of Gaussians (DoG), these extreme points are regarded as preliminary key point candidates, for the detected preliminary key points, the algorithm further accurately positions, through refining the key point position, including removing low contrast points and edge response unstable points, in order to achieve sub-pixel level accuracy, the algorithm uses Taylor series expansion to approximate the position of extreme points, finally, SIFT also calculates the direction features for each key point, so that these features remain invariant to image rotation.
[0040] In a preferred embodiment, feature matching: the key point information obtained by feature detection finds a set of corresponding key points and matches, then uses RANSAC (Random Sample Consensus) algorithm for geometric consistency verification, by estimating the fundamental matrix (Fundamental Matrix) or homography (Homography), RANSAC can identify and eliminate those incorrect matching points that do not meet the geometric consistency, i.e. so-called outliers.
[0041] In a preferred embodiment, pose estimation: According to the above process, COLMAP will select the image with the largest feature point overlap as the initial selection, and perform the above-mentioned camera perspective internal and external parameter conversion iteration by calculating the basic matrix between the two images until the end of the image group, so as to achieve the camera pose estimation effect. At the same time, this process is an iterative optimization process. By repeatedly adjusting the camera intrinsic parameters (such as focal length, optical center position) and external parameters (rotation matrix and translation vector), the projection error is gradually reduced to ensure that the perspective information of all images is accurate. After each optimization step, a bundle adjustment will be performed to globally optimize the camera parameters and the position of the three-dimensional points to further improve the accuracy of pose estimation;
[0042] COLMAP can stably and accurately extract the depth information lacking in multi-view RGB images, combining them with the captured RGB image set to achieve the same performance as a depth camera. Although COLMAP often suffers from feature loss in complex scenes, its stable and fast pose estimation solution for existing 3D models provides insights for this application.
[0043] In a preferred embodiment, Neural Radiance Field (NeRF) is a neural network that reconstructs 3D scenes / objects by learning the feature relationships between a set of 2D images. The core of NeRF is to output the color and volume density of the corresponding points through a small multi-layer perceptron (MLP) combined with the spatial coordinates and viewing direction, and finally construct an implicit representation of the 3D reconstruction of an object. That is, the input P (x, y, z) and the corresponding pose orientation D are output as the corresponding color c (r, g, b) and volume density σ, as shown in the attached figure. Figure 2 As shown in formula (2-4)
[0044] Nerf(P,D)=(c,σ) (2-4)
[0045] The detailed process of NeRF 3D reconstruction is as follows:
[0046] 1. Take the RGB information of a set of images as input, and combine it with the position and perspective (camera pose) of the corresponding image to pass it into the corresponding network. For ordinary digital products, this step can use COLMAP to perform depth estimation of the camera pose.
[0047] 2. Considering that the color reflected by an object in a certain direction is actually the result of integrating the point set information on the ray X emitted from that point, based on sampling theory, for the current ray X i As shown in formula (2-5):
[0048] X i =o+t id (2-5) This formula represents the projection of a ray from the camera position o along the line of sight direction d of each pixel, and uniformly sampling multiple points X on the ray i , t i is the corresponding parameter information.
[0049] 3. The sampling points of each position are input into the MLP for training, and the color value C(r) of a pixel is synthesized by using the volume rendering algorithm. The loss function is constructed by comparing the rendered image effect with the original image, so as to achieve the iterative effect. As shown in formulas (2-6) and (2-7):
[0050]
[0051]
[0052] where C(r) represents the final color of the ray direction, N is the number of sampling points, T i (1-exp(-σ i δ i )) represents the transmittance from the starting point of the ray to the Ith point, that is, the cumulative transmission function. δ I represents the distance between adjacent sampling points. C GT (r) in the loss function Loss represents the pixel color in the actual image, and R represents the set of all images.
[0053] In order to map the input spatial coordinates to high-frequency functions to capture more rich spatial details and improve the performance of the network, Nerf proposes a position encoding technology to solve the shortcomings of traditional neural networks in high-frequency details and complex scene reconstruction. That is, by mapping the coordinate information (x, y, z) to a high-dimensional space, the neural network can better learn and represent high-frequency details to enhance the representation ability of the coordinates, as shown in formula (2-8):
[0054] where γ(x) represents the high-dimensional vector after position encoding, and L is the number of position encoding layers (i.e., the number of frequencies).
[0055] Nerf guarantees the fast three-dimensional reconstruction scheme of real objects by virtue of the above end-to-end reconstruction process, but due to the use of volume rendering technology for new view synthesis, the result scheme presented by Nerf only stays in the implicit output (video stream) of the three-dimensional model. In order to support the connection of the subsequent design and printing process, the present application designs to improve the Nerf neural radiance field so that it can perform explicit output of the STL file.
[0056] In a preferred embodiment, the Nerf explicit output will input the RGB information of a set of two-dimensional images and the information after depth pose estimation by COLMAP into Nerf, and the volume density information and color information after the corresponding three-dimensional scene reconstruction can be obtained. However, this implicit output of three-dimensional reconstruction information cannot be directly applied to 3D printing, and due to the use of MLP evaluation method, the generalization ability of the training and inference speed of the NeRF model has decreased [55-7] . Therefore, in order to solve this problem, NeRF and its subsequent research reduces or eliminates MLP to optimize storage density. Although the speed is optimized and the rendering quality is improved, the explicit rendering method of NeRF in the downstream task still performs poorly. In view of this method, the goal of the present application is to realize the linkage of three-dimensional reconstruction and 3D printing by exploring the method of converting NeRF reconstruction into texture mesh, and the flow chart is as shown in the accompanying Figure 3 .
[0057] In a preferred embodiment, the present application combines COLMAP and NeRF neural radiation field architecture, inputs a set of RGB two-dimensional static object data without camera pose, uses remove-bg algorithm to realize fine removal of picture background, estimates reasonable camera pose by COLMAP, initializes NeRF network architecture with RGB image set, realizes implicit video stream three-dimensional reconstruction process of original NeRF, and for implicit information of volume rendering data, the present application extracts mesh through the following process:
[0058] 1. After generating color information and volume density after NeRF rendering, the volume density of each view is extracted using MarchingCubes algorithm to extract isosurface from density field σ(x), forming an initial surface mesh. That is, the volume density data is divided into small voxel information, and three-dimensional vertex index information is generated. MarchingCubes algorithm further generates triangle fragments according to index information, and these triangles together constitute the basic unit of surface mesh and build coarse triangle mesh data.
[0059] 2. Adaptive surface refinement. By setting the optimization goal to build a geometric loss function, in the present application, the difference between the target density and the original density information of each view is compared to gradually optimize, as shown in formula 9, where σ * (x) target density:
[0060]
[0061] The present application tests part of the Nerf-synthetic official data set and successfully reconstructs the STL triangle mesh of the corresponding reference object, and the test result is as shown in Figure 4as shown.
[0062] In a preferred embodiment, the NeRF network has good adaptability to the multi-view reconstruction scheme in the few-view reconstruction scheme, but the picture quality and the number of pictures strictly limit its theoretical development, and in the face of data sets with poor context picture features, ghosting effect will be greatly produced, that is, a considerable part of the reconstruction result has shadows, in order to solve the matching ability of NeRF in the few data set pictures and reduce the appearance of ghosting effect and affect the efficiency of body rendering grid extraction, the application proposes a few-view reconstruction scheme.
[0063] The emergence of the diffusion algorithm (Diffusion) ensures a huge leap from local data perception to similar data generation. Its essence is a kind of generation model based on probability distribution. Through the process of adding noise to data samples and then learning to return the original data, such as Figure 5 as shown.
[0064] In a preferred embodiment, the Diffusion process includes a forward noise adding process and a reverse noise removing process, that is, the learning process of mutual noise adding and removing between the original image X0 and the noise image X T The diffusion algorithm can predict pictures of different angles in the same scene, which has been verified in a series of studies on few-view diffusion models. Zero123++ is an excellent work of single-view diffusion theory, which provides a simple way to realize the prediction scheme of different angles of single view, such as Figure 6 as shown.
[0065] In a preferred embodiment, the application proposes to provide a few-view scheme, combine Zero123++ and NeRF network pipeline, form a double Stage stage to form a three-dimensional reconstruction theory of few-view diffusion, such as Figure 7 shows the process of experiment with Lego as a test case.
[0066] In a preferred embodiment, the module is divided into two parallel stages, namely the acquisition of camera orientation perspective through the initial pose information diffusion module and the few-view diffusion information module that diffuses the original RGB picture information. The diffusion results of the two are input into NeRF for initialization, and the geometric volume result of the body density rendering is extracted by the coarse network according to the above process. Through the comparison of the geometric loss function of each perspective, the grid is refined, and the surface excellent grid result is finally generated. In the light curing 3D printing, the texture grid of the color is not bound, and the final printing result is related to the photosensitive resin used. In this case, the rendering of the color texture is not performed to reduce the algorithm load. In this application, although the remove-bg algorithm is used to remove the background of the photographed object, part of the object reconstruction result still needs to be verified multiple times using the diffusion theory. The end-to-end few-view reconstruction algorithm will also be the future research direction of this application.
[0067] In a preferred embodiment, STL file preprocessing is a key step in 3D printing and numerical control machining, which usually includes reading, displaying, rotating, cutting, repairing and other processes. Through preprocessing, the support structure of the model file can be better analyzed, and the subsequent processing flow can be designed. The following is the detailed process of STL file preprocessing:
[0068] Reading: When processing different types of STL files, specific steps need to be performed, such as extracting normal vectors and vertex coordinates. These data are crucial for the geometric information of the model and are the basis for subsequent display and operation.
[0069] Display and rotation: By displaying and rotating the STL file, the forming direction of the model can be adjusted. Adjusting the direction not only helps to improve the printing quality of the model, but also can improve the printing success rate by reducing the projection area of the key position. For example, rotating the model to reduce the support requirements of the overhanging part, thereby optimizing the printing process.
[0070] Cutting: Cutting the STL file can divide the model into multiple parts, which helps to print complex structures in steps and reduces the overall support area. Cutting can also simplify the manufacturing process of large models, making them suitable for the physical limitations of the printer.
[0071] In this application, the state machine framework of OpenGL is used to realize the lighting, rendering, rotation and cutting operations of the STL file, and the Assimp library is used for optimization processing of STL information. In particular, we use an algorithm based on implicit cutting plane to realize the cutting of STL files through OpenGL. This algorithm allows users to move and rotate the implicit plane to flexibly set the cutting direction, thereby optimizing the support structure and reducing the use of support materials.
[0072] After cutting is completed, the program will automatically save the cut file and automatically repair the cutting surface to ensure the integrity of the model. Figure 8 The simulation object is demonstrated with Stanford Bunny (Bunny.stl) as an example, demonstrating the display and cutting effect of the model file. This preprocessing method significantly improves the efficiency of printing and processing, while reducing the use of support materials and optimizing the overall processing flow.
[0073] Working principle: according to SFM+MVS process, depth estimation and coordinate conversion are carried out through open source software COLMAP, the obtained sparse point cloud feature information and the shooting image are taken as input and transmitted into Nerf for initialization, meanwhile, feature extraction and grid rendering are carried out for the problem that Nerf itself does not support display STL model acquisition, a design idea suitable for three-dimensional explicit reconstruction of real world static objects is proposed, and the official data set of NeRF-synthetic is tested, and the final structure is realized, finally, aiming at the problem that the number of pictures of the real world static scene is not ideal, a few view reconstruction idea based on Diffusion diffusion is proposed, and finally it is verified.
[0074] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.
Claims
1. A 3D reconstruction method based on improved NeRF, including camera coordinate transformation, multi-view reconstruction scheme, few-view reconstruction scheme and STL file preprocessing, characterized in that: The camera coordinate transformation uses multiple two-dimensional images taken by the camera at different positions in space, and estimates the three-dimensional structure of the scene and the motion trajectory of the camera through the relative motion of these images. SFM uses feature points in the image group for matching, and estimates the motion trajectory of the camera and the spatial distribution of three-dimensional points through the relative motion of these feature points. That is, the posture information of the camera is inferred by analyzing the features of the two-dimensional image. The information obtained after converting the world coordinate system and the image coordinate system through the internal and external parameters of the camera of the SFM is passed as input to MVS for multi-view aggregation to form a dense point cloud, thereby constructing multi-view three-dimensional information.
2. The improved NeRF-based 3D reconstruction method according to claim 1, wherein: The camera conversion process is specifically as follows: Step 1: A point Pw (Xw, Yw, Zw) in the world coordinates represents the real coordinates of the object in the three-dimensional world. The world coordinates are converted to camera coordinates Pc (Xc, Yc, Zc) through the camera extrinsic rotation matrix R and translation vector T: Step 2: The camera coordinates Pc (Xc, Yc, Zc) are used to describe the position of the object relative to the camera. They are converted to image plane coordinates (xp, yp) through the camera's intrinsic parameters (internal parameters), namely the focal length f and the principal point position (cx, cy): Step 3: Convert the image plane coordinates to the image coordinate system, i.e. the UV plane, by pixel size and distortion correction: Among them, s x and s y is the pixel size, k1, k2 are the radial distortion coefficients, and p1, p2 are the tangential distortion coefficients.
3. The improved NeRF-based 3D reconstruction method according to claim 2, wherein: The multi-view reconstruction scheme includes COLMAP, neural radiance field and Nerf explicit output; by combining the SFM and MVS algorithms, pose estimation is introduced to realize dense point cloud fusion for three-dimensional reconstruction. The derived camera pose estimation information often maintains good stability. In the COLMAP, its pipeline pipeline process includes feature detection, feature matching and pose estimation.
4. The improved NeRF-based 3D reconstruction method according to claim 3, wherein: In the feature detection, COLMAP uses the SIFT algorithm to identify scale- and rotation-invariant key points from the image. It constructs a multi-scale pyramid for each image, detects extreme points at each scale and accurately locates them. Finally, it uses the Taylor expansion method to obtain the sub-pixel key point positions.
5. The improved NeRF-based 3D reconstruction method according to claim 4, wherein: In the pose estimation, COLMAP selects the image with the largest feature point overlap as the initial selection, and iterates the internal and external parameter conversion of the above-mentioned camera perspective by calculating the basic matrix between the two images until the end of the image group, so as to achieve the estimation effect of the camera pose. This process is an iterative optimization process by repeatedly adjusting the camera intrinsic and external parameters.
6. The improved NeRF-based 3D reconstruction method according to claim 5, characterized in that: The neural radiation field is a neural network that reconstructs three-dimensional scenes and objects by learning the feature relationships between a set of two-dimensional images. The core of the neural radiation field is to use a small multi-layer perceptron, combined with spatial coordinates and viewing direction, to output the color and volume density of corresponding points, and finally construct an implicit expression of the three-dimensional reconstruction of an object.
7. The improved NeRF-based 3D reconstruction method according to claim 6, wherein: The Nerf explicit output passes the RGB information of a set of two-dimensional images and the information of depth pose estimation performed by COLMAP into Nerf, and obtains the volume density information and color information of the corresponding three-dimensional scene reconstruction.
8. The improved NeRF-based 3D reconstruction method according to claim 7, characterized in that: In the few-view reconstruction scheme, noise is gradually added to the data sample through the diffusion algorithm and then the original number is returned. The Diffusion process includes a forward denoising process and a reverse denoising process, that is, the original image X0 is divided into a noise image X0 and a noise image X0. T The learning process of mutual denoising and denoising can use the Diffusion algorithm to predict pictures of different perspectives in the same scene.
9. The improved NeRF-based 3D reconstruction method according to claim 8, wherein: The STL file preprocessing is a key step in 3D printing and CNC machining, and generally includes reading, displaying, rotating, cutting, and repairing processes. Through preprocessing, the support structure of the model file can be better analyzed and subsequent machining processes can be designed.
10. The improved NeRF-based 3D reconstruction method according to claim 9, characterized in that: In the STL file preprocessing, the OpenGL state machine framework is used to implement the lighting, rendering, rotation and cutting operations of the STL file, the Assimp library is used to optimize the STL information, and an algorithm based on implicit cutting planes is adopted to implement the cutting of the STL file through OpenGL. The algorithm allows the user to flexibly set the cutting direction by moving and rotating the implicit plane, thereby optimizing the support structure and reducing the use of support materials. After the cutting is completed, the program automatically saves the cut file and automatically repairs the cut surface to ensure the integrity of the model.