Method for digitizing hub workpiece on basis of implicit three-dimensional reconstruction

By constructing a neural distance field model and optimizing the depth map, and combining the rotational symmetry of the wheel hub with point cloud completion technology, the problems of missing texture information and difficulty in bottom reconstruction in the 3D reconstruction of the wheel hub were solved, and high-precision wheel hub digitization was achieved.

WO2024234745A1PCT designated stage expired Publication Date: 2024-11-21HANGZHOU DIANZI UNIV

Patent Information

Application Number
PCT/CN2024/075682
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2024-02-04
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing 3D reconstruction methods struggle to accurately reconstruct the texture information and bottom region of wheel hubs when the original model is lacking. Furthermore, NeRF performs poorly on surfaces with indistinct textures, is prone to generating ghosting, and is difficult to accurately represent objects.

Method used

A neural distance field model is constructed, and combined with multilayer perceptron and depth map optimization, holes are filled by strong constraints and Gaussian probability distribution, and the rotational symmetry of the wheel hub is used to rotate and fill the point cloud. The normal and center point are optimized by combining size-invariant features and iterative nearest point algorithm to achieve accurate reconstruction of the wheel hub.

Benefits of technology

It improves the accuracy of 3D reconstruction of wheel hub workpieces, solves the problems of missing texture information and difficulty in bottom reconstruction, and realizes high-precision wheel hub digitization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024075682_21112024_PF_FP_ABST
    Figure CN2024075682_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed is a method for digitizing a hub workpiece on the basis of implicit three-dimensional reconstruction. The present invention comprises the following steps: step 1, constructing a neural distance field model; step 2, optimizing the neural distance field model; step 3, extracting point cloud information; step 4, acquiring a center point and a normal; and step 5, performing point cloud rotation supplementation. By means of the spatial rotatability of the hub, spatial points directly represented by the neural distance field model are supplemented by means of voting after point cloud rotation and expansion, and then a surface point cloud of the hub is finally obtained by means of spatial point corrosion. The present invention spans the fields of RF and NDdf, and spatial structure information constructed by a depth map is fully included in an RF training process in SDF format, thereby implementing intersection training of two types of modal data to make the point cloud more accurate. The present invention uses the deep learning field combined with exploring mathematical three-dimensional reconstruction methods and Lie algebra, having higher interpretability than deep learning methods.
Need to check novelty before this filing date? Find Prior Art

Description

A wheel hub workpiece digitization method based on implicit 3D reconstruction

[0001] The present invention claims priority to the Chinese patent application filed with the Patent Office of China on May 17, 2023, with application number 202310556740.2 and invention name “A method for digitizing a wheel hub workpiece based on implicit three-dimensional reconstruction”, the entire contents of which are incorporated by reference into the present invention. Technical Field

[0002] The present invention relates to the field of radiation field (RF), neural distance field (NSdf), and point cloud matching (PCR), and in particular to a wheel hub workpiece digitization method based on implicit three-dimensional reconstruction. Background Art

[0003] The field of 3D reconstruction studies the restoration of the original 3D model through other information such as images without the original model.

[0004] The Neural Signed Distance Field (NSDF) obtains an implicit spatial model by fitting the three-dimensional structure of real space. Essentially, the NSDF stores the closest distance from each point to the surface of an object, with distances outside the object being greater than 0 and distances inside being less than 0. Specifically, a multi-layer perceptron (MLP) is used as a model to predict signed distances.

[0005] NeRF achieves three-dimensional reconstruction in voxel space. The MLP inputs the coordinates of a spatial point (x, y, z) and the viewing direction (θ, φ) of the observation space point, and outputs the voxel at the spatial point and the corresponding color RGB value. Given the camera pose of a certain image (composed of the camera coordinates and viewing direction), the inferred image is obtained by integrating the color and voxel output by the MLP for each pixel under this pose, and the loss is obtained by pixel-by-pixel comparison with the real image for reverse optimization. Through the process of training with images from multiple perspectives as the true value, the MLP is fitted to a complex function so that any point in space can output the appropriate voxel and the corresponding color RGB value. There are also many works that improve on this basis, such as adding a distance cutoff as a constraint to the voxel accumulation process. These improvements have effectively improved the accuracy of implicit reconstruction, and their work has effectively improved the fineness of the reconstructed image.

[0006] Within the scope of this specification, point cloud matching refers to obtaining a rigid spatial transformation, including rotation and translation, such that the overlap between two point clouds is as high as possible. Point cloud matching is divided into two steps: coarse registration and fine registration. Coarse registration refers to the rough registration of the two point clouds when the transformation between them is completely unknown, obtaining the correspondence between the associated points in the point clouds. Fine registration refers to optimizing the transformation under the coarse registration point cloud correspondence. Fine registration can be achieved using the iterative closest point algorithm (ICP).

[0007] Problems with existing methods include:

[0008] 1. The SDF process does not learn texture information. Whether using traditional calculation methods or neural network fitting algorithms, the SDF process relies heavily on valid 3D structural information. When encountering areas where the original spatial structure is missing, the model is often inaccurate, resulting in an erroneous hole.

[0009] 2. NeRF performs poorly on surfaces with less pronounced textures. Strictly speaking, NeRF is not designed for 3D reconstruction. In particular, its volume density design can easily generate "ghosting" on surfaces and even in open areas, making it difficult to accurately represent objects.

[0010] 3. Existing methods have poor reconstruction performance for the bottom, which is difficult to capture. Both SDF and NeRF reconstruct based on the existing viewpoint. For parts not shown in the image or depth map, the algorithm simply guesses the shape and the predicted shape is far from the real object.

[0011] Summary of the Invention

[0012] In view of the shortcomings of the existing technology, the present invention proposes a wheel hub workpiece digitization method based on implicit 3D reconstruction, which can effectively overcome the above problems by making full use of the morphological characteristics of the wheel hub. The method includes the following steps:

[0013] Step 1: Construct a neural distance field model;

[0014] Construct a neural distance field model using a multi-layer perceptron MLP to represent the directed distance, taking the spatial point coordinates (x, y, z) and the viewing direction (θ, σ) as input and outputting the directed distance d out And the probability color c of the point in this direction out ;

[0015] Step 2: Optimize the neural distance field model;

[0016] With the help of camera Pose and depth map, the directed distance d out Add strong constraints to the predictions;

[0017] Step 3: Extract point cloud information;

[0018] By finding the directed distance d for the model out = 0, thereby extracting the smooth discrete surface of the hub model;

[0019] Step 4: Get the center point and normal;

[0020] Step 5: Point cloud rotation and completion;

[0021] The point cloud P is surrounded by T unknowThe obtained center point and normal are rotated and complemented, and the inconsistent spatial structure space is generated by completing the correct spatial structure information.

[0022] Furthermore, step 2 is implemented as follows:

[0023] 2-1. During sampling along the ray, the depth map provides a strict mandatory distance constraint value d true According to the design of the model, the midpoint in space passes through the directed distance d output by the neural distance field model out Close to the true constraint value d true , so construct the loss Loss d : Loss d =||d true -d out || 2 (1)

[0024] 2-2. For the holes in the depth map, the color estimation of the radiation field is used to fill the holes, and the approximate volume density v of the current point i is obtained through the Gaussian probability distribution on the signed distance field. i :

[0025] The density probability color c of the current point i in the viewing direction (θ, σ) in space i Expressed as:

[0026] The predicted color of a point on the image at the current viewing angle is obtained by summing the colors of the entire ray:

[0027] Structural Loss c : Loss c =||C predicition -C true || 2 (5)

[0028] Among them, C true Indicates the color of the corresponding point on the camera image; closest represents the closest position, which physically represents the starting position of the light from the camera when taking a picture; farthest represents the farthest position, which physically represents the position infinitely far away from the camera. Therefore, the value of closest is 0 and the value of farthest is +∞; C out Represents the probability color of the spatial point (x, y, z) in the viewing direction (θ, σ) in the neural distance field model constructed in step 1.

[0029] Furthermore, through Loss d and Loss cImplement neural distance field model optimization; assume that (a1, a2, a3) and (b1, b2, b3) in the virtual space represent the pixel point (x, y) in the nth set of depth and color maps corresponding to the current point (x, y, n); first determine whether the depth of (x, y, n) exists in the depth map. If the depth value corresponding to (b1, b2, b3) exists, then use both depth and neural predicted color to optimize the directed distance; if (a1, a2, a3) does not exist, then only use neural rendering predicted color to optimize the directed distance.

[0030] Furthermore, step 3 of extracting point cloud information is implemented as follows:

[0031] Due to image capture issues and noise interference in some areas, the directed distance fitted by the MLP has jumps, which neither conforms to the definition of a continuous smooth surface nor the actual shape of the wheel hub. Manual pre-calibration and voting methods are required to finally obtain the point cloud P.

[0032] Furthermore, the specific implementation of step 4 center point and normal acquisition is as follows:

[0033] 4-1. Use the SIFT algorithm to calibrate and obtain the SIFT features in the image. SIFT features are local features of the image. The specific operations are as follows:

[0034] Take a large front view of the wheel hub to remove the distortion a The connection points and the center of the spokes and the outer ring of the hub are marked as feature points. The SIFT algorithm is used to extract the features of the connection points and obtain the corresponding feature vectors. In the dataset used to optimize the neural distance field model, three front-facing images P containing the hub are found. b 、P c 、P d , and obtain the front image P respectively through the size-invariant feature transformation algorithm SIFT comparison b 、P c 、P d The plane coordinates of the connection point between the middle spoke and the outer ring and the center of the hub;

[0035] The number of the connection points is determined according to the spokes of the hub, and the number of the connection points is equal to the number of the spokes;

[0036] 4-2. Using the front image P b 、P c 、P d Obtain the wheel center and camera pose, calculate the normal and center point, and the specific operations are as follows:

[0037] Suppose the front picture P b The center of the plane is O b, with the help of camera pose Pose, through O b Establish a straight line L b , the straight line passes through O b Perpendicular to the front image P b plane;

[0038] Similarly, we get the straight line L c and L d ; Since the straight line L b , L c , L d Both pass through the center point O of the hub in real space, and L b , L c , L d They do not coincide, so the hub center point O must be the straight line L b , L c 、L d The intersection of

[0039] Similarly, determine the real space coordinates of the key points on the front of the hub. With the help of the space coordinates of the key points connecting the spokes and the hub, calculate the plane representation of the front of the hub and indirectly obtain the normal vector L of the front of the hub.

[0040] 4-3. Optimization of the wheel hub normal vector and center point.

[0041] The optimization method used for the wheel hub normal vector and center point uses the iterative closest point algorithm point cloud matching. The specific method is as follows:

[0042] 4-3-1. Remove the ground from the point cloud P extracted in step 3 and expand it to obtain the expanded point cloud P1. Rotate P1 by a specific angle α along the normal to obtain point cloud P2. The rotation process is recorded as T alpha ;

[0043] 4-3-2. Use the greedy algorithm between point clouds P1 and P2 to find the closest points as corresponding point pairs, calculate the Euclidean distance between the corresponding points, set a threshold and remove corresponding point pairs whose Euclidean distance is greater than the threshold, and construct Lose, as shown in Formula 6: Lose=||T unknow P origin -T alpha T unknow P origin || (6)

[0044] Among them, T unknow Represents an unknown spatial change and is the solution object; if and only if T unknow When the solution is reached, Loss is at its minimum value, T unknow P origin The obtained point cloud P trans Then through T alphaThe point cloud P obtained after transformation end Just like P trans Overlap; according to T unknow Obtain the true position and direction of the normal and center point, T unknow The physical meaning is that when the normal and center point are kept unchanged, the hub point cloud needs to be transformed to minimize the Loss; conversely, when it is known that the transformation fixes the point cloud, the inverse process of the spatial transformation can be applied to the normal and center point; P origin Represents the original point cloud expanded after removing the ground in step 4-3-1, which is numerically equal to P1;

[0045] Considering that the spatial point asymptotic process is nonlinear, the Lie algebra special orthogonal group SO(3) and special Euclidean group SE(3) are used to represent it. The spatial transformation of the point cloud is recorded as T and the perturbation model of Lie algebra is used for nonlinear optimization, as shown in Formula 7, where ΔT is the left multiplication perturbation. The specific method is to update ΔT into T after finding the zero value by taking the derivative of ΔT, as shown in Formula 8. Finally, ΔT becomes the E matrix, T unknow The value of is optimized; Lose=||ΔTT unknow P origin -T alpha ΔTT unknow P origin || (7) T t+1 ←ΔT, T t (8).

[0046] Among them, T t+1 represents the t+1th spatial transformation; T t represents the t-th spatial transformation.

[0047] Furthermore, the point cloud rotation completion in step 5 is specifically implemented as follows:

[0048] The point cloud P is surrounded by T unknow The obtained center point and normal are rotated and filled, and the inconsistent spatial structure space is generated by the correct spatial structure information. Assuming that the hub is N spokes, N pieces of spatial structure information can be obtained at the same angle through spatial rotation;

[0049] Perform an expansion operation on N pieces of spatial structure information and then superimpose them into the same space. Record the votes for the same spatial point and whether multiple pieces of spatial structure information exist at that point. Keep the points with more than half of the votes and discard the others.

[0050] Perform an erosion operation on the merged point cloud until it remains as a layer of point cloud;

[0051] Finally, the point cloud is the desired hub surface point cloud, thus completing the digitization of the hub workpiece based on implicit 3D reconstruction.

[0052] The beneficial effects of the present invention are as follows:

[0053] 1. This invention transcends the fields of RF and NDDF, fully integrating the spatial structure information constructed by the depth map into the RF training process in the form of SDF, and realizes the intersection training of the two modal data, making the point cloud more accurate.

[0054] 2. Unlike the existing visual methods that have difficulty in training symmetrical structures, the present invention actively utilizes the spatial characteristics of the scene to solve this problem.

[0055] 3. This invention not only uses the field of deep learning but also explores the method of mathematical three-dimensional reconstruction and Lie algebra, which is more interpretable than deep learning methods.

[0056] 4. The present invention proposes a method for optimizing point clouds with spatial symmetry and spatial rotational symmetry, which can be extended as a standard to other scenes with spatial symmetry and spatial rotational symmetry. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] FIG1 is a flow chart of the present invention;

[0058] FIG2 is a diagram of a neural distance field model proposed in the present invention;

[0059] FIG3 is a process diagram of training a neural distance field model according to the present invention;

[0060] Figure 4 is a schematic diagram of extracting point clouds in implicit space;

[0061] FIG5 is a schematic diagram of extracting corresponding feature points using the SIFT method;

[0062] FIG6 is a schematic diagram of finding the center point in the implicit space based on feature points;

[0063] FIG7 is a schematic diagram of optimizing normals and center points using the Lie algebra method;

[0064] FIG8 is a flow chart of obtaining the surface of the wheel hub model after performing expansion voting corrosion on the space.

[0065] Specific implementation methods

[0066] The present invention will be further described below with reference to the accompanying drawings and examples.

[0067] This invention provides a workpiece digitization method based on neural distance fields and implicit 3D reconstruction. The neural distance field can be used to obtain the smooth surface geometry of an object; the neural radiance field can be used to add color information to training, accelerating network training. This allows for the rapid and efficient generation of a model capable of representing a smooth workpiece surface.

[0068] The most important aspect is the utilization of rotational symmetry, which means that after a certain angle of rotation, a shape in a plane will overlap with the original shape. Wheels are manufactured under industrial conditions and also follow three-dimensional rotational symmetry, meaning that after a certain angle of rotation in a specific direction, the wheel will overlap with the original shape in space.

[0069] As shown in FIG1 , the method of the present invention mainly comprises the following steps:

[0070] Step 1: Construct a neural distance field model.

[0071] Construct a neural distance field model using a multi-layer perceptron MLP to represent the directed distance. The neural distance field model is shown in Figure 2. It takes the spatial point coordinates (x, y, z) and the viewing direction (θ, σ) as input and outputs the directed distance d out And the probability color c of the point in this direction out .

[0072] Step 2: Optimize the neural distance field model.

[0073] With the help of camera Pose and depth map, the directed distance d out Specifically, in the process of sampling along the ray, the depth map provides a strict mandatory distance constraint value dtrue; according to the design of the model, the directed distance dout output by the neural distance field model of the midpoint in space is close to the true constraint value dtrue, so the loss Lossd is constructed: Lossd=||dtrue-dout||2 (1)

[0074] In the hole part of the depth map, the radiation field is used to estimate the color to fill the hole, and the approximate volume density v of the current point i is obtained in the directed distance field through the Gaussian probability distribution. i :

[0075] The density probability color c of the current point i in the viewing direction (θ, σ) in space i Expressed as:

[0076] The predicted color of a point on the image at the current viewing angle is obtained by summing the colors of the entire ray:

[0077] Structural Loss c : Loss c =||C predicition -C true || 2 (5)

[0078] Among them, C trueIndicates the color of the corresponding point on the camera image; closest represents the closest position, which physically represents the starting position of the light from the camera when taking a picture; farthest represents the farthest position, which physically represents the position infinitely far away from the camera. Therefore, the value of closest is 0 and the value of farthest is +∞; C out Represents the probability color of the spatial point (x, y, z) in the viewing direction (θ, σ) in the neural distance field model constructed in step 1.

[0079] In particular, the metal wheel and the surrounding environment have significant color differences, especially in hue and brightness. RGB color cannot effectively represent this color and brightness difference. Therefore, the HSL color space is used to linearly convert the RGB color into hue, saturation, and lightness. The hue is calculated based on the counterclockwise angle between the predicted value and the true value and normalized. Furthermore, the high consistency of the hue of the metal surface can be used to constrain the cumulative color of the entire light. Specifically, a penalty function is used to ensure that the final predicted color is close to the metal surface.

[0080] By Loss d and Loss c To achieve optimization in the depth map network, as shown in Figure 3, (1, 2, 1) and (4, 1, 1) in the virtual space in the figure represent the (x, y) pixel in the nth set of depth and color maps corresponding to the current point (x, y, n). First, determine whether the depth of (x, y, n) exists in the depth map. If it does, such as d41 corresponding to (4, 1, 1), then use both depth and neural predicted color to optimize the directed distance; if it does not, such as (1, 2, 1), then only use neural rendering predicted color to optimize the directed distance.

[0081] Step 3: Extract point cloud information.

[0082] As introduced in the background, a positive SDF distance indicates that the point is outside the object, and a negative SDF distance indicates that the point is inside the object. As shown in Figure 4, by searching the model for a set with an SDF distance of 0, that is, a directed distance d out = 0, the smooth discrete surface of the wheel hub model can be extracted.

[0083] However, due to image capture issues and noise interference in some areas, the directed distance fitted by MLP has jumps, which neither conforms to the definition of a continuous smooth surface nor the actual shape of the actual wheel hub. This is a kind of noise introduced by the original data. For example, when there is a large range of reflections, the directed distance fitted by MLP easily generates a point cloud in space. Its physical meaning is a luminous object in space, which does not actually exist and needs to be discarded. The point cloud P can be finally obtained through manual pre-calibration and the voting method in step 5.

[0084] Step 4: Get the center point and normal.

[0085] 4-1. Use the Scale Invariant Feature Transform (SIFT) algorithm for calibration to obtain the SIFT features in the image. SIFT features are local features of the image that remain invariant to rotation, resizing, and brightness changes, and also maintain a certain degree of stability to perspective changes, affine transformations, and noise. The specific operations are as follows:

[0086] As shown in Figure 5, a large front view of the wheel hub is taken to remove the distortion. a The connection points and the center of the spokes and the outer ring of the hub are marked as feature points. The SIFT algorithm is used to extract the features of the connection points and obtain the corresponding feature vectors. In the dataset used to optimize the neural distance field model, three front-facing images P containing the hub are found. b 、P c 、P d , and obtain the front image P respectively through the size-invariant feature transformation algorithm SIFT comparison b 、P c 、P d The plane coordinates of the connection point between the middle spoke and the outer ring and the center of the hub.

[0087] The number of the connection points is determined according to the spokes of the hub, and the number of the connection points is equal to the number of the spokes.

[0088] 4-2. Using the front image P b 、P c 、P d Obtain the wheel center and camera pose, calculate the normal and center point, and the specific operations are as follows:

[0089] As shown in Figure 6, suppose the front image P b The center of the plane is O b , with the help of camera pose Pose, through O b Establish a straight line L b , the straight line passes through O b Perpendicular to the front image P b Similarly, we get the straight line L c and L d ; Since the straight line L b 、L c 、L d Both pass through the center point O of the hub in real space, and L b 、L c 、L d If they do not coincide, then the hub center point O must be the straight line L b 、L c 、L dSimilarly, determine the real space coordinates of the key points on the front of the hub. With the help of the spatial coordinates of the key points connecting the spokes and the hub, calculate the plane representation of the front of the hub and indirectly obtain the normal vector L of the front of the hub.

[0090] 4-3. Optimization of the wheel hub normal vector and center point.

[0091] Theoretically, O and L should be the normal vector in a certain space and the spatial center point of the wheel hub, but due to image capture errors, there may be some deviation. As shown in Figure 7, the optimization method used uses the iterative closest point algorithm for point cloud matching. The specific method is as follows:

[0092] In reality, wheels are three-dimensionally rotationally symmetric. This means that simply rotating the front of the wheel by the angle corresponding to the number of spokes (α = (360° / number of spokes)) yields a wheel that is identical to the original wheel. A perfect wheel point cloud is also rotationally symmetric. Therefore, using this property, the spatial transformation for the coarse point cloud matching is set to no transformation. The number of spokes can be determined manually or predicted using a simple convolutional neural network.

[0093] 4-3-2. Use the greedy algorithm between point clouds P1 and P2 to find the closest points as corresponding point pairs, calculate the Euclidean distance between the corresponding points, set a threshold and remove corresponding point pairs whose Euclidean distance is greater than the threshold, and construct Lose, as shown in Formula 6. Lose = || T unknow P origin -T alpha T unknow P origin || (6)

[0094] Among them, T unknow Represents an unknown spatial change and is the solution object; if and only if T unknow When it is the solution (Lose is the minimum value), T unknow P origin The obtained point cloud P trans Then through T alpha The point cloud P obtained after transformation end Just like P trans Overlap; according to T unknow Obtain the true position and direction of the normal and center point, T unknow The physical meaning is that when the normal and center point are kept unchanged, the hub point cloud needs to be transformed to minimize the Loss; conversely, when it is known that the transformation fixes the point cloud, the inverse process of the spatial transformation can be applied to the normal and center point; P originRepresents the original point cloud expanded after removing the ground in step 4-3-1, which is numerically equal to P1. Considering that the spatial point asymptotic process is nonlinear, the special orthogonal group SO(3) and special Euclidean group SE(3) of Lie algebra are used to represent it. The spatial transformation of the point cloud is recorded as T and the perturbation model of Lie algebra is used for nonlinear optimization, as shown in Formula 7, where ΔT is the left multiplication perturbation; the specific method is to update ΔT into T after finding the value of 0 by taking the derivative of ΔT, as shown in Formula 8, and finally ΔT becomes the E matrix, T unknow The value of is optimized. Lose=||ΔTT unknow P origin -T alpha ΔTT unknow P origin || (7) T t+1 ←ΔT, T t (8).

[0095] Among them, T t+1 represents the t+1th spatial transformation; T t represents the t-th spatial transformation.

[0096] Step 5: Rotate and complete the point cloud.

[0097] As shown in Figure 8, the point cloud P is surrounded by T unknow The center point and normal obtained are rotated and aligned. The correct spatial structure information is used to complete the generated inconsistent spatial structure space. Taking a 5-spoke wheel as an example, through spatial rotation, 5 copies of spatial structure information can be obtained at the same angle.

[0098] The multiple spatial structure information are expanded and then superimposed into the same space. The votes of the same spatial point are recorded (whether multiple spatial structure information exist at this point). The points with more than half of the votes are retained and the others are discarded.

[0099] The merged point cloud is eroded until only one layer of point cloud remains. This final point cloud is the wheel hub surface point cloud required by the specification. This completes the wheel hub workpiece digitization method based on implicit 3D reconstruction.

Claims

1. A wheel hub workpiece digitization method based on implicit 3D reconstruction, characterized in that: The steps include: Step 1: Construct a neural distance field model; Construct a neural distance field model that uses a multi-layer perceptron MLP to represent directed distances, taking spatial point coordinates (x, y, z) and viewing direction (θ, σ) as input and outputting the directed distance d out And the probability color c of the point in this direction out ; Step 2: Optimize the neural distance field model; With the help of camera Pose and depth map, the directed distance d out Add strong constraints to the predictions; Step 3: Extract point cloud information; By finding the directed distance d for the model out =0, thereby extracting the smooth discrete surface of the wheel hub model; Step 4: Get the center point and normal line; Step 5: Point cloud rotation and completion; With the help of the spatial rotation of the wheel hub, the spatial points directly represented by the neural distance field model are completed by rotating and expanding the point cloud and then voting, and then the surface point cloud of the wheel hub is finally obtained through spatial point erosion.

2. The method for digitizing a hub workpiece based on implicit 3D reconstruction according to claim 1, characterized in that: Step 2 is implemented as follows: 2-1. During sampling along the ray, the depth map provides a strict mandatory distance constraint value d true ; According to the design of the model, the midpoint of the space passes through the directed distance d output by the neural distance field model out Close to the true constraint value d true , so construct the loss Loss d : Loss d =||d true -d out || 2 (1) 2-2. For the holes in the depth map, the radiation field is used to estimate the color to fill the holes, and the approximate volume density v of the current point i is calculated on the signed distance field through the Gaussian probability distribution. i : The density probability color c of the current point i in the viewing direction (θ, σ) in space i It is expressed as: The predicted color of a point on the image at the current viewing angle is obtained by summing the colors of the entire ray: Construction loss Loss c : Loss c =||C predicition -C true || 2 (5) Among them, C true Indicates the color of the corresponding point on the camera image; closest represents the closest position, which physically represents the starting position of the light from the camera when taking a picture; farthest represents the farthest position, which physically represents the position infinitely far away from the camera, so the value of closest is 0 and the value of farthest is +∞; C out Represents the probability color of the spatial point (x, y, z) in the viewing direction (θ, σ) in the neural distance field model constructed in step 1.

3. The method for digitizing a hub workpiece based on implicit 3D reconstruction according to claim 2, characterized in that: By Loss d and Loss c Implement neural distance field model optimization; assume that (a1, a2, a3) and (b1, b2, b3) indicated in the virtual space represent the pixel point (x, y) in the nth group of depth map and color map corresponding to the current point (x, y, n); first determine whether the depth of (x, y, n) exists in the depth map. If the depth value corresponding to (b1, b2, b3) exists, use both depth and neural predicted color to optimize the directed distance; if (a1, a2, a3) does not exist, only use neural rendering predicted color to optimize the directed distance.

4. A hub workpiece digitization method based on implicit 3D reconstruction according to claim 2 or 3, characterized in that: Step 3 extracts point cloud information in detail as follows: Due to image shooting problems and noise interference in some areas, the directed distance of MLP fitting has jumps, which neither conforms to the definition of a continuous smooth surface nor the actual shape of the actual wheel hub. It is necessary to use manual calibration and voting method to finally obtain the point cloud P.

5. A hub workpiece digitization method based on implicit 3D reconstruction according to claim 2 or 3, characterized in that: Step 4 The specific implementation of obtaining the center point and normal is as follows: 4-1. Use the SIFT algorithm to calibrate and obtain the SIFT features in the image; SIFT features are local features of the image. The specific operations are as follows: Take a large picture of the front of the wheel hub to remove the distortion a , mark the connection points and the center of the wheel spokes and the outer ring of the wheel hub as feature points, and extract the features of the connection points using the SIFT algorithm to obtain the corresponding feature vectors; in the dataset used to optimize the neural distance field model, find three frontal images P containing the wheel hub. b , P c , P d , and obtain the front image P respectively through the SIFT algorithm. b , P c , P d The plane coordinates of the connection point between the middle spoke and the outer ring and the center of the hub; The number of the connection points is determined according to the spokes of the hub, and the number of the connection points is equal to the number of the spokes; 4-2. Using the front picture P b , P c , P d Obtain the wheel center and camera pose, calculate the normal and center The specific operations are as follows: Set the front picture P b The center of the plane is O b , with the help of camera pose Pose, through O b Establish a straight line L b , the straight line passes through O b Vertical to the front image P b plane; Similarly, we get the straight line L c and L d ; Since the straight line L b , L c , L d All pass through the center point O of the hub in real space, and L b , L c , L d They do not coincide, so the hub center point O must be the straight line L b , L c , L d The intersection of Similarly, determine the real space coordinates of the key points on the front of the hub, and use the space coordinates of the key points connecting the spokes and the hub to calculate the plane representation of the front of the hub, and indirectly obtain the normal vector L of the front of the hub; 4-3. Optimization of wheel hub normal vector and center point.

6. The method for digitizing a hub workpiece based on implicit 3D reconstruction according to claim 5, characterized in that: The optimization method used in step 4-3 for the wheel normal vector and center point uses the iterative closest point algorithm for point cloud matching. The specific method is as follows: 4-3-1. Remove the ground from the point cloud P extracted in step 3 and expand it to obtain the expanded point cloud P1. Rotate P1 by a specific angle α along the normal to obtain point cloud P2. The rotation process is recorded as T alpha ; 4-3-2. Use the greedy algorithm between point cloud P1 and point cloud P2 to find the closest point as the corresponding point pair, calculate the Euclidean distance between the corresponding points, set a threshold and remove the corresponding point pairs whose Euclidean distance is greater than the threshold, and construct Lose, as shown in Formula 6: Lose=||T unknow P origin -T alpha T unknow P origin || (6) Among them, T unknow represents an unknown spatial change and is the object to be solved; if and only if T unknow When the solution is reached, Lose is the minimum value, T unknow P origin The point cloud P obtained trans Then through T alpha The point cloud P obtained after transformation end Just like P trans Overlap; Based on T unknow Obtain the true position and direction of the normal and center point, T unknow The physical meaning is that when the normal and the center point are kept unchanged, the hub point cloud needs to be transformed to minimize the Loss; conversely, when it is known that the transformation will fix the point cloud, the inverse process of the spatial transformation can be applied to the normal and the center point; P origin It represents the original point cloud expanded after removing the ground in step 4-3-1, which is numerically equal to P1; Considering that the asymptotic process of spatial points is nonlinear, the special orthogonal group SO(3) and the special Euclidean group SE(3) of Lie algebra are used to represent it. The spatial transformation of the point cloud is recorded as T and the perturbation model of Lie algebra is used for nonlinear optimization, as shown in Formula 7, where ΔT is the left multiplication perturbation. The specific method is to update ΔT into T after finding the zero value by taking the derivative of ΔT, as shown in Formula 8. Finally, ΔT becomes the E matrix, T unknow The value of is optimized; Lose=||ΔTT unknow P origin -T alpha ΔTT unknow P origin || (7) T t+1 ←ΔT,T t (8)。 Among them, T t+1 represents the t+1th spatial transformation; T t represents the t-th spatial transformation.

7. The method for digitizing a hub workpiece based on implicit 3D reconstruction according to claim 6, characterized in that: Step 5: Point cloud rotation completion is implemented as follows: The point cloud P is surrounded by T unknow The obtained center point and normal are rotated and complemented, and the inconsistent spatial structure space is generated through the correct spatial structure information; Assuming the hub is N spokes, by rotating in space, N pieces of spatial structure information can be obtained at the same angle; The N pieces of spatial structure information are expanded and then superimposed into the same space, and the votes of the same spatial point are recorded to determine whether multiple pieces of spatial structure information exist at that point. Keep the points with more than half of the votes and discard the others; Perform an erosion operation on the merged point cloud until it is retained as a layer of point cloud; Finally, the point cloud is the desired wheel hub surface point cloud, thus completing the digitization of the wheel hub workpiece based on implicit 3D reconstruction.

Citation Information

Patent Citations

  • Three-dimensional model reconstruction method and device and computer readable storage medium

    CN115294275A

  • Dense synchronous positioning and mapping method based on voxel nerve implicit surface

    CN115619951A

  • Outdoor unbounded scene three-dimensional reconstruction method and system based on neural radiation field

    CN116051740A

  • Hub workpiece digitalization method based on implicit three-dimensional reconstruction

    CN116597082A

  • Figure-Ground Neural Radiance Fields For Three-Dimensional Object Category Modelling

    US20230130281A1

Cited By

  • Three-dimensional automatic modeling method for high-precision transformer substation

    CN119919578A

  • Large-scene point cloud fast up-sampling method based on implicit neural network and spatial hash

    CN119963767A

  • Landform surveying and mapping method, medium and system based on fusion of laser point cloud and optical image

    CN122335930A