Three-dimensional reconstruction method capable of being used for internal structure of cabin

Abstract

The invention relates to a three-dimensional reconstruction method capable of being used for an internal structure of a cabin, and solves the technical problem of how to improve the path planning precision in the process of performing automatic spraying operation on a ship by using a spraying robot. According to the method, a color image and a depth image are acquired by using a depth camera, and instance segmentation of the color image is realized by using a Mask R-CNN as a framework and Swin-Transform as a backbone network; performing feature extraction and feature matching on the color image to obtain a preliminary camera pose, optimizing the camera pose by using depth information and instance information, fusing the depth information and the camera pose to obtain a dense point cloud, and optimizing the point cloud by using 3D-Gaussian-patterning; comparing the instance information with the three-dimensional point cloud to obtain a point cloud with the instance information; and performing point cloud matching on the obtained component with the instance information and a standard component to obtain a complete and accurate point cloud. Target distinguishing and recognition between different components in the cabin and high-precision reconstruction of the inner surface of the cabin are achieved, and the requirement for subsequent customized and refined spraying tasks is met.

Description

Technical Field

[0001] The present invention relates to the technical field of ship spraying, and in particular to a three-dimensional reconstruction method that can be used for the internal structure of a cabin. Background Art

[0002] Ships are affected by seawater, atmosphere and marine organisms in the seawater. Anti-rust and painting measures can greatly extend the service life of ships. In the shipbuilding process, the interior of the cabin needs to be rust-free and painted. The hull rust prevention mainly relies on paint.

[0003] The painting process is mainly carried out manually, and spraying robots are also used. Spraying robots have the advantages of good spraying quality, high spraying efficiency, low operating cost, and avoiding toxic gases that harm human health. The use of painting robots can not only ensure the quality of the coating, but also avoid the damage of harmful spraying operating environments such as high temperature, high noise, and toxic gases.

[0004] The use of spray robots to realize the spraying process is the development trend of China's shipbuilding industry. The application of robots in ship spraying is of great significance to the intelligent development of China's shipbuilding industry. For the current structure and function of the spraying robot, you can refer to the invention patent application with the application publication number "CN115008435A" and the invention name "Large-scale multi-degree-of-freedom hybrid hull spraying robot", or the invention patent application with the application publication number "CN113751231A" and the name "Ultra-large space spraying equipment and spraying method for ship segmented non-structural surfaces". Track-type spraying robots are widely used, such as the invention patent application with the application publication number CN111013882A.

[0005] However, when using spray robots to achieve automatic spraying operations, the accuracy of path planning needs to be improved to meet the needs of high-precision, high-efficiency and refined spraying tasks. Summary of the invention

[0006] The present invention aims to solve the technical problem of how to improve the path planning accuracy during the automatic spraying operation of a ship using a spraying robot, and to provide a three-dimensional reconstruction method that can be used for the internal structure of a cabin, which is conducive to improving the path planning accuracy.

[0007] The present invention provides a three-dimensional reconstruction method that can be used for the internal structure of a cabin, comprising the following steps:

[0008] The first step is to use an RGB-D camera to capture images of the object to be reconstructed and obtain images from different perspectives;

[0009] The second step is to preprocess the RGB-D images collected by the RGB-D camera;

[0010] The third step is to perform instance segmentation on the preprocessed color image;

[0011] Instance segmentation is performed using the Mask R-CNN model, and the backbone network of the Mask R-CNN model uses the Swin-Transformer network. The Swin-Transformer network is trained using the COCO dataset.

[0012] The preprocessed color image is input into the trained Mask R-CNN model to perform instance segmentation on the object to be reconstructed and obtain the instance segmentation result;

[0013] The fourth step is to calculate the initial pose of the RGB-D camera;

[0014] Step S401, the preprocessed color image {F1, F2...F n}Use SIFT algorithm to extract features and get feature points. Next, use fast approximate nearest neighbor method to compare the two images F i and F j Perform feature matching to find corresponding feature points;

[0015] Step S402, obtaining the preliminary pose of the RGB-D camera through the PnP algorithm; first, the PnP algorithm is used to select the image F i and F j The transformation matrix is ​​calculated and converted from image F i The collected corresponding feature points are placed in the point set P m In the image F j The collected corresponding feature points are placed in the point set Q m middle,

[0016] P m ={p1,p2.....p n}∈F i

[0017] Q m ={q1,q2.....q n}∈F j

[0018]

[0019] Among them, {p1,p2.....p n} is the image F i The characteristic points of {q1,q2.....q n} is the image F j The characteristic points of R and t are point sets P m The corresponding feature points in are transformed into point set Q mThe rotation matrix and translation vector required when

[0020] Further nonlinear optimization is performed to solve the rotation matrix R and translation vector t by minimizing the error function:

[0021]

[0022] Then the preliminary pose of the RGB-D camera is obtained;

[0023] Step 5: Obtain a preliminary reconstructed point cloud based on the obtained preliminary pose and the preprocessed depth image;

[0024] Step 6: Optimize the initial camera pose;

[0025] Step 7: Obtain a dense point cloud based on the optimized camera pose and preprocessed depth image;

[0026] Step 8: Optimize the dense point cloud by 3D-Gaussion-Splatting to obtain the reconstructed point cloud;

[0027] In the ninth step, the reconstructed point cloud obtained in the eighth step is mapped with the instance segmentation result obtained in the third step to obtain a three-dimensional point cloud containing instance information;

[0028] Step 10: Create model point cloud;

[0029] Step S1001, using SolidWorks software to build a three-dimensional model of the object to be reconstructed;

[0030] Step S1002, using CloudCompare software to process the three-dimensional model in STL format, thereby obtaining an accurate model point cloud;

[0031] In the eleventh step, the three-dimensional point cloud containing instance information is registered with the model point cloud to obtain a three-dimensional point cloud for generating the work trajectory.

[0032] Preferably, the fifth step adopts the following process:

[0033] Step S501, initialize a 3D voxel grid, each unit of the voxel grid stores the TSDF value D(x) and the weight W(x), project each depth map pixel to the voxel grid in the 3D space, and for each depth map pixel (u, v, d), convert the pixel coordinates and depth value d into the 3D point coordinates p in the camera coordinate system through the camera intrinsic parameter K cam :

[0034] P cam =d.(K -1 .[u,v,1] T )

[0035] Use the obtained preliminary camera pose, i.e., rotation matrix R and translation vector t, to transform the 3D point in the camera coordinate system to the 3D point P in the world coordinate system. w :

[0036] P w =RP cam +t

[0037] Step S502: for each three-dimensional point coordinate P w , calculate its distance φ(p w )=||p w -c||, if the distance is less than the cutoff distance δ, then the distance is mapped to the TSDF value D(p w ):

[0038]

[0039] The calculated TSDF value D(p w ) value and the TSDF value D(x) of the existing voxel grid are weighted averaged:

[0040]

[0041] Where W(x) is the existing weight, ω p is the weight of the depth image;

[0042] Repeat steps S501 and S502 until all depth map pixels are processed, so that the voxel grid stores the TSDF values ​​after all depth images are fused; finally, extract the isosurface from the voxel grid and extract the vertices of the isosurface to obtain a preliminary reconstructed point cloud.

[0043] The sixth step uses the following process:

[0044] The preliminary reconstructed point cloud is projected onto the 2D image plane to obtain the rendered color image, depth value and instance features of the rendered color image. The rendered color image, depth value and instance features of the rendered color image are compared with the true value collected by RGB-D. The true value refers to the preprocessed color image. The camera pose is iteratively optimized through color loss, depth loss and instance information tracking loss.

[0045] The color loss Lc represents the weighted value of SSIM loss and L1 loss, and its formula is:

[0046] Lc=λL1(I r ,I gt )+(1-λ)[1-ssim(I r ,I gt )]

[0047] Among them, I r Indicates rendering of color images, I gt represents the true value, λ is the weighting parameter;

[0048] Depth loss L d The formula is:

[0049]

[0050] in, is the rendering depth value, is the depth value of the real value;

[0051] Total loss L total The formula is as follows:

[0052] L total =λ c L c +λ d L d +λ s L1(S render ,S head )

[0053] Among them, S render represents the instance feature of the rendered color image, S head The segmentation mask representing the true value, λ c , d , s Respectively represent the weight values ​​of color loss, depth loss, and instance loss;

[0054] Next, the gradient of the rotation matrix R and the translation vector t is calculated, and the pose is updated according to the learning rate η.

[0055]

[0056] After multiple iterations, the optimized camera pose is obtained, and the 4x4 pose transformation matrix T is used. opt express:

[0057]

[0058] Among them, R opt represents the optimized rotation matrix, t opt Represents the optimized translation vector.

[0059] Preferably, the seventh step adopts the following process: the preliminary reconstructed point cloud is updated and optimized according to the optimized camera pose, and the three-dimensional point coordinates p of each depth map pixel (u, v, d) in the camera coordinate system are cam , the optimized camera pose is the rotation matrix R opt and the translation vector topt , and then get the three-dimensional point P in the world coordinate system after optimizing the camera pose w ':

[0060] P′ w =R opt .P cam +t opt

[0061] For the three-dimensional point coordinates P w ', calculate its distance φ(p' w )=||p' w -c||, when the distance is less than the cutoff distance δ, the distance is mapped to the TSDF value D(p' w ):

[0062]

[0063] The calculated optimized TSDF value D(p' w ) is weighted averaged with the TSDF value D(x) in the existing voxel grid,

[0064]

[0065] Where W(x) is the existing weight, ω p is the weight of the depth image;

[0066] Until all depth map pixels are processed, the voxel grid stores all TSDF values ​​after depth image fusion under the optimized camera pose. By extracting the isosurface from the voxel grid and extracting the vertices of the isosurface, the optimized reconstructed point cloud is obtained, that is, the dense point cloud is obtained.

[0067] Preferably, the ninth step is performed using the following process:

[0068] Preprocessed color image F n Each pixel point (i, j) has corresponding color information and depth information, and the instance segmentation mask M is obtained by performing instance segmentation on the color image. i ∈{0,1} H×W and category information C i , classify each pixel (i, j), where M i (i,j)=1 means that this pixel belongs to a certain instance, M i (i, j) = 0 means that the pixel does not belong to the instance. The pixel position and category of each instance are determined by the mask and category information obtained after segmentation.

[0069] For the pixel point (i, j) belonging to the object in the segmentation mask, the corresponding depth information is obtained, and the pixel point is converted into a three-dimensional point in the camera coordinate system in combination with the camera intrinsic parameter Then, we can obtain the multi-view segmentation mask and the corresponding 3D point in the camera coordinate system, and then transform the 3D point in the camera coordinate system into the 3D point in the global coordinate system. Finally, a three-dimensional point cloud containing instance information is obtained.

[0070] Preferably, the eleventh step is performed using the following process:

[0071] Step S1101, coarse registration; using the RANSAC coarse matching algorithm based on FPFH features, step (1): downsampling the three-dimensional point cloud containing instance information and downsampling the model point cloud;

[0072] Step (2): extracting FPFH features of the downsampled 3D point cloud containing instance information, and extracting FPFH features of the downsampled model point cloud;

[0073] The process of extracting FPFH features is to construct features based on the normal angle between a point and its neighboring points and the angle between the lines between the points. i Find the normal n i , determine the radius r and search for the neighborhood point p ij , taking each point as the origin, calculate the local coordinate value of each point. The formula is:

[0074] u=n i

[0075]

[0076] w=u×v

[0077] Calculate the SPFH feature of each point in the point cloud, that is, p i With p ij The angle relationship (α, φ, θ) is:

[0078] α=vn i

[0079]

[0080] θ=arctan(wn i ,un i )

[0081] The FPFH feature of the point is calculated by weighted statistics of the SPFH features of the k adjacent points in the neighborhood. The formula is:

[0082]

[0083] Step (3): Use the RANSAC random sampling consensus algorithm to filter out abnormal points from the three-dimensional point cloud containing instance information. First, find the nearest matching point, use the FPFH feature to find the nearest neighbor points in the point cloud set P and the model point cloud set Q as corresponding points, and rotate the point cloud set P through the rotation matrix R c and the translation vector t c The transformed point cloud set P i ′:

[0084] P i '=R c P i +t c

[0085] Where i = 1, 2, 3, 4, R c is the rotation matrix, t c is the translation vector.

[0086] Calculate the distance projection error between the point cloud set P' after the rigid body rotation and translation transformation of the three-dimensional point cloud containing instance information and the corresponding points on the model point cloud set Q. The corresponding points with an error value less than the set threshold are added to the sample internal point set I, and the matching score S is calculated. i , matching score S i It is the ratio of the total number of point clouds of the internal points to the model point cloud set Q, that is:

[0087] d ⊥ =d 2 (p i ',Hp i )

[0088] Among them, H is the spatial transformation matrix of the corresponding points, d is the distance between the corresponding points, and d ⊥ is the distance error threshold;

[0089] Set the number of loops N and the matching score threshold S min , if the matching score S i Less than S min Jump out of the loop, otherwise until the number of iterations reaches N; then the point cloud set after rough matching is P b ;

[0090] Step S1102: fine registration: perform fine matching through the ICP algorithm, and continuously optimize the point cloud set P through iteration. b The matching error between the point cloud set Q is minimized until the error is minimized. The specific process is:

[0091] Find the point cloud set P b Each point in the point cloud set Q is regarded as the corresponding point, and the rotation matrix R of the corresponding point cloud registration is obtained. fand the translation vector t f , combined with p bi The normal η i , whose objective optimization function F(R f ,t f )for:

[0092]

[0093] The singular value decomposition method is used to calculate the optimal transformation matrix and define two sets of point clouds P b ={p bi},Q={q i The centroid and de-centroid coordinates of} are:

[0094]

[0095] Among them, p bc ,q c is the centroid coordinate, is the centroid coordinate;

[0096] Define the transformation matrix G, Perform SVD decomposition on G to obtain orthogonal matrices A and B, and finally obtain the rotation matrix R at this time f and the translation vector t f :

[0097] R f =AB T

[0098] t f =p bc -R f q c ;

[0099] Until the maximum number of iterations N is reached max Or the minimum threshold ε, the iteration ends, and the optimal rigid body transformation matrix is ​​returned to complete the ICP precise matching process, realizing the fusion of the three-dimensional point cloud containing instance information and the model point cloud, that is, obtaining the three-dimensional point cloud used to generate the operation trajectory.

[0100] The beneficial effect of the present invention is that the image information of the surface to be sprayed is obtained by means of a visual sensor, and an accurate three-dimensional model is obtained through three-dimensional reconstruction. This method has a higher degree of flexibility and intelligence, can meet the task requirements of high precision and high efficiency, and is very suitable for customized and fine spraying tasks in modern spraying systems.

[0101] It is helpful to plan the spraying trajectory according to the accurate three-dimensional model, realize the automatic planning of the spraying path, and then control the spraying robot to perform precise spraying operations according to the spraying trajectory.

[0102] Further features and aspects of the present invention will be clearly described in the following description of specific embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is an overall flow chart of a three-dimensional reconstruction method that can be used for the internal structure of a cabin;

[0104] Figure 2 This is the architecture diagram of the Mask R-CNN model;

[0105] Figure 3 It is the structural diagram of the Swin-Transformer network;

[0106] Figure 4 It is the flow chart of Swin Transformer Block in Swin-Transformer network;

[0107] Figure 5 is the result of instance segmentation;

[0108] Figure 6 is the result of instance segmentation;

[0109] Figure 7 The feature point result is obtained by extracting the features of the preprocessed color image through the SIFT algorithm;

[0110] Figure 8 is the feature point matching result;

[0111] Fig. 9 This is the schematic diagram of the 3D-Gaussion-Splatting algorithm;

[0112] Fig.10 It is the reconstructed point cloud obtained after 3D-Gaussion-Splatting optimization;

[0113] Fig.11 It is a diagram of the model to the model point cloud of the T-beam and rib plate inside the cabin;

[0114] Fig.12 is a flow chart of the registration process;

[0115] Fig.13 It is the result of the T-beam after rough matching during the registration process between the 3D point cloud containing instance information and the model point cloud;

[0116] Fig.14 It is the result of the T-beam after precise matching during the registration process of the 3D point cloud containing instance information and the model point cloud. DETAILED DESCRIPTION

[0117] The present invention will be further described in detail below with reference to the accompanying drawings by way of specific embodiments.

[0118] The three-dimensional reconstruction system disclosed in the present invention is a high-precision three-dimensional reconstruction system based on RGB-D images.

[0119] refer to Figure 1 The 3D reconstruction method that can be used for the internal structure of a cabin mainly includes the following steps:

[0120] In the first step, an RGB-D camera is used to capture images of the interior of the cabin and obtain images from different perspectives.

[0121] The second step is to preprocess the RGB-D images collected by the RGB-D camera.

[0122] The third step is to perform instance segmentation on the preprocessed color image.

[0123] In order to distinguish and identify different components inside the cabin, the color image is segmented by instance, which can identify and segment different components inside the cabin, and label different components with different instance labels. Mask R-CNN is used as the overall framework, and Swin-Transformer is used as the backbone network to implement instance segmentation operations, achieving parallel object detection and semantic segmentation.

[0124] The Mask R-CNN model consists of four main parts: backbone network, RPN network, RoI Align network, and FCN network. Figure 2 The backbone network uses the Swin-Transformer network, which performs feature extraction. The overall flow chart is as follows: Figure 3 As shown in the figure, the size of the feature map will gradually decrease through the four stages, and the number of channels will increase, so that the receptive field will continue to expand; when the image is input into the network, the block partitioning module divides the image into several independent blocks, and converts the pixels of each block into a one-dimensional vector, and then combines these vectors into a matrix. The matrix goes through four stages, each of which includes a block merging module and a Swin Transformer module. Block merging can reduce the resolution of the feature map. In the first stage, the block needs to be linearly encoded and then passed to the Swin Transformer module. The encoding method only changes the number of channels of the feature without reducing the resolution. The Swin Transformer module (Swin Transformer Block) is the core module of the Swin-Transformer network, and its process is as follows: Figure 4As shown in the figure, the Swin Transformer modules all appear in pairs, because in W-MSA (Window Multi-head Self Attention), attention calculation is performed separately in each window, which can effectively reduce the amount of calculation, but lacks the information connection between windows. Therefore, the shift window multi-head self-attention layer feature (Shift Window Multi-head Self Attention, SW-MSA) will be used in the next Swin-Transformer module to achieve information exchange between windows through window offset. Normalization (Layer Normalization, LN) and residual connection are used in the connection process with the multi-layer perceptron (MLP) and the module.

[0125] The Mask R-CNN model is trained, and the Swin-Transformer network is trained using the COCO dataset. The COCO dataset mainly calibrates the positions of objects in images through precise segmentation. Each image is manually labeled with the corresponding object area and the corresponding category. This dataset has great advantages in target detection.

[0126] After training, generate the corresponding weight file. Use the weight file of the final round, change the types in the python file for prediction execution to the type names in the training set, change the paths involved to the corresponding paths, and add the pth file for generating image information to the code for subsequent point cloud generation.

[0127] The preprocessed color image is input into the trained Mask R-CNN model, and instance segmentation is performed on the simulated environment inside the cabin to obtain instance segmentation results. The Mask R-CNN model recognizes various components inside the cabin, such as Figure 5 , 6 T-beams and ribs are shown.

[0128] The fourth step is to calculate the initial pose of the RGB-D camera.

[0129] Step S401: multiple pre-processed color images at different viewing angles are {F1, F2...F n}, the SIFT algorithm is used to extract the features of the preprocessed color image to obtain feature points, such as Figure 7 As shown in the figure, the feature points of each preprocessed color image are obtained. Next, feature matching is performed by fast approximate nearest neighbor method. The feature matching process is to match the two images F i and F j Find the corresponding feature points, such as matching images F1 and F2 to obtain the corresponding feature points (such as Figure 8 As shown), images F1 and F3 are matched to obtain corresponding feature points, and the corresponding feature points are matched to the other two images in a similar manner until all images are matched in pairs.

[0130] The fast approximate nearest neighbor method is adopted, and its dimension-divided search method greatly improves the matching efficiency, so it is often used in high-dimensional feature spaces. When used in the scenario of the present invention, the matching efficiency can be greatly improved.

[0131] Step S402: Obtain the preliminary pose of the RGB-D camera using the PnP algorithm. First, the PnP algorithm is used to obtain the initial pose of the selected image F i and F j Calculate the transformation matrix (that is, the corresponding camera pose transformation) and transform it from image F i The collected corresponding feature points are placed in the point set P m In the image F j The collected corresponding feature points are placed in the point set Q m middle.

[0132] P m ={p1,p2.....p n}∈F i

[0133] Q m ={q1,q2.....q n}∈F j

[0134]

[0135] In the above expression, {p1,p2.....p n} is the image F i The characteristic points of {q1,q2.....q n} is the image F j The characteristic points of R and t are point sets P m The corresponding feature points in are transformed into point set Q m The rotation matrix and translation vector required.

[0136] Further nonlinear optimization is performed to solve the rotation matrix R and translation vector t by minimizing the error function:

[0137]

[0138] Then the preliminary pose of the RGB-D camera is obtained.

[0139] The fifth step is to obtain the preliminary reconstructed point cloud based on the obtained preliminary pose and the preprocessed depth image.

[0140] The TSDF (Truncated Signed Distance Function) algorithm is a method used to represent the surface of an object in three-dimensional space. For each voxel (volume pixel), the TSDF value represents the distance from the center of the voxel to the surface of the object, and the distance value is truncated within a certain range from the surface. The TSDF algorithm is used to fuse the obtained preliminary pose with the preprocessed depth image. The specific process is as follows:

[0141] Step S501, initialize a 3D voxel grid, each unit of the voxel grid stores the TSDF value D(x) and the weight W(x). Project each depth map pixel to the voxel grid in the 3D space, and for each depth map pixel (u, v, d), convert the pixel coordinates and depth value d into the 3D point coordinates p in the camera coordinate system through the camera intrinsic parameter K. cam :

[0142] P cam =d.(K -1 .[u,v,1] T )

[0143] Use the obtained preliminary camera pose, i.e., rotation matrix R and translation vector t, to transform the 3D point in the camera coordinate system to the 3D point P in the world coordinate system. w :

[0144] P w =RP cam +t

[0145] Step S502: for each three-dimensional point coordinate P w , calculate its distance φ(p w )=||p w -c||, if the distance is less than the cutoff distance δ, then the distance is mapped to the TSDF value D(p w ):

[0146]

[0147] The calculated TSDF value D(p w ) value is weighted averaged with the TSDF value D(x) of the existing voxel grid.

[0148]

[0149] Where W(x) is the existing weight, ω p is the weight of the depth image.

[0150] Repeat steps S501 and S502 until all depth image pixels are processed, so that the voxel grid stores the TSDF values ​​of all depth image fusions. Finally, by extracting the isosurface from the voxel grid, the vertices of the isosurface are extracted to obtain a preliminary reconstructed point cloud.

[0151] In the sixth step, the camera pose is iteratively optimized based on the color loss, depth loss, and instance information tracking loss.

[0152] After obtaining the preliminary camera pose, it needs to be optimized. The preliminary 3D reconstructed point cloud has been obtained based on the preliminary pose using the TSDF algorithm. The preliminary reconstructed point cloud is projected onto the 2D image plane at these specific viewing angles of the acquired image, and the rendered color image, depth value, and instance features of the rendered color image at these viewing angles can be obtained. These rendered outputs are compared with the true values ​​collected through RGB-D ("true values" refer to the preprocessed color images), and the camera pose is iteratively optimized through color loss, depth loss, and tracking loss of instance information.

[0153] The color loss Lc represents the weighted value of SSIM loss and L1 loss, and its formula is:

[0154] Lc=λL1(Ir,I gt )+(1-λ)[1-ssim(I r ,I gt )]

[0155] Among them, I r Indicates rendering of color images, I gt represents the true value, and λ is the weighting parameter.

[0156] Depth loss L d The formula is:

[0157]

[0158] in, is the rendering depth value, is the depth value of the true value (that is, the depth value of the preprocessed color image).

[0159] Total loss L total The formula is as follows:

[0160] L total =λ c L c +λ d L d +λ s L1(S render ,S head )

[0161] Among them, Srender represents the instance features of the rendered color image, S head represents the segmentation mask of the true value (i.e., the segmentation mask of the preprocessed color image), λ c , d , s Respectively represent the weight values ​​of color loss, depth loss, and instance loss.

[0162] Next, the gradient of the rotation matrix R and the translation vector t is calculated, and the pose is updated according to the learning rate η.

[0163]

[0164] After multiple iterations, the optimized camera pose is obtained, and the 4x4 pose transformation matrix T is used. opt express:

[0165]

[0166] Among them, R opt represents the optimized rotation matrix, t opt Represents the optimized translation vector.

[0167] In the seventh step, a dense point cloud is obtained based on the optimized camera pose and the preprocessed depth image.

[0168] The preliminary reconstructed point cloud is updated and optimized according to the optimized camera pose. The three-dimensional point coordinates p of each depth map pixel (u, v, d) in the camera coordinate system are known. cam , and the optimized camera pose is known, that is, the rotation matrix R opt and the translation vector t opt , so we can get the three-dimensional point P in the world coordinate system after optimizing the camera pose w ':

[0169] P′ w =R opt .P cam +t opt

[0170] For the three-dimensional point coordinates P w ', calculate its distance φ(p' w )=||p' w -c||, when the distance is less than the cutoff distance δ, the distance is mapped to the TSDF value D(p' w ):

[0171]

[0172] The calculated optimized TSDF value D(p' w) is weighted averaged with the TSDF value D(x) in the existing voxel grid.

[0173]

[0174] Where W(x) is the existing weight, ω p is the weight of the depth image.

[0175] Repeat the above steps until all depth map pixels are processed, so that the voxel grid stores all TSDF values ​​after depth image fusion under the optimized camera pose. By extracting the isosurface from the voxel grid, the vertices of the isosurface are extracted to obtain the optimized reconstructed point cloud, that is, the dense point cloud.

[0176] The eighth step is to optimize the dense point cloud through 3D-Gaussion-Splatting.

[0177] The point cloud data in the scene is represented by using Gaussian distribution, and these point cloud data are converted into Gaussian volumes during the rendering process. At the same time, each data point not only represents a specific spatial position, but also contains attributes related to the point (such as density, color, temperature, etc.). These data points are regarded as samples carrying attribute information. The attributes of each point are "splatted" into the entire data volume through its Gaussian weight. The attribute value of each point is multiplied by its Gaussian weight, and then the result is accumulated to the volume unit (voxels) it affects. The accumulation stage integrates the influence of all points into a single volume data set. For each voxel in the volume data set, the contributions of all sample points that affect the voxel are summed to obtain the final attribute value. The overall process of the 3D-Gaussion-Splatting algorithm is as follows Fig. 9 As shown in Figure 2. 3D-Gaussion-Splatting can make the point cloud in space more continuous and smooth, reduce noise, and improve reconstruction accuracy. The reconstructed point cloud obtained by 3D-Gaussion-Splatting optimization is shown in Figure 2. Fig.10 shown.

[0178] In the ninth step, the reconstructed point cloud obtained in the eighth step is mapped with the instance segmentation result obtained in the third step to obtain a three-dimensional point cloud containing instance information.

[0179] Multiple images from different perspectives captured by the RGB-D camera, preprocessed color image F n Each pixel (i, j) has corresponding color information and depth information. In the third step, the color image is segmented to obtain the instance segmentation mask M. i ∈{0,1} H×W and category information C i, classify each pixel (i, j), where M i (i,j)=1 means that this pixel belongs to a certain instance, M i (i, j) = 0 means that the pixel does not belong to the instance. The pixel position and category of each instance are determined by the mask and category information obtained after segmentation.

[0180] For the pixel point (i, j) belonging to the object in the segmentation mask, the corresponding depth information can be obtained. Using the method of obtaining the three-dimensional point in the camera coordinate system mentioned above, the pixel point in the image corresponding to the instance is converted into a three-dimensional point in the camera coordinate system in combination with the camera intrinsic parameter By performing instance segmentation on images from different perspectives, we can obtain multi-perspective segmentation masks and corresponding 3D points in the camera coordinate system. The 3D points in the camera coordinate system are converted into 3D points in the global coordinate system through the camera pose obtained above. Finally, a three-dimensional point cloud containing instance information is obtained.

[0181] By mapping instance labels with reconstructed point clouds, point clouds with instance information are obtained, providing a good basis for subsequent matching with standard components in the parts library.

[0182] Step 10: Create model point cloud.

[0183] Step S1001, use SolidWorks software to create a 3D model of various components inside the cabin, and the model can reflect the detailed size of the components. Fig.11 , where (a) shows the T-beam on top and the rib plate on the bottom, and (b) shows the model stored in STL format.

[0184] Step S1002: Use CloudCompare software to process the 3D model in STL format to obtain an accurate model point cloud (such as Fig.11 As shown in the middle figure (c), the upper part of Figure (c) is the model point cloud of the T-beam, and the lower part is the model point cloud of the rib plate.

[0185] In the eleventh step, the three-dimensional point cloud containing instance information is registered with the model point cloud to obtain a three-dimensional point cloud for generating the work trajectory.

[0186] Due to the limited scanning range of the camera and the existence of occlusion and other problems, it is difficult to obtain the complete point cloud data of all components in the cabin at one time. However, the model point cloud contains complete point cloud data. By aligning the actual point cloud and the model point cloud, more accurate information about the components in the cabin can be obtained, thereby improving the accuracy and quality of visual 3D reconstruction. Therefore, it is particularly important to align the component point cloud and the model point cloud.

[0187] refer to Fig.12 .

[0188] Step S1101, coarse registration. The RANSAC coarse matching algorithm based on FPFH features is used. Step (1): First, downsample the three-dimensional point cloud containing instance information and downsample the model point cloud to speed up the calculation and reduce the calculation complexity;

[0189] Step (2): Secondly, extract the FPFH features of the three-dimensional point cloud containing instance information after downsampling, and extract the FPFH features of the model point cloud after downsampling;

[0190] The process of extracting FPFH features is to construct features based on the normal angle between a point and its neighboring points and the angle between the lines between the points. i Find the normal n i , determine the radius r and search for the neighborhood point p ij , taking each point as the origin, calculate the local coordinate value of each point. The formula is:

[0191] u=n i

[0192]

[0193] w=u×v

[0194] Calculate the SPFH feature of each point in the point cloud, that is, p i With p ij The angle relationship (α, φ, θ) is as follows:

[0195] α=vn i

[0196]

[0197] θ=arctan(wn i ,un i )

[0198] The FPFH feature of the point is calculated by weighted statistics of the SPFH features of the k adjacent points in the neighborhood. The formula is:

[0199]

[0200] The local geometric features of the point cloud are described by FPFH features, which are robust to local shape changes and noise and help improve the accuracy of matching.

[0201] Step (3): Then, the RANSAC random sampling consensus algorithm is used to filter out abnormal points from the three-dimensional point cloud containing instance information to complete the rough matching process. The specific process is: first find the nearest matching point, and use the FPFH feature to find the nearest neighbor points in the point cloud set P and the point cloud set Q as corresponding points. For example, select data points {p1, p2, p3, p4}∈P from the three-dimensional point cloud containing instance information, and rotate the point cloud set P through the rotation matrix R c and the translation vector t c The transformed point cloud set P i ′, the formula is:

[0202] P i '=R c P i +t c

[0203] Where i = 1, 2, 3, 4, R c is the rotation matrix, t c is the translation vector.

[0204] Select the corresponding data points {q1, q2, q3, q4} from the model point cloud Q. Calculate the distance projection error between the point cloud set P' after the three-dimensional point cloud containing instance information is transformed by rigid body rotation and translation and the corresponding points on the model point cloud set Q. The corresponding points with error values ​​less than the set threshold are added to the sample internal point set I, and the matching score S is calculated. i , matching score S i It is the ratio of the internal points to the total number of point clouds in the point cloud set Q, that is:

[0205] d ⊥ =d 2 (p i ',Hp i )

[0206] In the above formula, H is the spatial transformation matrix of the corresponding points, d is the distance between the corresponding points, and d ⊥ is the distance error threshold.

[0207] Set the number of loops N and the matching score threshold S min , if the matching score S i Less than S min Jump out of the loop, otherwise repeat the above steps until the number of iterations is N. Count the number of sample inliers under different rigid body transformation models, and the model with the largest number of inliers is taken as the best data model. The corresponding rigid body transformation matrix is ​​used for the point cloud rough matching operation, and the point cloud set after rough matching is P b .

[0208] Fig.13 Yes, the rough matching result of the T-beam.

[0209] Step S1102: Fine registration. Fine matching is performed using the ICP algorithm to improve matching accuracy. The processing process is to continuously optimize the point cloud set P in an iterative manner. b The matching error between the point cloud set Q is minimized. The specific process is:

[0210] Find the point cloud set P b Each point in the point cloud set Q is regarded as a corresponding point. To obtain the rotation matrix R for the corresponding point cloud registration f and the translation vector t f , combined with p bi The normal η i , whose objective optimization function F(R f ,t f )for:

[0211]

[0212] The singular value decomposition (SVD) method is used to calculate the optimal transformation matrix and define two sets of point clouds P b ={p bi},Q={q i The centroid and de-centroid coordinates of} are:

[0213]

[0214]

[0215] In the formula, p bc ,q c is the centroid coordinate, is the centroid coordinate.

[0216] Define the transformation matrix G, Perform SVD decomposition on G to obtain orthogonal matrices A and B, and finally obtain the rotation matrix R at this time f and the translation vector t f :

[0217] R f =AB T

[0218] t f =p bc -R f q c .

[0219] Repeat the above steps until the maximum number of iterations N is reached. max Or the minimum threshold ε, the iteration ends, returns the optimal rigid body transformation matrix, completes the ICP precise matching process, realizes the fusion of the three-dimensional point cloud containing instance information and the model point cloud, and obtains the three-dimensional point cloud used to generate the operation trajectory. The minimum threshold ε is: ε=Fk (R f ,t f )-F k-1 (R f ,t f ). Fig.14 It is the precise matching result of the T-beam.

[0220] It can be seen that, in view of the complex structure and weak texture of the cabin interior, the above method can efficiently perform high-precision 3D reconstruction of the cabin interior components and obtain accurate high-precision 3D point clouds. It has a higher degree of flexibility and intelligence, can meet the task requirements of high precision and high efficiency, and is very suitable for customized and fine spraying tasks in modern spraying systems.

[0221] Furthermore, the spraying trajectory is planned according to the three-dimensional point cloud, and the spraying robot is controlled to perform precise automatic spraying operations according to the spraying trajectory.

[0222] The above description is only for the preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes.

Claims

1. A three-dimensional reconstruction method that can be used for the internal structure of a cabin, characterized in that: The following steps are involved: The first step is to use an RGB-D camera to capture images of the object to be reconstructed and obtain images from different perspectives; The second step is to preprocess the RGB-D images collected by the RGB-D camera; The third step is to perform instance segmentation on the preprocessed color image; Instance segmentation is performed using the Mask R-CNN model, and the backbone network of the Mask R-CNN model uses the Swin-Transformer network. The Swin-Transformer network is trained using the COCO dataset. The preprocessed color image is input into the trained Mask R-CNN model to perform instance segmentation on the object to be reconstructed and obtain the instance segmentation result; The fourth step is to calculate the initial pose of the RGB-D camera; Step S401, the preprocessed color image {F1, F2...F n }Use SIFT algorithm to extract features and get feature points. Next, use fast approximate nearest neighbor method to compare the two images F i and F j Perform feature matching to find corresponding feature points; Step S402, obtaining the preliminary pose of the RGB-D camera through the PnP algorithm; first, the PnP algorithm is used to select the image F i and F j The transformation matrix is ​​calculated and converted from image F i The collected corresponding feature points are placed in the point set P m In the image F j The collected corresponding feature points are placed in the point set Q m middle, P m ={p1,p2.....p n }∈F i Q m ={q1,q2.....q n }∈F j Among them, {p1,p2.....p n } is the image F i The characteristic points of {q1,q2.....q n } is the image F j The characteristic points of R and t are point sets P m The corresponding feature points in are transformed into point set Q m The rotation matrix and translation vector required when Further nonlinear optimization is performed to solve the rotation matrix R and translation vector t by minimizing the error function: Then the preliminary pose of the RGB-D camera is obtained; Step 5: Obtain a preliminary reconstructed point cloud based on the obtained preliminary pose and the preprocessed depth image; Step 6: Optimize the initial camera pose; Step 7: Obtain a dense point cloud based on the optimized camera pose and preprocessed depth image; Step 8: Optimize the dense point cloud by 3D-Gaussion-Splatting to obtain the reconstructed point cloud; In the ninth step, the reconstructed point cloud obtained in the eighth step is mapped with the instance segmentation result obtained in the third step to obtain a three-dimensional point cloud containing instance information; Step 10: Create model point cloud; Step S1001, using SolidWorks software to build a three-dimensional model of the object to be reconstructed; Step S1002, using CloudCompare software to process the three-dimensional model in STL format, thereby obtaining an accurate model point cloud; In the eleventh step, the three-dimensional point cloud containing instance information is registered with the model point cloud to obtain a three-dimensional point cloud for generating the work trajectory.

2. The three-dimensional reconstruction method for the internal structure of a cabin according to claim 1, characterized in that: The fifth step adopts the following process: Step S501, initialize a 3D voxel grid, each unit of the voxel grid stores the TSDF value D(x) and the weight W(x), project each depth map pixel to the voxel grid in the 3D space, and for each depth map pixel (u, v, d), convert the pixel coordinates and depth value d into the 3D point coordinates p in the camera coordinate system through the camera intrinsic parameter K cam : P cam =d.(K -1 .[u,v,1] T ) Use the obtained preliminary camera pose, i.e., rotation matrix R and translation vector t, to transform the 3D point in the camera coordinate system to the 3D point P in the world coordinate system. w : P w =R.P cam +t Step S502: for each three-dimensional point coordinate P w , calculate its distance φ(p w )=||p w -c||, if the distance is less than the cutoff distance δ, then the distance is mapped to the TSDF value D(p w ): The calculated TSDF value D(p w ) value and the TSDF value D(x) of the existing voxel grid are weighted averaged: Where W(x) is the existing weight, ω p is the weight of the depth image; Repeat steps S501 and S502 until all depth map pixels are processed, so that the voxel grid stores the TSDF values ​​after all depth images are fused; finally, extract the isosurface from the voxel grid and extract the vertices of the isosurface to obtain a preliminary reconstructed point cloud. The sixth step adopts the following process: The preliminary reconstructed point cloud is projected onto the 2D image plane to obtain the rendered color image, depth value and instance features of the rendered color image. The rendered color image, depth value and instance features of the rendered color image are compared with the true value collected by RGB-D. The true value refers to the preprocessed color image. The camera pose is iteratively optimized through color loss, depth loss and instance information tracking loss. The color loss Lc represents the weighted value of SSIM loss and L1 loss, and its formula is: Lc=λL1(I r ,I gt )+(1-λ)[1-ssim(I r ,I gt )] Among them, I r Indicates rendering of color images, I gt represents the true value, λ is the weighting parameter; Depth loss L d The formula is: in, is the rendering depth value, is the depth value of the real value; Total loss L total The formula is as follows: L total λ c L c +λ d L d +λ s L1(S render ,S head ) Among them, S render represents the instance feature of the rendered color image, S head The segmentation mask representing the true value, λ c , d , s Respectively represent the weight values ​​of color loss, depth loss, and instance loss; Next, the gradient of the rotation matrix R and the translation vector t is calculated, and the pose is updated according to the learning rate η. After multiple iterations, the optimized camera pose is obtained, and the 4x4 pose transformation matrix T is used. opt express: Among them, R opt represents the optimized rotation matrix, t opt Represents the optimized translation vector.

3. The three-dimensional reconstruction method for the internal structure of a cabin according to claim 1 or 2, characterized in that: The seventh step adopts the following process: the preliminary reconstructed point cloud is updated and optimized according to the optimized camera pose, and the three-dimensional point coordinates p of each depth map pixel (u, v, d) in the camera coordinate system are cam , the optimized camera pose is the rotation matrix R opt and the translation vector t opt , and then get the three-dimensional point P in the world coordinate system after optimizing the camera pose w ': P' w =R opt .P cam +t opt For the three-dimensional point coordinates P w ', calculate its distance φ(p' w )=||p' w -c||, when the distance is less than the cutoff distance δ, the distance is mapped to the TSDF value D(p' w ): The calculated optimized TSDF value D(p' w ) is weighted averaged with the TSDF value D(x) in the existing voxel grid, Where W(x) is the existing weight, ω p is the weight of the depth image; Until all depth map pixels are processed, the voxel grid stores all TSDF values ​​after depth image fusion under the optimized camera pose. By extracting the isosurface from the voxel grid and extracting the vertices of the isosurface, the optimized reconstructed point cloud is obtained, that is, the dense point cloud is obtained.

4. The three-dimensional reconstruction method for the internal structure of a cabin according to claim 1 or 2, characterized in that: The ninth step is performed using the following process: Preprocessed color image F n Each pixel point (i, j) has corresponding color information and depth information, and the instance segmentation mask M is obtained by performing instance segmentation on the color image. i ∈{0,1} H×W and category information C i , classify each pixel (i, j), where M i (i,j)=1 means that this pixel belongs to a certain instance, M i (i, j) = 0 means that the pixel does not belong to the instance. The pixel position and category of each instance are determined by the mask and category information obtained after segmentation. For the pixel point (i, j) belonging to the object in the segmentation mask, the corresponding depth information is obtained, and the pixel point is converted into a three-dimensional point in the camera coordinate system in combination with the camera intrinsic parameter Then, we can obtain the multi-view segmentation mask and the corresponding 3D point in the camera coordinate system, and then transform the 3D point in the camera coordinate system into the 3D point in the global coordinate system. Finally, a three-dimensional point cloud containing instance information is obtained.

5. The three-dimensional reconstruction method for the internal structure of a cabin according to claim 1, characterized in that: The eleventh step is performed using the following process: Step S1101, coarse registration; using the RANSAC coarse matching algorithm based on FPFH features, step (1): downsampling the three-dimensional point cloud containing instance information and downsampling the model point cloud; Step (2): extracting FPFH features of the three-dimensional point cloud containing instance information after downsampling processing, and extracting FPFH features of the model point cloud after downsampling processing; The process of extracting FPFH features is to construct features based on the normal angle between a point and its neighboring points and the angle between the lines between the points. i Find the normal n i , determine the radius r and search for the neighborhood point p ij , taking each point as the origin, calculate the local coordinate value of each point. The formula is: Calculate the SPFH feature of each point in the point cloud, that is, p i With p ij The angle relationship (α, φ, θ) is: θ=arctan(w.n i ,u.n i ) The FPFH feature of the point is calculated by weighted statistics of the SPFH features of the k adjacent points in the neighborhood. The formula is: Step (3): Use the RANSAC random sampling consensus algorithm to filter out abnormal points from the three-dimensional point cloud containing instance information. First, find the nearest matching point, use the FPFH feature to find the nearest neighbor points in the point cloud set P and the model point cloud set Q as corresponding points, and rotate the point cloud set P through the rotation matrix R c and the translation vector t c The transformed point cloud set P i ': P i '=R c P i +t c Where i = 1, 2, 3, 4, R c is the rotation matrix, t c is the translation vector. Calculate the distance projection error between the point cloud set P' after the rigid body rotation and translation transformation of the three-dimensional point cloud containing instance information and the corresponding points on the model point cloud set Q. The corresponding points with an error value less than the set threshold are added to the sample internal point set I, and the matching score S is calculated. i , matching score S i It is the ratio of the total number of point clouds of the internal points to the model point cloud set Q, that is: d ⊥ =d 2 (p i ',Hp i ) Among them, H is the spatial transformation matrix of the corresponding points, d is the distance between the corresponding points, and d ⊥ is the distance error threshold; Set the number of loops N and the matching score threshold S min , if the matching score S i Less than S min Jump out of the loop, otherwise until the number of iterations reaches N; then the point cloud set after rough matching is P b ; Step S1102: fine registration: perform fine matching through the ICP algorithm, and continuously optimize the point cloud set P through iteration. b The matching error between the point cloud set Q is minimized until the error is minimized. The specific process is: Find the point cloud set P b Each point in the point cloud set Q is regarded as the corresponding point, and the rotation matrix R of the corresponding point cloud registration is obtained. f and the translation vector t f , combined with p bi The normal η i , whose objective optimization function F(R f ,t f )for: The singular value decomposition method is used to calculate the optimal transformation matrix and define two sets of point clouds P b ={p bi },Q={q i The centroid and de-centroid coordinates of} are: Among them, p bc ,q c is the centroid coordinate, is the centroid coordinate; Defining the transformation matrix Perform SVD decomposition on G to obtain orthogonal matrices A and B, and finally obtain the rotation matrix R at this time f and the translation vector t f : R f =AB T t f =p bc -R f q c ; Until the maximum number of iterations N is reached max Or the minimum threshold ε, the iteration ends, and the optimal rigid body transformation matrix is ​​returned to complete the ICP precise matching process, realizing the fusion of the three-dimensional point cloud containing instance information and the model point cloud, that is, obtaining the three-dimensional point cloud used to generate the operation trajectory.

Images