3D line reconstruction registration method based on improved plucker net

Abstract

The application discloses a 3D line reconstruction registration method based on an improved PluckerNet, which is a precise registration method for 3D line reconstruction based on an improved PluckerNet network. A data-driven deep learning network is used to solve the matching association problem between the two, which is an end-to-end method that avoids the defects of the traditional ICL method that requires setting the initial position. The network directly obtains a plurality of matching pairs with a higher matching probability in the two reconstructions. Unlike previous matching networks, the improved feature extraction local geometry module is used to extract initial features, and the matching existence discrimination module is used to guide the network to process noisy lines, so that the network obtains more competitive performance and improves the network's ability to obtain correct matching pairs between partially overlapping 3D line reconstructions. In two partially overlapping 3D line reconstructions mixed with noise data, as many correct matching pairs as possible are obtained to realize the precise registration of 3D line reconstruction.

Description

Technical Field

[0001] This invention relates to the technical field of 3D scene reconstruction and robot localization, and in particular to a 3D line reconstruction and registration method based on an improved PlückerNet. Background Technology

[0002] In applications such as 3D line-based odometry and scene reconstruction, matching and associating 3D lines is crucial. However, in real-world scenarios, changes in viewpoint and position lead to inconsistencies in the 3D line reconstructions acquired during each data acquisition (scene perception), resulting in a large amount of non-overlapping data. This non-overlapping data directly becomes noise data that interferes with matching the overlapping data. The presence of this noise data makes establishing accurate matching relationships between two partially overlapping 3D line reconstructions difficult. Establishing a large number of incorrect matching relationships can lead to odometry positioning errors, scene reconstruction failures, and other consequences.

[0003] Previous methods primarily used traditional ICL (Integrated Line Matching) for 3D line matching. This approach requires extensive iterative computation and is sensitive to initial pose settings. Furthermore, existing 3D line matching networks do not adequately address the unavoidable noise data within 3D line sets. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and propose a 3D line reconstruction registration method based on an improved PlückerNet. This method uses a data-driven deep learning network to solve the matching and association problem between two lines. This end-to-end approach avoids the deficiency of traditional ICL methods, which require setting initial positions, and directly obtains several matching pairs with high matching probabilities from the two reconstructions through the network. Furthermore, unlike previous matching networks, this invention uses an improved feature extraction local geometry module to extract initial features, and utilizes a matching existence discrimination module to guide the network in processing noisy lines. This results in more competitive performance and improves the network's ability to obtain correct matching pairs between partially overlapping 3D line reconstructions. It obtains as many correct matching pairs as possible in two partially overlapping 3D line reconstructions mixed with noisy data, and uses these matching pairs to achieve accurate registration of the 3D line reconstructions.

[0005] To achieve the above objectives, the technical solution provided by this invention is as follows: a 3D line reconstruction and registration method based on an improved PlückerNet. This method achieves accurate registration of 3D line reconstruction based on an improved PlückerNet network. The improved PlückerNet network improves the feature extraction local geometry module of the traditional PlückerNet network, adds a matching existence discrimination module, and improves the loss function used by the traditional PlückerNet network. Specifically, the improvement to the feature extraction local geometry module involves changing the two-branch feature extraction structure to a three-branch structure. By adding processing branches, more local geometric information of the lines is retained, improving the network's feature extraction capability. The newly added matching existence discrimination module is used to jointly estimate the global features obtained from the feature extraction module of the traditional PlückerNet network to obtain the existence probability vector of all lines in the source and target line reconstructions, improving the network's ability to distinguish outliers and real matching data, thereby enhancing the network's feature capture capability. The improvement to the loss function involves adding a new matching existence loss function to supervise network training, which, together with the matching existence discrimination module, improves the network's ability to handle noisy data.

[0006] The specific implementation of the 3D line reconstruction and registration method includes the following steps:

[0007] 1) Obtain the source 3D line reconstruction and target 3D line reconstruction data to be registered;

[0008] 2) The trained improved PlückerNet network is used to process the 3D line reconstruction data to be registered as follows:

[0009] The source 3D line reconstruction and target 3D line reconstruction data are input into the three-branch local geometry module in the feature extraction module to obtain the initial features F corresponding to the two reconstructions. source F target ;

[0010] The initial features F are processed by the global attention module in the feature extraction module. source F target Attention feature fusion is performed to obtain the enhanced global feature F′. source F′ target ;

[0011] The global feature F′ corresponding to the reconstruction of the source 3D lines is calculated by the matching existence discrimination module. source Score for the existence of matching lines in the middle source And the global feature F′ corresponding to the reconstruction of the target 3D lines target Score for the existence of matching lines in the middle targetAt the same time, using F′ source F′ target Calculate the similarity matrix Sim source Sim target Match the existence score. source and similarity matrix Sim source Multiplying them together yields the matching existence matrix E. source Then E source Row-wise pooling and channel concatenation are then fed into a multilayer perceptron block for processing to obtain the matching existence probability vector V. source Similarly, the existence score will be matched. target and similarity matrix Sim target Multiplication yields the matching existence matrix E target Then E target Row-wise pooling and channel concatenation are then fed into a multilayer perceptron block for processing to obtain the matching existence probability vector V. target ;

[0012] 3) Transfer global features F′ source F′ target And the existence probability vector V of the matching source V target The input is fed into the matching post-processing module of the improved PlückerNet network to estimate the relative pose of the final registration between the source 3D line reconstruction and the target 3D line reconstruction. Finally, the estimated relative pose is used to align the two 3D line reconstructions to achieve accurate registration.

[0013] Furthermore, step 1) includes the following steps:

[0014] 1.1) 2D line acquisition: For each frame of RGB image data in the public dataset, the fast line detection algorithm in the OpenCV library is used to obtain the 2D lines in each frame of RGB image;

[0015] 1.2) 3D line acquisition: For each 2D line detected from the RGB image data, a 3D line is fitted using a depth image that corresponds one-to-one with the RGB image; the depth image contains the depth information of each 2D line, thereby obtaining the 3D information corresponding to the line.

[0016] 1.3) Plücker Line Acquisition: Following steps 1.1) and 1.2), 3D lines describing the real-world scene are reconstructed from each frame of the RGB and depth images. Each 3D line in the 3D line reconstruction is transformed into the Plücker coordinate system to obtain the Plücker line, which serves as the input data for the improved PlückerNet network. For a given 3D line l = (p s ,p e ), where ps p e These represent the start and end points of the 3D line, respectively. The start point p... s =(x s ,y s ,z s End point p e =(x e ,y e ,z e ), where x s y s z s and x e y e z e These are the X, Y, and Z coordinates of the start and end points, respectively. Transforming the 3D line l into the Plücker coordinate system yields a six-dimensional vector l = (d, m), where d represents the direction vector of the 3D line, and m represents the torque vector of the 3D line. The direction vector d and the torque vector m are calculated using the following formulas:

[0017]

[0018]

[0019] In the formula, ||||2 represents the cross product of vectors, and ||||2 represents the L2 norm of the vector;

[0020] The source 3D line reconstruction to be registered is represented as follows:

[0021]

[0022] The target 3D line reconstruction is represented as:

[0023]

[0024] In the formula, Represents the reconstruction of source 3D lines, by N s A collection of 3D lines, l source_i d represents the i-th line in the source reconstruction. i and m i These are the corresponding direction vector and torque vector; Represents the reconstruction of the target 3D lines, by N t A collection of 3D lines, l target_j Denotes the j-th line in the target reconstruction, d j and m j These are the corresponding direction vector and moment vector; each 3D line in the two reconstructions is transformed into the Plücker coordinate system, consisting of a three-dimensional direction vector d and a three-dimensional moment vector m.

[0025] Furthermore, the feature extraction local geometry module uses the direction vector d, the moment vector m, and the overall vector l formed by these two as inputs to three branches to extract the initial feature F. source F target In each subspace of d and m, define the geometric nearest neighbor and construct linear KNNs respectively. Calculate the K geometric nearest neighbors in the Euclidean space based on Euclidean distance. Concatenate the residuals of the K neighbors with their own vectors and their own vectors to construct the corresponding local features. The specific calculation process is as follows:

[0026] F d =avg(mlp(concat) n (Neib d -d,d)))

[0027] F m =avg(mlp(concat) n (Neib m -m,m)))

[0028] In the formula, Neib d Neib m Let K be the geometric nearest neighbors of d and m respectively, and concat n This indicates a concatenation operation based on the neighborhood dimension, where MLP represents a multilayer perceptron block, AVG represents an average pooling layer, and F... d and F m These are the initial features corresponding to the direction vector and the moment vector, respectively;

[0029] The overall vector l is input into the third branch, which connects the direction vector d and the moment vector m, preserving the overall information of the line and extracting the features of their combined effect. The calculation process is as follows:

[0030] F l =ReLU(BatchNorm1d(Conv1d(l)))

[0031] In the formula, Conv1d represents a one-dimensional convolutional layer, BatchNorm1d represents a one-dimensional batch normalization layer, ReLU represents a non-linear activation function, and F... l This represents the initial features corresponding to the overall vector;

[0032] Finally, the outputs of the three branches are concatenated along the channel dimension, and then further processed by MLP. The calculation process is as follows:

[0033] F = mlp(concat) c (F d ,F m ,F l ))

[0034] In the formula, concat c This indicates a concatenation operation along the channel dimension. `mlp` represents a multilayer perceptron block, and `F` represents the initial features obtained from the local geometry module of feature extraction. Since the source 3D line reconstruction and the target 3D line reconstruction use the same processing procedure, `F` is F0. source With F target A unified representation.

[0035] Furthermore, the matching existence discrimination module will process the global enhanced feature F′ obtained after the two-stage feature extraction module. source F′ target After processing, the matching existence probability vector V of the source 3D line reconstruction is obtained. source The matching existence probability vector V corresponding to the target 3D line reconstruction target First, use feature F′ source and F′ target To calculate the matching existence score source and Score target The specific calculation process is as follows:

[0036] Score source =softmax(mlp(concat) c (max(F′ target ),mean(F′ target ),F′ source )))

[0037] Score target =softmax(mlp(concat) c (max(F′ source ),mean(F′ source ),F′ target )))

[0038] In the formula, softmax represents the normalization exponential function, and max and mean represent the operations of finding the maximum and minimum values ​​and the mean of the features in the first dimension, respectively.

[0039] Calculate the similarity matrix and similarity matrix N represents the set of real numbers. s and N t These represent the number of rows and columns of the matrix, and also the number of 3D lines in the corresponding reconstruction. The similarity between two lines is measured using the normalized dot product of the global features of the two reconstructed 3D lines. The specific calculation process is as follows:

[0040]

[0041]

[0042] In the formula, and F′ represents the corresponding element in the i-th row and j-th column of the corresponding similarity matrix, respectively. source_i F′ represents the global feature with index i in the source 3D line reconstruction. source_j F′ represents the global feature with index j in the source 3D line reconstruction. target_i F′ represents the global feature at index i in the reconstruction of the target 3D line. target_j represents the global feature with index j in the target 3D line reconstruction, · represents the vector dot product, × represents the numerical multiplication, and ||||2 represents the vector's L2 norm;

[0043] Multiplying the matching existence score and the similarity matrix yields the matching existence matrices corresponding to the 3D lines in the two reconstructions. Maximizing and averaging each row of these two matching existence matrices, concatenating them along the channel dimension, and then processing them using MLP yields the existence probability vector of the source reconstruction. The existence probability vector of the reconstructed target The specific calculation process is as follows:

[0044]

[0045]

[0046] In the formula, * denotes the maximum matrix multiplication. raw and mean raw These represent operations to find the maximum value and the mean value in each row of the matrix, respectively.

[0047] Furthermore, the post-matching processing module processes the input global features F′. source F′ target And the existence probability vector V of the matching source V target Obtain the matching probability matrix P; global features F′ source The feature vector F′ corresponding to the 3D line with index i in the middle. source_i and global features F′ target F′ with index j in the middle target_j Use L2 Euclidean distance to construct a distance matrix M between feature vectors, where M represents the values ​​of the feature vectors. ij Indicates at F′ source_i and F target_j The cost of establishing a match between them is calculated as follows:

[0048] M ij=||F′ source_i -F′ target_j ||2

[0049] To handle the probability of pairwise matching globally, the uncertainty of matching is described as an optimal transport problem, which is solved by solving the matching probability matrix P. The calculation process is as follows:

[0050]

[0051] In the formula, μ represents the parameter variable, log is the logarithmic function, and M... ij P represents the i-th row and j-th column of the distance matrix M. ij To match the i-th row and j-th column of the probability matrix P, argmin is the minimum function, and U(r,s) is the transmission polyhedron connecting the two prior probability vectors r and s, given as follows:

[0052]

[0053] In the formula, and These represent sizes N respectively. s and N t A singular vector, r and s, are prior probability vectors, representing the probability that a 3D line has a valid match and is not an outlier; the corresponding match existence probability vector V is reconstructed from the source 3D line using the output of the match existence discrimination module. source The matching existence probability vector V corresponding to the target 3D line reconstruction target Initialize r and s respectively; the matching probability matrix P obtained by the above formula contains the matching probability relationship between the source 3D line reconstruction and the target 3D line reconstruction. The values ​​in the matching probability matrix P represent the matching weights. The larger the value, the greater the probability of a match between the two 3D lines.

[0054] From the matching probability matrix P, select the top K′ pairs of potential 3D line matches with the highest weight scores. Use the RANSAC method to estimate the relative pose between the source and target 3D line reconstructions in the predicted top K′ pairs of potential line matches. Finally, use the estimated relative pose to align the two 3D line reconstructions, achieving accurate registration. The RANSAC method is implemented by cyclically and randomly selecting two matching pairs from the K′ pairs of potential matches. and in These are the two selected source 3D lines. These are two selected target 3D lines. These two matching pairs are used to calculate the relative pose required for registration. Since each 3D line consists of a direction vector d and a vector m, the specific calculation process is as follows:

[0055]

[0056] In the formula, R represents the rotation matrix, t represents the translation vector, t^ represents the antisymmetric matrix corresponding to the translation vector, and d source Let d be the direction vector of the source 3D lines in the two selected matching pairs. target To select the direction vector of the target 3D line, m source For the moment vectors of the source 3D lines in the two selected matching pairs, m target To select the moment vector of the target 3D line; the rotation matrix R and translation vector t calculated by two matching pairs can register and align the 3D lines in the source 3D line reconstruction with those in the target 3D line reconstruction. The alignment quality is defined by the fractional function S, and its calculation formula is:

[0057]

[0058] In the formula, T′ is the transformation matrix consisting of the rotation matrix R and the translation vector t, which transforms the source 3D lines l source Transform to target 3D lines target The system identifies nearby locations and measures the Euclidean distance between them. If the Euclidean distance is less than a predefined threshold, the corresponding 3D lines are considered internal pairs. The number of internal pairs is counted in each iteration, and after multiple iterations of the RANSAC method, the iteration with the most internal pairs is selected. The rotation matrix R obtained in this iteration is then used to determine the internal pair. best Translation vector t best As the final relative pose result, it enables precise registration between the source 3D line reconstruction and the target 3D line reconstruction.

[0059] Furthermore, in order to optimize the parameters of the feature extraction module and the matching existence discrimination module, and to make the matching probability matrix P approximate the true matching matrix P gt By using two loss functions L c and L e The network is trained, where the loss function L c The calculation is as follows:

[0060]

[0061] In the formula, the true matching matrix P gt This represents the truth value of the matching relationship. A value of 1 at the corresponding position in the matrix indicates a correct match, while a value of 0 indicates no match. By optimizing L... c This encourages the network to maximize the probability of correct matches while minimizing the probability of incorrect matches; this helps improve the network's ability to identify correct matches; another newly added loss function Le This is an improvement on the original loss function. A new matching existence loss function is added to supervise network training. Combined with the matching existence discrimination module, this enhances the network's ability to handle noisy data. Specifically, this is reflected in the network's output source 3D line reconstruction matching existence probability vector V. source And the matching existence probability vector V in the target 3D line reconstruction target The cross-entropy loss between the true matching existence vector and the actual matching vector is calculated as follows:

[0062]

[0063] In the formula, CrossEntropy represents the cross-entropy loss function. and These are the corresponding truth-valued matching existence vectors; to achieve accurate matching pair estimation, the two losses mentioned above are combined, and all parameters are trained jointly to optimize the entire objective L. total The specific calculation process is as follows:

[0064] L total =βL c +(1-β)L e

[0065] In the formula, β is the weight that balances the two losses.

[0066] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0067] 1. This invention improves the feature extraction capability of the network by changing the two-branch feature extraction structure in the local geometry module of the PlückerNet network to a three-branch feature extraction structure. By adding processing branches, more local geometric information of the lines is retained.

[0068] 2. Based on the PlückerNet network structure, a new matching existence discrimination module is added to jointly estimate the initial feature vectors obtained by the feature extraction module to obtain the matching existence probability vectors of all lines in the source line set and the target line set. This improves the network's ability to distinguish between outliers and real matching data, thereby enhancing the network's feature capture ability and explicitly guiding the network to process noisy data.

[0069] 3. In the process of improving the training of the PlückerNet network, a new matching existence loss function is added to supervise the network training. Together with the matching existence discrimination module, it enhances the network's ability to obtain the correct matching pair from two reconstructions of noisy data.

[0070] 4. By selecting the Top K potential matching pairs with different probability values ​​from the matching probability matrix predicted by the network, and evaluating the matching results, the improved PlückerNet network proposed in this invention has more correct matching pairs than the original PlückerNet network at each selection ratio. Furthermore, the improved PlückerNet network uses matching pairs to estimate the rotation and translation errors used for final registration, resulting in better 3D line reconstruction registration results. Attached Figure Description

[0071] Figure 1 A general framework diagram for improving the PlückerNet network.

[0072] Figure 2 The following is a scene example illustration demonstrating the 3D line reconstruction and registration of this invention.

[0073] Figure 3 The diagram shows the structure of the feature extraction local geometry module designed for this invention. In the diagram, KNN represents the geometric nearest neighbor operation, Conv1d represents a one-dimensional convolutional layer, BatchNorm1d represents a one-dimensional batch normalization layer, MLP represents a multilayer perceptron block, Avg represents an average pooling layer, and Concat represents a concatenation operation.

[0074] Figure 4 The diagram shows the structure of the matching existence discrimination module designed for this invention. In the diagram, MEAN represents the mean operation, MAX represents the maximum / minimum operation, Concat represents the concatenation operation, MLP represents the multilayer perceptron block, SOFTMAX represents the normalized exponential function, and Mul represents matrix multiplication. Detailed Implementation

[0075] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0076] This embodiment provides a 3D line reconstruction registration method based on an improved PlückerNet, such as... Figure 1As shown, this method achieves accurate registration for 3D line reconstruction based on an improved PlückerNet network. This improved PlückerNet network enhances the feature extraction local geometry module of the traditional PlückerNet network, adds a matching existence discrimination module, and improves the loss function used in the traditional PlückerNet network. The improvement to the feature extraction local geometry module involves changing the two-branch feature extraction structure to a three-branch structure. By adding processing branches, more local geometric information of the lines is retained, improving the network's feature extraction capability. The newly added matching existence discrimination module is used to jointly estimate the global features obtained from the traditional PlückerNet network's feature extraction module to obtain the existence probability vector of all lines in the source and target line reconstructions. This improves the network's ability to distinguish outliers and real matching data, thereby enhancing the network's feature capture capability. The improvement to the loss function involves adding a new matching existence loss function to supervise network training, working in conjunction with the matching existence discrimination module to improve the network's ability to handle noisy data. The method includes the following steps:

[0077] 1) Detect 2D and 3D lines from publicly available real-world RGB and depth images, and obtain source 3D line reconstructions and target 3D line reconstructions from two adjacent frames, such as... Figure 2 As shown. Each 3D line in the source and target 3D line reconstructions is transformed to the Plücker coordinate system, converting the 3D line representation composed of the start and end points into a 3D line representation composed of the direction vector d and the moment vector m. The specific calculation formula is as follows:

[0078]

[0079]

[0080] Where, p s p represents the starting and ending points of a 3D line. e Indicates the end point of a 3D line. represents the cross product of vectors, and ||||2 represents the L2 norm of the vector.

[0081] 2) The pre-trained improved PlückerNet network is used to perform the following processing on the 3D line reconstruction to be registered:

[0082] The source 3D line reconstruction and target 3D line reconstruction data are input into the three-branch local geometry module in the feature extraction module to obtain the initial features F corresponding to the two reconstructions. source F target ,like Figure 3As shown. Geometric nearest neighbors are defined in each subspace of d and m, and linear KNNs are constructed respectively. K geometric nearest neighbors in the Euclidean space are obtained based on Euclidean distance. The residuals of the K neighbors and their own vectors, along with the K neighbors' own vectors, are concatenated to construct the corresponding local features. The calculation process is as follows:

[0083] F d =avg(mlp(concat) n (Neib d -d,d)))

[0084] F m =avg(mlp(concat) n (Neib m -m,m)))

[0085] Among them, Neib d Neib m Let K be the geometric nearest neighbors of d and m respectively, and concat n This indicates a concatenation operation based on the neighborhood dimension, where MLP represents a multilayer perceptron block, AVG represents an average pooling layer, and F... d and F m These are the initial features corresponding to the direction vector and the moment vector, respectively;

[0086] The overall vector l is input into the third branch, which connects the direction vector d and the moment vector m, preserving the overall information of the line and extracting the features of their combined effect. The calculation process is as follows:

[0087] F l =ReLU(BatchNorm1d(Conv1d(l)))

[0088] Where Conv1d represents a one-dimensional convolutional layer, BatchNorm1d represents a one-dimensional batch normalization layer, ReLU represents a non-linear activation function, and F... l This represents the initial features corresponding to the overall vector;

[0089] Finally, the outputs of the three branches are concatenated along the channel dimension, and then further processed by MLP. The calculation process is as follows:

[0090] F = mlp(concat) c (F d ,F m ,F l ))

[0091] Among them, concat cThis indicates a concatenation operation along the channel dimension. `mlp` represents a multilayer perceptron block, and `F` represents the initial features obtained from the feature extraction geometry module. The source 3D line reconstruction and the target 3D line reconstruction use the same processing procedure at this stage, so `F` is F0. source With F target A unified representation;

[0092] The initial features F are processed by the global attention module in the feature extraction module. source F target Attention feature fusion is performed to obtain the enhanced global feature F′. source F′ target The existence determination module then processes it further, such as... Figure 4 As shown. First, the global feature F′ is used. source and F′ target To calculate the matching existence score source and Score target The specific calculation process is as follows:

[0093] Score source =softmax(mlp(concat) c (max(F′ target ),mean(F′ target ),F′ source )))

[0094] Score target =softmax(mlp(concat) c (max(F′ source ),mean(F′ source ),F′ target )))

[0095] Where softmax represents the normalized exponential function, mlp represents the multilayer perceptron block, and concat... c This indicates that the data is concatenated according to the channel dimension. Max and mean represent the maximum and minimum values ​​of the features in the first dimension, respectively.

[0096] Simultaneously calculate the similarity matrix. Similarity matrix with target N represents the set of real numbers. s and N t These represent the number of rows and columns of the matrix, and also the number of 3D lines in the corresponding reconstruction. The similarity between two lines is measured using the normalized dot product of the global features of the two reconstructed 3D lines. The specific calculation process is as follows:

[0097]

[0098]

[0099] In the formula, and F′ represents the corresponding element in the i-th row and j-th column of the corresponding similarity matrix, respectively. source_i F′ represents the global feature with index i in the source 3D line reconstruction. source_j F′ represents the global feature with index j in the source 3D line reconstruction. target_i F′ represents the global feature at index i in the reconstruction of the target 3D line. target_j represents the global feature with index j in the target 3D line reconstruction, · represents the vector dot product, × represents the numerical multiplication, and ||||2 represents the vector's L2 norm;

[0100] Multiplying the matching existence score and the similarity matrix yields the matching existence matrices corresponding to the 3D lines in the two reconstructions. Maximizing and averaging each row of these two existence matrices, concatenating them along the channel dimension, and then processing them using MLP yields the existence probability vector of the source reconstruction. The existence probability vector of the reconstructed target The specific calculation process is as follows:

[0101]

[0102]

[0103] Where * denotes matrix multiplication, E source and E target Let max represent the source matching existence matrix and the target matching existence matrix. raw and mean raw These represent finding the maximum and mean values ​​in each row of the matrix, respectively. c This indicates that the components are spliced ​​according to the channel dimension, mlp represents multilayer perceptron block, and softmax represents normalized exponential function.

[0104] The post-matching processing module processes the global features F′ of the input. source F′ target And the existence probability vector V of the matching source V target The matching probability matrix P is obtained. Global features F′ source The feature vector F′ corresponding to the 3D line with index i in the middle. source_i and global features F′ target F′ with index j in the middle target_j Use L2 Euclidean distance to construct a distance matrix M between feature vectors, where M represents the values ​​of the feature vectors.ij Indicates at F′ source_i and F′ target_j The cost of establishing a match between them is calculated as follows:

[0105] M ij =||F′ source_i -F′ target_j ||2

[0106] Where ||||2 represents the second norm of the vector;

[0107] To handle the probability of pairwise matching globally, the uncertainty of matching is described as an optimal transport problem, which is solved by solving the matching probability matrix P. The calculation process is as follows:

[0108]

[0109] Where μ represents the parameter variable, log is the logarithmic function, and M ij P represents the i-th row and j-th column of the distance matrix M. ij To match the i-th row and j-th column of the probability matrix P, argmin is the minimum function, and U(r,s) is the transmission polyhedron connecting the two prior probability vectors r and s, given as follows:

[0110]

[0111] Where * denotes matrix multiplication. N represents the set of real numbers. s and N t These represent the number of rows and columns of the matrix, respectively. and These represent sizes N respectively. s and N t A singular vector, r and s, are prior probability vectors, representing the probability that a 3D line has a valid match and is not an outlier; the corresponding match existence probability vector V is reconstructed from the source 3D line using the output of the match existence discrimination module. source The matching existence probability vector V corresponding to the target 3D line reconstruction target Initialize r and s respectively; the matching probability matrix P obtained by the above formula includes the matching probability relationship between the source 3D line reconstruction and the target 3D line reconstruction. The values ​​in the matching probability matrix P represent the matching weights. The larger the value, the greater the probability of matching between the two 3D lines.

[0112] From the matching probability matrix P, select the top K' pairs of potential 3D line matches with the highest weight scores. Then, use the RANSAC method to estimate the relative pose between the source and target 3D line reconstructions in the predicted top K' pairs of potential line matches. Finally, use the estimated relative pose to align the two 3D line reconstructions, achieving accurate registration. The implementation details of the RANSAC method are: cyclically and randomly select two matching pairs from the K' pairs of potential matches. and in These are the two selected source 3D lines. These are two selected target 3D lines. These two matching pairs are used to calculate the relative pose required for registration. Since each 3D line consists of a direction vector d and a vector m, the specific calculation process is as follows:

[0113]

[0114] Where R represents the rotation matrix, t represents the translation vector, t^ represents the antisymmetric matrix corresponding to the translation vector, and d source Let d be the direction vector of the source 3D lines in the two selected matching pairs. target To select the direction vector of the target 3D line, m source For the moment vectors of the source 3D lines in the two selected matching pairs, m target To select the moment vector of the target 3D line, the rotation matrix R and translation vector t obtained by calculating two matching pairs can be used to register and align the source 3D line reconstruction with the target 3D line reconstruction. The alignment quality is defined by the fractional function S, and its calculation formula is as follows:

[0115]

[0116] Where T′ is the transformation matrix consisting of rotation matrix R and translation vector t, which transforms the source 3D lines l source Transform to target 3D lines target The system identifies nearby locations and measures the Euclidean distance between them. If the Euclidean distance is less than a predefined threshold, the corresponding 3D lines are considered an inner pair. The number of inner pairs is counted in each iteration, and after multiple iterations of RANSAC, the iteration with the highest number of inner pairs is selected. The rotation matrix R obtained in this iteration is then used as the basis for the selection. best Translation vector t best As the final relative pose result, it enables precise registration between the source 3D line reconstruction and the target 3D line reconstruction.

[0117] To optimize the parameters of the feature extraction module and the matching existence discrimination module, and to make the matching probability matrix P approximate the true matching matrix P gt By using two loss functions Lc and L e The network is trained, where the loss function L c The calculation is as follows:

[0118]

[0119] In the formula, the true matching matrix P gt This represents the truth value of the matching relationship. A value of 1 at the corresponding position in the matrix indicates a correct match, while a value of 0 indicates no match. By optimizing L... c This encourages the network to maximize the probability of correct matches while minimizing the probability of incorrect matches; this helps improve the network's ability to identify correct matches; another newly added loss function L e This is an improvement on the original loss function. A new matching existence loss function is added to supervise network training. Combined with the matching existence discrimination module, this enhances the network's ability to handle noisy data. Specifically, this is reflected in the network's output source 3D line reconstruction matching existence probability vector V. source And the matching existence probability vector V in the target 3D line reconstruction target The cross-entropy loss between the true matching existence vector and the actual matching vector is calculated as follows:

[0120]

[0121] In the formula, CrossEntropy represents the cross-entropy loss function. and These are the corresponding truth-valued matching existence vectors; to achieve accurate matching pair estimation, the two losses mentioned above are combined, and all parameters are trained jointly to optimize the entire objective L. total The specific calculation process is as follows:

[0122] L total =βL c +(1-β)L e

[0123] In the formula, β is the weight that balances the two losses.

[0124] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A 3D line reconstruction and registration method based on an improved PlückerNet, characterized in that, Includes the following steps: 1) Obtain source 3D line reconstruction and target 3D line reconstruction data describing the real scene from each frame of RGB image and depth image; 2) The trained improved PlückerNet network is used to process the 3D line reconstruction data to be registered as follows: The source 3D line reconstruction and target 3D line reconstruction data are input into the three-branch local geometry module in the feature extraction module to obtain the initial features corresponding to the two reconstructions. , ; The initial features are processed by the global attention module in the feature extraction module. , Perform attention feature fusion. Obtain global features with feature enhancement , ; The global features corresponding to the source 3D line reconstruction are calculated by the matching existence discrimination module. Existence score of matching all lines in and the global features corresponding to the reconstruction of the target 3D lines. Existence score of matching all lines in At the same time, utilize , Calculate the similarity matrix , ; Match the existence score and similarity matrix Multiplying them together yields the matching existence matrix. Then Row-wise pooling and channel concatenation are then fed into a multilayer perceptron block for processing to obtain the matching existence probability vector. ; Similarly, the existence score of the match will be... and similarity matrix Multiplication yields the matching existence matrix Then Row-wise pooling and channel concatenation are then fed into a multilayer perceptron block for processing to obtain the matching existence probability vector. ; 3) Global features , Matching Existence Probability Vector , The input is fed into the matching post-processing module of the improved PlückerNet network to estimate the relative pose of the final registration between the source 3D line reconstruction and the target 3D line reconstruction. Finally, the estimated relative pose is used to align the two 3D line reconstructions to achieve accurate registration.

2. The 3D line reconstruction and registration method based on the improved PlückerNet according to claim 1, characterized in that, Step 1) includes the following steps: 1.1) 2D line acquisition: For each frame of RGB image data in the public dataset, the fast line detection algorithm in the OpenCV library is used to obtain the 2D lines in each frame of RGB image; 1.2) 3D line acquisition: For each 2D line detected from the RGB image data, a 3D line is fitted using a depth image that corresponds one-to-one with the RGB image; the depth image contains the depth information of each 2D line, thereby obtaining the 3D information corresponding to the line. 1.3) Plücker Line Acquisition: Following steps 1.1) and 1.2), 3D lines describing the real-world scene are reconstructed from each frame of the RGB and depth images. Each 3D line in the 3D line reconstruction is transformed into the Plücker coordinate system to obtain the Plücker line, which serves as input data for the improved PlückerNet network. For a given 3D line... ,in These represent the start and end points of the 3D line, respectively. End point ,in and These are the X, Y, and Z axis three-dimensional coordinates corresponding to the start and end points; 3D lines Transforming into the Plücker coordinate system yields a six-dimensional vector. ,in Represents the direction vector of a 3D line. The moment vector and direction vector of a 3D line. and torque vector It is calculated using the following formula: ； ； In the formula, Represents the cross product of vectors. The second norm of a vector; The source 3D line reconstruction to be registered is represented as follows: ； The target 3D line reconstruction is represented as: ； In the formula, Represents the reconstruction of source 3D lines, by A collection of 3D lines Indicating the first in source reconstruction line, and These are the corresponding direction vector and torque vector; This indicates the reconstruction of the target's 3D lines, by A collection of 3D lines Indicating the first step in the target reconstruction line, and These correspond to the direction vector and moment vector; each 3D line in the two reconstructions is transformed to the Plücker coordinate system, based on the three-dimensional direction vector. and three-dimensional torque vector composition.

3. The 3D line reconstruction and registration method based on the improved PlückerNet according to claim 2, characterized in that, The feature extraction local geometry module will extract the direction vector. Torque vector and the overall vector formed by these two. The inputs to the three branches are used to extract initial features. , ;exist and In each subspace, define the geometric nearest neighbor, construct a linear KNN, and calculate the nearest neighbors in the Euclidean space based on the Euclidean distance. Geometric nearest neighbor; The residuals of the neighbor's vector and its own vector are concatenated with the vector of the item itself to construct the corresponding local features. The specific calculation process is as follows: ； ； In the formula, , They represent and of Geometric nearest neighbor, This indicates a concatenation operation based on the neighborhood dimension. This represents a multilayer perceptron block. Indicates the average pooling layer. and These are the initial features corresponding to the direction vector and the moment vector, respectively; global vector The input is fed into the third branch, which will then input the direction vector. and torque vector By connecting the two, preserving the overall information of the lines, and extracting the features resulting from their combined effect, the calculation process is as follows: ； In the formula, This represents a one-dimensional convolutional layer. This represents a one-dimensional batch normalization layer. Represents a non-linear activation function. This represents the initial features corresponding to the overall vector; Finally, the outputs of the three branches are concatenated along the channel dimension, and then... Further processing is required, and the calculation process is as follows: ； In the formula, This indicates a concatenation operation based on the channel dimension. This represents a multilayer perceptron block. The initial features obtained from the local geometry module represent the features extracted. The source 3D line reconstruction and the target 3D line reconstruction use the same processing procedure, therefore... for and A unified representation.

4. The 3D line reconstruction and registration method based on the improved PlückerNet according to claim 3, characterized in that, The matching existence determination module will process the global enhanced features obtained after the two-stage feature extraction module. , The process is performed to obtain the matching existence probability vector of the reconstructed source 3D lines. The matching existence probability vector corresponding to the target 3D line reconstruction First, utilize features and To calculate the matching existence score and The specific calculation process is as follows: ； ； In the formula, Represents the normalized exponential function, and These represent the operations of finding the maximum and minimum values ​​and the mean of the feature in the first dimension, respectively. Calculate the similarity matrix and similarity matrix , Represents the set of real numbers. and These represent the number of rows and columns of the matrix, and also the number of 3D lines in the corresponding reconstruction. The similarity between two lines is measured using the normalized dot product of the global features of the two reconstructed 3D lines. The specific calculation process is as follows: ； ； In the formula, and Represent the first and second elements in the corresponding similarity matrix, respectively. row and number Column corresponding elements, Indicating the index in the source 3D line reconstruction global features Indicating the index in the source 3D line reconstruction global features Indicating the index in the target 3D line reconstruction global features Indicating the index in the target 3D line reconstruction global features Represents the dot product of vectors. Represents numerical multiplication. The second norm of a vector; Multiplying the match existence score and the similarity matrix yields two match existence matrices corresponding to the 3D lines in the reconstruction. The maximum and mean values ​​are then calculated for each row of these two match existence matrices, and the matrices are concatenated along the channel dimension before being processed... The existence probability vector of the source reconstruction is obtained through processing. Existence probability vector of target reconstruction The specific calculation process is as follows: ； ； In the formula, Representing matrix multiplication and These represent operations to find the maximum value and the mean value in each row of the matrix, respectively.

5. The 3D line reconstruction and registration method based on the improved PlückerNet according to claim 4, characterized in that, The post-matching processing module processes the input global features. , Matching Existence Probability Vector , Obtain the matching probability matrix ; Global features The index is The feature vector corresponding to the 3D lines and global features The index is of Use L2 Euclidean distance to construct the distance matrix between feature vectors. The values ​​in this distance matrix Indicates in and The cost of establishing a match between them is calculated as follows: ； The uncertainty of matching is described as an optimal transmission problem, and the matching probability matrix is ​​solved. To solve this, the calculation process is as follows: ； In the formula, Indicates parameter variables, It is a logarithmic function. Distance matrix The first in Line number List, For matching probability matrix The first in Line number List, It is a minimum value function. It is to combine two prior probability vectors and The interconnected transmission polyhedra are given in the following manner: ； In the formula, and They represent sizes of and A completely one vector, and Let be the prior probability vector, representing the likelihood that a 3D line has a valid match and is not an outlier; the corresponding match existence probability vector is reconstructed from the source 3D line using the output of the match existence discrimination module. The matching existence probability vector corresponding to the target 3D line reconstruction right , Initialize them separately; the matching probability matrix obtained by the above formula The matching probability matrix includes the matching probability relationship between the reconstructed source 3D lines and the reconstructed target 3D lines. The values ​​in the table represent the matching weights; the larger the value, the greater the probability that there is a match between the two 3D lines. From the matching probability matrix Select the top score with the highest weight. Matching potential 3D lines, in the predicted front The RANSAC method is used to estimate the relative pose of the source and target 3D line reconstructions for final registration in potential line matching. Finally, the estimated relative pose is used to align the two 3D line reconstructions, achieving accurate registration. The implementation of the RANSAC method is as follows: From... Two matching pairs are randomly selected cyclically from the potential matches. and ,in , These are the two selected source 3D lines. , These are two selected target 3D lines. The matching pairs are used to calculate the relative pose required for registration. Since each 3D line is composed of a direction vector... and vector Therefore, the specific calculation process is as follows: ； In the formula, Represents the rotation matrix. Represents the translation vector. This represents the antisymmetric matrix corresponding to the translation vector. The direction vectors of the source 3D lines in the two selected matching pairs. To select the direction vector of the target 3D line, The moment vectors of the source 3D lines in the two selected matching pairs. To select the moment vector of the target 3D line; the rotation matrix is ​​calculated using two pairs of matching lines. Translation vector It can register and align the 3D lines in the source 3D line reconstruction with the 3D lines in the target 3D line reconstruction, and the quality of the alignment is determined by a fractional function. To define it, its calculation formula is: ； In the formula, Rotation matrix Translation vector The transformation matrix formed will transform the source 3D lines Transform to target 3D lines The system identifies nearby locations and measures the Euclidean distance between them. If the Euclidean distance is less than a predefined threshold, the corresponding 3D lines are considered internal pairs. The number of internal pairs is counted in each iteration, and after multiple iterations of the RANSAC method, the iteration with the most internal pairs is selected. The rotation matrix obtained from this iteration is then used to determine the internal pair. Translation vector As the final relative pose result, it enables precise registration between the source 3D line reconstruction and the target 3D line reconstruction.

6. The 3D line reconstruction registration method based on the improved PlückerNet according to claim 5, characterized in that, To optimize the parameters of the feature extraction module and the matching existence discrimination module, and to make the matching probability matrix... Approximating the true matching matrix By using two loss functions and The network is trained, where the loss function is... The calculation is as follows: ； In the formula, the true matching matrix This represents the truth value of the matching relationship. A value of 1 at the corresponding position in the matrix indicates a correct match, while a value of 0 indicates no match. Another newly added loss function... This is an improvement on the original loss function. A new matching existence loss function is added to supervise network training. Combined with the matching existence discrimination module, this enhances the network's ability to handle noisy data. Specifically, this is reflected in the network's output of the source 3D line reconstruction matching existence probability vector. And the matching existence probability vector in the reconstruction of target 3D lines The cross-entropy loss between the true matching existence vector and the actual matching vector is calculated as follows: ； In the formula, Represents the cross-entropy loss function. and These are the corresponding truth-valued matching existence vectors; to achieve accurate matching pair estimation, the two losses mentioned above are combined, and all parameters are trained jointly to optimize the entire objective. The specific calculation process is as follows: ； In the formula, To balance the weights of the two losses.

Images