An industrial image feature matching processing method based on visual laser cooperation

Abstract

The application discloses a kind of industrial image feature matching processing methods of visual laser coordination, it is related to vehicle environment perception technical field;The application generates the lightweight cross-modal feature description vector of fusion texture and three-dimensional depth information, reduces the data representation difference of vision and laser, reduces initial mismatch from source;Combining semantic, neighborhood topological multidimensional calculation matching confidence, fully filter out inferior matching pair to reduce pose solving noise;To pixel re-projection error, point cloud registration error is respectively weighted to construct joint error function, six degrees of freedom pose is solved by nonlinear optimization, balance two kinds of sensor observation deviation, improve single-frame pose precision;Through feature priority matching neighborhood, suppress single-frame random error, then utilize bag-of-words closed loop detection and pose graph optimization to share time sequence cumulative error, solve long-term running trajectory drift problem, under the restriction of vehicle-mounted hardware, give consideration to matching precision and sequence trajectory global stability, adapt to the use demand of automatic driving environment perception.

Description

Technical Field

[0001] This invention relates to the field of vehicle environmental perception technology, and in particular to a visual-laser coordinated industrial image feature matching processing method. Background Technology

[0002] Autonomous driving relies on the fusion of vision and laser perception, and cross-modal feature matching is the core of pose calculation. Existing traditional methods have poor cross-modal feature discrimination, rely on single selection criteria which easily lead to mismatches, and cannot balance the errors of the two types of sensors in pose optimization. Furthermore, they can generate cumulative trajectory drift over long periods of time. Deep learning solutions, on the other hand, depend on annotation and consume a lot of onboard computing power. Current technologies cannot simultaneously improve on these multiple shortcomings, making it difficult to balance matching accuracy and long-term trajectory stability. Summary of the Invention

[0003] This invention proposes a vision-laser collaborative industrial image feature matching processing method to solve the problems mentioned in the prior art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a visual-laser coordinated industrial image feature matching processing method, comprising: S1. Construct a multi-scale Gaussian pyramid for the image, extract binary features with scale invariance and rotation invariance, distinguish between two types of 3D feature points, namely plane and edge, based on the local geometric attributes of the point cloud, project the 3D features onto the unified image plane obtained by joint calibration, and generate a fusion feature description vector with 3D depth information as the weighting factor. S2. Initial matching is completed based on feature descriptor similarity. Bidirectional consistency check combined with rigid body transformation constraints is used to remove coarse matching outliers. Pixel-level semantic segmentation is performed on the images corresponding to the coarse matching points. The unified matching confidence is calculated by fusing semantic category constraints and feature neighborhood topology constraints. Matching pairs are filtered according to the confidence level, and matching pairs that meet the preset standards are retained. S3. The image pixel matching correspondence and the laser point cloud spatial matching correspondence are fused together. At the same time, pixel reprojection error and point cloud registration error are introduced. A joint error function with weights assigned to the two types of errors is constructed. The optimal rigid body transformation matrix of six degrees of freedom is solved in one step through nonlinear optimization to obtain the relative pose of a single frame. S4. Define a priority matching search neighborhood for features that are stably tracked across multiple frames; use a bag-of-words model to detect loop closures and perform geometric verification; optimize and amortize the loop closure correction error through pose graph to correct the long-term cumulative drift caused by cross-modal feature matching, and output a globally unified sequence matching result.

[0005] Furthermore, before step S1, the process includes: identifying various dynamic moving targets in the image using a target detection algorithm, generating a dynamic target mask between consecutive frames using a target tracking algorithm, performing pixel dilation processing on the mask area, and then removing all feature points to be extracted within the mask coverage area.

[0006] Furthermore, the visual-laser collaborative industrial image feature matching processing method also includes: constructing a multi-scale Gaussian pyramid for the image and a multi-resolution voxel pyramid for the point cloud, respectively, adopting a coarse-to-fine hierarchical matching strategy, completing global matching and localization at the low-resolution level to obtain a coarse transformation relationship, passing the transformation result down layer by layer, and completing fine matching of local neighborhood features at the high-resolution level based on prior transformation. At the same time, a parallel computing architecture is adopted to divide the image into multiple sub-regions and allocate independent computing threads to process the feature matching task of each sub-region in parallel.

[0007] Furthermore, the formula for calculating the matching cost of the fused feature description vector in step S1 is as follows: Where D is the matching cost between the two sets of fused feature descriptors; N is the total number of bits in the descriptor; Let be the binary value of the i-th bit of the first feature descriptor; The second feature descriptor is the binary value of the i-th bit; is the depth weight coefficient for the i-th position.

[0008] Furthermore, in step S2, the bidirectional consistency check is a bidirectional mutual matching of image and point cloud features, and the matching pairs that only match in one direction are eliminated; the random sampling consistency algorithm is used to solve the rigid body transformation, and four pairs of non-collinear three-dimensional matching points are extracted in each iteration. Matching points with a pixel reprojection error of less than 3 pixels are determined as interior points. Interior points are filtered and geometric exterior points are eliminated to obtain a coarse matching point set.

[0009] Furthermore, in step S2, the unified matching confidence score is fused with two indicators: semantic consistency and neighborhood topological consistency. The corresponding calculation formula is as follows: Where C is the confidence score of the matching point pair; λ is the balancing weight coefficient; S is the semantic category consistency score; and T is the neighborhood topology consistency score.

[0010] Furthermore, in step S3, the Levenberg-Marquardt algorithm is used for iterative solution, with an error convergence threshold of 1e-6 and an upper limit of 50 iterations, so that the joint error function takes the minimum value; the six degrees of freedom parameters consisting of three-dimensional rotation and three-dimensional translation are used as optimization variables, and the optimal rigid body transformation matrix is ​​output after the iteration converges.

[0011] Furthermore, the loop closure detection and correction process in step S4 is as follows: extract the image features of each frame to generate corresponding bag-of-words vectors, compare the similarity between the bag-of-words vector of the current frame and the bag-of-words vectors of all historical frames, and select historical frames with a similarity higher than 0.7 as candidate loop closure frames; use rigid body transformation geometric constraints to verify the effectiveness of candidate loop closures; solve the loop closure correction error based on the matching point pairs of effective loops, and distribute the correction error to all historical frames through pose graph optimization to uniformly eliminate the cumulative drift of long-term cross-modal matching.

[0012] Compared with existing technologies, the beneficial effects of this invention are: According to the vision-laser collaborative industrial image feature matching processing method of this application, S1 to S4 work together to form a complete processing link. S1 generates a lightweight cross-modal feature description vector that fuses texture and 3D depth information, reducing the data representation difference between vision and laser, reducing initial mismatches from the source, and adapting to low-computing-power vehicle-mounted devices. On this basis, S2 calculates the matching confidence level in multiple dimensions, combining semantics and neighborhood topology, and fully filters out inferior matching pairs to reduce pose solution noise. S3 constructs a joint error function by weighting pixel reprojection error and point cloud registration error respectively, and then processes it through a nonlinear... The solution optimizes the six-DOF pose calculation, balances the observation bias of the two types of sensors, and improves the pose accuracy of a single frame. S4 suppresses random errors in a single frame by using feature-priority matching neighborhoods, and then uses bag-of-words loop closure detection and pose graph optimization to amortize the accumulated temporal error, solving the problem of long-term trajectory drift. The whole solution overcomes the shortcomings of existing technologies such as insufficient cross-modal discrimination, single dimension of mismatch elimination, difficulty in balancing dual-modal errors, long-term trajectory drift, and high computational cost of deep models. Under the constraints of vehicle hardware, it balances matching accuracy and global stability of sequence trajectory, and is suitable for the environmental perception needs of autonomous driving. Attached Figure Description

[0013] Figure 1 This is a flowchart of the overall process for visual laser-assisted industrial image feature matching proposed in this invention. Figure 2 This is a flowchart of the multi-source sensor data synchronization and spatial registration process of the present invention; Figure 3 This is a flowchart of the dual-modal multi-scale feature extraction and hierarchical matching process of the present invention; Figure 4 This is a flowchart of the multi-constraint fine matching and dynamic target filtering and elimination process of the present invention; Figure 5 This is a flowchart of the joint error pose calculation and global closed-loop correction of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Reference Figures 1 to 5 This application provides a vision-laser coordinated industrial image feature matching processing method, including the following steps: S1. Construct a multi-scale Gaussian pyramid for the image, extract binary features with scale invariance and rotation invariance, distinguish between two types of 3D feature points, namely plane and edge, based on the local geometric attributes of the point cloud, project the 3D features onto the unified image plane obtained by joint calibration, and generate a fusion feature description vector with 3D depth information as the weighting factor.

[0016] For example, a multi-layer Gaussian pyramid is constructed. Each layer of the image is smoothed using a 5×5 Gaussian kernel and then downsampled at a rate of 0.5. ORB binary features are extracted at each layer. These features naturally possess scale invariance and rotation invariance, and output a fixed-length 0 and 1 code. For each 3D point in the laser point cloud, the surface curvature is solved through the neighborhood point set, and planar feature points and edge feature points are divided according to a preset curvature threshold. Based on the intrinsic and extrinsic parameters obtained from the joint calibration of the camera and LiDAR, all 3D features are mapped to the same 2D image pixel plane through coordinate transformation. The binary code corresponding to the projected pixel position is extracted, and combined with the normalized 3D depth value of that point as a weighting coefficient, a feature description vector that simultaneously integrates image texture and 3D depth geometric information is generated. This weakens the data representation differences between the visual and laser modalities and reduces the number of mismatches in the initial matching stage.

[0017] For example, before step S1, a PPS pulse signal can be used to achieve microsecond-level time synchronization between the vision camera and the lidar, and two-dimensional image data and three-dimensional point cloud data can be collected respectively. Image distortion correction, grayscale normalization, Gaussian denoising and point cloud statistical filtering, voxel downsampling, and motion distortion correction are performed sequentially. A unified vehicle coordinate system is established through a pre-calibrated extrinsic parameter matrix to complete the spatial alignment of the dual-modal data.

[0018] S2. Initial matching is completed based on feature descriptor similarity. Bidirectional consistency check combined with rigid body transformation constraints is used to remove coarse matching outliers. Pixel-level semantic segmentation is performed on the images corresponding to the coarse matching points. The unified matching confidence is calculated by fusing semantic category constraints and feature neighborhood topological constraints. Matching pairs are filtered according to the confidence level, and matching pairs that meet the preset standards are retained.

[0019] In step S2, the similarity between the fused feature descriptors is calculated first to complete the initial matching. Bidirectional consistency verification involves searching for matches based on image features and laser features respectively, retaining only bidirectionally corresponding matching pairs. For example, in this embodiment, the FLANN fast nearest neighbor algorithm can be used to complete similarity retrieval and bidirectional matching verification. Subsequently, the spatial correspondence of matching point pairs is constrained based on the rigid body transformation model, and outliers deviating from the transformation model are eliminated. Ideally, the rigid body transformation matrix is ​​solved using the Random Sampling Consensus (RANSAC) algorithm, and valid matching points are selected. Pixel-level semantic segmentation is performed on the image region corresponding to the coarse matching point to distinguish the semantic category of the pixel. A lightweight semantic segmentation model is used to output the full-image pixel category label as the basis for semantic constraints. Simultaneously, the relative coordinates of the neighborhood features surrounding the matching point are extracted to form local topological constraints. The semantic category constraints and neighborhood topological constraints are fused to calculate the unified matching confidence of the matching pair. Finally, only matching pairs with a confidence level reaching a preset threshold are retained, filtering out low-quality mismatches from multiple dimensions.

[0020] S3. The image pixel matching correspondence and the laser point cloud spatial matching correspondence are fused together. At the same time, pixel reprojection error and point cloud registration error are introduced. A joint error function with weights assigned to the two types of errors is constructed. The optimal rigid body transformation matrix of six degrees of freedom is solved in one step through nonlinear optimization to obtain the relative pose of a single frame.

[0021] In step S3, based on the selected high-quality matching pairs, two sets of spatial matching correspondences are established for image pixels and laser point clouds, respectively. Pixel reprojection error refers to the deviation between the predicted pixel coordinates and the actual pixel coordinates after the three-dimensional features are projected back to the image through rigid body transformation. Point cloud registration error refers to the spatial distance deviation between the radar three-dimensional points and the target point cloud after transformation. Weighting coefficients are set independently for the two types of errors to construct a joint error function, balancing the influence weights of visual and laser observation data on the optimization results. In this embodiment, the Levenberg-Marquardt nonlinear optimization algorithm can be used to minimize the joint error function, outputting the optimal six-degree-of-freedom rigid body transformation matrix at one time, and solving the relative pose parameters of the vehicle body. This matrix contains three-dimensional rotation and three-dimensional translation components, thereby representing the single-frame pose of the current frame relative to the previous frame.

[0022] S4. Define a priority matching search neighborhood for features that are stably tracked across multiple frames; use a bag-of-words model to detect loop closures and perform geometric verification; optimize and amortize the loop closure correction error through pose graph to correct the long-term cumulative drift caused by cross-modal feature matching, and output a globally unified sequence matching result.

[0023] Specifically, for feature points that are continuously and stably tracked across frames, a local priority matching search neighborhood is defined with the pixel coordinates of the previous frame as the center, which narrows the single-frame matching retrieval range and reduces matching noise. The bag-of-words model clusters image features to generate a visual dictionary, and completes loop closure detection by comparing the similarity between the current frame and the historical frame dictionaries. After detecting the loop, the validity of the loop closure matching pair is verified by rigid body geometric constraints. A pose graph containing the poses and loop closure constraints of all historical frames is constructed. The loop closure correction error is uniformly distributed through graph optimization algorithms to offset the temporal error that is continuously accumulated in frame-by-frame matching, eliminate trajectory drift, and finally output a globally consistent complete image sequence matching and pose result.

[0024] According to the vision-laser collaborative industrial image feature matching processing method of this application, S1 to S4 work together to form a complete processing link. S1 generates a lightweight cross-modal feature description vector that fuses texture and 3D depth information, reducing the data representation difference between vision and laser, reducing initial mismatches from the source, and adapting to low-computing-power vehicle-mounted devices. On this basis, S2 calculates the matching confidence level in multiple dimensions, combining semantics and neighborhood topology, and fully filters out inferior matching pairs to reduce pose solution noise. S3 constructs a joint error function by weighting pixel reprojection error and point cloud registration error respectively, and then processes it through a nonlinear... The solution optimizes the six-DOF pose calculation, balances the observation bias of the two types of sensors, and improves the pose accuracy of a single frame. S4 suppresses random errors in a single frame by using feature-priority matching neighborhoods, and then uses bag-of-words loop closure detection and pose graph optimization to amortize the accumulated temporal error, solving the problem of long-term trajectory drift. The whole solution overcomes the shortcomings of existing technologies such as insufficient cross-modal discrimination, single dimension of mismatch elimination, difficulty in balancing dual-modal errors, long-term trajectory drift, and high computational cost of deep models. Under the constraints of vehicle hardware, it balances matching accuracy and global stability of sequence trajectory, and is suitable for the environmental perception needs of autonomous driving.

[0025] In one implementation, before step S1, the process may include: identifying various dynamic moving targets (such as vehicles, pedestrians, etc.) in the image using a target detection algorithm; generating a dynamic target mask between consecutive frames using a target tracking algorithm; and removing all feature points to be extracted within the mask coverage area after performing pixel dilation processing on the mask region. This way, only valid matching points in the static background region are retained for subsequent pose calculation, avoiding interference from dynamic targets on matching accuracy.

[0026] In one implementation, the visual-laser collaborative industrial image feature matching processing method may further include: constructing a multi-scale Gaussian pyramid for the image and a multi-resolution voxel pyramid for the point cloud, respectively; employing a coarse-to-fine hierarchical matching strategy; completing global matching and localization at the low-resolution level to obtain a coarse transformation relationship; passing the transformation result down layer by layer; and completing fine matching of local neighborhood features at the high-resolution level based on prior transformation. Simultaneously, a parallel computing architecture is used, dividing the image into multiple sub-regions and allocating independent computing threads to process the feature matching tasks of each sub-region in parallel. The top layer of the pyramid contains low-resolution data, obtained by downsampling and compressing the original data multiple times, resulting in fewer pixels and point clouds, simplified details, and faster processing speed but lower localization accuracy. The bottom layer contains high-resolution data, closely resembling the original acquired data, with complete geometric details and higher localization accuracy, but with a large global matching computational load.

[0027] In this embodiment, an image Gaussian pyramid and a point cloud voxel pyramid can be constructed first. The coarse-level data corresponding to the low-resolution data is used for large-scale global retrieval to quickly locate the coarse rigid body transformation relationship. Then, this transformation is used as a priori and passed to the high-resolution fine-level layer to narrow the matching search range of each layer and complete high-precision local feature matching. At the same time, a multi-threaded parallel architecture is adopted to divide the original image into several non-overlapping sub-regions and allocate independent threads to synchronously process the feature extraction and matching of each region. In this embodiment, the voxel pyramid is constructed by fixed-size voxel downsampling. Parallel processing can be achieved through GPU parallel operators, which can significantly reduce the global matching time and improve the overall speed of visual laser matching.

[0028] In one example, before step S1, each frame of point cloud data from the LiDAR corresponds to a high-precision timestamp, and each frame of image data from the camera corresponds to a hardware trigger moment. Linear interpolation is used to align the point cloud data and image data to the same time base, with the time synchronization error controlled at the microsecond level. For the image data, a Brownian distortion model is used to correct radial and tangential distortions, and the gray values ​​are normalized to the standard range of 0 to 255. A 5×5 Gaussian kernel is used to remove salt-and-pepper noise from the image. For the point cloud data, statistical filtering is used to remove outliers with a mean deviation greater than three times the standard deviation. A voxel grid with a side length of 0.2 meters is used to downsample and compress the data. The angular velocity and linear velocity data output by the vehicle inertial navigation system are combined to correct the point cloud distortion generated during vehicle movement. Finally, the three-dimensional point cloud is uniformly transformed to the camera coordinate system through the pre-calibrated extrinsic parameter matrices of the camera and LiDAR, completing the spatial alignment of the dual-modal data.

[0029] In one implementation, the formula for calculating the matching cost of fusing the feature description vector in step S1 is: Where D is the matching cost between the two sets of fused feature descriptors, with a dimension of 1; the smaller the value, the higher the similarity. N is the total number of bits in the descriptor, with a dimension of 1. Let be the binary value of the i-th bit of the first feature descriptor, with a dimension of 1; Let be the binary value of the i-th bit of the second feature descriptor, with a dimension of 1; Let be the depth weight coefficient for the i-th feature point, with a dimension of 1. The weight of the corresponding position in the center region of the feature point is higher than that in the edge region. By traversing and calculating the matching cost of all feature pairs, the modal feature point with the minimum cost for each feature point is selected as the initial matching result.

[0030] In this invention, step S2 involves bidirectional consistency verification, which involves two-way matching of image and point cloud features, eliminating matching pairs that only match in one direction. A random sampling consistency algorithm is used to solve the rigid body transformation. In each iteration, four pairs of non-collinear 3D matching points are extracted. Matching points with a pixel reprojection error of less than 3 pixels are identified as interior points. Interior points are filtered, and geometrical exterior points are eliminated to obtain a coarse matching point set. For example, in each iteration, four pairs of non-collinear matching points are randomly selected to calculate the transformation matrix. The number of interior points that conform to this transformation relationship among all matching points is counted. After iterating to the set maximum number of iterations, the transformation matrix with the most interior points is selected as the initial rigid body transformation matrix. All exterior points that do not conform to the geometric constraints are eliminated. During the iteration process, matching points with a pixel reprojection error of less than 3 pixels are identified as interior points. Finally, the initial matching point set containing all interior points is output, ensuring that the filtered matching point pairs satisfy the geometric consistency of the rigid body motion.

[0031] In this invention, in step S2, the unified matching confidence score is fused with two indicators: semantic consistency and neighborhood topological consistency. The corresponding calculation formula is as follows: Where C is the confidence score of the matched point pair, with a dimension of 1 and a value ranging from 0 to 1; a higher value indicates stronger matching reliability. λ is the balancing weight coefficient, with a dimension of 1 and a value ranging from 0 to 1. S is the semantic category consistency score, with a dimension of 1; a value of 1 for matching pairs within the same category and a value of 0.2 for matching pairs across categories. T is the neighborhood topology consistency score, with a dimension of 1, and is the ratio of the number of successfully matched point pairs in the neighborhood to the total number of feature points in the neighborhood. Matching point pairs with a confidence score higher than 0.7 are retained as the final valid matching set, further filtering out low-quality mismatches.

[0032] For example, a lightweight semantic segmentation network can be used to perform pixel-level semantic classification of images, marking different categories of areas such as roads, buildings, vegetation, and sky. Only matching point pairs belonging to the same semantic category are retained, while mismatched pairs across categories are eliminated. All feature points in the 8×8 neighborhood around each feature point are extracted to construct topological relationships. The proportion of successfully matched point pairs in the neighborhood to the total number of feature points is calculated as the topological consistency score. The comprehensive confidence C of the matching point pairs is calculated by combining semantic category consistency and neighborhood topological consistency.

[0033] In this invention, step S3 employs the Levenberg-Marquardt algorithm for iterative solution, setting an error convergence threshold of 1e-6 and an upper limit of 50 iterations to minimize the joint error function. The six degrees of freedom parameters, consisting of three-dimensional rotation and translation, are used as optimization variables. After iterative convergence, the optimal rigid body transformation matrix is ​​output. For example, the two-dimensional pixel correspondence obtained from visual feature matching and the three-dimensional point correspondence obtained from laser point cloud matching can be fused to construct a joint error function containing pixel reprojection error and point cloud registration error. The Levenberg-Marquardt nonlinear optimization algorithm is then used to iteratively solve for the optimal rigid body transformation matrix, minimizing the weighted sum of the two types of errors. During optimization, a convergence threshold of 1e-6 and an upper limit of 50 iterations are set. Finally, the rotation matrix and translation vector of the vehicle body between adjacent frames are output, completing the accurate calculation of the relative pose and ensuring the accuracy and stability of the pose estimation.

[0034] In this invention, the loop closure detection and correction process in step S4 specifically involves: extracting image features from each frame to generate corresponding bag-of-words vectors; comparing the similarity between the current frame's bag-of-words vector and all historical frame bag-of-words vectors; selecting historical frames with a similarity higher than 0.7 as candidate loop closure frames; verifying the validity of candidate loop closures using rigid body transformation geometric constraints; solving the loop closure correction error based on the matching point pairs of effective loops; and distributing the correction error across all historical frames through pose graph optimization to uniformly eliminate long-term cross-modal matching cumulative drift. Here, the bag-of-words vector is a feature numerical representation obtained by clustering image features and mapping them to a unified visual dictionary; historical frames with a vector similarity higher than a preset threshold are candidate loop closure frames. For example, the DBoW2 framework can be used to construct the visual dictionary and generate bag-of-words vectors, and the pose graph can be optimized using the g2o tool for global error distribution.

[0035] In one example, an 8-layer Gaussian pyramid for the image and an 8-layer multi-resolution voxel pyramid for the point cloud are constructed, with a scaling factor of 0.8 for each layer. Global feature matching is performed starting from the highest layer, which is also the lowest resolution layer, to obtain a coarse transformation relationship. The transformation result is passed down layer by layer, and fine matching is performed in the local neighborhood based on the coarse transformation result at the next level. This process is iterated until the lowest layer, which is also the highest resolution layer, is reached to obtain the final matching result. The matching process adopts a GPU parallel computing architecture, which divides the image into multiple non-overlapping sub-regions and allocates different computing threads to process the feature matching tasks in the sub-regions in parallel, which greatly improves the real-time performance of the overall matching process and meets the low latency requirements of automotive scenarios.

[0036] The following two examples further illustrate the specific implementation of this system: Example 1: Implementation of Visual-Library Cooperative Feature Matching in Urban Road Autonomous Driving Scenarios This embodiment applies to the environmental perception system of an autonomous passenger vehicle on urban roads. The vehicle is equipped with a forward-looking industrial area array camera, a 16-line mechanical LiDAR, and a high-precision inertial navigation module. The data processing unit is an onboard intelligent domain controller with a built-in parallel computing GPU unit to accelerate image and point cloud processing tasks. This embodiment achieves high-precision pose estimation and environmental perception of the vehicle in urban road scenarios through feature matching that combines visual and LiDAR modalities, providing data support for autonomous driving positioning and navigation.

[0037] The system employs a PPS (Pulse Per Second) signal output from the inertial navigation system (INS) to achieve hardware-level time synchronization between the camera and the LiDAR. The camera's exposure time is precisely triggered by the PPS signal, and each sampling point in each frame of the LiDAR point cloud is accompanied by a corresponding INS timestamp. During data processing, the timestamps of the LiDAR point cloud are aligned to the same reference time using linear interpolation, with the time synchronization error controlled within 2 microseconds, eliminating time misalignment caused by motion. In the image preprocessing stage, based on a pre-calibrated camera intrinsic Brownian distortion model, radial and tangential distortions of the image are corrected. The original 12-bit grayscale image is normalized to a standard 8-bit grayscale range of 0 to 255 through linear mapping. A 5×5 Gaussian kernel with a standard deviation of 1.5 is used to perform convolution filtering on the image to remove sensor thermal noise and salt-and-pepper noise interference. In the point cloud preprocessing stage, a statistical filtering algorithm is used to calculate the average distance of the 50 nearest neighbors of each point, removing outlier noise points whose distance is greater than the mean plus three standard deviations. A cubic voxel grid with a side length of 0.2 meters is used to downsample the point cloud, retaining the centroid point within each voxel as a representative, compressing approximately 30,000 original points per frame to about 8,000, reducing subsequent computational load. Combining the 100Hz angular velocity and linear velocity data output from the inertial navigation system, each sampled point in each frame is back-projected to the coordinate system of the frame's start time through rigid body transformation based on its time difference relative to the frame's start time, correcting the point cloud motion distortion caused by vehicle movement. After preprocessing, using pre-calibrated extrinsic matrices of the LiDAR and camera, including 3D rotation and translation vectors, all 3D point clouds are uniformly transformed to the camera coordinate system, and then projected onto the 2D image plane through the camera intrinsic matrix to obtain the pixel coordinates corresponding to each point cloud, completing the spatial alignment of the dual-modal data.

[0038] Two-dimensional image feature extraction employs an improved multi-scale FAST corner detection algorithm, constructing an 8-layer Gaussian image pyramid with a scaling ratio of 0.8 for each layer. Corner features are detected at each scale layer: within a 9-pixel circular neighborhood centered on a pixel, if the grayscale value of 9 consecutive pixels differs from that of the center pixel by a set threshold, the point is identified as a corner. Non-maximum suppression is used to remove adjacent overlapping corners, retaining approximately 200 stable feature points at each scale layer, resulting in approximately 1200 feature points extracted from the entire image, ensuring scale and rotation invariance of the features. Feature descriptor generation uses the BRIEF binary descriptor algorithm. Within a 31×31-pixel neighborhood around each feature point, 256 pairs of pixels are randomly selected, and a 256-bit binary feature descriptor is generated by comparing the grayscale values ​​of each pair of pixels. The 3D point cloud feature extraction employs a curvature clustering method. For each point, the 20 nearest neighbors are selected, and the covariance matrix and three eigenvalues ​​of the local neighborhood are calculated using principal component analysis. The ratio of the smallest eigenvalue to the sum of the three eigenvalues ​​is used as the local curvature value. Points with curvature greater than 0.1 are clustered as edge feature points, and points with curvature less than 0.02 are clustered as planar feature points. Approximately 1000 feature points are extracted from a single frame of point cloud.

[0039] After mapping the 3D feature points onto the image plane through a projection matrix, the mean and variance of the grayscale values ​​of the 8×8 neighborhood around the corresponding pixel position are extracted. Combined with the depth values ​​of the feature points, a 64-dimensional depth-weighted feature description vector is generated, where the feature points with larger depth values ​​have lower weights, which is adapted to the characteristic of decreased accuracy of distant point clouds.

[0040] The weighted matching cost between all image feature descriptors and point cloud projection feature descriptors is calculated. For each feature point, the corresponding point with the minimum matching cost is selected as the initial matching pair. A total of about 800 initial matching points are obtained for a single frame image.

[0041] A random sampling consensus algorithm combined with iterative elimination of erroneous outliers based on rigid body transformation geometric constraints is employed. The algorithm is set to a maximum of 1000 iterations. In each iteration, four pairs of non-collinear matching points are randomly selected to solve for the rigid body transformation matrix. The number of interior points among all matching points that conform to the transformation relationship is counted. The criterion for interior points is a pixel reprojection error of less than 3 pixels. After iteration, the transformation matrix with the largest number of interior points is selected as the initial rigid body transformation matrix. All outliers that do not conform to the geometric constraints are eliminated. The final output is an initial set of valid matching points containing approximately 500 pairs of interior points and the corresponding initial transformation matrix.

[0042] A lightweight semantic segmentation network is employed for pixel-level semantic classification of images. The network uses MobileNetV2 as its backbone and incorporates the DeepLabv3 segmentation architecture, achieving an inference speed of up to 30 frames per second per frame. Images are divided into six semantic regions: roads, buildings, vegetation, sky, vehicles, and pedestrians. Only matching point pairs belonging to the same semantic category are retained, while mismatched pairs across categories are discarded. For each feature point, all feature points within an 8×8 pixel neighborhood are extracted to construct a local topological relationship. The proportion of successfully matched point pairs within the neighborhood to the total number of feature points in the neighborhood is calculated as the neighborhood topological consistency score. The combined confidence of matching point pairs is calculated by combining semantic category consistency and neighborhood topological consistency, with semantic consistency accounting for 0.6 and topological consistency accounting for 0.4. Matching point pairs with a confidence score higher than 0.7 are retained as the final valid matching set, yielding approximately 350 high-precision matching points per frame, further filtering out low-quality mismatches.

[0043] By fusing pixel correspondences obtained from visual 2D feature matching with spatial point correspondences obtained from laser 3D point cloud matching, a joint error function is constructed, incorporating pixel reprojection error and point cloud registration error. The pixel error weight is set to 0.4, and the point cloud spatial error weight is set to 0.6. A Levonburg-Marquardt nonlinear optimization algorithm is used iteratively to solve for the optimal rigid body transformation matrix. The optimization variables are six degrees of freedom parameters: 3D rotation and 3D translation. The algorithm has a maximum of 50 iterations and a convergence threshold of 1e-6. Iteration stops when the error change between two iterations is less than the convergence threshold. The final output is high-precision relative pose parameters between adjacent frames, with translation estimation accuracy reaching the centimeter level and rotation estimation accuracy reaching the 0.1-degree level, ensuring the accuracy and stability of pose estimation.

[0044] The feature matching template is dynamically updated based on the pose estimation results between consecutive frames. Feature points that are stably tracked for more than three consecutive frames are given a higher matching priority. In the next frame, the search is prioritized in the neighborhood around the predicted position of these feature points, improving matching speed and stability. A pre-trained bag-of-words model is used for loop closure detection. The bag-of-words dictionary is trained based on ORB features and contains 10,000 visual words. After extracting features from each frame, the corresponding bag-of-words vector is calculated and compared with the bag-of-words vectors of historical frames using cosine similarity. Scenes with a similarity greater than 0.8 are considered loop closure candidates. When a loop is detected, the feature matching result between the current frame and the corresponding historical frame is calculated to obtain the loop closure correction error. A pose graph optimization algorithm is used to evenly distribute the loop closure error across all frames of the motion sequence, correcting the cumulative drift error caused by long-term matching, and outputting a globally consistent sequence image feature matching result.

[0045] This embodiment synchronously performs a dynamic target perception matching and filtering step. A lightweight target detection model is used to identify dynamic moving targets such as vehicles and pedestrians in the image, generating a rectangular mask for the corresponding target region. All feature matching pairs within the mask region are removed, retaining only valid matching points from the static background region for subsequent pose calculation, thus avoiding interference from dynamic moving targets on matching accuracy. Simultaneously, a multi-scale hierarchical matching acceleration optimization strategy is employed, combined with a GPU parallel computing architecture to improve processing speed. The total time for feature matching per frame is less than 40 milliseconds, meeting the real-time requirements of in-vehicle applications.

[0046] Example 2: Implementation of Visual-Library Collaborative Feature Matching in Industrial Warehousing AGV Scenarios This embodiment applies to the environmental perception system of automated guided vehicles (AGVs) in industrial warehousing parks. The vehicle is equipped with a front-facing global shutter camera, a 2D LiDAR, and a low-cost MEMS inertial navigation system. The data processing unit is an embedded industrial controller with a built-in small GPU acceleration module. This embodiment achieves high-precision positioning and navigation of AGVs in indoor warehousing environments through feature matching that combines vision and LiDAR, making it suitable for industrial warehousing scenarios with dense shelving and large variations in lighting.

[0047] The system uses PPS pulse signals output from the inertial navigation system to achieve time synchronization between the camera and the LiDAR. The camera employs a global shutter mode to avoid motion blur, and each frame of LiDAR scan data is accompanied by a precise timestamp. The point cloud timestamps are aligned using linear interpolation based on the camera's exposure time, with the time synchronization error controlled within 5 microseconds. In the image preprocessing stage, image distortion is corrected based on calibrated intrinsic parameters, grayscale values ​​are normalized to a standard range, and image noise is removed using a 3×3 Gaussian kernel. Considering the large variations in lighting conditions in warehouse environments, a histogram equalization algorithm is used to enhance image contrast and improve the stability of feature extraction in low-light environments. In the point cloud preprocessing stage, statistical filtering is used to remove outlier noise points, and a voxel grid with a side length of 0.1 meters is used for downsampling to adapt to the high density of point clouds at close range indoors. Motion distortion of the point cloud is corrected using motion data output from the inertial navigation system. Due to the planar scanning characteristics of the 2D LiDAR, only horizontal motion distortion is corrected. After preprocessing, the two-dimensional laser point cloud is transformed to the camera coordinate system using a pre-calibrated extrinsic parameter matrix and projected onto the image plane to complete spatial alignment, with the alignment error controlled within 2 pixels.

[0048] The 2D image employs a multi-scale FAST corner detection algorithm, constructing a 6-layer Gaussian pyramid with a scaling factor of 0.8 for each layer. Approximately 150 stable corner points are extracted from each scale layer, resulting in about 800 feature points extracted from the entire image, generating a 256-bit BRIEF binary descriptor. Considering the rich planar features characteristic of warehouse scenes, 3D point cloud feature extraction focuses on extracting straight edge features and planar features. Hough transform is used to extract straight edge feature points from the 2D laser point cloud, while simultaneously extracting planar feature points corresponding to walls and the ground, resulting in approximately 600 feature points extracted per frame. After projecting these feature points onto the image plane, a depth-weighted descriptor vector is generated by combining neighborhood grayscale information, adapting to the feature distribution characteristics of indoor scenes.

[0049] The weighted matching cost of all feature pairs is calculated, resulting in approximately 500 initial matching pairs. The rigid body transformation matrix is ​​solved iteratively using a random sampling consensus algorithm, with 800 iterations. The criterion for determining interior points is a reprojection error of less than 2 pixels. After the iteration, outliers are removed, and approximately 320 interior points are retained to form the initial set of valid matching points. The initial transformation matrix is ​​then output.

[0050] A lightweight semantic segmentation network is used to perform pixel-level classification of warehouse scene images, dividing them into four semantic regions: shelves, floors, pallets, and walls. The inference speed per frame can reach 40 frames per second. Matching point pairs within the same semantic category are retained, while mismatched pairs across categories are removed. Feature points in the 6×6 neighborhood around each feature point are extracted to construct topological relationships. The neighborhood matching success rate is calculated as the topological consistency score. The combined confidence score is calculated by combining semantic consistency and topological consistency, with semantic weight accounting for 0.5 and topological weight accounting for 0.5. Matching point pairs with a confidence score higher than 0.75 are retained, ultimately yielding approximately 220 high-precision matching points.

[0051] By integrating visual 2D matching results and laser 3D matching results, a joint error function is constructed with a pixel error weight of 0.5 and a point cloud error weight of 0.5. The Levenberg-Marquardt algorithm is used for nonlinear optimization, with a maximum of 40 iterations and a convergence threshold of 1e-6. The relative pose parameters of adjacent frames of the AGV are output, and the planar positioning accuracy can reach the centimeter level, while the heading angle accuracy can reach the 0.2 degree level.

[0052] Setting matching priorities for continuously and stably tracked feature points improves matching efficiency. A bag-of-words model adapted to warehouse scenarios is used for loop closure detection. The dictionary is trained based on warehouse scene images and contains 8,000 visual words. After loop closure is detected, the cumulative error is corrected through pose graph optimization to ensure global consistency in long-distance navigation.

[0053] This embodiment synchronously executes a dynamic target matching and filtering step to identify dynamic targets such as workers and handling robots in the warehouse scene, generating masks to remove matching points in corresponding areas and avoid interference from dynamic targets. Employing a multi-scale hierarchical matching and GPU parallel acceleration strategy, the image is divided into 12 sub-regions for parallel processing, with a total matching time of less than 50 milliseconds per frame, meeting the real-time requirements of embedded platforms.

[0054] Reference Figure 2 This diagram provides an in-depth analysis of the synchronization, collaboration, and noise reduction alignment mechanisms of multi-source sensors during the front-end data access phase. After the microsecond-level synchronization trigger signal is sent, the image acquisition and laser scanning branches each enter independent data preprocessing pipelines. The vision branch focuses on eliminating physical distortions in the radial and tangential directions of the lens and uses Gaussian convolution kernels to filter out high-frequency thermal and salt-and-pepper noise; the laser branch focuses on statistical filtering to remove outlier noise points and uses cubic voxel rasterization to achieve lightweight point cloud dimensionality reduction. More importantly, to address the point cloud trailing distortion caused by the high-speed motion of the carrier, the system performs back-projection compensation by frequently acquiring and fusing inertial motion state data. Finally, the two cleaned data streams intersect based on the rigid body transformation matrix obtained through joint calibration, and the 3D point cloud is accurately projected onto the 2D imaging plane, completing the deep anchoring of the dual-modal underlying data.

[0055] Reference Figure 3 This diagram details the bimodal feature extraction and coarse-to-fine accelerated matching logic under a multi-level pyramid architecture. To meet the stringent real-time requirements of industrial scenarios, the system abandons flattened global search and instead constructs image Gaussian scaling pyramids with a set number of layers and point cloud resolution decreasing voxel pyramids. At the feature extraction level, the visual algorithm extracts corner points with scale and rotation invariance and generates binary descriptors, while the point cloud algorithm solves local curvature through principal component analysis to extract edge and planar structural features. In the matching computation domain, the system utilizes a multi-core parallel computing architecture at the bottom layer to divide independent computation sub-regions. A global coarse scan is performed at the low-resolution top layer to obtain initial transformation associations. Subsequently, this association result is used as a priori condition to be deposited layer by layer downwards, completing high-precision feature pairing of local neighborhoods at the high-resolution bottom layer, greatly reducing the system's memory usage and matching time.

[0056] Reference Figure 4This section focuses on the deep cleaning and multi-dimensional constraint purification process of the initial matching feature set. After the original coarse matching point pairs are input into the system, they first undergo bidirectional consistency verification, and unidirectionally isolated matching pairs are directly discarded. The system simultaneously introduces a dynamic target perception bypass, using a lightweight network to generate an inflated mask for moving targets, forcibly blocking the matching process of feature points within the mask coverage area, thus eliminating dynamic interference at the source. For the remaining point pairs in the static background, the system further superimposes semantic and topological dual cross-examination: on the one hand, it determines whether the matching pair crosses the boundaries of different semantic categories, and on the other hand, it evaluates the topological similarity of the feature network structure in its local neighborhood, calculating a comprehensive confidence score. Finally, the high-score matching subset is sent to the random sampling consensus algorithm, which strictly delineates the interior point boundaries using pixel reprojection error during the iterative convergence process, thoroughly eliminating abnormal exterior points that disrupt the consistency of rigid body motion.

[0057] Reference Figure 5 This figure illustrates the mathematical optimization solution and global spatiotemporal consistency guarantee mechanism for high-precision six-DOF pose estimation. The system performs heterogeneous data fusion between the selected 2D pixel correspondences and 3D spatial point correspondences to construct a composite joint objective function that simultaneously encompasses the pixel reprojection residual and the point cloud spatial registration residual. Under the constraints of a preset maximum number of iterations and convergence threshold, the nonlinear optimization solver continuously approximates and outputs the optimal rotation and translation matrices. To prevent track drift caused by long-distance operation, the system introduces a dynamic optimization and update strategy for feature templates in the temporal dimension and relies on the bag-of-words model to encode and verify the global descriptive features of each frame of image. Once a highly similar closed-loop candidate scene is retrieved from the historical database and geometric confirmation is completed, the backend graph optimization algorithm immediately intervenes, evenly distributing the local closed-loop error to the entire temporal trajectory network and outputting an absolutely consistent global matching topology.

[0058] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A visual-laser coordinated industrial image feature matching processing method, characterized in that, include: S1. Construct a multi-scale Gaussian pyramid for the image, extract binary features with scale invariance and rotation invariance, distinguish between two types of 3D feature points, namely plane and edge, based on the local geometric attributes of the point cloud, project the 3D features onto the unified image plane obtained by joint calibration, and generate a fusion feature description vector with 3D depth information as the weighting factor. S2. Initial matching is completed based on feature descriptor similarity. Bidirectional consistency check combined with rigid body transformation constraints is used to remove coarse matching outliers. Pixel-level semantic segmentation is performed on the images corresponding to the coarse matching points. The unified matching confidence is calculated by fusing semantic category constraints and feature neighborhood topology constraints. Matching pairs are filtered according to the confidence level, and matching pairs that meet the preset standards are retained. S3. The image pixel matching correspondence and the laser point cloud spatial matching correspondence are fused together. At the same time, pixel reprojection error and point cloud registration error are introduced. A joint error function with weights assigned to the two types of errors is constructed. The optimal rigid body transformation matrix of six degrees of freedom is solved in one step through nonlinear optimization to obtain the relative pose of a single frame. S4. Define a priority matching search neighborhood for features that are stably tracked across multiple frames; use a bag-of-words model to detect loop closures and perform geometric verification; optimize and amortize the loop closure correction error through pose graph to correct the long-term cumulative drift caused by cross-modal feature matching, and output a globally unified sequence matching result.

2. The visual-laser coordinated industrial image feature matching processing method according to claim 1, characterized in that, Before step S1, the following is included: The target detection algorithm identifies various dynamic moving targets in the image, and the target tracking algorithm generates a dynamic target mask between consecutive frames. After pixel dilation processing of the mask area, all feature points to be extracted within the mask coverage area are removed.

3. The visual laser-assisted industrial image feature matching processing method according to claim 1, characterized in that, Also includes: A multi-scale Gaussian pyramid for images and a multi-resolution voxel pyramid for point clouds are constructed respectively. A coarse-to-fine hierarchical matching strategy is adopted. At the low-resolution level, global matching and localization are completed to obtain a coarse transformation relationship. The transformation results are passed down layer by layer. At the high-resolution level, local neighborhood feature matching is completed based on prior transformation. At the same time, a parallel computing architecture is adopted, dividing the image into multiple sub-regions and allocating independent computing threads to process the feature matching task of each sub-region in parallel.

4. The visual-laser coordinated industrial image feature matching processing method according to claim 1, characterized in that, The formula for calculating the matching cost of the fused feature description vector in step S1 is as follows: Where D is the matching cost between the two sets of fused feature descriptors; N is the total number of bits in the descriptor; Let be the binary value of the i-th bit of the first feature descriptor; The second feature descriptor is the binary value of the i-th bit; is the depth weight coefficient for the i-th position.

5. The visual laser-assisted industrial image feature matching processing method according to claim 1, characterized in that, In step S2, the bidirectional consistency check is a bidirectional mutual matching of image and point cloud features, and the matching pairs that only match in one direction are eliminated; the random sampling consistency algorithm is used to solve the rigid body transformation, and four pairs of non-collinear three-dimensional matching points are extracted in each iteration. Matching points with a pixel reprojection error of less than 3 pixels are determined as interior points. Interior points are filtered and geometric exterior points are eliminated to obtain a coarse matching point set.

6. The visual laser-assisted industrial image feature matching processing method according to claim 1, characterized in that, In step S2, the unified matching confidence score is fused with two metrics: semantic consistency and neighborhood topological consistency. The corresponding calculation formula is as follows: Where C is the confidence score of the matching point pair; λ is the balancing weight coefficient; S is the semantic category consistency score; and T is the neighborhood topology consistency score.

7. The visual-laser coordinated industrial image feature matching processing method according to claim 1, characterized in that, In step S3, the Levenberg-Marquardt algorithm is used for iterative solution. The error convergence threshold is set to 1e-6 and the upper limit of the number of iterations is 50, so that the joint error function takes the minimum value. The six degrees of freedom parameters consisting of three-dimensional rotation and three-dimensional translation are used as optimization variables. After the iteration converges, the optimal rigid body transformation matrix is ​​output.

8. The visual-laser coordinated industrial image feature matching processing method according to claim 1, characterized in that, The loop closure detection and correction process in step S4 is as follows: extract the image features of each frame to generate corresponding bag-of-words vectors, compare the similarity between the bag-of-words vector of the current frame and the bag-of-words vectors of all historical frames, and select historical frames with a similarity higher than 0.7 as candidate loop closure frames; use rigid body transformation geometric constraints to verify the effectiveness of candidate loop closures; solve the loop closure correction error based on the matching point pairs of effective loops, and distribute the correction error to all historical frames through pose graph optimization to uniformly eliminate the long-term cross-modal matching cumulative drift.

Images