A large road maintenance machine all-around splicing method and system

Abstract

The application discloses a kind of large-scale road maintenance machinery all-around view splicing method and system.The method comprises the following steps: establishing multiple sensor unit groups, each group comprising a camera and a structured light sensor, the internal relative pose is fixed, and the field of view of adjacent groups overlaps to form a common view area;Respectively extract feature points from the RGB images and depth images collected by each group to obtain RGB feature point sets and depth feature point sets;Respectively filter bad points to obtain optimized feature point sets;Match the feature points of adjacent groups located in the common view area, estimate the relative pose between the groups, obtain the initial mapping matrix of RGB and the initial mapping matrix of depth;Construct an evaluation mechanism and iteratively optimize the feature point sets and initial mapping matrix based on it, and converge the target mapping matrix used for all-around view image splicing.The application realizes self-adaptive all-around view splicing without relying on calibration cloth, has strong anti-shock interference capability, and has high splicing precision.

Description

Technical Field

[0001] This invention belongs to the field of panoramic monitoring technology for large-scale road maintenance machinery, specifically involving a panoramic stitching method and system for large-scale road maintenance machinery based on multi-sensor fusion, so as to achieve adaptive panoramic image stitching that does not rely on fixed calibration objects. Background Technology

[0002] Traditional vehicle surround-view calibration requires placing calibration cloths and sequentially mapping the world coordinate system of the surrounding calibration cloths to the image coordinate system to obtain a fixed BEV mapping matrix for each camera, thus achieving multi-camera surround-view stitching. For large road maintenance machinery (such as tamping machines), the vehicle body is much larger than that of ordinary vehicles, making accurate measurement using calibration cloths more time-consuming; moreover, the amplitude of vehicle body vibration is much greater than that of ordinary vehicles, making it difficult to maintain the relative pose between cameras, requiring fine-tuning of the mapping matrix after the operation is completed.

[0003] In the existing technology, there is a lack of an adaptive, calibration cloth-independent panoramic calibration method that can automatically restore the relative pose between cameras under the vibration environment of large road maintenance machinery and achieve accurate stitching of panoramic images.

[0004] Therefore, providing a method and system for panoramic splicing of large-scale road maintenance machinery that can achieve adaptive calibration, does not rely on fixed calibration objects, and is resistant to vibration interference is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for panoramic stitching of large-scale road maintenance machinery based on multi-sensor fusion, thereby overcoming the technical defects of existing technologies, such as reliance on calibration cloth, inability to adapt, and significant impact from vibration.

[0006] This invention constructs a sensor unit group consisting of a camera and a structured light sensor, and utilizes feature point fusion matching between RGB images and depth images to achieve adaptive estimation of relative pose that does not depend on a calibration cloth.

[0007] The system adopts a modular architecture, consisting of multiple sensor unit groups, feature extraction module, bad pixel filtering module, pose estimation module, and iterative optimization module. The modules work together to form a complete closed-loop system.

[0008] Specifically, this invention establishes multiple sensor unit groups, each including a camera and a structured light sensor. The relative poses between the camera and the structured light sensor are pre-fixed, and adjacent sensor unit groups share a common field of view. When the large track maintenance machinery stops operating, RGB and depth image data from each group are collected to obtain the SIFT feature point set of the RGB image and the ORB feature point set of the depth image. For bad points affected by lighting and occasional objects moving relative to the vehicle, a filtering strategy is designed to ensure a sufficient number of inliers remain. The large track maintenance machinery is moved approximately 5 meters along the track, and the above process is repeated to obtain multiple feature point sets, constructing epipolar constraints. Bad points are filtered from the feature point sets through cross-validation and a screening strategy, forming an optimized feature point set. The relative pose mapping matrix of adjacent sensor groups is estimated using the optimized feature point set, and an evaluation mechanism is constructed for iterative processing, ultimately converging to the optimal mapping matrix. Particularly noteworthy is the re-optimization process of the composite mapping matrix integrating the SIFT feature point set, the ORB feature point set, and the module's prior knowledge.

[0009] Thus, this invention achieves a complete closed loop from sensor deployment to feature extraction to pose estimation and then to mapping matrix optimization, significantly improving the adaptive capability and anti-vibration interference capability of large-scale road maintenance machinery surround view splicing.

[0010] According to one aspect of the present invention, a method for assembling a panoramic view of large road maintenance machinery is provided, comprising the following steps: Multiple sensor unit groups are established, each sensor unit group including a camera and a structured light sensor, and the relative poses between the camera and the structured light sensor in all sensor unit groups are pre-fixed. The fields of view of adjacent sensor unit groups partially overlap to form a common field of view for subsequent feature point matching. Feature points are extracted from the RGB images and depth images acquired by each of the sensor unit groups to obtain RGB feature point sets and depth feature point sets; Bad point filtering is performed on the RGB feature point set and the depth feature point set respectively to obtain the optimized RGB feature point set and the optimized depth feature point set; Based on the optimized RGB feature point set and the optimized depth feature point set, feature points located within the common viewing area between adjacent sensor unit groups are matched, and the inter-group relative pose between adjacent sensor unit groups is estimated to obtain the RGB initial mapping matrix and the depth initial mapping matrix. An evaluation mechanism is constructed, and the RGB feature point set, the depth feature point set, the RGB initial mapping matrix, and the depth initial mapping matrix are iteratively optimized based on the evaluation mechanism to converge the target mapping matrix for surround view image stitching.

[0011] Another aspect of the present invention provides a large-scale road maintenance machinery surround view splicing system, comprising: Multiple sensor unit groups, each sensor unit group including a camera and a structured light sensor, and the relative poses between the camera and the structured light sensor in all sensor unit groups are pre-fixed. The fields of view of adjacent sensor unit groups partially overlap to form a common field of view for subsequent feature point matching. The feature extraction module is used to extract feature points from the RGB images and depth images acquired by each of the sensor unit groups, and obtain the RGB feature point set and the depth feature point set. The bad pixel filtering module is used to perform bad pixel filtering on the RGB feature point set and the depth feature point set respectively, so as to obtain the optimized RGB feature point set and the optimized depth feature point set; The pose estimation module is used to match the feature points located in the common field of view between adjacent sensor unit groups based on the optimized RGB feature point set and the optimized depth feature point set, and to estimate the inter-group relative pose between adjacent sensor unit groups to obtain the RGB initial mapping matrix and the depth initial mapping matrix. An iterative optimization module is used to construct an evaluation mechanism and iteratively optimize the RGB feature point set, the depth feature point set, the RGB initial mapping matrix, and the depth initial mapping matrix based on the evaluation mechanism to converge the target mapping matrix for surround view image stitching.

[0012] Compared with the prior art, the present invention has the following beneficial effects: No need for calibration cloth: By fusing and matching feature points of RGB images and depth images, adaptive relative pose estimation is achieved without the need to place a calibration cloth, which greatly reduces deployment time and cost.

[0013] Vibration resistance: By using cross-validation of structured light depth maps and RGB images, the relative pose between cameras can still be accurately estimated even under the vibration environment of large track maintenance machinery, and there is no need to readjust the mapping matrix after the operation is completed.

[0014] High stitching accuracy: By constructing an evaluation mechanism to iteratively optimize feature point selection and pose estimation, a high-precision target mapping matrix is ​​finally converged, ensuring the stitching quality of the panoramic images. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the overall architecture of the large-scale road maintenance machinery surround view splicing system provided in this embodiment of the invention; Figure 2 This is a flowchart of the panoramic splicing method for large road maintenance machinery provided in this embodiment of the invention; Figure 3 This is a flowchart of the bad pixel filtering process provided in an embodiment of the present invention; Figure 4 This is an iterative optimization flowchart provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The embodiments of this invention are written in a progressive manner.

[0019] Terminology Explanation To facilitate understanding of the technical solution of this invention, some technical terms appearing in the specification are explained as follows: SIFT (Scale-Invariant Feature Transform) is a computer vision algorithm used to detect and describe local features of images. It is invariant to changes in image scale, rotation, and illumination, and is often used for image matching and object recognition.

[0020] ORB (Oriented FAST and Rotated BRIEF) is an image feature extraction algorithm that combines FAST corner detection and BRIEF feature description. It features fast computation speed and good rotation invariance, making it particularly suitable for scenarios with high real-time requirements.

[0021] Ransac (Random Sample Consensus) is an iterative algorithm that estimates mathematical model parameters through random sampling and model validation. It can effectively remove outliers (outliers) in the data and is widely used for removing mismatched points in image matching.

[0022] Hungarian matching is a combinatorial optimization algorithm used to solve the maximum matching problem in bipartite graphs. In this invention, it is used to match the set of feature points collected at adjacent time points and establish the correspondence between the feature points.

[0023] Reprojection error: refers to the pixel distance between the projected point of a three-dimensional point onto a two-dimensional image plane and the actual detected feature point. It is used to evaluate the accuracy of the mapping matrix. The smaller the error, the more accurate the mapping.

[0024] Bundle Adjustment is a global optimization algorithm that minimizes reprojection error by simultaneously optimizing 3D point coordinates and camera parameters, significantly improving multi-view performance. Figure 3 The accuracy of dimensional reconstruction and camera pose estimation.

[0025] The Levenberg-Marquardt algorithm is a nonlinear least squares optimization algorithm that combines the advantages of the Gauss-Newton method and gradient descent. It features fast convergence and good robustness and is widely used in parameter estimation and curve fitting problems.

[0026] The terms used above are only for explaining specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art will understand that other existing technologies with the same or similar functions can be employed without departing from the concept of the present invention.

[0027] Example 1: System Architecture like Figure 1 As shown, this invention provides a large-scale road maintenance machinery surround view stitching system. Its core lies in constructing a sensor unit group consisting of a "camera + structured light sensor," utilizing feature point fusion matching between RGB and depth images to achieve adaptive relative pose estimation independent of the calibration fabric. The system adopts a modular architecture, composed of multiple sensor unit groups, a feature extraction module, a bad pixel filtering module, a pose estimation module, and an iterative optimization module, working collaboratively to achieve a complete closed loop from sensor deployment to target mapping matrix output.

[0028] Specifically, the system includes the following functional modules: Multiple sensor unit groups are used, each including a camera and a structured light sensor. The relative poses of the camera and the structured light sensor in all sensor unit groups are pre-fixed, and the fields of view of adjacent sensor unit groups partially overlap to form a common viewing area for subsequent feature point matching. The sensor unit groups are deployed on the crossbeam of a large track maintenance machine to ensure 360-degree field of view coverage. The camera and structured light sensor are installed according to the principle of parallel optical axes and minimal distance between optical axes. The camera is an industrial-grade global shutter RGB camera with a resolution of no less than 1920×1080 and a frame rate of no less than 30fps; the structured light sensor is an active depth camera with a depth resolution of no less than 640×480 and a depth detection range covering 0.5m~10m to ensure complete acquisition of depth information of the environment around the tamping machine. The baseline distance between the camera and the structured light sensor is controlled within the range of 10cm~30cm to ensure depth measurement accuracy while maintaining a compact overall installation size.

[0029] The feature extraction module is used to extract feature points from the RGB images and depth images acquired by each of the sensor unit groups, respectively, to obtain an RGB feature point set and a depth feature point set. Specifically, SIFT feature points are extracted from the RGB images, and the SIFT feature point set is used as the RGB feature point set; ORB feature points are extracted from the depth images, and the ORB feature point set is used as the depth feature point set.

[0030] The bad pixel filtering module is used to filter bad pixels from the RGB feature point set and the depth feature point set respectively, resulting in optimized RGB feature point sets and optimized depth feature point sets. This module employs a cross-validation strategy, performing Hungarian matching between SIFT feature point sets acquired at adjacent time points and between ORB feature point sets acquired at adjacent time points, and constructs an energy function as the cost evaluation function for the Ransac algorithm to filter interior points.

[0031] The pose estimation module is used to match the feature points located in the common viewing area between adjacent sensor unit groups based on the optimized RGB feature point set and the optimized depth feature point set, and to estimate the inter-group relative pose between adjacent sensor unit groups to obtain the RGB initial mapping matrix and the depth initial mapping matrix.

[0032] An iterative optimization module is used to construct an evaluation mechanism and iteratively optimize the RGB feature point set, the depth feature point set, the RGB initial mapping matrix, and the depth initial mapping matrix based on the evaluation mechanism to converge the target mapping matrix for surround view image stitching.

[0033] The five functional modules described above work together to form a complete closed-loop system. After the sensor unit group completes data acquisition, the feature extraction module extracts feature points; the bad point filtering module removes invalid feature points; the pose estimation module calculates the initial mapping matrix; and the iterative optimization module converges to the target mapping matrix through an evaluation mechanism. Since the relative poses of the cameras and structured light within the n modules are fixed, the prior fixed mapping relationship from ORB feature points to SIFT feature points can be expressed as follows: This is used for subsequent cross-validation optimization.

[0034] Example 2: General Process like Figure 2 As shown, the present invention provides a method for panoramic splicing of large road maintenance machinery, comprising the following steps S1 to S5: S1: Sensor Unit Group Establishment Multiple sensor unit groups are established, each including a camera and a structured light sensor. The relative poses of the camera and the structured light sensor in all sensor unit groups are pre-fixed, and the fields of view of adjacent sensor unit groups partially overlap to form a common viewing area for subsequent feature point matching. The sensor unit groups are deployed at the crossbeams of large road maintenance machinery to ensure 360-degree field of view coverage.

[0035] When large road maintenance machinery stops operating, RGB image data and depth map data of each group are collected to obtain the set of SIFT feature points of the RGB images. ORB feature point set of depth image The large track maintenance machinery was moved approximately 5 meters along the track and then stopped. This 5-meter distance was chosen based on the following considerations: it comprehensively considered the effective panoramic coverage required for image stitching and the camera's field of view (FOV), ensuring sufficient common viewing area between the current and previous frames after the machinery's movement. This means that some feature points from the previous frame can still be observed in the current frame, thus meeting the data association requirements for subsequent feature point matching and pose estimation. This process was repeated to obtain multiple sets of feature point arrays. and .

[0036] S2: Feature Point Extraction Feature points are extracted from the RGB and depth images acquired by each of the sensor unit groups to obtain RGB feature point sets and depth feature point sets. Specifically, SIFT feature points are extracted from the RGB images, and the SIFT feature point set is used as the RGB feature point set; ORB feature points are extracted from the depth images, and the ORB feature point set is used as the depth feature point set.

[0037] The adjacent points in the Dlist set are sequentially filtered using the Hungarian matching method. The initial matching point inside is denoted as and Initial matching point inside This corresponds to the initial mapping matrix H.

[0038] S3: Dead Pixel Filtering Bad point filtering is performed on the RGB feature point set and the depth feature point set respectively to obtain the optimized RGB feature point set and the optimized depth feature point set.

[0039] like Figure 3 As shown, bad pixel filtering specifically includes: A1. Using a cross-validation strategy, Hungarian matching is performed between the SIFT feature point sets collected at adjacent time points to obtain initial SIFT matching points, and Hungarian matching is performed between the ORB feature point sets collected at adjacent time points to obtain initial ORB matching points. The specific implementation of the Hungarian matching is as follows: A Euclidean distance cost matrix is ​​constructed between feature point descriptors. With the goal of minimizing the overall matching distance, the optimal bipartite graph matching is solved using the Hungarian algorithm, thereby establishing a one-to-one correspondence between feature points at adjacent time points. During the matching process, a distance threshold is set to eliminate mismatched point pairs with excessively large distances, retaining matching points that meet the threshold condition as initial matching points. Simultaneously, a bidirectional consistency check is adopted, meaning that a matching point pair is retained as a valid match only if the matching result from set A to set B is consistent with the matching result from set B to set A, further improving matching accuracy.

[0040] A2. Construct an energy function as the cost evaluation function for the Ransac algorithm, and perform interior point filtering on the initial SIFT matching points and the initial ORB matching points respectively to obtain the optimized SIFT feature point set and the optimized ORB feature point set.

[0041] The energy function is used to efficiently evaluate the effectiveness of initial matching points. This energy function, used as the cost evaluation in Ransac, can be used to filter out interior points. It is expressed as: ; in Let the coordinates be the points. The labels for the points are {0,1}, where 1 indicates an interior point and 0 indicates an exterior point; Represented as a mapping matrix The reprojection error with the current point is calculated; based on this, a smoothing factor is added to ensure the continuity of labels between adjacent points. .

[0042] The corresponding Ransac cost function is: ; in, express and Adjacent.

[0043] Use the Ransac process to select each set one by one. dot matrix and A lattice can be used to obtain a new lattice set with minimal reprojection error. , .

[0044] S4: Pose Estimation and Initial Mapping Matrix Based on the optimized RGB feature point set and the optimized depth feature point set, feature points located within the common viewing area between adjacent sensor unit groups are matched, and the inter-group relative pose between adjacent sensor unit groups is estimated to obtain the RGB initial mapping matrix and the depth initial mapping matrix.

[0045] because and This data was obtained during the movement process; here, it is extracted sequentially from each module. and The dot matrix ensures sequential correspondence; for all modules, the set of dot matrices for the filtered SIFT features... and the set of points for ORB features. , where n represents the number of modules and i represents the number of photos taken during the motion process.

[0046] For SIFT lattices of adjacent modules, due to the shared viewing area, the Ransac matching point search method also applies. Sampling is performed between vertically adjacent elements to select matching point pairs, denoted as Similarly, the ORB matrix of adjacent modules has matching results from vertically adjacent modules. .

[0047] Specifically, the optimized SIFT feature point sets of adjacent sensor unit groups are matched to obtain SIFT matching point pairs, and the RGB initial mapping matrix is ​​calculated based on the SIFT matching point pairs; the optimized ORB feature point sets of adjacent sensor unit groups are matched to obtain ORB matching point pairs, and the depth initial mapping matrix is ​​calculated based on the ORB matching point pairs.

[0048] S5: Iterative Optimization An evaluation mechanism is constructed, and based on this mechanism, the RGB feature point set, the depth feature point set, the RGB initial mapping matrix, and the depth initial mapping matrix are iteratively optimized to converge the target mapping matrix for surround view image stitching. The evaluation mechanism uses the reprojection error loss function (Loss) as the core evaluation index. It determines the quality of the current mapping matrix by comprehensively evaluating the magnitude of the reprojection error between adjacent groups of feature points, the proportion of inliers, and the consistency of prior mapping relationships. During the iterative optimization process, the optimal mapping matrix parameters are gradually selected, guided by minimizing the loss function value.

[0049] like Figure 4 As shown, iterative optimization specifically includes: B1. By utilizing the pre-fixed relative poses between the camera and the structured light sensor within the same sensor unit group, a priori mapping relationship between the depth feature point set and the RGB feature point set is constructed. B2. Based on the prior mapping relationship, convert the initial depth mapping matrix into the desired RGB mapping matrix; B3. Construct a reprojection error loss function, and use the desired RGB mapping matrix as a reference to verify and optimize the initial RGB mapping matrix.

[0050] for and Each element can be computed with its own internal concatenation mapping matrix, denoted as . and In theory there is , here will Let be the desired internal splicing mapping matrix. For the internal stitching mapping matrix of the observations, a reprojection error loss can be constructed here: ; The cutoff condition for the Ransac method is used to extract all concatenated interior points that satisfy the cutoff condition in round i, and to obtain the corresponding concatenation mapping matrix. .

[0051] A global optimization process based on MLE is constructed. Since image stitching of multiple modules is ultimately involved, bundle adjustment (nonlinear least squares) is used to minimize the reprojection error of feature points in all images, thereby improving the global consistency and visual quality of the stitching. For the above process, all stitching mapping matrices satisfying the ransac process termination condition are obtained. , None of these may be the global optimal solution, but they can be used as initial seed points for the gradient descent process. The corresponding objective function is: ; This process uses the Levenberg-Marquardt method to solve for... and The incremental BA optimization solution.

[0052] Iterative optimization also includes: employing a bundle adjustment method, using the verified and optimized RGB initial mapping matrix and depth initial mapping matrix as initial seed points, and solving for the globally optimal target mapping matrix using the Levenberg-Marquardt algorithm. The convergence condition is set as follows: when the change in the loss function value between two consecutive iterations is less than a preset threshold... ( The value is 1× When the number of iterations reaches the preset maximum number of iterations (set to 100), the optimization process is determined to have converged, the iteration is stopped and the current target mapping matrix is ​​output; at the same time, when the average reprojection error of all feature points is less than 0.5 pixels, it is also considered to have met the convergence condition.

[0053] Example 3: Application of tamping machine panoramic splicing Taking a tamping machine (a typical example of large-scale road maintenance machinery) as an example, this system and method are applied to the stitching of panoramic images before and after the operation of large-scale road maintenance machinery.

[0054] After large road maintenance machinery completes its work, the vehicle body may vibrate, causing changes in the relative pose between the cameras. At this point, the system initiates the adaptive calibration process: The sensor unit groups are deployed at the crossbeams of large road maintenance machinery. A total of 4 sensor unit groups are deployed, covering the front, rear, left and right directions of the vehicle body respectively. The fields of view of adjacent unit groups partially overlap, forming a common field of view area.

[0055] After the large track maintenance machinery stops operating, each sensor unit group simultaneously acquires RGB and depth images. The feature extraction module extracts SIFT feature points from the RGB images and ORB feature points from the depth images. The large track maintenance machinery moves approximately 5 meters along the track (this distance is selected based on a comprehensive consideration of the coverage required for panoramic stitching and the camera's field of view to ensure sufficient shared feature area between adjacent acquisition positions), repeatedly acquiring multiple sets of data.

[0056] The bad point filtering module adopts a cross-validation strategy, performing Hungarian matching between SIFT feature point sets collected at adjacent time points and between ORB feature point sets collected at adjacent time points. An energy function is constructed as the cost evaluation function of the Ransac algorithm to filter interior points, resulting in an optimized feature point set.

[0057] The pose estimation module matches feature points within the common viewing area between adjacent unit groups based on the optimized feature point set, estimates the inter-group relative pose between adjacent unit groups, and obtains the RGB initial mapping matrix and the depth initial mapping matrix.

[0058] The iterative optimization module utilizes the pre-fixed relative poses between the camera and the structured light sensor within the same unit group to construct a priori mapping relationship, verifies and optimizes the initial RGB mapping matrix, and then solves for the globally optimal target mapping matrix through beam adjustment and the Levenberg-Marquardt algorithm.

[0059] Ultimately, the system uses the converged target mapping matrix to complete the surround view image stitching without needing to rearrange the calibration cloth, thus achieving adaptive and vibration-resistant surround view stitching.

[0060] The specific fusion method for panoramic image stitching is as follows: First, based on the intrinsic parameter matrix calibrated by each camera and the converged target mapping matrix, the RGB images captured by each camera are projected onto a unified BEV (bird's-eye view) plane coordinate system to form multiple local top views; Second, within the common viewing area of ​​each adjacent image, a fade-in / fade-out weighted fusion algorithm is used for image fusion. The fusion weight is dynamically calculated based on the distance from the pixel to the boundary of the overlapping area to smoothly eliminate stitching seams; Then, for the fused panoramic image, a multi-band fusion algorithm is used to perform consistency correction on global brightness and color to ensure that the brightness and hue of each area of ​​the panoramic image are coordinated and unified; Finally, the fused image is edge-cropped to output the final complete 360-degree panoramic image.

[0061] This invention is applicable not only to large road maintenance machinery, but also to surround view monitoring systems for other large engineering vehicles, and has broad application prospects.

[0062] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

[0063] If flowcharts are used in this application, they are used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these flows, or one or more steps can be removed from these flows.

[0064] The foregoing provides a detailed description of a method and system for panoramic stitching of large-scale road maintenance machinery. The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for assembling panoramic views of large road maintenance machinery, characterized in that, Includes the following steps: Multiple sensor unit groups are established, each sensor unit group including a camera and a structured light sensor, and the relative poses between the camera and the structured light sensor in all sensor unit groups are pre-fixed. The fields of view of adjacent sensor unit groups partially overlap to form a common field of view for subsequent feature point matching. Feature points are extracted from the RGB images and depth images acquired by each of the sensor unit groups to obtain RGB feature point sets and depth feature point sets; Bad point filtering is performed on the RGB feature point set and the depth feature point set respectively to obtain the optimized RGB feature point set and the optimized depth feature point set; Based on the optimized RGB feature point set and the optimized depth feature point set, feature points located within the common viewing area between adjacent sensor unit groups are matched, and the inter-group relative pose between adjacent sensor unit groups is estimated to obtain the RGB initial mapping matrix and the depth initial mapping matrix. Using the pre-fixed relative poses between the camera and the structured light sensor within the same sensor unit group, a priori mapping relationship between the depth feature point set and the RGB feature point set is constructed; based on the priori mapping relationship, the initial depth mapping matrix is ​​converted into the desired RGB mapping matrix; An evaluation mechanism is constructed, which determines the quality of the mapping matrix by comprehensively evaluating the reprojection error of feature points between adjacent groups, the proportion of interior points, and the consistency of the prior mapping relationship. Based on the evaluation mechanism, the initial RGB mapping matrix is ​​verified and optimized with the expected RGB mapping matrix as a reference. The convergence condition for iterative optimization is: the change in the loss function value between two consecutive iterations is less than 1× Or the number of iterations reaches 100; or the average reprojection error of all feature points is less than 0.5 pixels; Using the verified and optimized RGB initial mapping matrix and depth initial mapping matrix as initial seed points, the globally optimal target mapping matrix is ​​solved by the bundle adjustment method.

2. The method for panoramic splicing of large road maintenance machinery according to claim 1, characterized in that, The step of extracting feature points from the RGB images and depth images acquired by each of the sensor unit groups to obtain an RGB feature point set and a depth feature point set includes the following steps: extracting SIFT feature points from the RGB images and using the SIFT feature point set as the RGB feature point set; ORB feature points are extracted from the depth image, and the set of ORB feature points is used as the set of depth feature points.

3. The method for panoramic splicing of large road maintenance machinery according to claim 2, characterized in that, The step of filtering out bad pixels from the RGB feature point set and the depth feature point set to obtain the optimized RGB feature point set and the optimized depth feature point set includes the following steps: A cross-validation strategy is adopted to perform Hungarian matching between the SIFT feature point sets collected at adjacent time points to obtain the initial SIFT matching points, and to perform Hungarian matching between the ORB feature point sets collected at adjacent time points to obtain the initial ORB matching points. An energy function is constructed as the cost evaluation function of the Ransac algorithm. Interior points are filtered for the initial SIFT matching points and the initial ORB matching points respectively to obtain the optimized SIFT feature point set and the optimized ORB feature point set.

4. The method for panoramic splicing of large-scale road maintenance machinery according to claim 3, characterized in that, The expression for the energy function is: ； in, The coordinates of the feature point, These are the labels for feature points, where 1 represents an interior point and 0 represents an exterior point. This represents the reprojection error between the mapping matrix and the current feature point. Indicates pairs of adjacent feature points. Indicates the distance between adjacent feature points. This is a smoothing factor.

5. The method for panoramic splicing of large-scale road maintenance machinery according to claim 3, characterized in that, The matching of feature points located within the shared field of view between adjacent sensor unit groups includes the following steps: The optimized SIFT feature point sets of adjacent sensor unit groups are matched to obtain SIFT matching point pairs; The optimized ORB feature point set of adjacent sensor unit groups is matched to obtain ORB matching point pairs.

6. The method for panoramic splicing of large road maintenance machinery according to claim 5, characterized in that, The estimation of the inter-group relative pose includes the following steps: Calculate the initial RGB mapping matrix based on the SIFT matching point pairs; Calculate the initial depth mapping matrix based on the ORB matching point pairs.

7. The method for panoramic splicing of large road maintenance machinery according to claim 1, characterized in that, The expression for the reprojection error loss function is: ； in, The observed RGB initial mapping matrix, The desired RGB mapping matrix is ​​obtained by transformation based on the prior mapping relationship.

8. The method for panoramic splicing of large road maintenance machinery according to claim 1, characterized in that, The iterative optimization of the RGB feature point set, the depth feature point set, the RGB initial mapping matrix, and the depth initial mapping matrix based on the evaluation mechanism to converge the target mapping matrix for surround view image stitching also includes the following steps: The beam adjustment method is adopted, using the verified and optimized RGB initial mapping matrix and depth initial mapping matrix as initial seed points, and the globally optimal target mapping matrix is ​​solved by the Levenberg-Marquardt algorithm.

9. A large-scale road maintenance machinery surround view splicing system, characterized in that, include: Multiple sensor unit groups, each sensor unit group including a camera and a structured light sensor, and the relative poses between the camera and the structured light sensor in all sensor unit groups are pre-fixed. The fields of view of adjacent sensor unit groups partially overlap to form a common field of view for subsequent feature point matching. The feature extraction module is used to extract feature points from the RGB images and depth images acquired by each of the sensor unit groups, and obtain the RGB feature point set and the depth feature point set. The bad pixel filtering module is used to perform bad pixel filtering on the RGB feature point set and the depth feature point set respectively, so as to obtain the optimized RGB feature point set and the optimized depth feature point set; The pose estimation module is used to match the feature points located in the common field of view between adjacent sensor unit groups based on the optimized RGB feature point set and the optimized depth feature point set, and to estimate the inter-group relative pose between adjacent sensor unit groups to obtain the RGB initial mapping matrix and the depth initial mapping matrix. An iterative optimization module is used to construct a priori mapping relationship between the depth feature point set and the RGB feature point set using the pre-fixed relative poses between the camera and the structured light sensor within the same sensor unit group. Based on this priori mapping relationship, the initial depth mapping matrix is ​​converted into the desired RGB mapping matrix. An evaluation mechanism is constructed, which determines the quality of the mapping matrix by comprehensively evaluating the reprojection error of feature points between adjacent groups, the proportion of interior points, and the consistency of the priori mapping relationship. Based on this evaluation mechanism, the initial RGB mapping matrix is ​​verified and optimized using the desired RGB mapping matrix as a reference. The convergence condition for iterative optimization is that the change in the loss function value between two adjacent iterations is less than 1× If the iteration count reaches 100 times, or the average reprojection error of all feature points is less than 0.5 pixels; the RGB initial mapping matrix and the depth initial mapping matrix after verification and optimization are used as initial seed points, and the globally optimal target mapping matrix is ​​solved by the bundle adjustment method.

Images