Self-calibration method, device and equipment of ray multi-source sensor and storage medium

Abstract

This application discloses a self-calibration method, device, equipment, and storage medium for radar-visual multi-source sensors, relating to the field of extrinsic parameter calibration technology. The method includes: acquiring lidar point cloud data collected by a vehicle-mounted mobile inspection platform during inspection, as well as road images captured by a distributed multi-camera array; extracting lidar point cloud features and image semantic features corresponding to calibration reference objects in the inspected road scene; constructing multi-dimensional decoupled geometric constraints; optimizing the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, obtaining a benchmark rigid extrinsic parameter matrix; defining a time-varying extrinsic parameter function based on the benchmark rigid extrinsic parameter matrix and a micro-deformation compensation matrix, maintaining a variable-length sliding time window, optimizing the micro-deformation compensation matrix of the time-varying extrinsic parameter function within the window, and outputting a corrected extrinsic parameter matrix. This application improves the accuracy of radar-visual data fusion.

Description

Technical Field

[0001] This application relates to the field of external parameter calibration technology, and in particular to a self-calibration method, apparatus, equipment and storage medium for a multi-source sensor. Background Technology

[0002] With the rapid growth in demand for digital operation and maintenance of Intelligent Transportation Systems (ITS) and road infrastructure, vehicle-mounted mobile inspection systems based on multi-source sensor fusion have become core equipment for detecting road defects (such as cracks, potholes, and ruts) and conducting road asset surveys. These inspection systems typically employ a "one master, many slaves" sensor layout, where a high-precision LiDAR is mounted on the roof beam as a spatial reference, and multiple industrial cameras (covering different perspectives such as forward, side, and downward views) are distributed throughout the system to acquire high-resolution texture information.

[0003] In practical operations, to accurately correspond the 2D image defect information captured by the camera with the 3D spatial position acquired by the lidar (i.e., to achieve RGB-D fusion), it is essential to precisely determine the rotation and translation relationship of the camera relative to the lidar (i.e., the extrinsic parameter matrix). However, the road inspection environment is complex, with vehicles constantly subjected to high-speed driving and vibration. The sensor extrinsic parameters are not static, and any minute calibration error can lead to misalignment in multi-source data fusion, directly affecting the accuracy of millimeter-level defect measurement and the pass rate of project acceptance.

[0004] In related technologies, multi-sensor extrinsic parameter calibration typically employs either offline calibration or online calibration methods based on lane line planar features. The former usually involves collecting data in a stationary state using a checkerboard, corner reflectors, or specific calibration boards, either manually or through automated control, before the vehicle leaves the factory or during routine maintenance, and then solving for the extrinsic parameters using the Perspective-n-Point (PnP) algorithm. The latter utilizes common road features such as lane lines and ground markings, matching lane line intensity information from laser point clouds with lane line edges in images to construct optimization equations and update the extrinsic parameters online. However, the above methods still have significant shortcomings: offline calibration can only reflect the static parameters at the time of calibration, and cannot adapt to the continuous vibration and environmental changes of the inspection vehicle in actual operation. It also requires a special site and manual operation, resulting in high maintenance costs. On the other hand, online calibration methods based on lane line features are difficult to effectively constrain the degrees of freedom such as pitch angle, roll angle, and vertical displacement. Once the vehicle experiences pitch and bumps, the calibration algorithm is prone to getting trapped in local optima, leading to an exponential amplification of long-distance projection errors. The algorithm is also prone to failure in road environments where the lane markings are worn or missing. In addition, current solutions usually assume that the sensor system is a rigid structure, which makes it difficult to compensate for the millimeter-level non-rigid deflection deformation that the distributed cameras at both ends of the vehicle's crossbeam will produce relative to the central radar under dynamic loads, thus affecting the multi-source data fusion and high-precision road defect measurement results. Summary of the Invention

[0005] The main purpose of this application is to provide a self-calibration method, apparatus, device and storage medium for a multi-source sensor, aiming to solve the technical problem of low self-calibration accuracy of multi-source sensors in related technologies.

[0006] Firstly, to achieve the above objectives, this application provides a self-calibration method for a multi-source sensor, the method comprising:

[0007] Acquire lidar point cloud data collected by the vehicle-mounted mobile inspection platform during the inspection process, as well as road images captured by a distributed multi-camera array;

[0008] Based on lidar point cloud data and road images, the lidar point cloud features and image semantic features corresponding to the calibration reference objects in the inspection road scene are extracted; the calibration reference objects include the road surface, curb, and pole-shaped objects of the inspection road.

[0009] The 3D radar point cloud features of the calibration reference object are projected onto the image space of each distributed camera to determine the positional relationship with the corresponding calibration reference object in each distributed camera, and a multi-dimensional decoupling geometric constraint is constructed. The multi-dimensional decoupling geometric constraint includes road surface normal vector constraint, line-surface projection constraint, and line-line projection constraint.

[0010] Based on multidimensional decoupled geometric constraints, a nonlinear optimization objective function is constructed, and the extrinsic parameters to be optimized are optimized until the nonlinear optimization objective function converges, thereby obtaining the benchmark rigid extrinsic parameter matrix.

[0011] Based on the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, a time-varying extrinsic parameter function is defined, and a sliding time window of variable length is maintained. Within the window, the micro-deformation compensation matrix of the time-varying extrinsic parameter function is optimized, and the corrected extrinsic parameter matrix is ​​output. The length of the sliding time window is negatively correlated with the type of calibration reference in the current inspection section within the predetermined range.

[0012] In one embodiment, the steps of extracting radar point cloud features and image semantic features corresponding to calibration reference objects in the inspected road scene based on lidar point cloud data and road images include:

[0013] Determine the road surface point cloud set, roadside point cloud set, and rod-shaped object point cloud set in the lidar point cloud data;

[0014] Based on the road surface point cloud set, the roadside point cloud set, and the rod-shaped object point cloud set, the road surface normal vector, the roadside vertical elevation equation, and the rod-shaped object centerline equation of the point cloud corresponding to the lidar are determined, and the lidar point cloud features are obtained.

[0015] Semantic segmentation is performed on road images to determine the semantic regions of road surface, roadside, and pole-shaped objects;

[0016] Image semantic features are obtained by determining the image road surface normal vector, the bottom edge contour of the road edge, and the vertical edge of the rod corresponding to the road image from the road surface semantic region, the road edge semantic region, and the rod semantic region.

[0017] In one embodiment, the steps of determining the road surface normal vector, roadside vertical elevation equation, and rod-shaped object centerline equation of the point cloud corresponding to the lidar based on the road surface point cloud set, roadside point cloud set, and rod-shaped object point cloud set, and obtaining the radar point cloud features include:

[0018] The road surface point cloud is divided into several local sub-blocks using a gridding strategy;

[0019] For each local sub-block, planar fitting is performed on the road surface point cloud data within the local sub-block, and the road surface normal vector of the local sub-block is calculated;

[0020] Weighted clustering and smoothing are performed on the road surface normal vectors of all local sub-blocks to obtain the point cloud road surface normal vectors;

[0021] By using the least squares method, the point cloud sets of the roadside and the point cloud sets of the rod are fitted to obtain the equations of the vertical elevation of the roadside and the central axis of the rod.

[0022] In one embodiment, the steps of projecting the three-dimensional radar point cloud features of the calibration reference object onto the image space of each distributed camera, determining the positional relationship with the corresponding calibration reference object in each distributed camera, and constructing multi-dimensional decoupled geometric constraints include:

[0023] Establish the initial extrinsic parameter mapping relationship between the lidar coordinate system and the coordinate systems of each distributed camera;

[0024] Based on the initial extrinsic parameter mapping relationship and combined with the roadside vertical elevation equation, the roadside vertical elevation in the lidar point cloud data is projected onto the image plane of the corresponding distributed camera.

[0025] Determine the vertical distance from the bottom edge contour of the road edge in the image plane to the projection of the road edge vertical elevation, and use it as the line-plane projection constraint;

[0026] Based on the initial extrinsic parameter mapping relationship and combined with the equation of the central axis of the rod, the central axis of the rod in the lidar point cloud data is projected onto the image plane of the corresponding distributed camera;

[0027] Determine the distance error between the projection of the vertical edge of the rod and the central axis of the rod in the image plane, and use it as a line-to-line projection constraint;

[0028] Based on the rotational transformation relationship between the point cloud road surface normal vector and the image road surface normal vector, a geometric equation of the rotation matrix is ​​constructed as a constraint on the road surface normal vector.

[0029] In one embodiment, the steps of constructing a nonlinear optimization objective function based on multidimensional decoupled geometric constraints, optimizing the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, and obtaining the baseline rigid extrinsic parameter matrix include:

[0030] Based on multidimensional decoupled geometric constraints, a nonlinear objective optimization function is constructed. The nonlinear objective optimization function consists of the first projection residual determined by the line-plane projection constraint, the second projection residual determined by the line-line projection constraint, the angle deviation residual determined by the road surface normal vector constraint, and the time smoothing constraint term.

[0031] The Levenberg-Marquardt algorithm is used to iteratively update the extrinsic parameters in the Lie algebra space until the nonlinear optimization objective function converges, thereby determining the baseline rigid extrinsic parameter matrix.

[0032] In one embodiment, based on the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, a time-varying extrinsic parameter function is defined, a sliding time window of variable length is maintained, and the micro-deformation compensation matrix of the time-varying extrinsic parameter function is optimized within the window to output the corrected extrinsic parameter matrix. The steps include:

[0033] The modified extrinsic parameter matrix is ​​defined as the product of the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, and the time-varying extrinsic parameter function is established.

[0034] Using the micro-deformation compensation matrix as the optimization variable, a local adjustment model is constructed based on the radar point cloud features and image semantic features of multiple calibration reference objects accumulated within the sliding time window;

[0035] Solve the local adjustment model, iteratively update the six-degree-of-freedom increment of the micro-deformation compensation matrix, and correct the baseline rigid extrinsic parameter matrix based on the updated micro-deformation compensation matrix, and output the corrected extrinsic parameter matrix.

[0036] In one embodiment, the step of correcting the reference rigidity extrinsic parameter matrix based on the updated micro-deformation compensation matrix and outputting the corrected extrinsic parameter matrix includes:

[0037] If the increment of the six degrees of freedom exceeds the preset threshold, the reference rigid extrinsic parameter matrix is ​​updated through a first-order low-pass damping filter, and the corrected extrinsic parameter matrix is ​​output.

[0038] Secondly, to achieve the above objectives, this application further provides a self-calibration device for a multi-source sensor, the device comprising:

[0039] The data acquisition module is used to acquire lidar point cloud data collected by the vehicle-mounted mobile inspection platform during the inspection process, as well as road images captured by a distributed multi-camera array.

[0040] The feature extraction module is used to extract radar point cloud features and image semantic features corresponding to calibration reference objects in the inspection road scene based on LiDAR point cloud data and road images; the calibration reference objects include the road surface, curb and pole-shaped objects of the inspection road;

[0041] The constraint construction module is used to project the 3D radar point cloud features of the calibration reference object onto the image space of each distributed camera, determine the positional relationship with the corresponding calibration reference object in each distributed camera, and construct multi-dimensional decoupling geometric constraints. The multi-dimensional decoupling geometric constraints include road surface normal vector constraints, line-surface projection constraints, and line-line projection constraints.

[0042] The extrinsic parameter optimization module is used to construct a nonlinear optimization objective function based on multidimensional decoupled geometric constraints, optimize the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, and obtain the benchmark rigid extrinsic parameter matrix.

[0043] The deformation compensation module is used to define a time-varying extrinsic function based on the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, maintain a sliding time window of variable length, optimize the micro-deformation compensation matrix of the time-varying extrinsic function within the window, and output the corrected extrinsic parameter matrix; wherein, the length of the sliding time window is negatively correlated with the type of calibration reference in the current inspection section within the predetermined range.

[0044] Thirdly, to achieve the above objectives, this application further provides a Ravis multi-source sensor self-calibration device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the above-described Ravis multi-source sensor self-calibration method.

[0045] Fourthly, to achieve the above objectives, this application further provides a storage medium, which is a computer-readable storage medium, and stores a computer program on the storage medium. When the computer program is executed by a processor, it implements the steps of the above-described self-calibration method for multi-source sensors.

[0046] One or more technical solutions proposed in this application have at least the following technical effects:

[0047] This application achieves fully automatic online self-calibration of multi-source sensors on the inspection platform under complex dynamic environments by extracting multi-dimensional structural features of calibration reference objects in road scenes and constructing a multi-dimensional decoupled geometric constraint model. It also ensures pose stability under continuous road vibration excitation and guarantees accurate locking of the six degrees of freedom of external parameters in the entire space. By introducing a sliding time window and a micro-deformation compensation mechanism, the non-rigid deflection deformation problem of the inspection beam under dynamic loads is solved. Combined with low-pass damping filtering, the robustness and continuity of external parameter updates are ensured, significantly improving the accuracy of radar-visual data fusion and providing a highly reliable spatial reference guarantee for millimeter-level road defect measurement. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the self-calibration method for multi-source sensors of Rayvision, as described in an embodiment of this application.

[0051] Figure 2 This is a schematic diagram of the structure of the Leishi multi-source sensor self-calibration device of this application.

[0052] Figure 3 This is a schematic diagram of the structure of the Leishi multi-source sensor self-calibration device of this application.

[0053] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0055] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of this application embodiment is as follows: By simultaneously acquiring lidar point cloud data and road image data collected by a distributed multi-camera array during vehicle-mounted mobile inspection, the lidar point cloud features and image semantic features corresponding to calibration reference objects such as road surface, curb, and poles in the road scene are extracted. The three-dimensional lidar point cloud features are projected onto the image space of each distributed camera to construct multi-dimensional decoupled geometric constraints, including road surface normal vector constraints, line-plane projection constraints, and line-line projection constraints. On this basis, a nonlinear optimization objective function is established, and the extrinsic parameters between the camera and lidar are iteratively solved to obtain the reference rigid extrinsic parameter matrix. A micro-deformation compensation matrix is ​​introduced to construct a time-varying extrinsic parameter function. The micro-deformation compensation matrix is ​​optimized by using multi-frame observation data within a sliding time window, thereby dynamically correcting the sensor extrinsic parameters and realizing online self-calibration and high-precision updating of the extrinsic parameters of the lidar multi-source sensors during the inspection process.

[0057] Specifically, this application provides a self-calibration method for a multi-source radar sensor, referring to... Figure 1 , Figure 1 This is a schematic flowchart of the first embodiment of the Leishi multi-source sensor self-calibration method of this application. In this embodiment, the Leishi multi-source sensor self-calibration method includes steps S10 to S50:

[0058] Step S10: Acquire the lidar point cloud data collected by the vehicle-mounted mobile inspection platform during the inspection process, as well as the road images captured by the distributed multi-camera array.

[0059] It should be noted that this embodiment uses a vehicle-mounted mobile inspection platform as the physical payload, acquires the three-dimensional spatial structure information of the inspected road scene through LiDAR, and collects high-resolution image information of the road surface and surrounding facilities through a distributed multi-camera array at different perspectives such as front, side or bottom.

[0060] In one example, a vehicle-mounted mobile inspection platform is used as the hardware architecture, with a LiDAR as the geometric center node. A distributed array of cameras—front-view, left-side-view, right-side-view, and road-view cameras—is mounted on a crossbeam. The PPS (Pulse Per Second) signal output by a high-precision GNSS / IMU integrated navigation system serves as the master clock source, triggering synchronous acquisition by the LiDAR and the distributed multi-camera array (including front-view, side-view, and road-view cameras). The start and end times of each frame of the LiDAR point cloud, as well as the exposure center time of each image frame, are recorded and assigned a unified UTC timestamp.

[0061] Furthermore, addressing the "tailing" and "distortion" phenomena of point clouds caused by vehicle movement during high-speed inspection of LiDAR, this example utilizes high-frequency pose data from the IMU (typically above 100Hz) and employs linear interpolation or spherical linear interpolation (SLERP) algorithms to calculate the transformation matrix of the instantaneous pose of each laser point scanned by the LiDAR relative to the pose at the start of the frame. All laser points are then uniformly transformed to the coordinate system at the start of the frame, eliminating motion distortion and obtaining geometrically accurate point cloud data for the current frame.

[0062] Specifically, assuming the scanning period of one frame of point cloud by the lidar is Δ t For any laser point within this frame P i Its relative to the start time of the frame t start Time offset is δt i 。

[0063] According to the integrated navigation system t start and t start +Δ t The pose sequence between them is obtained using spherical linear interpolation (SLERP). P i Compensation pose during sampling T i .

[0064] This process converts all point clouds that have undergone displacement within a single frame to a unified form. t start In the coordinate system of time, eliminate the geometric stretching phenomenon caused by the high speed of the inspection vehicle (e.g., 80km / h).

[0065] Step S20: Based on lidar point cloud data and road images, extract the lidar point cloud features and image semantic features corresponding to the calibration reference objects in the inspected road scene.

[0066] The reference objects for calibration include the road surface, curb, and pole-shaped objects of the inspection road.

[0067] Step S30: Project the 3D radar point cloud features of the calibration reference object onto the image space of each distributed camera, determine the positional relationship with the corresponding calibration reference object in each distributed camera, and construct multi-dimensional decoupling geometric constraints; the multi-dimensional decoupling geometric constraints include road surface normal vector constraints, line-surface projection constraints, and line-line projection constraints.

[0068] It should be noted that this embodiment constructs and runs a dual-stream parallel feature mining architecture, which performs heterogeneous feature extraction at the spatial manifold of the laser point cloud and the semantic topology of the image, respectively.

[0069] LiDAR point cloud data is used to describe the three-dimensional spatial structure of the road scene, while road images reflect the texture and visual edge information of the road scene. By jointly analyzing the two types of data, target objects with stable geometric structures are identified in the same inspection scene and used as calibration references in the subsequent extrinsic parameter solving process. In this embodiment, the calibration references include the road surface, curb, and pole-like objects of the inspected road.

[0070] Calibration techniques in related technologies often rely on planar features such as lane lines, which have weak constraint capabilities for rotational drift perpendicular to the road surface. This embodiment constructs a physically meaningful decoupled constraint model by mining multi-level structural features in the road environment, such as "road surface manifold normal vectors," "roadside vertical elevations," and "vertical rods." By using normal vector alignment to forcibly lock the Roll / Pitch angle and using vertical reprojection to lock the Yaw angle, the pose stability of the system under continuous road vibration excitation is greatly enhanced.

[0071] In one feasible implementation, step S20 includes steps A10 to A40:

[0072] Step A10: Determine the road surface point cloud set, roadside point cloud set, and pole-shaped object point cloud set in the lidar point cloud data.

[0073] Step A20: Based on the road surface point cloud set, the roadside point cloud set, and the rod-shaped object point cloud set, determine the road surface normal vector, the roadside vertical elevation equation, and the rod-shaped object centerline equation corresponding to the lidar point cloud to obtain the radar point cloud features.

[0074] Step A30: Perform semantic segmentation on the road image to determine the semantic regions of the road surface, the roadside, and the pole-shaped objects.

[0075] Step A40: Determine the image road surface normal vector, the bottom edge contour of the road edge, and the vertical edge of the rod corresponding to the road image from the road surface semantic region, the road edge semantic region, and the rod-shaped semantic region to obtain the image semantic features.

[0076] Step A20 includes steps A21 to A24:

[0077] Step A21: Divide the road surface point cloud set into several local sub-blocks using a gridding strategy.

[0078] Step A22: For each local sub-block, perform planar fitting on the road surface point cloud data within the local sub-block and calculate the road surface normal vector of the local sub-block.

[0079] Step A23: Perform weighted clustering and smoothing on the road surface normal vectors of all local sub-blocks to obtain the point cloud road surface normal vectors.

[0080] Step A24: Using the least squares method, the point cloud set of the roadside and the point cloud set of the rod are obtained by fitting, and the equations of the vertical elevation of the roadside and the central axis of the rod are obtained.

[0081] Specifically, this involves extracting the geometric manifold and vertical structure of the laser point cloud:

[0082] The steps for extracting road surface normal vectors from point cloud are as follows: First, define the Region of Interest (ROI) and divide the road surface point cloud into several local sub-blocks using a meshing strategy. Within each sub-block, iteratively fit the local plane equation using a random sampling consensus algorithm and calculate the local normal vector. Weighted clustering and smoothing are then performed on all local normal vectors to obtain the global road surface normal vector for the current frame. This feature strongly constrains the vehicle's pitch and roll attitudes.

[0083] For example, the distortion-free point cloud is first segmented into ground regions, and a gridding smoothing strategy is used to divide the road surface point cloud into local regions of 0.5m × 0.5m. Within each grid, a local plane is fitted using Principal Component Analysis (PCA) or RANSAC algorithm. The local normal vector is obtained by calculating the eigenvector corresponding to the smallest eigenvalue of the covariance matrix. To eliminate interference from road surface damage, water accumulation, or debris, a Gaussian mixture model is used to cluster the normal vectors of the entire frame, and the cluster centers are extracted as the point cloud road surface normal vectors at the current moment. This feature is rotationally invariant, providing an absolute reference for attitude angles.

[0084] It is worth mentioning that this embodiment calculates the road surface normal vector in the point cloud. In addition to utilizing the current frame data, a point cloud preprocessing mechanism based on the attitude change rate of a high-frequency IMU is also incorporated. By identifying and eliminating instantaneous 'pseudo-bump' noise generated by the inspection vehicle during high-speed travel, this point cloud data preprocessing mechanism ensures the smoothness of the normal vector features in the time dimension. This provides a high signal-to-noise ratio attitude reference for subsequent calculation of the low-frequency rotational deformation of the crossbeam caused by thermal stress, which is impossible to achieve with techniques that rely solely on single-frame point cloud fitting.

[0085] For the vertical elevation equations of the roadside and the central axis equations of poles, firstly, based on the beam distribution and spatial gradient characteristics of the laser point cloud, height abrupt change points are detected to determine the three-dimensional structural point cloud data in the road. Then, Euclidean clustering algorithm is used to segment the non-ground point cloud into independent clusters. Based on the geometric morphological characteristics of the clusters (such as aspect ratio, height, linearity, and point density), the point cloud sets of the roadside and poles (such as lampposts and signposts) are identified and classified. Finally, the least squares method is used to fit the vertical elevation equations of the roadside and the central axis equations of the poles.

[0086] Semantic awareness and edge refinement for images:

[0087] The distortion-free road image is input into a pre-trained semantic segmentation network, which outputs pixel-level semantic masks. Category masks belonging to the categories of "drivable area", "curb", and "guardrail / post" are extracted to determine the semantic regions of the road surface, curb, and poles.

[0088] Within the semantic boundary region of the category mask, the Canny edge detection operator or the Sobel operator is used to extract the sub-pixel level roadside bottom contour and the vertical edge of the pole. The road surface normal vector in the image coordinate system is then deduced using the lane vanishing point in the image or using camera intrinsics and horizon detection results.

[0089] Furthermore, by combining the Canny edge detection operator or the Sobel operator, the sub-pixel level contours of the curb bottom edge and the vertical edges of the rod-shaped object are extracted, specifically including:

[0090] First, the semantic mask is projected onto an initial horizontal plane using initial extrinsic parameters. Edge extraction is then performed in the initial horizontal plane coordinate system, and finally, it is back-projected back to the image coordinate system.

[0091] This processing eliminates the image edge 'stretching' error caused by the distributed layout of the inspection vehicles, enabling the positioning accuracy of the bottom edge of the roadside to jump from the pixel level to the millimeter level (corresponding to a physical world measured accuracy of ≤1mm).

[0092] In addition, the distributed multi-camera array adopts a star topology structure with the lidar as the central node.

[0093] Specifically, to address the challenge of completely non-overlapping fields of view of distributed camera arrays, this application also constructs a spatiotemporal bridging constraint system with lidar as the spatial link and vehicle trajectory as the time dimension through back projection.

[0094] In practice, the system records the time. Features observed by the left-side camera Utilizing the precise pose provided by integrated navigation and the features acquired by the lidar at this time Three-dimensional coordinates in the radar local coordinate system The system maps this feature to a global spatial reference frame.

[0095] After the inspection vehicle has traveled a certain distance, at a certain time Features that were originally invisible Entering the observation range of the right-view or downward-view camera, the system establishes a virtual co-view relationship across frames and cameras through back-projection simulation. Through this 'space-for-time' strategy, the system unifies the fields of view of multiple physically isolated cameras into a single 'star-shaped topology' constraint grid with the lidar as the origin. This mechanism not only verifies the extrinsic parameter accuracy of a single camera relative to the radar but also, through the consistency residuals across the fields of view, keenly detects the asymmetric torsional displacement of the beam generated during long-term inspections. This represents a significant creative advantage over existing technologies that only perform single-frame feature matching.

[0096] In this embodiment, step S30 includes steps B10 to B60:

[0097] Step B10: Establish the initial extrinsic parameter mapping relationship between the lidar coordinate system and the coordinate systems of each distributed camera.

[0098] Step B20: Based on the initial extrinsic parameter mapping relationship and combined with the roadside vertical elevation equation, project the roadside vertical elevation in the lidar point cloud data onto the image plane of the corresponding distributed camera.

[0099] Step B30: Determine the vertical distance from the bottom edge contour of the road edge in the image plane to the vertical elevation projection of the road edge, as a line-plane projection constraint.

[0100] Step B40: Based on the initial extrinsic parameter mapping relationship and combined with the equation of the central axis of the rod, project the central axis of the rod in the lidar point cloud data onto the image plane of the corresponding distributed camera.

[0101] Step B50: Determine the distance error between the projection of the vertical edge of the rod and the central axis of the rod in the image plane, and use it as a line projection constraint.

[0102] Step B60: Based on the rotational transformation relationship between the point cloud road surface normal vector and the image road surface normal vector, construct the geometric equation of the rotation matrix as a constraint on the road surface normal vector.

[0103] Specifically, this step is mainly used to construct a multi-dimensional geometric constraint model to achieve deep correlation between 2D image features and 3D point cloud features:

[0104] Construction of line and surface constraints for "point cloud road edge elevation - image road edge bottom edge":

[0105] Based on the current initial extrinsic parameter estimation, the roadside vertical elevation extracted from the laser point cloud is projected onto the image plane.

[0106] On the image plane, the vertical distance from the extracted bottom edge pixel of the roadside to the projected elevation (represented as a straight line or strip area on the image) is calculated to determine the depth factor s.

[0107] This constraint utilizes the "elevation change" characteristic of the roadside as a road boundary, which can effectively constrain the lateral translation of the camera. ), height translation ( The rotation about the optical axis provides strong geometric locking for the relevant extrinsic parameters.

[0108] Construction of line constraints for "point cloud rod-image vertical edge":

[0109] The central axis (3D straight line) of the rod-shaped object fitted in the laser point cloud is projected onto the image plane to obtain a 2D projection line.

[0110] Search for vertical edge segments within the corresponding semantic region in the image, calculate the distance error from points on the image edge to the 2D projection line, and determine the translation vector. .

[0111] This constraint utilizes the "towering" characteristic of the rod in the vertical direction to precisely lock the camera's yaw angle and horizontal position (tx, ty). This constraint can effectively lock the inspection vehicle's yaw angle and horizontal translation parameters.

[0112] Constructing a "point cloud-image" ground normal vector consistency constraint:

[0113] Establish point cloud road surface normal vector With the image road surface normal vector Rotational transformation relationship between them:

[0114] .

[0115] This constraint directly constructs the rotation matrix. The geometric equations greatly reduce the solution space of the rotation parameters, preventing the optimization process from getting trapped in local minima.

[0116] After determining the above three constraints, 3D-to-2D projection can be achieved. Specifically, let the 3D feature points in the lidar coordinate system be... Its corresponding pixel coordinates on the camera image plane are p img Considering that inspection cameras typically use wide-angle or telephoto lenses, the system uses a distortion model to determine the pixel coordinates before performing the projection transformation. p img Radial and tangential distortion correction is performed, followed by projection transformation. The mathematical relationship between the two follows the pinhole camera model, and the projection relationship is expressed as:

[0117] .

[0118] in, For rotation matrix, It is a translation vector. For depth factor, K This is the camera intrinsic parameter matrix.

[0119] Step S40: Based on multidimensional decoupled geometric constraints, construct a nonlinear optimization objective function, optimize the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, and obtain the benchmark rigid extrinsic parameter matrix.

[0120] Specifically, based on the multidimensional geometric feature association established in step S30, this step designs a global objective function that includes a geometric flatness constraint term, a vertical structure reprojection error term, and a time smoothness regularization term. Based on this objective function, extrinsic parameter optimization is performed.

[0121] In one feasible implementation, step S40 includes steps C10 to C20:

[0122] Step C10: Based on multidimensional decoupled geometric constraints, construct a nonlinear objective optimization function. The nonlinear objective optimization function consists of the first projection residual determined by the line-plane projection constraint, the second projection residual determined by the line-line projection constraint, the angle deviation residual determined by the road surface normal vector constraint, and the time smoothing constraint term.

[0123] Step C20: The Levenberg-Marquardt algorithm is used to iteratively update the extrinsic parameters to be optimized in the Lie algebra space until the nonlinear optimization objective function converges, thereby determining the baseline rigid extrinsic parameter matrix.

[0124] Specifically, the angular deviation residual between the point cloud road surface normal vector and the image road surface normal vector after rotation by external parameters is defined as the road surface smoothness constraint term (angular deviation residual). This term is used to strongly constrain the rotational degrees of freedom around the X-axis and Y-axis, solving the problem of vertical drift in traditional methods.

[0125] After projecting the roadside elevation point cloud and the central axis of the rod-shaped structure extracted from the laser point cloud onto the image plane, the Euclidean or Mahalanobis distance between them and the corresponding edge features in the image is defined as the vertical structure reprojection error term (first reprojection residual and second reprojection residual). This term is used to refine the constraint translation vector ( ) and heading angle (Yaw).

[0126] In addition, the objective function introduces a penalty term and a time smoothness regularization term for changes in extrinsic parameters at adjacent time points. This assumes that the changes in extrinsic parameters of the vehicle are smooth over a short period of time, preventing sudden changes in single-frame calibration results caused by severe road bumps.

[0127] Finally, the global cost function is defined. The weighted sum of squares of the geometric residuals:

[0128]

[0129] in, For angular deviation residuals, For the first projection residual, This is the second projection residual. For time smoothness regularization, ~ These are the weighting coefficients.

[0130] Subsequently, the objective function is transformed into a nonlinear least squares problem. This is achieved using Lie algebras. The pose transformation matrix to be optimized is parameterized to avoid gimbal lock. The Levenberg-Marquardt algorithm or Gauss-Newton algorithm is used for iterative solution until the objective function converges or the maximum number of iterations is reached, thereby obtaining a high-precision global rigid extrinsic parameter matrix at the current time step.

[0131] Specifically, in solving the above optimization problem, to avoid gimbal lock and singularity problems existing in Euler angles, this application innovatively applies the pose to be calibrated to a Lie group. The parameterization is performed on the above, and its corresponding Lie algebra is used. Perform a tangent space perturbation update.

[0132] Specifically, let the estimated value of the extrinsic parameters at the current time be... In each iteration, the system adds a small perturbation to the left multiplication. Update pose:

[0133]

[0134] in, δξ is the six-degree-of-freedom increment vector; ^ is the hat operator, which means mapping the six-degree-of-freedom increment vector δξ in the Lie algebra se(3) space to its corresponding matrix form.

[0135] Specifically, to avoid gimbal lock and singularity issues in Euler angle calibration, in Lie algebra... The extrinsic parameters to be optimized are perturbed and parameterized within the space. The Levenberg-Marquardt algorithm is employed, introducing a Huber robust kernel function in each iteration.

[0136]

[0137] in, This is the kernel function used to automatically identify and reduce the weight of outlier matching points. The Jacobian matrix is ​​calculated iteratively. And update the increment This continues until the objective function converges.

[0138] Experiments show that, under complex road surface excitation, the convergence accuracy of the attitude angle can reach [percentage missing]. The reprojection error can be stably kept within 1.2 pixels, providing a solid external parameter benchmark for millimeter-level pavement distress measurement.

[0139] Furthermore, during the external parameter update process, this embodiment also updates the weights of the objective function. In this embodiment, the following mechanism is introduced in step S30:

[0140] In one example, pixel coordinates are calculated. p img Relative to the Jacobian matrix of a six-DOF extrinsic parameter;

[0141] Based on the magnitude of the partial derivatives in the Jacobian matrix, the external parameter degrees of freedom that are most sensitive to deformation at the current feature point are identified.

[0142] Increase the weight of the residual constraints corresponding to the most sensitive external parameter degrees of freedom.

[0143] Specifically, during the optimization iteration process, the reprojection residual is derived using the chain rule. (Right now (General term) on disturbance quantity Analytical Jacobian Matrix The Jacobian matrix directly quantifies the sensitivity of different features to each extrinsic degree of freedom. Based on the projection sensitivity analysis results completed in step S30, the optimization solver applies stronger convergence constraints to the axis of the most sensitive extrinsic degree of freedom when updating the iteration step size. This directional solution performed in the Lie algebra tangent space can greatly improve the convergence stability of the optimization algorithm under high-frequency road surface excitation, enabling it to achieve sub-pixel-level residual convergence even when the vehicle is bumping at high speed.

[0144] For example, the projection displacement of the curb feature of the side-view camera at the bottom of the image affects the vertical deflection of the beam. The image edge has the largest gradient, while the vertical bars are more sensitive to torsional deformation (Yaw). This matching logic based on geometric sensitivity provides a physical-level prior guidance for the decoupling optimization in step S40 and forms a two-layer weight optimization strategy, which effectively prevents the problem of mutual coupling and non-convergence of parameter solution spaces.

[0145] In another example, this application can also update the weight coefficients w based on the dynamic weight allocation of observation quality by constructing a dynamic weight allocation model and adopting a dynamic allocation strategy based on information balance. i Its dynamic weight allocation model is expressed as:

[0146]

[0147] Where, σ i ² represents the variance of the current i-th type of feature observations, reflecting the accuracy of feature extraction; R i This refers to the real-time statistics of the reprojection residuals for this type of feature. The physical meaning of this strategy is: when the extraction accuracy of a certain type of feature (such as curb) is high (σ... i When the data is small and abundant, the system will automatically increase its corresponding weight (e.g., w2) to ensure that the optimization process always tilts towards the most reliable observation data.

[0148] Understandably, this embodiment introduces a hierarchical cost function in the feature matching stage. First, a coarse association of "line-surface-point cloud" is performed to lock the principal pose residual, followed by a fine association of sub-pixel edges. To address the issue of mismatches caused by strong light or shadow interference at the inspection site, the system incorporates a Huber robust kernel operator to downsample the weights of abnormal association points during the matching process. This application constructs a directional gradient constraint term by analyzing the consistency between the normal vector direction of the point cloud road edge and the tangent direction of the extracted image edge. This dual verification mechanism combining geometric topology and semantic gradient ensures that even in worn road sections lacking high-definition lane lines, this application can still maintain pixel-level feature association accuracy, thus providing high-quality input data for millimeter-level online self-calibration.

[0149] Step S50: Based on the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, a time-varying extrinsic parameter function is defined, and a sliding time window of variable length is maintained. Within the window, the micro-deformation compensation matrix of the time-varying extrinsic parameter function is optimized, and the corrected extrinsic parameter matrix is ​​output. The length of the sliding time window is negatively correlated with the type of calibrated reference objects in the current inspection section within the predetermined range.

[0150] Specifically, in the process of dynamic extrinsic parameter calibration, this embodiment defines a time-varying extrinsic parameter model based on the extrinsic parameters determined in step S40:

[0151]

[0152] in As a reference rigid external parameter, For a moment The deformation compensation matrix is ​​calculated; a sliding window of variable length is constructed, assuming the deformation compensation matrix is ​​within the window. Maintain a constant or linear variation; utilize multi-frame heterogeneous feature observation data accumulated within the window to fix Only for The six-DOF parameters are optimized using bundle adjustment; when the sliding window is updated, the corrected extrinsic parameters at the current time are output. This is used for subsequent disease measurement projection.

[0153] It is worth mentioning that this embodiment introduces a feature confidence scoring mechanism for the sensitivity of inspection conditions to adjust the sliding window size in real time.

[0154] System-defined scoring function Based on the original geometric features, a deflection constraint contribution factor has been added:

[0155]

[0156] in, and The linearity (e.g., the straightness of a roadside line) and the point cloud density of the geometric feature extracted from the point cloud are used to evaluate its geometric quality. This characterizes the constraint strength of the feature point on the six-degree-of-freedom deformation of the camera at the end of the crossbeam; α, β, and γ are preset weighting coefficients used to balance the importance of different factors. For example, the vertical member located directly in front of the crossbeam has a stronger constraint on the yaw angle, while the long roadside curb has a stronger constraint on the roll angle and height offset. The constraints are stronger.

[0157] Furthermore, the system not only evaluates the quality of individual features, but also assesses the overall observability of the current feature set by calculating the condition number of the Jacobian matrix of the optimization problem. If the feature distribution of the current road segment lacks sufficient geometric diversity (for example, the scene only has straight road surfaces and no vertical structures), resulting in an ill-conditioned Jacobian matrix (i.e., the condition number is too high), the system will determine that the current state is "under-constrained".

[0158] To address this situation, the system will automatically increase the observation length of the sliding window. This self-aware feature filtering logic, by accumulating more and richer observation data over the time dimension, ensures that even in environments with feature degradation such as complex greening or road marking wear, the feature set that ultimately enters the optimizer can still specifically and unambiguously lock the non-rigid micro-deformation generated by the crossbeam.

[0159] In one feasible implementation, step S50 includes steps D10 to D30:

[0160] Step D10: Define the modified extrinsic parameter matrix as the product of the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, and establish the time-varying extrinsic parameter function;

[0161] Step D20: Using the micro-deformation compensation matrix as the optimization variable, a local adjustment model is constructed based on the radar point cloud features and image semantic features of multiple calibration reference objects accumulated within the sliding time window.

[0162] Step D30: Solve the local adjustment model, iteratively update the six-degree-of-freedom increment of the micro-deformation compensation matrix, and correct the baseline rigid extrinsic parameter matrix based on the updated micro-deformation compensation matrix, and output the corrected extrinsic parameter matrix.

[0163] Step D30 includes step D31:

[0164] Step D31: If the six-degree-of-freedom increment exceeds the preset threshold, the reference rigid extrinsic parameter matrix is ​​updated by first-order low-pass damping filter, and the corrected extrinsic parameter matrix is ​​output.

[0165] Specifically, addressing the non-rigid deformation problem caused by the long crossbeam distributed layout unique to traffic inspection vehicles under long-term operation and road surface excitation, this embodiment establishes a time-varying model, abandoning the traditional all-rigid-body assumption and defining the sensor extrinsic parameters as time-varying functions. .

[0166] in, The reference rigidity extrinsic parameters obtained in step S40, This is the time-varying small deformation compensation matrix.

[0167] Set a time sliding window, and fix the reference extrinsic parameters within the sliding window. The local optimization map is constructed by taking the feature observation data (point cloud features and image features) of all frames within the window as input. Only the compensation amount for micro-deformation is considered. The six degrees of freedom parameters are combined and adjusted to solve the problem.

[0168] As the vehicle moves, the sliding window moves forward, providing real-time calculations of the latest... This parameter is then applied to a reference extrinsic parameter to generate the current real-time calibration parameter. This parameter is fed back in real-time to the multi-sensor fusion module of the inspection system to correct the projection position of the laser point cloud on the image, ensuring pixel-level alignment of the defect measurement.

[0169] For example, this applies to the long-span mechanical crossbeams (typically long) unique to traffic inspection vehicles. To address the non-rigid failure problem under long-term dynamic loads and extreme temperature differences, this application proposes a time-varying extrinsic parameter tracking mechanism based on Lie algebra state perturbation.

[0170] This example abandons the limitation of traditional calibration methods that treat extrinsic parameters as constant values, and instead considers the sensor's real-time pose. It is broken down into "global static installation component" and "local dynamic deflection component".

[0171] System defines real-time extrinsic parameter matrix

[0172]

[0173] in, The reference rigid external parameter is obtained by solving step S40; δξ(t) ∈ se(3) is a small physical deformation vector with six degrees of freedom that varies with time t. It should be emphasized that δξ(t) here represents the real dynamic physical deformation and should be clearly distinguished from the mathematical increment vector δξ in the optimization iterative solution process.

[0174] Furthermore, this example maintains an independent state variable for each camera in the distributed camera array. Furthermore, a beam structure stiffness constraint factor was introduced. This factor is used to limit the correlation of deformation between adjacent cameras located on the same rigid beam, thereby realistically simulating the continuous deformation characteristics of the mechanical structure at the mathematical level. This level of refined physical modeling is completely unavailable in existing technologies.

[0175] In practical processing, this example maintains a time sliding window of length N, establishes an observation chain of structured features through cross-frame feature tracking, and constructs a composite cost function containing a priori terms of temporal continuity within this sliding window:

[0176]

[0177] in, For the comprehensive reprojection residual term (first reprojection residual) Second projection residual (the sum) λ is the deformation smoothing regularization term, used to penalize drastic changes in the deformation vector between consecutive frames, and λ is the weight of the regularization term.

[0178] Here, Ω is a 6×6 symmetric positive definite matrix, serving as the weighting matrix for this regularization term. This matrix encodes prior information about the thermodynamic properties of the beam material and the structural stiffness (for example, based on mechanical analysis, a larger penalty weight can be assigned to the axial tensile direction, which is less prone to deformation, while a smaller penalty weight can be assigned to the vertical deflection, which is more prone to deformation). By introducing this weighted regularization term based on physical priors, the system can effectively distinguish between "random vibrations caused by instantaneous road bumps" and "quasi-static trend displacements of the beam caused by uneven heating."

[0179] In solving the composite cost function, the system uses the Lie algebra left perturbation model to calculate the analytical Jacobian matrix of the residual with respect to the deformation vector, which greatly improves the optimization efficiency and ensures that even under high-speed driving conditions (such as 80km / h), the calculation and update frequency of the calibration parameters is not less than 10Hz.

[0180] To ensure the geometric consistency of the disease measurement data stream, the system executes the following closed-loop control logic:

[0181] Threshold activation determination: The system monitors the calculated deformation increment in real time. Activation is only triggered when... The translation modulus value exceeded for 5 consecutive frames. Or the cumulative rotation offset angle exceeds Only then is it determined that "physical pose drift" caused by mechanical fatigue or environmental stress has occurred.

[0182] Damped smoothing strategy: After activation is detected, the system does not jump directly to the new extrinsic parameters, but instead uses a first-order low-pass damped filter for updating. The damping coefficient .

[0183] Experiments have shown that this mechanism can accurately compensate for the approximately [missing information - likely referring to a specific problem or effect] generated at the ends of the beam under high temperature and sunlight. The "warping deformation". This dynamic self-healing capability, designed to address the unique physical defects of inspection, ensures the system's ability to repair even with a width of only [missing information]. The overlap error between the micro-cracks in the image and the point cloud is less than The increased pixel count significantly enhances the system's millimeter-level measurement consistency in long-distance, temperature-differential operation scenarios.

[0184] It should be noted that all the above examples are only for understanding this application and do not constitute a limitation on the self-calibration method of multi-source sensors of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0185] This application also provides a self-calibration device for a multi-source sensor, please refer to... Figure 2 The Ravis multi-source sensor self-calibration device includes:

[0186] The data acquisition module 10 is used to acquire lidar point cloud data collected by lidar during the inspection process of the vehicle-mounted mobile inspection platform, as well as road images captured by a distributed multi-camera array.

[0187] The feature extraction module 20 is used to extract radar point cloud features and image semantic features corresponding to calibration reference objects in the inspection road scene based on lidar point cloud data and road images; the calibration reference objects include the road surface, curb and pole-shaped objects of the inspection road.

[0188] The constraint construction module 30 is used to project the 3D radar point cloud features of the calibration reference object onto the image space of each distributed camera, determine the positional relationship with the corresponding calibration reference object in each distributed camera, and construct multi-dimensional decoupling geometric constraints; the multi-dimensional decoupling geometric constraints include road surface normal vector constraints, line-surface projection constraints, and line-line projection constraints.

[0189] The extrinsic parameter optimization module 40 is used to construct a nonlinear optimization objective function based on multidimensional decoupled geometric constraints, optimize the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, and obtain the benchmark rigid extrinsic parameter matrix.

[0190] The deformation compensation module 50 is used to define a time-varying extrinsic function based on the reference rigid extrinsic parameter matrix and the micro-deformation compensation matrix, maintain a sliding time window of variable length, optimize the micro-deformation compensation matrix of the time-varying extrinsic function within the window, and output the corrected extrinsic parameter matrix; wherein, the length of the sliding time window is negatively correlated with the type of calibration reference in the current inspection section within the predetermined range.

[0191] The self-calibration device for multi-source sensors provided in this application employs the self-calibration method for multi-source sensors described in the above embodiments, which can solve the technical problem of low accuracy in the self-calibration of multi-source sensors in related technologies. Compared with related technologies, the beneficial effects of the self-calibration device for multi-source sensors provided in this application are the same as those of the self-calibration method for multi-source sensors provided in the above embodiments, and other technical features in the self-calibration device for multi-source sensors are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0192] This application provides a multi-source sensor self-calibration device for radar vision, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-source sensor self-calibration method in the above embodiments.

[0193] The following is for reference. Figure 3 This document illustrates a structural schematic diagram of a multi-source sensor self-calibration device suitable for implementing embodiments of this application. The multi-source sensor self-calibration device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 3 The Leishi multi-source sensor self-calibration device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0194] like Figure 3As shown, the Rayvision multi-source sensor self-calibration device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 (ROM) or a program loaded from the storage device 1003 into the random access memory 1004 (RAM). The random access memory 1004 also stores various programs and data required for the operation of the Rayvision multi-source sensor self-calibration device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 (I / O interface) is also connected to the bus 1005. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the Rayvision multi-source sensor self-calibration device to communicate wirelessly or wiredly with other devices to exchange data. Although the Rayvision multi-source sensor self-calibration device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0195] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0196] The self-calibration device for multi-source sensors provided in this application employs the self-calibration method for multi-source sensors described in the above embodiments, which can solve the technical problem of low accuracy in the self-calibration of multi-source sensors in related technologies. Compared with related technologies, the beneficial effects of the self-calibration device for multi-source sensors provided in this application are the same as those of the self-calibration method for multi-source sensors provided in the above embodiments, and other technical features in this self-calibration device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0197] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0198] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0199] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the Ravis multi-source sensor self-calibration method in the above embodiments.

[0200] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0201] The aforementioned computer-readable storage medium may be included in the Ravis multi-source sensor self-calibration device; or it may exist independently and not assembled into the Ravis multi-source sensor self-calibration device.

[0202] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0204] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0205] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described self-calibration method for multi-source radar sensors, thereby solving the technical problem of low accuracy in self-calibration of multi-source radar sensors in related technologies. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the self-calibration method for multi-source radar sensors provided in the above embodiments, and will not be repeated here.

[0206] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described self-calibration method for multi-source sensors.

[0207] The computer program product provided in this application can solve the technical problem of low self-calibration accuracy of Ravis multi-source sensors in related technologies. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the Ravis multi-source sensor self-calibration method provided in the above embodiments, and will not be repeated here.

[0208] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for self-calibration of a lightning visual multi-source sensor, characterized in that, The method includes: Acquire lidar point cloud data collected by the vehicle-mounted mobile inspection platform during the inspection process, as well as road images captured by a distributed multi-camera array; Based on the lidar point cloud data and the road image, the lidar point cloud features and image semantic features corresponding to the calibration reference objects in the inspection road scene are extracted; the calibration reference objects include the road surface, curb, and pole-shaped objects of the inspection road. The three-dimensional radar point cloud features of the calibration reference are projected onto the image space of each distributed camera to determine the positional relationship with the corresponding calibration reference in each distributed camera, and a multi-dimensional decoupling geometric constraint is constructed; the multi-dimensional decoupling geometric constraint includes road surface normal vector constraint, line-surface projection constraint and line-line projection constraint. Based on the multidimensional decoupled geometric constraints, a nonlinear optimization objective function is constructed, and the extrinsic parameters to be optimized are optimized until the nonlinear optimization objective function converges, thereby obtaining the benchmark rigid extrinsic parameter matrix. Based on the aforementioned benchmark rigid extrinsic parameter matrix and micro-deformation compensation matrix, a time-varying extrinsic parameter function is defined, and a sliding time window of variable length is maintained. Within the window, the micro-deformation compensation matrix of the time-varying extrinsic parameter function is optimized, and a corrected extrinsic parameter matrix is ​​output. The length of the sliding time window is negatively correlated with the type of calibrated reference objects in the current inspection section within a predetermined range. The steps of defining a time-varying extrinsic function based on the benchmark rigid extrinsic parameter matrix and the micro-deformation compensation matrix, maintaining a sliding time window of variable length, optimizing the micro-deformation compensation matrix of the time-varying extrinsic parameter function within the window, and outputting the corrected extrinsic parameter matrix include: The modified extrinsic parameter matrix is ​​defined as the product of the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, and the time-varying extrinsic parameter function is established. Using the micro-deformation compensation matrix as the optimization variable, a local adjustment model is constructed based on the radar point cloud features and image semantic features of multiple frames of calibration references accumulated within the sliding time window; Solve the local adjustment model, iteratively update the six-degree-of-freedom increment of the micro-deformation compensation matrix, and correct the reference rigid extrinsic parameter matrix based on the updated micro-deformation compensation matrix, and output the corrected extrinsic parameter matrix; In the process of optimizing the micro-deformation compensation matrix of the time-varying extrinsic function within the window, a composite cost function is also introduced, expressed as: in, For a six-degree-of-freedom increment vector, For the comprehensive reprojection residual term, λ is the deformation smoothing regularization term, used to penalize drastic changes in the deformation vector between consecutive frames. λ is the weight of the regularization term, and Ω is a 6×6 symmetric positive definite matrix that encodes the thermodynamic properties of the beam material and the prior information on the structural stiffness.

2. The method of self-calibration of a multi-source sensor of lightning according to claim 1, characterized in that, The step of extracting radar point cloud features and image semantic features corresponding to calibration reference objects in the inspected road scene based on the lidar point cloud data and the road image includes: Determine the road surface point cloud set, the roadside point cloud set, and the rod-shaped object point cloud set in the lidar point cloud data; Based on the road surface point cloud set, the roadside point cloud set, and the rod-shaped object point cloud set, the point cloud road surface normal vector, the roadside vertical elevation equation, and the rod-shaped object centerline equation corresponding to the lidar are determined, and the lidar point cloud features are obtained. The road image is semantically segmented to determine the road surface semantic region, the roadside semantic region, and the pole-shaped object semantic region; The image semantic features are obtained by determining the image road surface normal vector, the bottom edge contour of the road edge, and the vertical edge of the rod corresponding to the road image from the road surface semantic region, the road edge semantic region, and the rod semantic region.

3. The method of self-calibration of a multi-source sensor of lightning according to claim 2, characterized in that, The step of determining the road surface normal vector, roadside vertical elevation equation, and rod-shaped object centerline equation of the point cloud corresponding to the lidar based on the road surface point cloud set, the roadside point cloud set, and the rod-shaped object point cloud set, and obtaining the lidar point cloud features, includes: The road surface point cloud set is divided into several local sub-blocks using a gridding strategy; For each local sub-block, the road surface point cloud data within the local sub-block is fitted with a plane, and the road surface normal vector of the local sub-block is calculated. Weighted clustering and smoothing are performed on the road surface normal vectors of all the local sub-blocks to obtain the point cloud road surface normal vectors; By fitting the point cloud set of the roadside and the point cloud set of the rod-shaped object using the least squares method, the equations of the vertical elevation of the roadside and the central axis of the rod-shaped object are obtained.

4. The method of self-calibration of a multi-source sensor of claim 2, wherein, The step of projecting the three-dimensional radar point cloud features of the calibration reference object onto the image space of each distributed camera, determining the positional relationship with the corresponding calibration reference object in each distributed camera, and constructing multi-dimensional decoupled geometric constraints includes: Establish the initial extrinsic parameter mapping relationship between the lidar coordinate system and the coordinate systems of each distributed camera; Based on the initial extrinsic parameter mapping relationship and combined with the roadside vertical elevation equation, the roadside vertical elevation in the lidar point cloud data is projected onto the image plane of the corresponding distributed camera. The vertical distance from the bottom edge contour of the road edge in the image plane to the projection of the road edge vertical elevation is determined as the line-plane projection constraint; Based on the initial extrinsic parameter mapping relationship and combined with the equation of the central axis of the rod, the central axis of the rod in the lidar point cloud data is projected onto the image plane of the corresponding distributed camera; The distance error between the projection of the vertical edge of the rod-shaped object and the projection of the central axis of the rod-shaped object in the image plane is determined and used as the line projection constraint; Based on the rotational transformation relationship between the point cloud road surface normal vector and the image road surface normal vector, a geometric equation for the rotation matrix is ​​constructed as a constraint on the road surface normal vector.

5. The method of self-calibration of a multi-source sensor of lightning according to claim 4, characterized in that, The steps of constructing a nonlinear optimization objective function based on the multidimensional decoupled geometric constraints, optimizing the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, and obtaining the benchmark rigid extrinsic parameter matrix include: Based on the multidimensional decoupling geometric constraints, a nonlinear objective optimization function is constructed; the nonlinear objective optimization function consists of a first projection residual determined based on the line-plane projection constraints, a second projection residual determined based on the line-line projection constraints, an angle deviation residual determined based on the road surface normal vector constraints, and a time smoothing constraint term. The Levenberg-Marquardt algorithm is used to iteratively update the extrinsic parameters to be optimized in the Lie algebra space until the nonlinear optimization objective function converges, thereby determining the baseline rigid extrinsic parameter matrix.

6. The method of self-calibration of a multi-source sensor of lightning according to claim 1, characterized in that, The step of correcting the reference rigid extrinsic parameter matrix based on the updated micro-deformation compensation matrix and outputting the corrected extrinsic parameter matrix includes: If the six-degree-of-freedom increment exceeds a preset threshold, the reference rigid extrinsic matrix is ​​updated by a first-order low-pass damping filter, and the corrected extrinsic matrix is ​​output.

7. A device for self-calibration of a multi-source sensor of a lightning vision system, characterized in that it comprises: The device includes: The data acquisition module is used to acquire lidar point cloud data collected by the vehicle-mounted mobile inspection platform during the inspection process, as well as road images captured by a distributed multi-camera array. The feature extraction module is used to extract radar point cloud features and image semantic features corresponding to calibration reference objects in the inspection road scene based on the lidar point cloud data and the road image; the calibration reference objects include the road surface, curb, and pole-shaped objects of the inspection road. The constraint construction module is used to project the three-dimensional radar point cloud features of the calibration reference object onto the image space of each distributed camera, determine the positional relationship with the corresponding calibration reference object in each distributed camera, and construct multi-dimensional decoupling geometric constraints; the multi-dimensional decoupling geometric constraints include road surface normal vector constraints, line-surface projection constraints, and line-line projection constraints. The extrinsic parameter optimization module is used to construct a nonlinear optimization objective function based on the multidimensional decoupled geometric constraints, optimize the extrinsic parameters to be optimized until the nonlinear optimization objective function converges, and obtain the benchmark rigid extrinsic parameter matrix. The deformation compensation module is used to define a time-varying extrinsic function based on the reference rigid extrinsic parameter matrix and the micro-deformation compensation matrix, maintain a sliding time window of variable length, optimize the micro-deformation compensation matrix of the time-varying extrinsic function within the window, and output a corrected extrinsic parameter matrix; wherein, the length of the sliding time window is negatively correlated with the type of calibration reference in the current inspection section within a predetermined range; The deformation compensation module is also used for: The modified extrinsic parameter matrix is ​​defined as the product of the baseline rigid extrinsic parameter matrix and the micro-deformation compensation matrix, and the time-varying extrinsic parameter function is established. Using the micro-deformation compensation matrix as the optimization variable, a local adjustment model is constructed based on the radar point cloud features and image semantic features of multiple frames of calibration references accumulated within the sliding time window; Solve the local adjustment model, iteratively update the six-degree-of-freedom increment of the micro-deformation compensation matrix, and correct the reference rigid extrinsic parameter matrix based on the updated micro-deformation compensation matrix, and output the corrected extrinsic parameter matrix; In the process of optimizing the micro-deformation compensation matrix of the time-varying extrinsic function within the window, a composite cost function is also introduced, expressed as: in, For a six-degree-of-freedom increment vector, For the comprehensive reprojection residual term, λ is the deformation smoothing regularization term, used to penalize drastic changes in the deformation vector between consecutive frames. λ is the weight of the regularization term, and Ω is a 6×6 symmetric positive definite matrix that encodes the thermodynamic properties of the beam material and the prior information on the structural stiffness.

8. A device for self-calibration of a multi-source sensor of a lightning vision system, characterized in that it comprises: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the Rayvision multi-source sensor self-calibration method as described in any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the self-calibration method for the multi-source sensor as described in any one of claims 1 to 6.

Images