A visual-laser radar target parameter repairing method, system and computer program product

Abstract

The application belongs to the technical field of visual detection, and discloses a vision-laser radar target parameter repair method, a system and a computer program product. The method establishes a cross-modal target pair of a vision detection result and a laser radar detection result, identifies an external parameter drift or a time offset based on an attribution target group, and verifies a global correction by a check target group. A parameter item validity mask is generated for an abnormally corrected target, an unaffected parameter item is locked, and a candidate repair value is generated. The candidate repair value is written only when the projection residual and the time sequence motion residual are both reduced and the locked parameter item does not change beyond the limit, otherwise the parameter before repair is restored. The application can reduce the effective parameter miscoverage caused by local degradation and improve the stability of target parameter output.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor environmental perception and target detection technology, specifically to a visual-lidar target parameter repair method, system, and computer program product. Background Technology

[0002] With the development of autonomous driving, mobile robots, and intelligent transportation systems, environmental perception devices are typically equipped with both cameras and LiDAR. Cameras provide information on the target's color, texture, category, and two-dimensional contour, while LiDAR provides information on the target's spatial location, distance, size, and point cloud geometry. By performing spatiotemporal registration and correlation of the outputs from these two types of sensors, the information deficiencies of a single sensor under conditions such as low light, occlusion, long distance, or sparse point clouds can be compensated for.

[0003] Chinese patent CN113985445A discloses a 3D target detection algorithm based on camera and LiDAR data fusion. It uses a 2D target detection model to obtain the target's 2D bounding box, establishes a mapping relationship between image data and point cloud data, projects the foreground point cloud onto the camera's imaging plane, and extracts the target's 3D bounding box based on the point cloud candidate region within the 2D bounding box. This scheme can utilize image detection results to limit the point cloud search area, achieving joint target detection using camera and LiDAR information. Chinese patent CN113327296A discloses a depth-weighted online joint calibration method for LiDAR and camera. It extracts edge features from point cloud and image respectively, constructs a depth-weighted edge consistency objective function, and optimizes the sensor's extrinsic parameters. This scheme is used to correct the spatial transformation relationship between LiDAR and camera during sensor use.

[0004] Existing vision-LiDAR fusion methods typically assume that the two modalities have already achieved time synchronization and spatial calibration, or directly adjust sensor extrinsic parameters, reduce the overall weight of the corresponding modality, or replace the entire target bounding box when detection results deviate. However, inconsistencies between the visual bounding box and the LiDAR projection box can be caused by various reasons. For example, when multiple targets in different spatial locations show consistent projection offsets, it may be caused by extrinsic parameter drift between the camera and LiDAR; when the projection offset of a moving target changes with its speed, it may be caused by acquisition time offset; when only a single target exhibits anomalies, it may be caused by image blurring, local occlusion, sparse point clouds, or abnormal reflections. If global spatiotemporal anomalies are not distinguished from local degradation of a single target, it is easy to mistakenly attribute global deviations to abnormal target parameters, or to replace the entire detection result when only some parameters are affected.

[0005] Furthermore, existing correction processes typically do not set modification permissions for parameters such as target category, image plane center position, spatial center position, target size, and heading angle. When a single parameter in a modality becomes abnormal, global fusion or global bounding box replacement may simultaneously alter other parameters that were originally valid. Some methods use the projection consistency of the current frame to evaluate the correction results, but even if candidate parameters reduce the projection error of the current frame, they may not conform to the target's motion patterns in consecutive frames. If the calculation and verification of the correction amount use the same set of targets, the correction results may only be suitable for the targets involved in the calculation, and not for other spatial regions in the image.

[0006] Therefore, there is a need for a target parameter repair scheme that can distinguish between global and local degradation, limit the modification range of target parameter items, and determine whether candidate repair values ​​are allowed to be written by using mutually independent spatial and temporal residuals. Summary of the Invention

[0007] The purpose of this invention is to provide a visual-LiDAR target parameter repair method, system, and computer program product based on degradation attribution and dual residual verification, so as to distinguish between global and local causes of cross-modal detection deviation and reduce erroneous modification of unaffected parameter items during the target parameter repair process.

[0008] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution.

[0009] In a first aspect, the present invention provides a method for repairing target parameters of a vision-lidar radar, comprising:

[0010] S1. Obtain visual detection results and lidar detection results for the same observation period, project the 3D bounding box onto the image plane, and establish a cross-modal target pair for the same target; generate trajectory prediction state based on at least two historical frames, and calculate cross-modal projection residual and temporal motion residual;

[0011] S2. Divide the spatially dispersed cross-modal target pairs into non-overlapping attribution target groups and verification target groups; based on the relationship between the projection residual directions of multiple targets in the attribution target group and the target motion directions, identify extrinsic parameter drift or time offset and determine the corresponding correction amount; retain the corresponding global correction only when the projection residual of the verification target group decreases after applying the correction amount.

[0012] S3. For any target whose residual still exceeds the corresponding anomaly threshold after global correction, generate a parameter validity mask based on image quality, point cloud support status and local occlusion status, isolate the affected target parameter items and lock the remaining target parameter items.

[0013] S4. While keeping the locked target parameter items unchanged, generate candidate repair values ​​for the isolated target parameter items using the effective parameters of another modality, the trajectory prediction state, and the target category prior.

[0014] S5. Substitute the candidate repair values ​​into the corresponding isolated target parameter items in sequence; only write the candidate repair values ​​when both the cross-modal projection residual and the temporal motion residual relative to the trajectory prediction state decrease, and the residual of the locked target parameter item does not exceed the allowable increment; otherwise, restore the parameters before repair.

[0015] Specifically, in step S1: the visual detection result includes the target category, visual bounding box, and visual confidence score; the lidar detection result includes the target category, 3D bounding box, lidar confidence score, and point cloud support information within the 3D bounding box.

[0016] Based on the calibration parameters between the camera and the lidar, the 3D bounding box is projected as the lidar projection box;

[0017] For detection results where the target categories are consistent and the intersection-union ratio between the visual bounding box and the LiDAR projection box reaches the first matching threshold, a target pair is established;

[0018] For detection results where the intersection-union ratio does not reach the first matching threshold, when both the visual bounding box and the lidar projection box intersect with the prediction region corresponding to the trajectory prediction state, and the center distance between the two boxes does not exceed the distance threshold, a pair of targets to be verified is established.

[0019] The target pairs to be verified are matched one-to-one according to the degree of class consistency, intersection-union ratio, and center distance to obtain the cross-modal target pairs.

[0020] Specifically, the spatially dispersed cross-modal target pairs refer to cross-modal target pairs whose projection centers are located in different preset image partitions, or whose distance between any two projection centers reaches a spatial dispersion threshold.

[0021] The attribution target group and the verification target group each include at least two cross-modal target pairs, and the same cross-modal target pair is only assigned to one of the target groups;

[0022] Cross-modal target pairs are selected in different preset image partitions, and the selected cross-modal target pairs are alternately assigned to the attribution target group and the verification target group;

[0023] The verification target group does not participate in determining the spatial transformation correction amount corresponding to the extrinsic parameter drift and the time compensation amount corresponding to the time offset.

[0024] Specifically, in step S2: for each cross-modal target pair, a projection residual vector is generated based on the directional offset between the center of the visual bounding box and the center of the lidar projection box, and the target motion direction is determined based on the changes in the target's spatial position in historical frames.

[0025] When the projection residual vectors of multiple targets in the attribution target group have the same directional component, and the correlation between the same directional component and the motion direction of each target is lower than the directional correlation threshold, an extrinsic parameter drift identifier is generated.

[0026] When the component of the projection residual vector of a moving target in the direction of target motion increases with the target's velocity, and the components of different moving targets have the same sign, a time offset identifier is generated.

[0027] If neither the extrinsic parameter drift flag nor the time offset flag is generated, global correction is not performed, and targets with residuals exceeding the corresponding anomaly thresholds are transferred to step S3.

[0028] More specifically, in step S2:

[0029] When generating extrinsic drift identifiers, a spatial transformation correction amount, including rotation correction and translation correction, is determined based on the projection residual vector in the attribution target group, so that the projection residual of the attribution target group after applying the spatial transformation correction amount is reduced.

[0030] When generating the time offset identifier, the time compensation amount is determined based on the component of the projection residual vector in the target motion direction and the target motion velocity in the attribution target group, and the visual detection result or lidar detection result is extrapolated to a unified observation time based on the time compensation amount.

[0031] When both time offset flags and extrinsic parameter drift flags are generated simultaneously, time compensation is performed first, and then the spatial transformation correction amount is determined based on the cross-modal target pair after time compensation.

[0032] The corresponding correction amount is applied to the verification target group. The corresponding correction amount is retained when the median value of the projected residual of the verification target group decreases by at least the first improvement threshold compared with the original value. Otherwise, the corresponding correction amount is withdrawn.

[0033] Specifically, in step S3: the target parameter items include the image plane center position, spatial center position, target size, and heading angle; the parameter item validity mask includes valid state, repairable state, and unrepairable state corresponding one-to-one with each target parameter item;

[0034] The image quality is determined based on image sharpness, brightness uniformity, and target contour integrity; the point cloud support state is determined based on the number of point clouds within the target area, reflection intensity stability, and point cloud geometric integrity.

[0035] When the image quality is below the image quality threshold and the point cloud support status is normal, the center position of the image plane or the target size determined by the visual inspection results will be set to the state to be repaired.

[0036] When the point cloud support status is below the point cloud support threshold and the image quality is normal, the spatial center position, target size or heading angle determined by the lidar detection results will be set to a state to be repaired.

[0037] When neither the visual region nor the point cloud region corresponding to the target parameter item meets the valid conditions, the target parameter item is set to an unrepairable state; except for the target parameter items in the unrepairable state, all other target parameter items are given a locked attribute.

[0038] Specifically, in step S4: for the center position of the image plane or the target size in the state to be repaired, the visual confidence, the LiDAR confidence, the four coordinates of the visual bounding box, the four coordinates of the LiDAR projection box, and the four coordinates of the minimum bounding rectangle of the two boxes are normalized to form a fourteen-dimensional joint code.

[0039] The fourteen-dimensional joint encoding input parameter repair model is used to obtain candidate values ​​for the center position of the image plane, candidate values ​​for the target size, and repair confidence.

[0040] For the spatial center position, target size or heading angle that is in a state to be repaired, candidate repair values ​​are generated based on the target parameter items that are in a valid state in another mode, the trajectory prediction state and the size range or attitude range of the same type of target, respectively.

[0041] The target category prior includes the size range and heading angle variation range of the same category of targets obtained from historical samples where the sensor is in normal condition;

[0042] When another modal parameter used to generate candidate repair values ​​is in an unrepairable state, that other modal parameter is not used to generate candidate repair values.

[0043] Specifically, in step S5: the cross-modal projection residual is determined based on the center distance, size difference, and degree of non-overlap of the contours between the 3D bounding box projection result and the visual bounding box;

[0044] The temporal motion residual is determined based on the difference between the target's image plane center position, spatial center position, target size or heading angle and the corresponding predicted value in the trajectory prediction state;

[0045] The trajectory prediction state is determined before the candidate repair value is generated in the current observation period and remains unchanged until the candidate repair value verification is completed;

[0046] In descending order of repair confidence, the candidate repair values ​​are substituted into the corresponding target parameter items to be repaired.

[0047] The candidate repair value is accepted when the cross-modal projection residual after the candidate repair value is reduced by at least the second improvement threshold compared to before the substitution, the temporal motion residual is reduced by at least the third improvement threshold compared to before the substitution, and the increase in the residual component calculated by the locked target parameter does not exceed the allowable increment.

[0048] If any condition is not met, the current candidate repair value is revoked, while the previously verified candidate repair values ​​remain unchanged.

[0049] In a second aspect, the present invention also provides a vision-lidar target parameter repair system for implementing the vision-lidar target parameter repair method described in the first aspect, comprising:

[0050] The target pair establishment module is used to correlate visual detection results and lidar detection results, establish cross-modal target pairs, and generate the basic state information required for target parameter repair.

[0051] The global degradation attribution module is used to identify global degradation based on the overall deviation characteristics between cross-modal target pairs, and to correct and verify the global degradation.

[0052] The local parameter isolation module is used to determine the status of parameter items for targets that still have anomalies after global correction, and to identify the target parameter items to be repaired and the corresponding locked target parameter items.

[0053] The candidate repair module is used to generate candidate repair values ​​for the target parameter items to be repaired.

[0054] The residual verification module is used to verify the candidate repair values ​​and write them into the candidate repair values ​​or keep the original parameters of the corresponding target parameter items based on the verification results.

[0055] Thirdly, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the visual-lidar target parameter repair method described in the first aspect.

[0056] This invention distinguishes between extrinsic parameter drift and temporal offset based on the projection residual direction of spatially dispersed targets and its relationship with the target's motion state. It also determines whether to retain global correction by using a verification target group that does not participate in the correction calculation, thereby reducing the likelihood that the correction is only applicable to attribution samples. After completing global correction, a parameter validity mask is generated based on image quality, point cloud support state, and local occlusion state. Only affected target parameter items are set to the state to be repaired, while write protection is implemented for the remaining parameter items, reducing the overlap of valid parameters caused by local degradation. Before writing candidate repair values, the cross-modal projection residual and the temporal motion residual relative to the pre-generated trajectory prediction state are compared simultaneously, and the increase in residual of locked parameter items is limited. If the admission conditions are not met, the parameters before repair are restored, thereby improving the continuity and verifiability of the output parameters such as target position, size, and heading angle in degraded scenarios. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the overall process of the visual-lidar target parameter repair method of the present invention.

[0058] Figure 2 This is a schematic diagram illustrating the establishment of cross-modal target pairs based on the visual detection results and the lidar detection results of this invention.

[0059] Figure 3 This is a schematic diagram of the process of global degradation attribution and correction based on attribution target group and verification target group in this invention.

[0060] Figure 4 This is a schematic diagram of parameter validity mask generation and parameter repair control in this invention.

[0061] Figure 5 This is a flowchart of the present invention for performing dual residual verification of candidate repair values ​​and performing parameter writing or parameter rollback.

[0062] Figure 6 This is a structural block diagram of the vision-lidar target parameter repair system of the present invention. Detailed Implementation

[0063] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. It should be understood that the following embodiments are used to illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can adjust the corresponding parameters according to the camera type, the number of LiDAR lines, the target detection model, the vehicle-mounted computing platform, and the application scenario without changing the basic processing logic of the present invention.

[0064] This implementation focuses on vehicles, pedestrians, and cyclists in road traffic scenarios. A visual sensor is used to collect the target's color, texture, and two-dimensional contour information, while a lidar sensor is used to collect the target's spatial location, size, and surface reflection information. The visual detection model and the lidar detection model each output detection results. This invention performs correlation, degradation attribution, and target parameter repair on the two types of outputs at the detection result level, without requiring changes to the basic network structure of the front-end detection model.

[0065] I. Terminology and Data Structure Description

[0066] In this embodiment, the same observation period refers to the time range corresponding to the visual image and the lidar point cloud configured for one cross-modal correlation processing. When the lidar scanning frequency is 10 Hz, one lidar scanning period can be used as an observation period; when the camera acquisition frequency is higher than the lidar scanning frequency, the image frame with the closest time distance before and after the corresponding lidar frame timestamp is selected as the current visual frame.

[0067] The visual detection results include at least the target category, the visual bounding box, and the visual confidence score. The visual bounding box is represented by the coordinates of its top-left and bottom-right corners, i.e.:

[0068]

[0069] In the formula, Indicates the first A visual bounding box; , These represent the x and y coordinates of the top-left corner of the visual bounding box, respectively. , These represent the x and y coordinates of the lower right corner of the visual bounding box, respectively.

[0070] The LiDAR detection results should include at least the target category, the 3D bounding box, the LiDAR confidence score, and the point cloud support information within the 3D bounding box. The 3D bounding box is represented as:

[0071]

[0072] In the formula, Indicates the first A 3D bounding box; , , Indicates the spatial center position of the 3D bounding box; , , These represent the length, width, and height of the target, respectively. Indicates the target heading angle.

[0073] Cross-modal target pairs refer to visual detection results and LiDAR detection results that are determined to correspond to the same physical target after verification by category, projection position, trajectory prediction region, and matching score. Cross-modal target pairs can be represented as:

[0074]

[0075] The trajectory prediction state is a target prediction state generated based on at least two historical frames prior to the current observation period. The trajectory prediction state includes at least the predicted spatial center position, the predicted image plane center position, the predicted target size, the predicted heading angle, and the predicted motion velocity.

[0076] Projection residuals refer to the difference between the visual bounding box and the LiDAR 3D bounding box projected onto the image plane. Temporal motion residuals refer to the difference between the current detection parameters or candidate repair parameters and the corresponding parameters in the trajectory prediction state.

[0077] Global correction includes spatial transformation correction and temporal compensation. Spatial transformation correction is used to handle extrinsic parameter drift between the camera and the LiDAR; temporal compensation is used to handle the acquisition time offset between visual data and LiDAR data.

[0078] The parameter validity mask is used to record the status of each target parameter. Parameter status includes valid, pending repair, and unrepairable states. For parameter items not set to the pending repair state, a locking attribute is further assigned to ensure that the parameter item remains unchanged during the generation, verification, and writing of candidate repair values. Therefore, the locking attribute is a write protection attribute and is not considered a fourth data quality state that is mutually exclusive with the valid, pending repair, and unrepairable states.

[0079] II. System Overall Structure

[0080] like Figure 6 As shown, the vision-lidar target parameter repair system includes a target pair establishment module, a global degradation attribution module, a local parameter isolation module, a candidate repair module, and a residual verification module.

[0081] The target pair establishment module receives visual detection results, lidar detection results, and historical frame data, projects the lidar 3D bounding box, establishes cross-modal target pairs, and generates trajectory prediction states, projection residuals, and temporal motion residuals.

[0082] The global degradation attribution module is used to divide spatially dispersed cross-modal target pairs into attribution target groups and verification target groups. Based on the commonality of residual directions of multiple targets in the attribution target group, the target motion direction and motion velocity, it identifies extrinsic parameter drift or time offset, determines the corresponding spatial transformation correction or time compensation amount, and uses the verification target group to determine whether to retain the corresponding global correction.

[0083] The local parameter isolation module is used to handle individual targets whose projection residuals or temporal motion residuals still exceed the abnormal threshold after global correction. The local parameter isolation module generates a parameter validity mask based on image quality, point cloud support state, and local occlusion state, sets the affected target parameter items to a state to be repaired, and locks the remaining target parameter items.

[0084] The candidate repair module is used to generate candidate repair values ​​by utilizing the effective parameters of another modality, the trajectory prediction state, and the prior of the target category, while keeping the target parameter terms unchanged.

[0085] The residual verification module is used to sequentially substitute candidate repair values, compare the projection residuals and temporal motion residuals before and after repair, and calculate the residual increase value of the locked target parameter item; if the candidate repair value meets the admission conditions, it is written; otherwise, the current candidate repair value is canceled and the parameters before repair are restored.

[0086] III. The Visual-LiDAR Target Parameter Repair Method of the Invention

[0087] like Figure 1 As shown, the method of this embodiment includes steps S1 to S5.

[0088] 3.1 Establish cross-modal target pairs and calculate residuals

[0089] The system acquires visual and lidar detection results within the same observation period. The visual detection model can employ a one-stage target detection model; in this embodiment, a lightweight target detection network (YOLOv8n) is used. The lidar detection model employs a point cloud target detection network (PointPillars). The visual detection model outputs a 2D bounding box, target category, and visual confidence score; PointPillars outputs a 3D bounding box, target category, lidar confidence score, and the point cloud covered by the target bounding box.

[0090] 3.1.1 3D Bounding Frame Projection

[0091] like Figure 2 As shown, based on the camera intrinsic parameters and the camera-LiDAR extrinsic parameters, the eight vertices of the 3D bounding box are transformed from the LiDAR coordinate system to the camera coordinate system, and then projected onto the image plane. For points in 3D space... The projection satisfies:

[0092]

[0093] In the formula, , Represents the image coordinates of the projection point; Indicates the scale factor; Represents the camera intrinsic parameter matrix; This represents the homogeneous transformation matrix from the lidar coordinate system to the camera coordinate system; , , The coordinates of a point in the LiDAR coordinate system are represented by the coordinates of a point in space. The two values ​​1 in the formula are the homogeneous coordinate components of the three-dimensional spatial point and the two-dimensional image point, respectively. They are used to uniformly represent spatial rotation, translation and perspective projection in matrix form, and do not represent the parameters of the target to be detected.

[0094] The smallest bounding rectangle of the eight projection vertices is used to obtain the lidar projection frame:

[0095]

[0096] When some vertices of the 3D bounding box are located behind the camera or outside the image field of view, only projection vertices with effective positive depth and located within the extended image boundary are used to generate the projection box; when the number of effective projection vertices is less than the preset number, the corresponding LiDAR detection result is marked as not participating in the cross-modal matching of the current image.

[0097] 3.1.2 Determine the target pair and the target pair to be verified.

[0098] Calculate the intersection-union ratio (IUGR) of the visual bounding box and the LiDAR projection box respectively:

[0099]

[0100] In the formula, This represents the area of ​​the intersection of the two frames; This represents the area of ​​the union of the two frames.

[0101] Also calculate the center-to-center distance between the two frames:

[0102]

[0103] In the formula, , Indicates the coordinates of the center of the visual bounding box; , This indicates the center coordinates of the lidar projection frame.

[0104] In this embodiment, when the target categories of the two detection results are the same and the intersection-union ratio is not less than the first matching threshold... At this point, a target pair is established. The first matching threshold can be set to 0.7. For distant targets, partially occluded targets, or targets with sparse point clouds, the intersection-union ratio may be lower than the first matching threshold. In this case, the trajectory prediction state is further used to establish a target pair to be verified.

[0105] A prediction region is generated on the image plane based on the historical trajectory prediction status. If both the visual bounding box and the LiDAR projection box intersect the prediction region, and the center distance does not exceed a distance threshold, a target pair to be verified is established. The distance threshold can be set to 0.05 to 0.15 times the length of the image diagonal; in this embodiment, it is set to 0.08 times.

[0106] When a visual detection result meets the candidate criteria with multiple LiDAR detection results, or when a LiDAR detection result meets the candidate criteria with multiple visual detection results, a matching score is calculated based on class consistency, intersection-over-union ratio, center distance, and trajectory prediction overlap. The matching score can be greedily assigned one-to-one in descending order. Detection results that have already entered one cross-modal target pair will not enter other cross-modal target pairs.

[0107] 3.1.3 Trajectory Prediction Status

[0108] For targets with established tracking markers, a trajectory prediction state is generated based on at least two historical frames. In this implementation, a joint state model using constant velocity and constant angular velocity is employed, and the target state vector is:

[0109]

[0110] In the formula, , , Indicates the location of the center of the target space; , , Represents the target velocity component; Indicates the heading angle; Indicates the rate of change of heading angle; , , Indicates the three-dimensional dimensions of the target.

[0111] The trajectory prediction state for the current observation period is generated using a target trajectory prediction method, including Kalman prediction, exponential smoothing, extrapolation based on historical pose differences, or other methods capable of predicting the current target state based on historical target states. The trajectory prediction state should be determined before the generation of candidate repair values ​​for the current observation period and should remain unchanged until all candidate repair values ​​have been verified, thus preventing candidate repair values ​​from participating in the generation of their own time-series verification benchmark.

[0112] 3.1.4 Projected Residuals and Temporal Motion Residuals

[0113] Generate a projection residual vector for each cross-modal target pair:

[0114]

[0115] In the formula, , These represent the horizontal and vertical offsets between the center of the visual bounding box and the center of the LiDAR projection box, respectively. , These represent the normalized width difference and height difference between the two frames, respectively; This indicates the intersection-to-union ratio of the two frames.

[0116] The temporal motion residual is determined by the difference between the current target parameters and the corresponding parameters of the trajectory prediction state. Before repair is performed, the initial temporal motion residual is calculated using the original detection parameters; during candidate repair value verification, the temporal motion residual is recalculated using the parameters after candidate repair.

[0117] 3.2 Global Degradation Attribution and Correction

[0118] This step is used to distinguish between global anomalies caused by changes in the overall spatiotemporal relationship between sensors and local anomalies that only affect a single target. If global extrinsic parameter drift or time offset is not addressed first, and the parameters of a single target are directly repaired, the same global error may be repeatedly written into multiple target parameters. Therefore, this implementation first performs global degradation attribution and correction, and then performs local parameter repair.

[0119] 3.2.1 Division of Attribution Target Group and Verification Target Group

[0120] like Figure 3 As shown, spatially dispersed cross-modal target pairs are divided into non-overlapping attribution target groups and verification target groups. Spatial dispersion refers to the fact that the projection centers of the selected targets are located in different preset image partitions, or the distance between any two projection centers reaches the spatial dispersion threshold.

[0121] One implementation involves dividing the image into three columns and three regions or three rows and three columns (nine regions in total), and selecting targets from different regions where cross-modal target pairs exist. The selected targets are then alternately assigned to the attribution target group and the verification target group according to their region numbers. Each group includes at least two cross-modal target pairs. Cross-modal target pairs in the verification target group are not included in the calculation of spatial transformation corrections or temporal compensations.

[0122] When the number of cross-modal target pairs in the current observation period is insufficient to simultaneously form an attribution target group and a verification target group, spatially dispersed targets can be accumulated over multiple adjacent observation periods. However, the accumulation time should not exceed the time period allowed by the calibration parameters to remain stable. In this embodiment, the accumulation time does not exceed 1 second.

[0123] 3.2.2 Determination of External Parameter Drift

[0124] The residual direction is determined based on the projection residual vector of each target in the attribution target group, and the target motion direction is determined based on the target position changes in historical frames.

[0125] For any two targets and The degree of consistency of the projection residual direction can be determined by the following formula:

[0126]

[0127] In the formula, ; ; Let cosine similarity be the direction of the projection residuals of the two targets;

[0128] symbol" " denotes the Euclidean norm of a vector; The range of values ​​is The closer the value is to 1, the more consistent the directions of the projection residuals of the two targets are.

[0129] When the projected residuals of multiple spatially dispersed targets in an attribution target group have the same directional component, and the correlation between the directional component and the motion direction of each target is low, it is determined that there is extrinsic parameter drift. In actual implementation, the residual direction consistency threshold can be set to 0.8 to 0.95, and the correlation threshold between the residual direction and the motion direction can be set to 0.2 to 0.4.

[0130] In this embodiment, when at least two-thirds of the targets within the attribution target group are paired... When the correlation coefficient between the normalized component of the projection residual in the direction of target motion and the target velocity is less than 0.3, an extrinsic parameter drift indicator is generated.

[0131] 3.2.3 Spatial Transformation Correction

[0132] The spatial transformation correction includes rotation and translation corrections. Based on the center of the visual bounding box and the center of the LiDAR projection box of the attribution target group, the spatial transformation correction that reduces the projection error is calculated:

[0133]

[0134] In the formula, This represents the obtained spatial transformation correction; it represents the variable value that minimizes the subsequent objective function. Indicate the attribution target group; Indicates the first The center of a visual bounding box; This represents the projection process of a 3D bounding box. This represents a robust loss function used to reduce the impact of anomalous targets; This represents the original camera-to-LiDAR extrinsic parameter transformation matrix; This represents the extrinsic transformation matrix after applying the candidate correction; Indicates the first A 3D bounding box for a LiDAR; This represents the projection center obtained by projecting the three-dimensional bounding box onto the image plane using the corrected extrinsic parameters. The L2 norm of a vector.

[0135] Spatial transformation corrections can be solved using Gauss-Newton iterations. To avoid excessively large single corrections, upper limits are set for both rotation and displacement corrections. For example, the allowable rotation correction for a single observation period is no more than 1.5°, and the allowable displacement correction is no more than 0.15 m; if the upper limit is exceeded, the external parameters are not updated directly, but a sensor recalibration prompt is generated.

[0136] 3.2.4 Time Offset Determination and Time Compensation Amount

[0137] For the For each moving target, the component of the projection center residual in the direction of motion on the target image plane is denoted as:

[0138]

[0139] In the formula, Represents the projection center residual vector; This indicates the direction of planar motion of the normalized target image.

[0140] When multiple moving targets Having the same symbol, and With the target image plane velocity When the value increases, a time offset is determined to exist. The time compensation amount can be estimated by the following formula:

[0141]

[0142] In the formula, This indicates the amount of time compensation between visual data and LiDAR data. Indicates the first The velocity of the target in the image plane; This indicates the median value operation.

[0143] To avoid scale instability caused by targets with excessively low velocity, the time compensation amount is determined only for targets with image plane velocities greater than a velocity threshold. After determining the time compensation amount, the visual detection results or lidar detection results are extrapolated to a unified observation time based on the target velocity.

[0144] 3.2.5 Global Correction Verification

[0145] When both the extrinsic parameter drift and time offset conditions are met, time compensation is performed first, and then the spatial transformation correction amount is determined based on the attribution target group after time compensation.

[0146] The time compensation or spatial transformation correction is applied to the target group for verification, and the projection residuals of the target group are recalculated. If the median value of the projection residuals of the target group decreases by at least the first improvement threshold compared to before the correction, the corresponding global correction is retained; otherwise, the corresponding global correction is revoked.

[0147] In this embodiment, the first improvement threshold is set to 15%. The target group is not included in the correction calculation, so the verification result is used to determine whether the correction is equally effective for spatial locations that are not involved in the solution.

[0148] After retaining the global correction, the 3D bounding boxes of all LiDAR cameras in the current observation period are reprojected, and cross-modal target pairs are re-established and residuals are calculated. Targets whose residuals no longer exceed the anomaly threshold after global correction are directly output as detection results; targets that still have anomalies proceed to step S3.

[0149] 3.3 Generate a parameter validity mask and isolate affected parameter items.

[0150] like Figure 4 As shown, this step analyzes the image quality, point cloud support status, and local occlusion status of a single target that is still abnormal after global correction.

[0151] 3.3.1 Image Quality

[0152] Image quality is determined at least by image sharpness, brightness uniformity, and target outline integrity.

[0153] Image sharpness can be determined by the Laplacian response variance within the target region; a lower Laplacian response variance indicates a more blurred target region. Brightness uniformity can be determined by the mean brightness, standard deviation of brightness, and proportion of highlight pixels within the target region. Target contour integrity can be determined by the ratio of the detected target contour length within the visual bounding box to the estimated length of the complete target contour.

[0154] After normalizing the various indicators, an image quality score can be generated:

[0155]

[0156] In the formula, Indicates the image quality score; Indicates the clarity index; Indicates the brightness uniformity index; Indicates the completeness of the target outline; , , This represents the weights, and the sum of the three is 1.

[0157] 3.3.2 Point Cloud Support Status

[0158] The point cloud support state is determined at least based on the number of point clouds within the 3D bounding box, the stability of reflection intensity, and the geometric integrity of the point cloud.

[0159] The number of point clouds is determined by the number of valid points within the 3D bounding box. Reflection intensity stability can be determined by the reflection intensity dispersion coefficient. Point cloud geometric completeness can be determined by dividing the 3D bounding box into multiple spatial units, based on the proportion of spatial units with valid points to the total number of spatial units.

[0160] The point cloud support score can be expressed as:

[0161]

[0162] In the formula, This represents the point cloud support score; This represents the number of point clouds. Indicates the stability index of reflection intensity; Represents the geometric completeness of the point cloud; where, All are non-negative weight coefficients, and satisfy the following: .

[0163] 3.3.3. Partial Occlusion State

[0164] The visual partial occlusion state can be determined based on the overlap area between the current visual bounding box and other visual bounding boxes, the degree of target contour truncation, and the proportion of historical prediction regions not covered by the current contour.

[0165] The partial occlusion state of the point cloud can be determined based on the proportion of point cloud occupied on the side of the 3D bounding box facing the lidar and the proportion of point cloud missing on the side facing away from the lidar.

[0166] When an anomaly exists only on a single target, and other spatially dispersed targets maintain cross-modal alignment, while the target has obvious visual contour truncation or local point cloud missingness, the anomaly is attributed to local occlusion, and no adjustments are made to the global calibration parameters or temporal parameters.

[0167] 3.3.4. Parameter Item Status Settings

[0168] In this embodiment, the target parameters include the image plane center position, the spatial center position, the target size, and the heading angle. The target size includes two-dimensional width and two-dimensional height in the image plane, and length, width, and height in three-dimensional space.

[0169] When the image quality is below the image quality threshold and the point cloud support status is normal, the image plane center position or two-dimensional target size, which is mainly determined by the visual inspection results, is set to the state to be repaired.

[0170] When the point cloud support status is below the point cloud support threshold and the image quality is normal, the spatial center position, three-dimensional target size, or heading angle, which are mainly determined by the LiDAR detection results, will be set to a state to be repaired.

[0171] When neither the visual region nor the point cloud region corresponding to the same parameter item meets the valid conditions, the parameter item is set to an unrepairable state. Parameter items in the unrepairable state do not generate candidate repair values; instead, the original parameters are retained with an additional "to be reviewed" flag.

[0172] Parameters in both valid and unrepairable states are assigned a locking attribute; within the current observation period, only parameter items in states to be repaired are allowed to generate and write candidate repair values.

[0173] 3.4. Generate candidate repair values

[0174] The sources of candidate repair values ​​include valid parameters of another modality, trajectory prediction state, target category prior, and parameter repair model output.

[0175] 3.4.1 Image Planar Parameter Restoration Model

[0176] For the center position of the image plane or the size of the two-dimensional target in the image to be repaired, bounding box joint encoding is used to generate candidate repair values.

[0177] The input parameters include visual confidence, LiDAR confidence, four coordinates of the visual bounding box, four coordinates of the LiDAR projection box, and four coordinates of the minimum bounding rectangle of the two boxes, forming a total of fourteen-dimensional joint encoding:

[0178]

[0179] In the formula, Indicates visual confidence level; Indicates the confidence level of the lidar; Represents the four coordinates of the visual bounding box; This represents the four coordinates of the lidar projection frame; The four coordinates represent the minimum bounding rectangle of the two frames.

[0180] All coordinates are normalized by dividing by either the image width or the image height.

[0181] The parameter repair model employs a multilayer perceptron with an input dimension of 14. The number of neurons in the three hidden layers are 56, 112, and 56 respectively, and the output dimension is 5. The hidden layers use the Leaky ReLU activation function, and the output layer uses the Sigmoid activation function. The output vector is represented as follows:

[0182]

[0183] In the formula, , This parameter represents the scale of the center position of the candidate bounding box. , Indicates the candidate bounding box size ratio parameter; This indicates the confidence level of the repair.

[0184] Let the coordinates of the top left corner of the smallest bounding rectangle be... The width and height are respectively and The candidate bounding box can then be decoded using the following formula:

[0185]

[0186] In the formula, , Indicates the center coordinates of the candidate bounding box; , This indicates the width and height of the candidate bounding box.

[0187] 3.4.2 Candidate values ​​for spatial center location

[0188] When the spatial center position of the lidar is set to a state to be repaired, the following candidate values ​​can be generated:

[0189] The first candidate value is the center position of the predicted space in the trajectory prediction state;

[0190] The second candidate value is the spatial position obtained from the bottom center of the visual bounding box, the camera imaging ray, and the road plane;

[0191] The third candidate value is the spatial location estimated based on the visual bounding box height, the prior object category height, and perspective relationships.

[0192] The road plane can be obtained by fitting the current LiDAR ground points; if the current point cloud support state is insufficient to fit the road plane, the most recent valid road plane can be used.

[0193] 3.4.3 Candidate values ​​for target size

[0194] Candidate values ​​for target size include the effective size of another modality, the trajectory prediction size, and the prior size of the target category.

[0195] The target category prior is obtained from statistical analysis of historical samples where the sensor is in normal condition. For vehicles, size ranges can be established separately for small passenger cars, large vehicles, and other vehicles; for pedestrians and cyclists, height, width, and length ranges can be established separately.

[0196] When a candidate size exceeds the range of target sizes in the same category, the candidate size is truncated to the corresponding range boundary, or the corresponding repair confidence is reduced.

[0197] 3.4.4 Candidate values ​​for heading angle

[0198] The candidate heading angle values ​​include the trajectory prediction heading angle, the effective point cloud principal direction, and the visual motion direction of consecutive frames. The point cloud principal direction is only used when the point cloud geometric integrity meets the requirements; when the target velocity is lower than the heading determination velocity threshold, the visual motion direction is not used alone to determine the heading angle.

[0199] For angle parameters, the difference is calculated using sine and cosine forms to avoid... and Nearby periodic jumps.

[0200] 3.4.5. Candidate Value Sorting

[0201] Each parameter to be repaired can generate one or more candidate repair values. The repair confidence of the candidate repair values ​​is determined based on the validity of the data source, the output of the parameter repair model, the initial difference from the trajectory prediction state, and the degree of conformity with the prior of the target category.

[0202] Candidate repair values ​​are arranged from highest to lowest repair confidence for verification in step S5. If the other modal parameter used to generate candidate repair values ​​is in an unrepairable state, that other modal parameter is not used to generate candidate repair values.

[0203] 3.5 Dual Residual Verification and Parameter Writing

[0204] like Figure 5 As shown, candidate repair values ​​are substituted into the corresponding parameters to be repaired in sequence. Only one candidate repair value is substituted at a time, while the other parameters to be repaired retain their current values, and the locked parameters retain their original values.

[0205] 3.5.1 Cross-modal projection residuals

[0206] The cross-modal projection residuals are determined based on the center distance, size differences, and the degree of contour non-overlap.

[0207]

[0208] In the formula, Represents the cross-modal projection residual; This indicates the distance between the center of the visual bounding box and the center of the LiDAR projection box; Indicates the length of the image diagonal; This indicates the difference in normalized dimensions between the two frames; , , This represents the weights, and the sum of the three is 1.

[0209] 3.5.2 Temporal Motion Residuals

[0210] The temporal motion residual is determined based on the difference between the current parameters and the corresponding predicted parameters in the trajectory prediction state:

[0211]

[0212] In the formula, Represents the temporal motion residual; Indicates the current location of the center of space; Indicates the spatial center location of the trajectory prediction; Indicates the normalized distance scale; This represents the minimum periodic angle difference between the current heading angle and the predicted heading angle; This represents the normalized difference between the current size and the predicted trajectory size; , , Indicates the weight.

[0213] For cases where only two-dimensional target parameters are repaired, the corresponding temporal motion residuals can be calculated using the image plane center position and two-dimensional dimensions.

[0214] 3.5.3, Locking parameter residual increment value

[0215] Record the residual components calculated using the locking parameter term before and after substituting the candidate repair value, and obtain the increase in the residual value of the locking parameter term:

[0216]

[0217] In the formula, This represents the residual of the locked parameter term after substituting the candidate repair values; This represents the residual of the locked parameter term before the candidate repair value is substituted.

[0218] 3.5.4. Eligibility Criteria

[0219] Candidate repair values ​​are written when the following conditions are met simultaneously:

[0220]

[0221] In the formula, This represents the minimum improvement percentage of the projected residual; This represents the minimum improvement percentage of the temporal motion residual; This indicates that the residual of the locked parameter item is allowed to increment. This represents the projected residual of the current cross-modal target pair before the candidate repair value is substituted; This represents the projected residual obtained by recalculating after substituting the candidate repair values ​​into the corresponding parameter items to be repaired; This represents the minimum relative improvement ratio of the projected residual, satisfying... .

[0222] This represents the temporal motion residual between the current target state and the trajectory prediction state in the current observation period before the candidate repair values ​​are substituted. This represents the temporal motion residual between the candidate repair target state and the trajectory prediction state after substituting the candidate repair value into the corresponding parameter item to be repaired; Represents the minimum relative improvement ratio of the time-series motion residuals, satisfying .

[0223] This represents the overall increase in the residual of the locked target parameter before and after substituting the candidate repair value; This indicates the upper limit of the allowable increase in the composite residual of the locked target parameter item, and The locked target parameter items are those parameters marked as valid and prohibited from being overridden by candidate repair values ​​in the parameter item validity mask.

[0224] In this embodiment, Set to 10%, Set to 8%, Set to 3% of the total residual before repair. The corresponding threshold can be adjusted based on the validation data.

[0225] If a candidate repair value does not meet any of the admission criteria, the current candidate repair value is revoked and the parameters before repair are restored. When the current candidate value is revoked, other parameter items that have been verified and written remain unchanged.

[0226] After processing all parameters to be repaired, output the target detection results. The output results should include at least the target identifier, the parameter values ​​of each target parameter, the parameter source identifier, the global degradation attribution results, the parameter repair status, and the parameter uncertainty level.

[0227] IV. Model Structure, Training Methods, Parameter Selection, and Dataset

[0228] 4.1 Dataset and Partitioning

[0229] This implementation uses publicly available road scene data and continuously acquired data to train and validate the model. The 2D target detection, 3D target detection, and bounding box parameter repair models can use KITTI target detection data; trajectory prediction, time offset recognition, and cross-frame verification use KITTI tracking sequences or continuously acquired sequences with equivalent sensor configurations.

[0230] The KITTI target detection training component includes images, LiDAR point clouds, camera-LiDAR calibration parameters, and target annotations. In one implementation, the 7481 frames of samples in the KITTI target detection training component are divided into a training set, a validation set, and a test set, with 4489 frames in the training set, 1496 frames in the validation set, and 1496 frames in the test set. For KITTI tracking sequences or self-collected continuous sequences with continuous temporal relationships, the division is performed on a complete sequence basis, ensuring that adjacent frames within the same continuous sequence are not separately assigned to the training and test sets.

[0231] The original annotations include target category, 2D bounding box, 3D dimensions, spatial location, heading angle, truncation state, and occlusion state. During the training of the bounding box parameter repair model, the visual detection results, LiDAR detection results, and real 2D bounding boxes are mapped to each other by frame number.

[0232] The main data types and uses used in this implementation are shown in Table 1.

[0233] Table 1. Composition of Training and Validation Data

[0234]

[0235] 4.2 Front-end detection model

[0236] The visual detection model uses a YOLOv8n pre-trained model and is fine-tuned using road target data. The input image is scaled to the size required by the model, and the output is a 2D bounding box and confidence score for vehicles, pedestrians, and cyclists.

[0237] The LiDAR detection model uses PointPillars. The 3D space is divided into columnar units along the horizontal plane. Points in each columnar unit are sampled or zero-filled. A pseudo-image is generated using a columnar feature encoder. Finally, a 3D bounding box, category, and confidence score are output through a 2D convolutional backbone network and a detection head.

[0238] This invention does not require joint training of the front-end detection model. The front-end model can have its parameters fixed after training is complete, and the target parameter repair system only processes its output.

[0239] 4.3 Training of the Parameter Repair Model

[0240] The parameter repair model adopts a 14-56-112-56-5 multilayer perceptron structure. The model input is a 14-dimensional joint encoding, and the output consists of four bounding box repair parameters and repair confidence scores.

[0241] The model loss function includes the complete intersection-over-union loss and the confidence binary cross-entropy loss:

[0242]

[0243] In the formula, Indicates the total loss; This represents the complete intersection-union loss between the candidate repair box and the ground truth bounding box; This indicates the repair of the binary cross-entropy loss between the confidence score and the true matching label; , This represents the loss weight.

[0244] In one set of implementations, and All values ​​are set to 50; the optimizer used is Adam, and the initial learning rate is... The weight decays to The learning rate is updated with a decay factor of 0.95 each round; the training rounds are 200; the random number seed is set to 3407. All cross-modal target pairs in a single frame are used as a variable-length batch for forward computation, and then the loss of the effective target pairs in that frame is averaged.

[0245] During training, the initial matching threshold between the visual bounding box and the LiDAR projection box was set to 0.7. The minimum intersection-union (IU) threshold used to determine whether the repaired box corresponds to the ground truth bounding box was set to 0.2. The confidence-supervised label was determined based on the matching result between the minimum bounding rectangle and the ground truth bounding box, and not by directly using the larger value of the visual confidence score and the LiDAR confidence score as the ground truth label.

[0246] 4.4 Construction of Degraded Samples

[0247] To obtain image quality threshold, point cloud support threshold, global degradation attribution threshold, and residual improvement threshold, a controllable degradation is applied to samples with normal sensor conditions.

[0248] Image degradation includes reduced brightness, uneven brightness, motion blur, and local occlusion. Reduced brightness can be achieved by multiplying the image pixels by a factor of 0.25 to 0.70; motion blur can be achieved using a linear blur kernel with a length of 5 to 15 pixels, or a Gaussian blur with a standard deviation of 1 to 3 pixels; local occlusion can be set to 20% to 50% of the target area.

[0249] Point cloud degradation includes random point pruning, distance-dependent point pruning, reflection intensity perturbation, and spatial noise. The random point pruning ratio is set to 30% to 70%; the standard deviation of spatial noise is set to 0.02 m to 0.08 m; a higher point pruning ratio can be used for distant targets.

[0250] The extrinsic parameter drift samples are constructed by applying rotational and translational perturbations to the original extrinsic parameters. The rotational perturbation range is set to... to The translation disturbance range is set to m to 0.15 m.

[0251] The time-off sample is constructed by offsetting the corresponding times of visual frames and LiDAR frames, with the time offset range set to... ms to 150 ms.

[0252] Controlled degradation can be used to obtain known degradation types and amounts, which can be used to verify the judgment rules for extrinsic drift, temporal offset, visual degradation, point cloud degradation, and local occlusion.

[0253] 4.5 Threshold Selection

[0254] The first matching threshold, distance threshold, orientation consistency threshold, velocity threshold, image quality threshold, point cloud support threshold, and various residual abnormality thresholds are all selected on the validation set.

[0255] For the projection residuals and temporal motion residuals under normal sensor conditions, the median and median absolute deviation can be calculated. The median plus three times the median absolute deviation can be used as the initial anomaly threshold. Anomaly thresholds can be set separately for different target categories and different distance ranges to reduce the scale effect caused by the small projection size of distant targets.

[0256] V. Examples

[0257] 5.1 Example 1: Attribution and Correction of External Parameter Drift

[0258] In this embodiment, the camera acquisition frequency is 30 Hz, and the LiDAR acquisition frequency is 10 Hz. The front-end visual detection model uses YOLOv8n, and the LiDAR detection model uses PointPillars.

[0259] In a continuous data set of urban roads, a yaw angle perturbation of 0.7° and a lateral translation perturbation of 0.06 m were applied to the camera-lidar extrinsic parameters. A total of 12 spatially dispersed cross-modal target pairs were obtained in the current observation period and adjacent observation periods, including 8 vehicles, 3 pedestrians, and 1 cyclist.

[0260] Eight cross-modal target pairs were assigned to the attribution target group, and four cross-modal target pairs were assigned to the verification target group. The projection residuals of the seven targets in the attribution target group share a common component pointing towards the lower right of the image, and the median cosine similarity of the pairwise residual directions is 0.91; the absolute value of the correlation coefficient between the projection residuals and the motion direction of each target is 0.18. Therefore, extrinsic parameter drift indicators are generated.

[0261] The yaw angle correction was obtained based on the attribution target set. The lateral translation correction is m. After applying the spatial transformation correction to the verification target group, the median distance between the centers of the verification target group decreased from 14.6 pixels to 5.3 pixels, and the overall projection residual decreased by 57.1%, exceeding the first improvement threshold. Therefore, the global correction was retained.

[0262] After reprojection, the projection residuals of 10 targets are below the anomaly threshold, and the detection results are output directly; the remaining 2 targets have local occlusion and sparse point cloud respectively, and enter the local parameter isolation process.

[0263] In this embodiment, the global correction is solved using the attribution target group, and the verification target group is only used to verify the correction results. The spatial transformation correction amount is not refitted using the verification target group.

[0264] 5.2 Example 2: Time Offset Attribution and Compensation

[0265] In this embodiment, the lidar detection results are delayed by 80 ms relative to the visual detection results to simulate sensor timestamp misalignment. The test scenario includes 6 moving targets and 3 low-speed or stationary targets.

[0266] For moving targets, the component of the projection residual in the direction of target motion is calculated. For 5 out of 6 moving targets, this component is positive, and its correlation coefficient with the image plane velocity is 0.86; the center residuals of stationary targets are all lower than those of moving targets. Therefore, a time offset label is generated.

[0267] The time compensation calculated based on the attribution target group is 76 ms. The lidar detection results are extrapolated along the target movement direction by 76 ms and then applied to the verification target group. The median value of the projection residual of the verification target group decreases from 13.2 pixels to 5.7 pixels, a reduction of 56.8%, therefore the time compensation is retained.

[0268] In another observation period, an 80 ms time offset and a 0.4° extrinsic yaw disturbance are applied simultaneously. The system first performs time compensation, and then solves for the spatial transformation correction based on the time-compensated target pair. This processing order is used to reduce the interference of residuals caused by target motion in the solution of extrinsic corrections.

[0269] 5.3 Example 3: Two-dimensional parameter restoration under visual degradation

[0270] This embodiment selects a nighttime road image and applies brightness reduction and motion blur to a pedestrian target area. After processing, the image sharpness and brightness uniformity of the target area are below the corresponding thresholds, the center of the visual bounding box is shifted to the left by 9 pixels, and the width is increased by 12%; the point cloud quantity, geometric integrity, and reflection intensity stability of the corresponding LiDAR target are within the normal range.

[0271] The system sets the image plane center position and two-dimensional target size of the target to a state to be repaired, and sets the spatial center position, three-dimensional target size and heading angle to a valid state and assigns a locking attribute.

[0272] The visual confidence score is 0.43, and the LiDAR confidence score is 0.84. A 14-dimensional joint encoding is formed by the visual bounding box, the LiDAR projection box, and the minimum bounding rectangle of both boxes. Candidate repair values ​​and their repair confidence scores are obtained through a parametric repair model. These candidate repair values ​​are then substituted into the corresponding parameters to be repaired, and double residual verification is performed.

[0273] After the candidate repair value was substituted, the projection residual decreased from 0.284 to 0.151, a reduction of 46.8%; the temporal motion residual decreased from 0.119 to 0.082, a reduction of 31.1%; and the increase in the residual of the locked parameter term was 0.006, which was lower than the allowable increment, so the first candidate repair value was accepted.

[0274] During the repair process, the spatial center position, the three-dimensional target size, and the heading angle remained unchanged.

[0275] 5.4 Example 4: Spatial parameter repair and rollback under sparse point cloud conditions

[0276] In this embodiment, 65% of the point cloud data of a vehicle target 42 m in front of the vehicle is randomly removed, and spatial noise with a standard deviation of 0.05 m is added to the remaining points. The image quality of the target area is normal, but the number of point clouds and geometric integrity are below the corresponding thresholds.

[0277] The system sets the spatial center position and heading angle to a state of pending repair, and sets the image plane center position and 2D target size to a state of validity and assigns them a locked attribute. Candidate values ​​for the spatial center position include the trajectory prediction spatial center, the intersection of the visual bottom edge center and the road plane, and the spatial position estimated based on prior vehicle size.

[0278] The first candidate value is the spatial center predicted by the trajectory. After substitution, the projection residual decreased by 21%, the temporal motion residual decreased by 26%, and the increase in the residual of the locked parameter term was less than the allowable increment, so this candidate value was accepted.

[0279] The first candidate value for the heading angle was the main direction of the remaining point cloud. After substitution, the projection residual decreased by 4%, which did not reach the second improvement threshold, so this candidate value was withdrawn. The second candidate value for the heading angle was the heading angle predicted by the trajectory. After substitution, the projection residual decreased by 13%, the temporal motion residual decreased by 19%, and the residual of the locked parameter did not exceed the allowable increment, so the second candidate value was accepted.

[0280] This process shows that different parameter items generate candidate values ​​and are verified one by one, and the cancellation of a candidate value does not affect other parameter items that have already passed the verification.

[0281] VI. System Deployment and Program Implementation

[0282] The target parameter repair system of this invention can be deployed on an onboard computing platform, roadside sensing device, mobile robot controller, edge computing node, or cloud-edge collaborative sensing platform. The system can be implemented by one or more processors executing a computer program in memory, or it can be implemented collaboratively by a camera processing unit, a lidar processing unit, a target tracking unit, and a fusion repair unit.

[0283] In one implementation, visual detection results and lidar detection results are generated by corresponding front-end detection units and input into the target parameter repair system. The system establishes cross-modal target pairs according to the aforementioned method, generates trajectory prediction states, performs global degradation attribution, parameter isolation, candidate repair value generation, and dual residual verification, and outputs the repaired target parameters, repair status identifier, and corresponding confidence level. Each processing step can be executed sequentially or in parallel, provided that data dependencies are satisfied.

[0284] Functions such as target pair establishment, global degradation attribution, parameter validity mask generation, candidate repair value generation, and residual verification can be implemented by software modules, or by processors, graphics processors, neural network accelerators, field-programmable gate arrays, or combinations thereof. Modules can exchange detection results, trajectory status, residual data, parameter status, and repair results via shared memory, message queues, data buses, or interface calls.

[0285] The system can save the original detection parameters, global correction values, parameter validity masks, candidate repair values, admission judgment results, and rollback records for subsequent fault analysis, parameter calibration, and model updates. When the input data is insufficient to complete the global degradation attribution, or when the candidate repair value fails the admission verification, the system retains the parameters before repair and outputs the corresponding status indicator.

[0286] The corresponding computer program product includes a computer program that, when executed by a processor, causes the processor to implement the aforementioned target parameter repair method. The computer program can be stored in a non-transitory computer-readable storage medium or transmitted to the target computing device via a wired or wireless network.

Claims

1. A method for repairing target parameters in a vision-lidar system, characterized in that, include: S1. Obtain the visual detection results and lidar detection results for the same observation period, project the 3D bounding box onto the image plane, and establish a cross-modal target pair for the same target; The trajectory prediction state is generated based on at least two historical frames, and the cross-modal projection residual and temporal motion residual are calculated. S2. Divide the spatially dispersed cross-modal target pairs into non-overlapping attribution target groups and verification target groups; based on the relationship between the projection residual directions of multiple targets in the attribution target group and the target motion directions, identify extrinsic parameter drift or time offset and determine the corresponding correction amount; retain the corresponding global correction only when the projection residual of the verification target group decreases after applying the correction amount. S3. For any target whose residual still exceeds the corresponding anomaly threshold after global correction, generate a parameter validity mask based on image quality, point cloud support status and local occlusion status, isolate the affected target parameter items and lock the remaining target parameter items. S4. While keeping the locked target parameter items unchanged, generate candidate repair values ​​for the isolated target parameter items using the effective parameters of another modality, the trajectory prediction state, and the target category prior. S5. Substitute the candidate repair values ​​into the corresponding isolated target parameter items in turn; Candidate repair values ​​are written only when both the cross-modal projection residual and the temporal motion residual relative to the trajectory prediction state decrease, and the residual of the locked target parameter does not exceed the allowable increment; otherwise, the parameters before repair are restored.

2. The visual-lidar target parameter repair method according to claim 1, characterized in that, In step S1: The visual detection results include target category, visual bounding box, and visual confidence score; the lidar detection results include target category, 3D bounding box, lidar confidence score, and point cloud support information within the 3D bounding box. Based on the calibration parameters between the camera and the lidar, the 3D bounding box is projected as the lidar projection box; For detection results where the target categories are consistent and the intersection-union ratio between the visual bounding box and the LiDAR projection box reaches the first matching threshold, a target pair is established; For detection results where the intersection-union ratio does not reach the first matching threshold, when both the visual bounding box and the lidar projection box intersect with the prediction region corresponding to the trajectory prediction state, and the center distance between the two boxes does not exceed the distance threshold, a pair of targets to be verified is established. The target pairs to be verified are matched one-to-one according to the degree of class consistency, intersection-union ratio, and center distance to obtain the cross-modal target pairs.

3. The visual-lidar target parameter repair method according to claim 1, characterized in that: The spatially dispersed cross-modal target pair refers to a cross-modal target pair whose projection centers are located in different preset image partitions, or whose distance between any two projection centers reaches the spatial dispersion threshold. The attribution target group and the verification target group each include at least two cross-modal target pairs, and the same cross-modal target pair is only assigned to one of the target groups; Cross-modal target pairs are selected in different preset image partitions, and the selected cross-modal target pairs are alternately assigned to the attribution target group and the verification target group; The verification target group does not participate in determining the spatial transformation correction amount corresponding to the extrinsic parameter drift and the time compensation amount corresponding to the time offset.

4. The visual-lidar target parameter repair method according to claim 1, characterized in that, In step S2: For each cross-modal target pair, a projection residual vector is generated based on the directional offset between the center of the visual bounding box and the center of the lidar projection box, and the target motion direction is determined based on the changes in the target's spatial position in historical frames. When the projection residual vectors of multiple targets in the attribution target group have the same directional component, and the correlation between the same directional component and the motion direction of each target is lower than the directional correlation threshold, an extrinsic parameter drift identifier is generated. When the component of the projection residual vector of a moving target in the direction of target motion increases with the target's velocity, and the components of different moving targets have the same sign, a time offset identifier is generated. If neither the extrinsic parameter drift flag nor the time offset flag is generated, global correction is not performed, and targets with residuals exceeding the corresponding anomaly thresholds are transferred to step S3.

5. The visual-lidar target parameter repair method according to claim 4, characterized in that, In step S2: When generating extrinsic drift identifiers, a spatial transformation correction amount, including rotation correction and translation correction, is determined based on the projection residual vector in the attribution target group, so that the projection residual of the attribution target group after applying the spatial transformation correction amount is reduced. When generating the time offset identifier, the time compensation amount is determined based on the component of the projection residual vector in the target motion direction and the target motion velocity in the attribution target group, and the visual detection result or lidar detection result is extrapolated to a unified observation time based on the time compensation amount. When both time offset flags and extrinsic parameter drift flags are generated simultaneously, time compensation is performed first, and then the spatial transformation correction amount is determined based on the cross-modal target pair after time compensation. The corresponding correction amount is applied to the verification target group. The corresponding correction amount is retained when the median value of the projected residual of the verification target group decreases by at least the first improvement threshold compared with the original value. Otherwise, the corresponding correction amount is withdrawn.

6. The visual-lidar target parameter repair method according to claim 1, characterized in that, In step S3: The target parameter items include the image plane center position, spatial center position, target size, and heading angle; the parameter item validity mask includes valid state, repairable state, and unrepairable state corresponding to each target parameter item. The image quality is determined based on image sharpness, brightness uniformity, and target contour integrity; the point cloud support state is determined based on the number of point clouds within the target area, reflection intensity stability, and point cloud geometric integrity. When the image quality is below the image quality threshold and the point cloud support status is normal, the center position of the image plane or the target size determined by the visual inspection results will be set to the state to be repaired. When the point cloud support status is below the point cloud support threshold and the image quality is normal, the spatial center position, target size or heading angle determined by the lidar detection results will be set to a state to be repaired. When neither the visual region nor the point cloud region corresponding to the target parameter item meets the valid conditions, the target parameter item is set to an unrepairable state; except for the target parameter items in the unrepairable state, all other target parameter items are given a locked attribute.

7. The visual-lidar target parameter repair method according to claim 1, characterized in that, In step S4: For the center position or target size of the image plane in the state to be repaired, the visual confidence, LiDAR confidence, four coordinates of the visual bounding box, four coordinates of the LiDAR projection box, and four coordinates of the minimum bounding rectangle of the two boxes are normalized to form a fourteen-dimensional joint encoding. The fourteen-dimensional joint encoding input parameter repair model is used to obtain candidate values ​​for the center position of the image plane, candidate values ​​for the target size, and repair confidence. For the spatial center position, target size or heading angle that is in a state to be repaired, candidate repair values ​​are generated based on the target parameter items that are in a valid state in another mode, the trajectory prediction state and the size range or attitude range of the same type of target, respectively. The target category prior includes the size range and heading angle variation range of the same category of targets obtained from historical samples where the sensor is in normal condition; When another modal parameter used to generate candidate repair values ​​is in an unrepairable state, that other modal parameter is not used to generate candidate repair values.

8. The visual-lidar target parameter repair method according to claim 1, characterized in that, In step S5: The cross-modal projection residual is determined based on the center distance, size difference, and non-overlapping contours between the 3D bounding box projection result and the visual bounding box. The temporal motion residual is determined based on the difference between the target's image plane center position, spatial center position, target size or heading angle and the corresponding predicted value in the trajectory prediction state; The trajectory prediction state is determined before the candidate repair value is generated in the current observation period and remains unchanged until the candidate repair value verification is completed; In descending order of repair confidence, the candidate repair values ​​are substituted into the corresponding target parameter items to be repaired. The candidate repair value is accepted when the cross-modal projection residual after the candidate repair value is reduced by at least the second improvement threshold compared to before the substitution, the temporal motion residual is reduced by at least the third improvement threshold compared to before the substitution, and the increase in the residual component calculated by the locked target parameter does not exceed the allowable increment. If any condition is not met, the current candidate repair value is revoked, while the previously verified candidate repair values ​​remain unchanged.

9. A vision-lidar target parameter repair system, used to implement the method according to any one of claims 1 to 8, characterized in that, include: The target pair establishment module is used to correlate visual detection results and lidar detection results, establish cross-modal target pairs, and generate the basic state information required for target parameter repair. The global degradation attribution module is used to identify global degradation based on the overall deviation characteristics between cross-modal target pairs, and to correct and verify the global degradation. The local parameter isolation module is used to determine the status of parameter items for targets that still have anomalies after global correction, and to identify the target parameter items to be repaired and the corresponding locked target parameter items. The candidate repair module is used to generate candidate repair values ​​for the target parameter items to be repaired. The residual verification module is used to verify the candidate repair values ​​and write them into the candidate repair values ​​or keep the original parameters of the corresponding target parameter items based on the verification results.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual-lidar target parameter repair method according to any one of claims 1 to 8.

Images