An online calibration and evaluation method for a vehicle-mounted camera based on a high-precision map

Abstract

This application provides an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps, relating to the field of vehicle-mounted camera calibration. The method includes: retrieving a high-precision point cloud map based on the vehicle's initial inertial navigation system (INS) information; determining the vehicle's precise pose through point cloud matching and graph optimization; extracting 3D point and line features within the field of view from the high-precision point cloud map based on the precise pose and camera extrinsic parameters; extracting corresponding 2D point and line features; performing absolute depth inference on the image using a monocular depth estimation model to obtain depth values; comparing the depth values ​​to determine whether the 3D point and line features are visible in the image; projecting the visible 3D point and line features onto the image plane and matching them with the 2D point and line features; and evaluating the camera calibration status based on the matching residuals. By using tightly coupled graph optimization with high-precision map point clouds and vehicle INS information, the accumulated error of a single sensor is overcome, providing a long-term stable, centimeter-level absolute pose reference for evaluation.

Description

Technical Field

[0001] This application relates to the field of vehicle-mounted camera calibration, and in particular to an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps. Background Technology

[0002] As a core sensor in Advanced Driver Assistance Systems (ADAS) and autonomous driving systems, the calibration accuracy of vehicle-mounted cameras directly affects the accuracy of environmental perception, target detection, and localization. Currently, the calibration of vehicle-mounted cameras mainly adopts offline calibration methods, which involve one-time calibration in a laboratory or fixed location using a calibration board or a known 3D scene, or calibration and calibration before the vehicle leaves the factory.

[0003] However, offline calibration methods have the following drawbacks: It relies on specific calibration environments and equipment, resulting in high costs and poor flexibility; It cannot adapt to dynamic changes during vehicle operation, such as vibration, temperature changes, and slight shifts in camera mounting position, which can cause calibration drift. The failure to consider time synchronization errors and sensor cumulative errors during vehicle movement leads to a decrease in the reliability of calibration results in real driving environments. The lack of online monitoring and real-time evaluation mechanisms makes it impossible to detect calibration failures and correct them in a timely manner, thus affecting driving safety. Summary of the Invention

[0004] The purpose of this invention is to provide an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps, in order to solve the problems that existing technologies cannot evaluate the calibration accuracy of vehicle-mounted cameras in real time and dynamically during vehicle operation, and that rely on offline calibration environments and cannot adapt to calibration drift problems during long-term use.

[0005] The above-mentioned objective of this application is achieved through the following technical solution: S1: Based on the vehicle's initial inertial navigation system information, recall a high-precision point cloud map, and determine the vehicle's precise pose through point cloud matching and map optimization; S2: Based on the precise pose and camera external parameters, extract 3D point and line features within the field of view from the high-precision point cloud map; and use a deep learning model to extract corresponding 2D point and line features from the images captured by the vehicle camera. S3: Use a monocular depth estimation model to perform absolute depth inference on the image to obtain a depth value; based on the depth value, compare and determine whether the 3D point and line features are visible in the image, and filter out invisible 3D point and line features. S4: Project the visible 3D point and line features onto the image plane and match them with the 2D point and line features. Evaluate the camera calibration status based on the matching residual.

[0006] Optionally, step S4 includes: First, 2D line features are matched with 3D line features to obtain 2D-3D line matching results; the matching criteria for line features include line segment angle, overlap, and projection distance. The 2D-3D line matching results are used to guide the matching of 2D point features and 3D point features. Only point matching pairs with corresponding line matching are retained, and the ratio of the number of successfully matched point matching pairs to the total number of point matching pairs that should theoretically be matched is calculated, i.e., the matching ratio. The calibration status is evaluated based on the matching ratio and the mean residual. When the residual exceeds the threshold or the matching ratio is lower than the preset value, a recalibration alarm is triggered.

[0007] As an example, when the camera's intrinsic and extrinsic parameters are completely accurate, when the known three-dimensional point and line features in the map are projected onto the image plane through the pinhole camera model, they will inevitably maintain a high degree of consistency with the corresponding two-dimensional features in the image in terms of position and direction. This geometric mapping relationship is strictly determined, so the projection residual can directly reflect the degree of deviation of the calibration parameters, thereby providing a reliable quantitative basis for evaluation.

[0008] In practice, the visible 3D line features are first projected onto the image plane using the camera's intrinsic and extrinsic parameters to be evaluated. These projected line segments are then matched with 2D feature line segments based on overlap, angle, and distance. A single projected line segment can have multiple 2D matches, but a single feature line segment can only have one 3D match. The resulting 2D-3D line segment matches can guide 2D-3D point matching. 3D map points are typically endpoints of 3D map line segments; therefore, 2D feature points are also endpoints of 2D feature line segments. The average line matching distance can be dynamically set to a threshold for point matching distance. Based on cross-modal point matching according to the distance threshold, only point matches with corresponding line matches are retained. By calculating the matching ratio, it can be determined whether the current calibration has a significant deviation. Within the deviation range, the average of the matching residuals can be used to quantitatively represent the current calibration accuracy, enabling online monitoring of the camera calibration status. The current calibration is the object being evaluated, namely the set of camera parameters upon which the vehicle's real-time perception depends. This method continuously monitors the "health status" of these parameters while the vehicle is in motion.

[0009] Optionally, step S2 includes: The deep learning model for extracting two-dimensional point and line features is a model specifically trained for road scenes, used to extract semantic consistency features of lane lines, road edges, and traffic sign edges.

[0010] Optionally, step S3 includes: The steps to determine whether 3D point and line features are visible in an image include: The depth difference threshold for each projected pixel is calculated using the following formula:

[0011] in, For the threshold, For adaptive coefficients, The angle between the ray and the normal. To estimate depth; If the difference between the projected depth and the estimated depth is greater than If the pixel is not visible, then it is determined that the pixel is not visible. 3D point and line features include: 3D point features and 3D line features; For 3D line features, a line is considered visible when the proportion of visible pixels exceeds a preset threshold.

[0012] Optionally, the 3D point and line features include: 3D point features and 3D line features; the 2D point and line features include: 2D point features and 2D line features.

[0013] Optionally, step S4 includes: First, 2D line features are matched with 3D line features to obtain 2D-3D line matching results; the matching criteria for line features include line segment angle, overlap, and projection distance. The 2D-3D line matching results are used to guide the matching of 2D point features and 3D point features. Only point matching pairs with corresponding line matching are retained, and the ratio of the number of successfully matched point matching pairs to the total number of point matching pairs that should theoretically be matched is calculated, i.e., the matching ratio. The calibration status is evaluated based on the matching ratio and the mean residual. When the residual exceeds the threshold or the matching ratio is lower than the preset value, a recalibration alarm is triggered.

[0014] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps.

[0015] A computer-readable storage medium storing instructions that, when executed, perform an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps.

[0016] The beneficial effects of the technical solution provided in this application are: Using high-precision point cloud maps as a stable and semantically rich 3D geometric benchmark, this system employs precise vehicle localization, multimodal feature point / line extraction and matching, and 3D feature visibility filtering combined with monocular absolute depth estimation. Finally, projection residuals are used to perform real-time, quantitative evaluation of camera calibration accuracy. This achieves online automatic evaluation without relying on special sites or equipment, significantly reducing calibration and maintenance costs. Visibility filtering and multi-feature matching greatly improve the robustness and accuracy of the evaluation, effectively distinguishing between occlusion and calibration failure. Furthermore, it provides continuous and proactive calibration status monitoring capabilities for the long-term reliability of in-vehicle vision systems and the safety of autonomous driving. Attached Figure Description

[0017] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a step diagram of an embodiment of this application; Figure 2 This is a schematic diagram of the electronic device structure in the embodiments of this application. Detailed Implementation

[0018] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] The embodiments of this application provide an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps.

[0020] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps, as described in an embodiment of this application, including: S1: Based on the vehicle's initial inertial navigation system information, recall a high-precision point cloud map, and determine the vehicle's precise pose through point cloud matching and map optimization; S2: Based on the precise pose and camera external parameters, extract 3D point and line features within the field of view from the high-precision point cloud map; and use a deep learning model to extract corresponding 2D point and line features from the images captured by the vehicle camera. S3: Use a monocular depth estimation model to perform absolute depth inference on the image to obtain a depth value; based on the depth value, compare and determine whether the 3D point and line features are visible in the image, and filter out invisible 3D point and line features. S4: Project the visible 3D point and line features onto the image plane and match them with the 2D point and line features. Evaluate the camera calibration status based on the matching residual.

[0021] Step S1 includes: The initial inertial navigation system information of the vehicle is filtered to obtain smooth trajectory information; By using smooth trajectory information, point cloud registration and frame stacking are performed on key frames and non-key frames; High-precision point cloud maps are retrieved based on smooth trajectory information; The keyframes after frame stacking are registered with the ultra-high precision point cloud map to obtain the registered pose and the judgment result of whether the current scene is a failure scene; After all keyframes have been registered, the pose of each frame is optimized using the inter-frame registration information and the pose of the non-failure scene as constraints, and finally the optimized precise pose is output.

[0022] As one example, based on the initial INS information (initial inertial navigation system information of the vehicle), high-precision point cloud map point cloud data is recalled to determine the precise location of the vehicle. However, due to issues such as the quality of point cloud maps and single-frame laser point clouds, cumulative errors and jumps in inertial navigation, and GPS lock-off, simply fusing these positioning information can result in vehicle positioning deviations and positioning jumps, making it impossible to accurately obtain the vehicle's location.

[0023] In practice, the initial INS information is first filtered to obtain a smooth trajectory. Then, the processed trajectory information is used to perform point cloud registration and frame stacking on keyframes, and non-keyframes are also cross-registered. Simultaneously, the trajectory information is used to recall an ultra-high-precision point cloud map. Then, using the trajectory information as initial values, the stacked keyframes are registered with the ultra-high-precision point cloud map to obtain the registered pose and whether the current scene is a failure scenario. After all keyframes have been registered, the inter-frame registration information and the absolute pose of non-failure scenarios are used as constraints to perform graph optimization on the pose of each frame, finally outputting the graph-optimized vehicle pose.

[0024] Step S2 includes: The deep learning model for extracting two-dimensional point and line features is a model specifically trained for road scenes, used to extract semantic consistency features of lane lines, road edges, and traffic sign edges.

[0025] As one example, a deep learning model is used to extract 2D point and line features from camera images. In an in-vehicle environment, although LiDAR can directly acquire 3D point clouds, extracting easily matching features consistent with the image from the point cloud is not easy. Moreover, for telephoto and fisheye cameras, there is a lack of shared field of view with LiDAR. In contrast, high-precision maps are pre-constructed with high-precision surveying, covering rich 3D point and line features such as road arrows, lane lines, road edges, and sign edges. They have advantages in global consistency, semantic integrity, and long-term stability, and are not affected by real-time weather and lighting changes, making them more suitable for camera calibration and evaluation.

[0026] As one embodiment, the camera pose is first obtained based on the precise vehicle pose acquired in previous steps and the extrinsic parameters of the camera to be evaluated. Then, 3D point and line features within a certain range are retrieved from a high-precision map database based on the camera position. Next, only point and line features within the field of view are retained according to the camera orientation and field of view. Simultaneously, road images synchronously acquired by the vehicle-mounted camera are used for forward inference using a deep learning model trained on a large-scale labeled dataset to extract 2D point and line features from the images. Unlike general point and line feature extraction models, this model focuses on extracting point and line features of traffic elements in road scenes, ensuring consistency with the semantic categories of map 3D features. Through the above process, structurally consistent and semantically related set of point and line features can be obtained at both the map and image ends, providing material for subsequent calibration and evaluation.

[0027] Step S3 includes: The steps to determine whether 3D point and line features are visible in an image include: The depth difference threshold for each projected pixel is calculated using the following formula:

[0028] in, For the threshold, For adaptive coefficients, The angle between the ray and the normal. To estimate depth; If the difference between the projected depth and the estimated depth is greater than If the pixel is not visible, then it is determined that the pixel is not visible. 3D point and line features include: 3D point features and 3D line features; For 3D line features, a line is considered visible when the proportion of visible pixels exceeds a preset threshold.

[0029] As one example, the absolute depth of the camera image is inferred, and the visibility of the 3D feature is determined by comparing the image depth with the 3D feature depth.

[0030] As one example, due to factors such as road curvature, roadside obstacles, and dynamic objects, some 3D features of the map are actually occluded. After projection, there are no corresponding pixels in the image, and therefore no corresponding 2D features. Matching failures or large residuals cannot be determined whether they are caused by occlusion or calibration failure. Therefore, evaluating camera calibration based on high-precision maps and filtering out invisible 3D features is a crucial step.

[0031] In practice, the monocular depth estimation model is first used to perform forward inference on the images captured by the vehicle-mounted camera to obtain the absolute depth value of each pixel in the image. Then, the 3D point and line features of the high-precision map within the field of view are projected onto the image plane using the current camera's intrinsic and extrinsic parameters, resulting in the corresponding 2D point and line projections and their corresponding projection depths. For each pixel, a threshold is adaptively calculated based on the cosine of the angle between the ray and the object's normal. If the difference between the projected depth and the estimated depth is greater than this threshold, the pixel is considered invisible. 3D line feature projections are also composed of pixels; only when the proportion of visible pixels exceeds a certain threshold is the line feature considered visible and retained.

[0032] The 3D point and line features include: 3D point features and 3D line features; the 2D point and line features include: 2D point features and 2D line features.

[0033] Step S4 includes: First, 2D line features are matched with 3D line features to obtain 2D-3D line matching results; the matching criteria for line features include line segment angle, overlap, and projection distance. The 2D-3D line matching results are used to guide the matching of 2D point features and 3D point features. Only point matching pairs with corresponding line matching are retained, and the ratio of the number of successfully matched point matching pairs to the total number of point matching pairs that should theoretically be matched is calculated, i.e., the matching ratio. The calibration status is evaluated based on the matching ratio and the mean residual. When the residual exceeds the threshold or the matching ratio is lower than the preset value, a recalibration alarm is triggered.

[0034] As an example, when the camera's intrinsic and extrinsic parameters are completely accurate, when the known three-dimensional point and line features in the map are projected onto the image plane through the pinhole camera model, they will inevitably maintain a high degree of consistency with the corresponding two-dimensional features in the image in terms of position and direction. This geometric mapping relationship is strictly determined, so the projection residual can directly reflect the degree of deviation of the calibration parameters, thereby providing a reliable quantitative basis for evaluation.

[0035] In practice, the visible 3D line features are first projected onto the image plane using the camera's intrinsic and extrinsic parameters to be evaluated. These projected line segments are then matched with 2D feature line segments based on overlap, angle, and distance. A single projected line segment can have multiple 2D matches, but a single feature line segment can only have one 3D match. The resulting 2D-3D line segment matches can guide 2D-3D point matching. 3D map points are typically endpoints of 3D map line segments; therefore, 2D feature points are also endpoints of 2D feature line segments. The average line matching distance can be dynamically set to a threshold for point matching distance. Based on cross-modal point matching according to the distance threshold, only point matches with corresponding line matches are retained. By calculating the matching ratio, it can be determined whether the current calibration has a significant deviation. Within the deviation range, the average of the matching residuals can be used to quantitatively represent the current calibration accuracy, enabling online monitoring of the camera calibration status. The current calibration is the object being evaluated, namely the set of camera parameters upon which the vehicle's real-time perception depends. This method continuously monitors the "health status" of these parameters while the vehicle is in motion.

[0036] In one embodiment, during vehicle operation, local point cloud data matching the current position is first retrieved from a high-precision point cloud map database based on the vehicle's initial inertial navigation system (INS) information, and the precise spatial pose of the vehicle is calculated using a point cloud matching algorithm, thereby establishing a stable spatial reference for subsequent feature extraction and registration.

[0037] After determining the vehicle's location, and combining the current camera calibration parameters, 3D point and line features within the vehicle's field of view are obtained from a high-precision point cloud map. Simultaneously, road images captured by the vehicle-mounted camera are used to extract corresponding 2D point and line features using a deep learning model, thus forming a set of geometrically corresponding features at both ends of the map and the image. To improve the effectiveness and reliability of the matching, a monocular depth estimation model is further used to infer the absolute depth information of each pixel in the image. This depth is then compared with the known depth of the map's 3D features to determine whether the 3D features are visible in the current image, retaining only the visible 3D features for subsequent matching. Subsequently, based on the current camera's intrinsic and extrinsic parameters, the visible 3D point and line features are projected onto the image plane, generating corresponding projection points and lines. These are then matched with the 2D point and line features extracted from the image. By calculating the distance residual between the projection result and the corresponding 2D feature, the calibration accuracy of the camera's intrinsic and extrinsic parameters is comprehensively evaluated. When the residual exceeds a set threshold or the number of matches decreases significantly, calibration drift is determined, and an alarm is issued or recalibration is triggered.

[0038] In another embodiment, this application has the following technical effects: Dynamic spatial reference establishment: By using high-precision map point cloud and vehicle INS information for tight-coupled graph optimization, the cumulative error of a single sensor is overcome, providing a long-term stable, centimeter-level absolute pose reference for evaluation.

[0039] Visibility-driven feature filtering: Innovatively introduces absolute-scale monocular depth estimation, and through depth comparison and adaptive thresholding, effectively distinguishes whether "feature matching failure" is due to calibration drift or simply occlusion, greatly improving the robustness and reliability of the evaluation.

[0040] Structured cross-modal matching evaluation: A hierarchical matching strategy of "line-guided point" is adopted. The rich structured line features in the road scene (such as lane lines) are used to establish robust geometric constraints, which then guide point feature matching. Finally, quantitative evaluation from "whether it has failed" to "how much deviation" is achieved through matching ratio and projection residual.

[0041] This application also discloses an electronic device. (See reference...) Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0042] The communication bus 502 is used to enable communication between these components.

[0043] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0044] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0045] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the above-described online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps.

[0046] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure.

[0047] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for online calibration and evaluation of vehicle-mounted cameras based on high-precision maps, characterized in that, The method includes the following steps: S1: Based on the vehicle's initial inertial navigation system information, recall a high-precision point cloud map, and determine the vehicle's precise pose through point cloud matching and map optimization; S2: Based on the precise pose and camera external parameters, extract 3D point and line features within the field of view from the high-precision point cloud map; and use a deep learning model to extract corresponding 2D point and line features from the images captured by the vehicle camera. S3: Use a monocular depth estimation model to perform absolute depth inference on the image to obtain a depth value; based on the depth value, compare and determine whether the 3D point and line features are visible in the image, and filter out invisible 3D point and line features. S4: Project the visible 3D point and line features onto the image plane and match them with the 2D point and line features. Evaluate the camera calibration status based on the matching residual.

2. The online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in claim 1, characterized in that, Step S1 includes: The initial inertial navigation system information of the vehicle is filtered to obtain smooth trajectory information; By using smooth trajectory information, point cloud registration and frame stacking are performed on key frames and non-key frames; High-precision point cloud maps are retrieved based on smooth trajectory information; The keyframes after frame stacking are registered with the ultra-high precision point cloud map to obtain the registered pose and the judgment result of whether the current scene is a failure scene; After all keyframes have been registered, the pose of each frame is optimized using the inter-frame registration information and the pose of the non-failure scene as constraints, and finally the optimized precise pose is output.

3. The online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in claim 1, characterized in that, Step S2 includes: The deep learning model for extracting two-dimensional point and line features is a model specifically trained for road scenes, used to extract semantic consistency features of lane lines, road edges, and traffic sign edges.

4. The online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in claim 1, characterized in that, Step S3 includes: The steps to determine whether 3D point and line features are visible in an image include: The depth difference threshold for each projected pixel is calculated using the following formula: in, For the threshold, For adaptive coefficients, The angle between the ray and the normal. To estimate depth; If the difference between the projected depth and the estimated depth is greater than If the pixel is not visible, then it is determined that the pixel is not visible. 3D point and line features include: 3D point features and 3D line features; For 3D line features, a line is considered visible when the proportion of visible pixels exceeds a preset threshold.

5. The online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in claim 1, characterized in that, The 3D point and line features include: 3D point features and 3D line features; the 2D point and line features include: 2D point features and 2D line features.

6. The online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in claim 5, characterized in that, Step S4 includes: First, 2D line features are matched with 3D line features to obtain 2D-3D line matching results; the matching criteria for line features include line segment angle, overlap, and projection distance. The 2D-3D line matching results are used to guide the matching of 2D point features and 3D point features. Only point matching pairs with corresponding line matching are retained, and the ratio of the number of successfully matched point matching pairs to the total number of point matching pairs that should theoretically be matched is calculated, i.e., the matching ratio. The calibration status is evaluated based on the matching ratio and the mean residual. When the residual exceeds the threshold or the matching ratio is lower than the preset value, a recalibration alarm is triggered.

7. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform the online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the online calibration and evaluation method for vehicle-mounted cameras based on high-precision maps as described in any one of claims 1-6.

Images