Online calibration method and device based on multi-information fusion for multiple panoramic cameras on a vehicle and storage medium

Abstract

The application provides a kind of online calibration method based on multi-information fusion for multiple panoramic cameras on vehicle, device and storage medium, comprising: obtaining lane line and feature point;Respectively, the original external parameter of front and rear camera is used as initial value and is traversed in the respective setting range, to obtain the optimal solution of the weighted sum of the parallelism of lane line in vehicle coordinate system, the lane line satisfying standard width, and the minimum error of the moving distance of interframe feature point and the calculated distance obtained according to vehicle odometer information;The feature points detected by left and right cameras are matched with the feature points in the overlapping area of front and rear cameras;Construct feature point map in setting time period, obtain final calibration parameter according to the mapping relationship between physical coordinates and image pixel coordinates of fused feature points. By using the online calibration method provided by the application, online calibration can be realized based on the multi-camera of vehicle itself and the vehicle odometer information without the aid of other sensing devices.

Description

Technical Field

[0001] This application relates to the field of online camera calibration technology, and in particular to an online calibration method, device and storage medium based on multi-information fusion for multiple panoramic cameras mounted on a vehicle. Background Technology

[0002] With the development of automotive technology, many intelligent vehicles are now equipped with functions such as automatic parking. Vehicles that implement automatic parking typically have four panoramic cameras (such as fisheye cameras) installed at predetermined positions on the front, rear, left, and right sides, which can capture a view of the vehicle's surroundings without blind spots. The basic principles and processes of the in-vehicle panoramic surround view function during automatic parking include: 1. Using the ground as a reference plane, after obtaining the camera's extrinsic parameters through calibration (the camera's intrinsic parameters are known at the factory), the four cameras can be mapped onto the ground around the vehicle. 2. The four images are merged to form a composite image.

[0003] The above process requires a special calibration environment: the vehicle is stationary; there are checkerboard or other markers at fixed positions around the vehicle body.

[0004] If the positions of multiple cameras change, such as replacing all or some of them, the stitching quality of the panoramic view will deteriorate. To improve stitching quality, recalibration is required to obtain the camera extrinsic parameters. Conventional offline calibration is extremely cumbersome. To address this issue, some current algorithmic work focuses on online calibration, i.e., calibrating multiple cameras while the vehicle is in motion. This approach can significantly reduce after-sales workload.

[0005] However, existing multi-camera online calibration methods, in order to achieve online calibration, typically require not only the use of multiple cameras on the vehicle itself (such as fisheye cameras) to acquire images, but also the addition of other sensors on the vehicle (such as LiDAR) or additional fixed perspective sensors on the road surface to simultaneously acquire image data. For example, when some vehicles are not equipped with sufficient features and only have multiple cameras, attempting to use the above methods results in long-standing problems such as high cost and cumbersome implementation. Furthermore, calibration while the vehicle is in motion is unstable, leading to poor accuracy of camera visual information and low calibration robustness, such as increased false detection and mismatch of feature points. Additionally, for multi-camera calibration of panoramic surround view systems, the common practice is to calibrate each camera individually, which results in deficiencies in calibration stability and robustness.

[0006] Therefore, this application aims to provide a method for online calibration of multiple onboard panoramic cameras without relying on other sensing devices (sensors), based solely on the vehicle's original multi-camera system and the vehicle's built-in odometer information. Summary of the Invention

[0007] In view of this, embodiments of this application provide an online calibration method, apparatus and storage medium based on multi-information fusion for multiple panoramic cameras mounted on a vehicle, so as to at least solve one of the problems in the prior art.

[0008] In a first aspect, embodiments of this application provide an online calibration method based on multi-information fusion for multiple vehicle-mounted panoramic cameras, the online calibration method comprising: Lane lines and feature points are obtained from images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion. The original extrinsic parameters determined during the calibration of the front and rear cameras are used as initial values ​​and traversed within their respective set ranges to obtain the optimal solution that minimizes the parallelism of the lane lines in the vehicle coordinate system, the standard width of the lane lines, and the error between the movement distance of the feature points between frames and the calculated distance obtained from the vehicle odometer information. Based on this, the corrected extrinsic parameters of the front and rear cameras are obtained. After obtaining the corrected extrinsic parameters of the front and rear cameras, the feature points of the overlapping areas of the front and rear cameras with the left and right cameras are mapped to the ground plane and their physical coordinates are obtained. The feature points detected by the left and right cameras are matched with the feature points of the overlapping areas of the front and rear cameras. The corrected extrinsic parameters of the left and right cameras are obtained based on the matched feature points. Based on the corrected extrinsic parameters of the front, rear, left, and right cameras, a feature point map of each panoramic camera within a set time period is constructed. The feature points acquired from multiple frames are tracked and fused to correct the physical coordinates. Based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates, the final calibration parameters of the four panoramic cameras are obtained simultaneously.

[0009] Secondly, embodiments of this application also provide an online calibration device based on multi-information fusion for multiple vehicle-mounted panoramic cameras, the online calibration device comprising: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the online calibration method of the above-described technical solution.

[0010] Thirdly, embodiments of this application also provide a storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the online calibration method of the above-described technical solution.

[0011] According to the online calibration method of this application, compared with the traditional panoramic surround view calibration method which requires a calibration site and a chessboard with obvious features, this application only relies on the vehicle's own multiple cameras and vehicle odometer information. By utilizing information such as lane lines and feature points on the road, a feature point map is constructed, and online calibration is completed by integrating time domain information. The method is simple, accurate, requires few resources, and has small errors.

[0012] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0013] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a flowchart of an online calibration method according to an embodiment of this application; Figure 2 This is a schematic diagram of lane line detection in an online calibration method according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of obtaining lane width in an online calibration method according to an embodiment of this application; Figure 4 This is a schematic diagram of the feature point detection and matching results of adjacent frames of the front camera in an online calibration method according to an embodiment of this application; Figure 5 This is a schematic diagram of the feature point detection and matching results of the left camera and the front and rear cameras in an online calibration method according to an embodiment of this application; Figure 6 This is a schematic diagram showing the stitching effect before and after using the online calibration method according to an embodiment of this application; wherein, Figure 6 (a) is a schematic diagram of the splicing effect before using the online calibration method according to an embodiment of this application. Figure 6(b) is a schematic diagram of the splicing effect after using the online calibration method according to an embodiment of this application; Figure 7 This is a schematic diagram of an online calibration device according to an embodiment of this application; Figure 8 This is a schematic diagram of an online calibration system according to an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0016] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0017] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0018] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0019] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0020] First, refer to Figure 1 This application describes an online calibration method 100 based on multi-information fusion for multiple vehicle-mounted panoramic cameras according to an embodiment of the present application.

[0021] like Figure 1 As shown, the online calibration method 100 may include steps S110 to S140, as detailed below: In step S110, lane lines and feature points are obtained from images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion.

[0022] In step S120, the original extrinsic parameters determined during the calibration of the front and rear cameras are used as initial values ​​and traversed within their respective set ranges to obtain the optimal solution that minimizes the parallelism of the lane lines in the vehicle coordinate system, the standard width of the lane lines, and the error between the movement distance of the inter-frame feature points and the calculated distance obtained from the vehicle odometer information. The corrected extrinsic parameters of the front and rear cameras are thus obtained.

[0023] In step S130, after obtaining the corrected extrinsic parameters of the front and rear cameras, the feature points of the overlapping areas of the front and rear cameras with the left and right cameras are mapped to the ground plane and their physical coordinates are obtained. The feature points detected by the left and right cameras are matched with the feature points of the overlapping areas of the front and rear cameras, and the corrected extrinsic parameters of the left and right cameras are obtained based on the matched feature points.

[0024] In step S140, based on the corrected extrinsic parameters of the front and rear cameras and the left and right cameras, a feature point map of each panoramic camera within a set time period is constructed. The feature points acquired from multiple frames are tracked and fused to correct the physical coordinates. Based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates, the final calibration parameters of the four panoramic cameras are obtained simultaneously.

[0025] According to the online calibration method 100 of this application embodiment, compared with the traditional panoramic surround view calibration method which requires a calibration site and a chessboard with obvious features, this application only relies on the vehicle's own multiple cameras and vehicle odometer information. By utilizing information such as lane lines and feature points on the road, a feature point map is constructed, and online calibration is completed by fusing time domain information. The method is simple, accurate, requires few resources, and improves robustness. In addition, the traditional panoramic surround view calibration method is based on a single camera and does not consider the stitching performance between multiple cameras. This application optimizes multiple cameras simultaneously, and in addition to considering the accuracy of individual cameras, it also optimizes the stitching effect.

[0026] It should be noted that, given that the camera's intrinsic parameters are known, the online calibration of the camera in this embodiment refers to acquiring the camera's extrinsic parameters.

[0027] Furthermore, in the following text, steps S110 to S130 in this embodiment can be referred to as the first stage of online calibration, and step S140 can be referred to as the second stage of online calibration, the general contents of which are as follows: Phase 1: During vehicle operation, after acquiring images from four panoramic cameras, fisheye image correction is first performed, and the corrected images are mapped to the ground plane based on the camera intrinsic and extrinsic parameters obtained from the underline calibration. Lane line features and feature point information (such as arrows, road markings, etc.) are extracted based on the corrected image and the image mapped to the ground plane. Based on the standard width of the road lines, the parallelism of the road lines (the vanishing point can be obtained), the inter-frame motion distance of feature points (obtained by the vehicle odometer ODO), and the extrinsic parameters obtained from the underline calibration as the initial extrinsic parameters, the four cameras are calibrated sequentially to obtain the camera correction extrinsic parameters.

[0028] The second stage: The first stage is a single calibration result, which does not consider temporal information, resulting in low robustness and large errors for some features. Therefore, the calibration results obtained in the first stage are used as initial parameters to map feature points acquired by multiple cameras back to physical space to construct a feature point map (global map, v-SLAM). Feature points acquired from multiple frames are then tracked and fused. Finally, the final camera calibration parameters are obtained based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates.

[0029] The following will describe in detail the above steps of the online calibration method 100 according to an embodiment of this application.

[0030] In the embodiments of this application, in step S110, lane lines and feature points are obtained based on the images captured by the four panoramic cameras at the front, rear, left, and right of the vehicle while the vehicle is in motion.

[0031] Specifically, before proceeding to step S110, it is necessary to acquire images captured by four panoramic cameras located at predetermined positions in front, behind, left, and right of the vehicle while it is in motion.

[0032] Specifically, the images captured by the four panoramic cameras include those from the front camera, rear camera, left camera, and right camera.

[0033] Then, fisheye image correction can be performed on the images captured by the four panoramic cameras to obtain a corrected image.

[0034] The extrinsic parameters of the camera at the offline calibration time are used to map the corrected image onto the ground plane.

[0035] Lane lines and feature points on the road are extracted based on the corrected image and the image mapped to the ground plane. Feature points include arrows and road markings.

[0036] Lane lines and feature points are obtained from images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion. This is mainly achieved through computer vision algorithms, which can employ existing technologies and algorithms.

[0037] For example, see Figure 2To obtain lane lines, for example, the following steps may be included: (1) Image acquisition and region of interest (ROI): The vehicle-mounted camera captures images of the road ahead. To improve processing efficiency, only the region in the image related to the lane lines can be focused on, i.e., the region of interest (ROI).

[0038] (2) Image preprocessing and edge detection: The acquired images undergo preprocessing such as noise reduction and contrast enhancement. Edge detection algorithms (such as the Canny algorithm) can be used to extract regions in the image with drastic brightness changes; these regions typically correspond to lane line edges.

[0039] (3) Inverse Perspective Transformation (IPM) / Bird's-eye View (BEV) Transformation: Because cameras are typically mounted from above, the images suffer from perspective distortion, causing lane lines to appear as trapezoids. Inverse perspective mapping (IPM) can be used to transform the image from the camera's viewpoint to a bird's-eye view (BEV). In the BEV, lane lines more closely resemble their true geometry (approximately parallel lines), greatly simplifying subsequent feature extraction and curve fitting.

[0040] (4) Feature extraction: In the bird's-eye view, lane lines are filtered based on their physical characteristics (such as a relatively constant width, typically 4-5 pixels). Methods such as line-by-line scanning can be used to identify consecutive edge point pairs that match the lane line width characteristics, while isolated noise points are filtered out. These filtered edge points are the features used to fit the lane lines.

[0041] (5) Feature fitting and lane line determination: Due to noise and occlusion, the extracted features may be incomplete or contain scattered points. Robust fitting algorithms (such as RANSAC - Random Sample Consensus) can be used to fit these features and find the curve model (such as a polynomial curve) that best represents the true lane lines. RANSAC effectively eliminates the interference of outliers, improving the accuracy of the fitting. Finally, based on the fitted curve model, the position and direction of the lane lines can be determined.

[0042] In the embodiments of this application, in step S120, the original extrinsic parameters determined when the front and rear cameras are offline are used as initial values ​​and traversed within their respective set ranges to obtain the optimal solution that minimizes the weighted sum of the optimal parallelism of the lane lines in the vehicle coordinate system, the standard width of the lane lines, and the error between the moving distance of the inter-frame feature points and the calculated distance obtained from the vehicle odometer information. The corrected extrinsic parameters of the front and rear cameras are thus obtained.

[0043] Specifically, since the front and rear cameras can capture complete lane line pairs and obtain the vanishing point of the lane lines due to their parallelism, the front and rear cameras are calibrated first.

[0044] The initial values ​​are the original external parameters (camera external parameters obtained at the factory) determined during the calibration of the front and rear cameras respectively, including the camera position ( ), camera angle ( And, iterate within their respective defined ranges, for example, within a given certain range ( ), , , , , We can iterate through the steps stepx, stepy, and stepz to find the optimal solution that makes the weighted sum of the following three terms optimal: First condition: Lane lines have optimal parallelism in the vehicle coordinate system; Second requirement: Lane markings meet standard width; The third criterion is that the error between the movement distance of the feature points between frames and the calculated distance obtained from the vehicle odometer information is minimized.

[0045] That is, the optimal solution obtained minimizes the weighted sum of the above three terms. Weighted sum = *Lane lines are optimally parallel in the vehicle coordinate system. *Lane markings meet standard width+ The error between the movement distance of feature points between frames and the calculated distance obtained from vehicle odometer information is minimized.

[0046] , and These are the weights, and by default, the weights are equal.

[0047] In this embodiment, both lane lines and feature points are assumed to be located on the ground plane (which is flat). Based on the ground plane assumption and with the Z-coordinate of the ground plane being zero, the lane lines and feature points can be mapped back to the ground in the vehicle coordinate system.

[0048] Among them, the optimal parallelism of lane lines in the vehicle coordinate system specifically means: Obtain the vanishing points of lane lines detected in all image frames during traversal, and determine the maximum distance among all vanishing points as a measure of parallelism; the smaller the distance, the better the parallelism. and These are the coordinates of any two vanishing points.

[0049] The vanishing point is the point where, in perspective projection, the projection lines of a set of parallel lines in three-dimensional space meet on a two-dimensional image plane.

[0050] Simply put, when you observe an object extending into the distance (such as a railway track, highway, or the side of a building), the originally parallel lines will appear to gradually move away from each other in the image and eventually intersect at a point, which is called the vanishing point.

[0051] Among them, lane lines meeting the standard width specifically means: First, obtain the width of the current lane line.

[0052] See Figure 3 Taking the left lane line as an example, obtaining the width of the current lane line can include the following process: Based on the corrected image after image correction of the captured image, the lane line edges are obtained through Hough transform. ; On the corrected image, the horizontal line with a fixed height of pixel H is aligned with... Intersect at points ; straight line and points Mapped to the vehicle coordinate system, respectively ; Passing point beg distance past the point beg distance ; Get lane width .

[0053] Then, the current lane width is subtracted from the standard lane widths of various lanes in the current national standard (usually 10cm, 15cm, and 20cm), and the minimum deviation is taken as the optimization amount, which is the final optimization term (the unit here is mm). in, This represents the width of the current lane line.

[0054] Considering the first optimization term (optimal parallelism of lane lines in the vehicle coordinate system), the optimization unit is pixels, while this optimization term is in mm, the final optimization term is multiplied by a coefficient. , .

[0055] Among them, the movement distance of inter-frame feature points has the smallest error compared to the calculated distance obtained from vehicle odometer information, specifically: By matching ground plane feature points between frames, under the assumption Given the extrinsic parameters at time step, calculate the feature points at... relative time The movement distance, and minimizing the error between this distance and the distance calculated based on vehicle odometer information, specifically includes: Get in Time of the first Coordinates of the feature points in the vehicle coordinate system , .

[0056] exist At any given moment, considering the vehicle's movement on the ground plane, the vehicle's coordinate system is relative to... The rotation and translation matrix at time t is, in, Let be the vehicle's rotation angle on the ground plane. It is a translation vector.

[0057] because Time and At any given moment, the camera and the vehicle body are rigid bodies in motion, but the camera does not move relative to the vehicle's coordinate system. in, For the first Each feature point in The coordinates in the current vehicle coordinate system at any given time. for Time of the first Image pixel coordinates of feature points.

[0058] When the number of point pairs exceeds 4, the rotation angle is obtained by solving the equation. as well as .

[0059] Time and The distance the vehicle traveled at any given time was .

[0060] Assuming the vehicle travel distance obtained based on vehicle odometer information is The minimum error between the movement distance of the feature points between frames and the calculated distance obtained from the vehicle odometer information is expressed as follows: For simplicity, minimizing angle changes is not considered here.

[0061] Considering the first optimization term (optimal parallelism of lane lines in the vehicle coordinate system), the optimization unit is pixels, while this optimization term is in mm, the final optimization term is multiplied by a coefficient. , .

[0062] In the embodiments of this application, after obtaining the corrected extrinsic parameters of the front and rear cameras in step S130, the feature points of the overlapping areas of the front and rear cameras and the left and right cameras are mapped to the ground plane and the physical coordinates are obtained. The feature points detected by the left and right cameras are matched with the feature points of the overlapping areas of the front and rear cameras, and the corrected extrinsic parameters of the left and right cameras are obtained based on the matched feature points.

[0063] Specifically, see Figure 4 and Figure 5 After obtaining the corrected extrinsic parameters of the front and rear cameras, the feature points in the overlapping areas of the front and rear cameras with the left and right cameras can be mapped to the ground plane and their physical coordinates obtained. Then, the feature points detected by the left and right cameras are matched with the feature points in the overlapping areas of the front and rear cameras. The matched feature points (including the physical coordinates calculated using the corrected extrinsic parameters of the front and rear cameras) are used to calculate the corrected extrinsic parameters of the left and right cameras. If the number of matched feature points is less than a set threshold, the current frame information is discarded, and the next frame image is obtained to re-match the feature points.

[0064] The process of obtaining the corrected extrinsic parameters for the left and right cameras based on the matched feature points typically involves using the mapping relationship between the image pixel coordinates and physical coordinates of the matched feature points. Existing algorithms, such as the PnP (Perspective-n-Point) algorithm, can be employed, but will not be detailed here.

[0065] In the embodiments of this application, in step S140, based on the corrected extrinsic parameters of the front and rear cameras and the left and right cameras, a feature point map of each panoramic camera within a set time period is constructed. The feature points acquired from multiple frames are tracked and fused to correct the physical coordinates. Based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates, the final calibration parameters of the four panoramic cameras are obtained simultaneously.

[0066] Specifically, after obtaining the corrected extrinsic parameters of the front and rear cameras and the left and right cameras through steps S120 and S130, the error is relatively large due to the lack of temporal information.

[0067] Therefore, based on the corrected extrinsic parameters of the front and rear cameras and the left and right cameras, the feature points acquired by the four panoramic cameras are mapped back to the physical space, and a feature point map (global map) of each panoramic camera within a set time period is constructed.

[0068] For example, a feature point map (v-slam) for each panoramic camera within a set time period can be constructed based on the v-slam method.

[0069] Specifically, the core idea of ​​constructing a feature point map for each panoramic camera within a set time period based on the vslam method is to represent the environment as a collection of sparse 3D point clouds with significant image features. In this embodiment, the application environment specifically refers to a road surface environment. Some core concepts will be explained below.

[0070] Core Concepts 1. Feature points: These are local points (such as corner points and edge points) in an image that are easily detected, distinguished, and matched. Commonly used features include: 1.1 Traditional features: SIFT, SURF, ORB (most popular, fast), BRISK, AKAZE, etc.

[0071] 1.2 Key Attributes: Location (u, v), scale, orientation, and most importantly—the descriptor. A descriptor is a high-dimensional vector used to uniquely describe the image patch information surrounding that point for matching purposes.

[0072] 2. Map Points: Map points are the corresponding points of feature points in three-dimensional space. They are not merely 3D coordinates (x, y, z), but also contain rich relational information. In this embodiment, they mainly correspond to points on the ground, i.e., z=0.

[0073] 2.1 3D Position: Coordinates in the world coordinate system. Unlike the vehicle coordinate system, it does not change with vehicle movement.

[0074] 2.2 Observation information: Which keyframes and which feature points observed this map point (i.e., co-view relationship).

[0075] 2.3 Descriptor: Usually, the "average" or "most representative" of all observed descriptors is taken and used for subsequent matching.

[0076] 2.4 Statistical information: such as average observation direction, number of successful / failed matches, etc., used to evaluate its quality.

[0077] 3. Keyframes: Not every frame is used for map construction. Typically, representative frames with significant field-of-view changes and high tracking quality are selected as keyframes. Keyframes are the foundation of the map; map points are generated by triangulation between keyframes. In this embodiment, keyframes are selected based on a fixed duration and a sufficient number of matching feature points (exceeding a set threshold).

[0078] Based on the above core concepts, taking ORB-SLAM as an example, the process of constructing a feature point map can include: Phase 1: Tracking (Front-end) Input: The image of the current frame.

[0079] Objective: Estimate the current camera pose.

[0080] Specific process: 1. Feature extraction: Extract ORB feature points in the current frame.

[0081] 2. Initial pose estimation: If it is the first frame, set it as the first keyframe, and leave the map empty. Otherwise, estimate the current pose by performing feature matching with the previous frame and using a motion model or a constant velocity model.

[0082] 3. Relocalization (if tracking is lost): Match with map points in the global map, and solve the pose using the PnP algorithm.

[0083] 4. Local map tracking: Project local map points within the current frame's observation range (view frustum) onto the current image plane. Find feature points corresponding to these projected points for matching (smaller search range, more robust). Using all matching points, the current camera pose can be refined through optimization (such as bundle adjustment, BA).

[0084] Phase Two: Local Map Building (Mid-stage, Independent Thread) Input: New keyframe.

[0085] Objective: To expand and optimize the local map.

[0086] Specific process: 1. Keyframe Insertion: Add new keyframes and their feature points to the map.

[0087] 2. Map point removal: Remove map points of poor quality (such as those with few observations or large triangulation errors).

[0088] 3. New map point creation (triangulation): Find neighboring keyframes with high co-visibility with the current keyframe. Match unmatched feature points in the current keyframe and neighboring keyframes. For successfully matched feature point pairs, calculate their 3D coordinates through triangulation and create new map points.

[0089] 4. Local Bundle Adjustment: This method jointly optimizes the pose of the current keyframe, its co-view keyframes, and all map points observed by them, thus improving the 3D positions of the map points. This is the core of ensuring local map consistency.

[0090] Phase 3: Loop Detection and Global Optimization (Backend, Independent Thread) Objective: To eliminate accumulated errors and ensure global consistency.

[0091] Specific process: 1. Loop closure detection: For new keyframes, a fast image retrieval is performed in the database using their bag-of-words model (generated from feature descriptors) to find similar past keyframes. Geometric verification is then performed to confirm whether a loop has been formed (returning to a previously visited location).

[0092] 2. Loop correction: If a loop closure is detected, calculate the relative transformation between the current keyframe and the loop closure frame. The error of the entire loop closure can be evenly distributed across the poses of all keyframes through "pose graph optimization" or "core graph optimization," resulting in a global pose adjustment and a feature point map.

[0093] In addition, after loop closure correction, a global BA can be initiated to optimize the poses of all keyframes and the positions of all map points, resulting in the optimal feature point map (global map).

[0094] Then, due to calibration deviations, the physical coordinates of the same feature point from different cameras are not equal. Feature points acquired from multiple frames can be tracked and fused to correct the physical coordinates. For example, the four physical coordinates of the same feature point constructed in images captured by different cameras can be weighted and summed to obtain its final physical coordinates for physical coordinate correction, so that the four cameras have the same physical coordinates for the same feature point.

[0095] The final calibration parameters of the four panoramic cameras are obtained simultaneously based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates. It is worth noting that unifying and fusing feature points in the temporal domain and across multiple cameras based on the feature point map can improve the robustness of the calibration.

[0096] Online calibration using feature point maps deals with sparse point data, resulting in significantly lower computational and storage costs compared to dense maps. It is computationally efficient, mature, stable, and relatively robust to dynamic objects.

[0097] Finally, the image stitching result based on the calibrated extrinsic parameters is as follows: Figure 6 As shown in the image, before online calibration, the stitching was misaligned due to changes in the camera's position (as indicated by the red circle). After online calibration, the camera's extrinsic parameters were re-acquired, resulting in a significant improvement in the stitching quality.

[0098] Based on the above description, the innovations of the online calibration method 100 according to the embodiments of this application include: 1. By fusing lane line information (lane line detection), feature point information, motion information (optical flow), positioning information (vehicle odometer, ODO), and temporal information (multi-frame) to jointly obtain camera extrinsic parameters, road line calibration is performed. 2. Simultaneous optimization of multi-camera calibration, rather than stepwise calibration of a single camera; 3. Different calibration methods are adopted for different attributes and characteristics of cameras. The front and rear cameras are calibrated by traversing lane lines and feature points, while the left and right cameras are calibrated by mapping physical coordinates to image pixel coordinates to obtain camera extrinsic parameters.

[0099] 4. During the calibration process, a feature point map is constructed to integrate time-domain information for calibration, thereby improving the robustness of the calibration.

[0100] 5. A phased calibration method (first stage and second stage) can improve calibration robustness and accuracy.

[0101] Based on the aforementioned innovations, the online calibration method 100 of this application, due to the unstable overall state caused by calibration during vehicle movement, suffers from poor accuracy of camera visual information, resulting in low calibration robustness, such as increased false detection and mismatch of feature points. This application integrates lane line information (lane line detection), feature point information, motion information (optical flow), location information (ODO), and temporal information (multi-frame) to jointly calculate camera extrinsic parameters for road line calibration. Furthermore, for multi-camera calibration in panoramic surround view, the common practice is to calibrate each camera individually, without considering the discrepancies between different cameras in the fusion area after individual calibration, leading to ghosting in the stitching effect. To address this issue, this application optimizes the calibration of the four cameras by grouping and sequentially. For example, the front and rear cameras are calibrated based on lane line information and motion information, and then the calibration results of the front and rear cameras are combined with the feature point tracking and matching information of the left and right cameras to calibrate the left and right cameras. In addition, to better optimize multiple cameras, this application constructs a feature point map centered on the vehicle body. The advantage of this map is that each camera can be calibrated independently using ODO, and the corresponding feature points are mapped onto the map. Considering the temporal nature of the data, the same feature point can be seen by multiple cameras, each with corresponding physical coordinates on the map (one physical coordinate per camera view). The four physical coordinates of the same feature point can be averaged or a weighted average can be calculated based on the distance to the installation location. Through these methods, the online calibration method of this application is simple, accurate, resource-efficient, and more robust.

[0102] refer to Figure 7 This application also provides an online calibration device 200 for implementing the online calibration method 100 according to the embodiments of this application. The online calibration device 200 includes a processor 210 and a memory 220. The online calibration device 200 may include one or more processors 210 and one or more memories 220. The memory 220 stores an executable program that is run by the processor 210. When the executable program is run by the processor 210, it causes the processor 210 to execute the online calibration method 100 described above according to the embodiments of this application.

[0103] The processor 210 may be a central processing unit (CPU) or other processing units with data processing capabilities and / or instruction execution capabilities.

[0104] The memory 220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 210 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of this application described herein, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0105] The online calibration device 200 may also include input and output devices, which are interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that... Figure 7 The components and structure of the online calibration device 200 shown are merely exemplary and not limiting. The online calibration device 200 may also have other components and structures as needed.

[0106] The input device can be a device used by a user to input commands, and can include one or more of a keyboard, mouse, microphone, and touchscreen. Furthermore, the input device can also be any interface for receiving information.

[0107] The output device can output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Furthermore, the output device can also be any other device with output functionality.

[0108] For example, the example online calibration device 200 for implementing the online calibration method 100 according to the embodiments of this application can be applied to terminal devices (such as mobile phones), tablet computers, laptops, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable devices (such as smartwatches, smart glasses, or smart helmets), augmented reality (AR) devices, virtual reality (VR) devices, smart home devices, in-vehicle computers, and other electronic devices. The embodiments of this application do not impose any limitations on this.

[0109] Those skilled in the art can understand the specific operation of the online calibration device 200 for implementing the online calibration method 100 according to the embodiments of this application in conjunction with the content described above. For the sake of brevity, the specific details will not be repeated here, but only some main operations of the processor 210 will be described.

[0110] In one embodiment of this application, when the executable program is run by the processor 210, the processor 210 performs the following steps: Lane lines and feature points are obtained from images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion. Initial values ​​are determined using the original extrinsic parameters from the front and rear cameras during their calibration. The system iterates within its defined range to obtain the optimal solution that minimizes the weighted sum of the parallelism of the lane lines in the vehicle coordinate system, the standard width of the lane lines, and the error between the inter-frame feature point movement distance and the calculated distance obtained from the vehicle's odometer. This yields the corrected extrinsic parameters for the front and rear cameras. After obtaining these corrected extrinsic parameters, feature points in the overlapping areas of the front and rear cameras with the left and right cameras are mapped to the ground plane, and their physical coordinates are obtained. Feature points detected by the left and right cameras are matched with those in the overlapping areas of the front and rear cameras, and the corrected extrinsic parameters for the left and right cameras are obtained based on the matched feature points. Based on the corrected extrinsic parameters of the front and rear cameras and the left and right cameras, a feature point map for each panoramic camera within a defined time period is constructed. Feature points acquired from multiple frames are tracked and fused to correct the physical coordinates. Finally, the final calibration parameters for all four panoramic cameras are obtained based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates.

[0111] The above exemplarily illustrates an online calibration method 100 according to an embodiment of this application. The following, in conjunction with... Figure 8 This application describes an online calibration system 300 provided in another embodiment.

[0112] Reference Figure 8 This document describes an example online calibration system 300 for implementing the online calibration method 100 of the embodiments of this application. The online calibration system 300 may include an acquisition module 310, a first correction module 320, a second correction module 330, and a final calibration module 340. Wherein: The acquisition module 310 is used to: acquire lane lines and feature points based on images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion.

[0113] The first correction module 320 is used to: take the original extrinsic parameters determined when the front and rear cameras are offline and traverse within their respective set ranges as initial values, respectively, to obtain the optimal solution that minimizes the weighted sum of the parallelism of the lane lines in the vehicle coordinate system, the standard width of the lane lines, and the error between the moving distance of the inter-frame feature points and the calculated distance obtained from the vehicle odometer information, thereby obtaining the corrected extrinsic parameters of the front and rear cameras.

[0114] The second correction module 330 is used to: after obtaining the correction extrinsic parameters of the front and rear cameras, map the feature points of the overlapping area of ​​the front and rear cameras with the left and right cameras to the ground plane and obtain the physical coordinates, match the feature points detected by the left and right cameras with the feature points of the overlapping area of ​​the front and rear cameras, and obtain the correction extrinsic parameters of the left and right cameras based on the matched feature points.

[0115] The final calibration module 340 is used to: construct a feature point map of each panoramic camera within a set time period based on the corrected extrinsic parameters of the front and rear cameras and the left and right cameras; track and fuse the feature points acquired from multiple frames to correct the physical coordinates; and simultaneously obtain the final calibration parameters of the four panoramic cameras based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates.

[0116] The online calibration system 300 proposed in this application embodiment can achieve online calibration without relying on other sensing devices, based solely on the vehicle's own multiple cameras.

[0117] Furthermore, according to embodiments of this application, this application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, it performs corresponding steps of the online calibration method 100 of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0118] Furthermore, according to embodiments of this application, this application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the online calibration method 100 of embodiments of this application.

[0119] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0120] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0122] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0123] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0124] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. An online calibration method based on multi-information fusion for multiple vehicle-mounted panoramic cameras, characterized in that, The online calibration method includes: Lane lines and feature points are obtained from images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion. The original extrinsic parameters determined during the calibration of the front and rear cameras are used as initial values ​​and traversed within their respective set ranges to obtain the optimal solution that minimizes the parallelism of the lane lines in the vehicle coordinate system, the standard width of the lane lines, and the error between the movement distance of the feature points between frames and the calculated distance obtained from the vehicle odometer information. Based on this, the corrected extrinsic parameters of the front and rear cameras are obtained. After obtaining the corrected extrinsic parameters of the front and rear cameras, the feature points of the overlapping areas of the front and rear cameras with the left and right cameras are mapped to the ground plane and their physical coordinates are obtained. The feature points detected by the left and right cameras are matched with the feature points of the overlapping areas of the front and rear cameras. The corrected extrinsic parameters of the left and right cameras are obtained based on the matched feature points. Based on the corrected extrinsic parameters of the front, rear, left, and right cameras, a feature point map of each panoramic camera within a set time period is constructed. The feature points acquired from multiple frames are tracked and fused to correct the physical coordinates. Based on the mapping relationship between the physical coordinates of the fused feature points and the image pixel coordinates, the final calibration parameters of the four panoramic cameras are obtained simultaneously.

2. The online calibration method according to claim 1, characterized in that, The optimal solution is obtained by maximizing the weighted sum of the optimal values ​​that ensures the lane lines have the best parallelism in the vehicle coordinate system, meet the standard width requirements, and minimize the error between the inter-frame feature point movement distance and the calculated distance obtained from the vehicle odometer information. The optimal parallelism of lane lines in the vehicle coordinate system specifically means: Obtain the vanishing points of lane lines detected in all image frames during traversal, and determine the maximum distance between all vanishing points as a measure of parallelism; the smaller the distance, the better the parallelism. ； Lane markings meeting standard width specifically means: Get the width of the current lane line; The current lane width is calculated by subtracting it from the standard lane width of each lane in the current national standard, and the minimum deviation is taken as the optimization amount. in, The width of the current lane line; The minimum error between the movement distance of inter-frame feature points and the calculated distance obtained from vehicle odometer information is specifically: By matching ground plane feature points between frames, under the assumption Given the extrinsic parameters at time step, calculate the feature points at... relative time The movement distance, and minimizing the error between this distance and the distance calculated based on vehicle odometer information, specifically includes: Get in Time of the first Coordinates of the feature points in the vehicle coordinate system , ； exist At any given moment, considering the vehicle's movement on the ground plane, the vehicle's coordinate system is relative to... The rotation and translation matrix at time t is, in, Let be the vehicle's rotation angle on the ground plane. It is a translation vector; because Time and At any given moment, the camera and the vehicle body are rigid bodies in motion, but the camera does not move relative to the vehicle's coordinate system. in, For the first Each feature point in The coordinates in the current vehicle coordinate system at any given time. for Time of the first Image pixel coordinates of each feature point; When the number of point pairs exceeds 4, the rotation angle is obtained by solving the equation. as well as ; Time and The distance the vehicle traveled at any given time was ; Assuming the vehicle travel distance obtained based on vehicle odometer information is The minimum error between the movement distance of the feature points between frames and the calculated distance obtained from the vehicle odometer information is expressed as follows: .

3. The online calibration method according to claim 2, characterized in that, The process of obtaining the width of the current lane line includes: Based on the corrected image after image correction of the captured image, the lane line edges are obtained through Hough transform. ; On the corrected image, the horizontal line with a fixed height of pixel H is aligned with... Intersect at points ; straight line and points Mapped to the vehicle coordinate system, respectively ; Passing point beg distance past the point beg distance ; Get lane width .

4. The online calibration method according to claim 2, characterized in that, The optimal solution is obtained by maximizing the weighted sum of the factors that ensure optimal parallelism of lane lines in the vehicle coordinate system, meet standard lane line width requirements, and minimize the error between the inter-frame feature point movement distance and the calculated distance obtained from the vehicle odometer information. Specifically, this means: The optimal solution obtained minimizes the weighted sum of the following three terms: Weighted sum = *Lane lines are optimally parallel in the vehicle coordinate system. *Lane markings meet standard width+ The error between the movement distance of feature points between frames and the calculated distance obtained from vehicle odometer information is minimized.

5. The online calibration method according to claim 1, characterized in that, The feature points of the overlapping areas of the front and rear cameras with the left and right cameras are mapped to the ground plane and their physical coordinates are obtained. The feature points detected by the left and right cameras are matched with the feature points of the overlapping areas of the front and rear cameras. Based on the matched feature points, the corrected extrinsic parameters of the left and right cameras are obtained. Specifically, this means: The corrected extrinsic parameters of the left and right cameras are obtained based on the mapping relationship between the image pixel coordinates and physical coordinates of the matched feature points.

6. The online calibration method according to claim 5, characterized in that, If the number of matched feature points is less than the set threshold, the information of the current frame is discarded, and the next frame image is obtained to re-match feature points.

7. The online calibration method according to claim 1, characterized in that, The process of tracking and fusing feature points acquired from multiple frames to correct physical coordinates specifically refers to: For the same feature point, the four physical coordinates constructed in images captured by different cameras are weighted and summed to obtain its final physical coordinates for physical coordinate correction.

8. The online calibration method according to claim 1, characterized in that, Lane lines and feature points are obtained from images captured by four panoramic cameras (front, rear, left, and right) while the vehicle is in motion, specifically including: Fisheye image correction is performed on the images captured by the four panoramic cameras to obtain the corrected images; The corrected image is mapped to the ground plane based on the camera extrinsic parameters during offline calibration; Lane lines and feature points are extracted based on the corrected image and the image mapped to the ground plane; Among them, feature points include arrows and road markings; The online calibration method further includes: acquiring images captured by four panoramic cameras located at predetermined positions in front, behind, left, and right of the vehicle while it is in motion; The images captured by the four panoramic cameras include those from the front camera, rear camera, left camera, and right camera.

9. An online calibration device based on multi-information fusion for multiple vehicle-mounted panoramic cameras, characterized in that, The online calibration device includes: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the online calibration method according to any one of claims 1 to 8.

10. A storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the online calibration method according to any one of claims 1 to 8.

Images