Camera external parameter calibration method and device

Abstract

The invention provides a camera external parameter calibration method and equipment. The method comprises the following steps: acquiring image data in a vehicle movement process, wherein the image data comprises image data acquired by a to-be-calibrated camera and image data acquired by an anchoring camera; according to the image data, determining image data frames collected by the to-be-calibrated camera and the anchoring camera under the same timestamp; determining a relative pose of the to-be-calibrated camera corresponding to the image data frame and the anchoring camera; and calibrating the external parameters of the to-be-calibrated camera according to the relative pose and the external parameters of the anchored camera. The method is used for achieving the effect of improving the external parameter calibration precision of the camera.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a camera extrinsic parameter calibration method and device. Background Art

[0002] Camera extrinsic calibration is one of the basic and core technologies of robotic application technologies such as autonomous driving. It can determine the position and rotation information of the camera in the vehicle coordinate system.

[0003] In related technologies, when the camera extrinsic parameters are self-calibrated, the camera motion constraints are used to capture surrounding objects and extract parallel or orthogonal information in the surrounding scene. Then, the feature information of the object and the scene is extracted through digital image processing algorithms, and the feature information is analyzed and processed to obtain the camera's extrinsic parameters.

[0004] However, using the above calibration method, the calibration result has large error and low accuracy. Summary of the invention

[0005] The present application provides a camera extrinsic parameter calibration method and device to solve the problem of low accuracy of camera extrinsic parameter calibration methods in the prior art.

[0006] In a first aspect, the present application provides a camera extrinsic calibration method, comprising:

[0007] Acquire image data during the vehicle movement process, the image data including image data acquired by the camera to be calibrated and image data acquired by the anchor camera;

[0008] According to the image data, determine the image data frames collected by the camera to be calibrated and the anchor camera at the same timestamp;

[0009] Determine the relative pose of the camera to be calibrated and the anchor camera corresponding to the image data frame;

[0010] Calibrate the extrinsic parameters of the camera to be calibrated based on the relative pose and the extrinsic parameters of the anchor camera.

[0011] In some embodiments, determining the relative position of the camera to be calibrated and the anchor camera corresponding to the image data frame includes:

[0012] Determine the vehicle pose corresponding to the image data frame;

[0013] Determine the pose of the camera to be calibrated according to the pose of the vehicle and the initial external parameters of the camera to be calibrated;

[0014] Determine the position of the anchor camera based on the vehicle position and the external parameters of the anchor camera;

[0015] According to the pose of the camera to be calibrated and the pose of the anchor camera, the relative pose of the camera to be calibrated and the anchor camera is determined.

[0016] In some embodiments, determining the relative pose of the camera to be calibrated and the anchor camera according to the pose of the camera to be calibrated and the pose of the anchor camera includes:

[0017] The pose of the camera to be calibrated and the pose of the anchor camera are respectively rotationally averaged to obtain the first optimized pose of the camera to be calibrated and the first optimized pose of the anchor camera;

[0018] The bundle adjustment method is used to optimize the first optimized pose of the camera to be calibrated and the first optimized pose of the anchor camera respectively, so as to obtain the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera;

[0019] The relative pose of the camera to be calibrated and the anchor camera is determined according to the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera.

[0020] In some embodiments, calibrating the external parameters of the camera to be calibrated according to the relative pose and the external parameters of the anchor camera includes:

[0021] Determine a feature point map according to the image data frame, the feature point map including a plurality of map points;

[0022] According to the feature point map, the relative pose of the camera to be calibrated and the anchor camera is optimized to obtain the optimized relative pose;

[0023] According to the optimized relative pose and the external parameters of the anchor camera, the external parameters of the camera to be calibrated are calibrated.

[0024] In some embodiments, the relative position of the camera to be calibrated and the anchor camera is optimized according to the feature point map, including:

[0025] According to the curvatures of the multiple map points, the multiple map points are divided into edge points and plane points;

[0026] According to the sub-image, edge points and plane points of the camera to be calibrated in the image data frame, point-line residuals and point-surface residuals are constructed, wherein the sub-image of the camera to be calibrated includes map points within a preset range of the camera to be calibrated;

[0027] According to the point-line residual and point-surface residual, the relative pose of the camera to be calibrated and the anchor camera is optimized.

[0028] In some embodiments, the relative position of the camera to be calibrated and the anchor camera is optimized according to the feature point map, including:

[0029] The feature point map is optimized by using the bundle adjustment method to obtain an optimized feature point map;

[0030] According to the optimized feature point map, the relative pose of the camera to be calibrated and the anchor camera is optimized.

[0031] In a second aspect, the present application provides a camera extrinsic calibration device, comprising:

[0032] An acquisition module is used to acquire image data during the movement of the vehicle, wherein the image data includes image data acquired by the camera to be calibrated and image data acquired by the anchor camera;

[0033] A determination module, used to determine, based on the image data, image data frames acquired by the camera to be calibrated and the anchor camera at the same timestamp;

[0034] The determination module is also used to determine the relative pose of the camera to be calibrated and the anchor camera corresponding to the image data frame;

[0035] The calibration module is used to calibrate the external parameters of the camera to be calibrated according to the relative pose and external parameters of the anchor camera.

[0036] In a third aspect, the present application provides an electronic device, including: a memory and a processor;

[0037] The memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory to implement the camera extrinsic parameter calibration method in the first aspect and any one of the embodiments of the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the camera extrinsic parameter calibration method in the first aspect and any one of the embodiments of the first aspect is implemented.

[0039] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the camera extrinsic parameter calibration method in the first aspect and any one of the embodiments of the first aspect.

[0040] The camera extrinsic parameter calibration method and device provided in the present application obtain image data during the movement of the vehicle, determine the image data frames collected by the camera to be calibrated and the anchor camera at the same timestamp based on the image data, and determine the relative posture of the camera to be calibrated and the anchor camera corresponding to the image data frame. The external parameters of the camera to be calibrated are calibrated according to the relative posture and the external parameters of the anchor camera. Since the external parameters of the anchor camera are known data, there is no need to refer to the parallel or orthogonal information in the surrounding scene when calibrating the external parameters of the camera to be calibrated, which improves the calibration accuracy of the external parameters of the camera to be calibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 A flowchart of a camera extrinsic calibration method provided in one embodiment of the present application;

[0043] Figure 2 A schematic diagram of a vehicle motion trajectory provided in an embodiment of the present application;

[0044] Figure 3 Another schematic diagram of a vehicle motion trajectory provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a camera extrinsic calibration device provided in one embodiment of the present application;

[0046] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances. For example, without departing from the scope of this document, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0049] Camera extrinsic calibration is one of the foundations and core technologies of robotics applications such as autonomous driving. It is used to determine the position and rotation information of the camera in the vehicle coordinate system. According to different calibration scenarios, camera extrinsic calibration methods can be divided into production line calibration, self-calibration and active vision calibration.

[0050] Among them, the production line calibration method requires a calibrated reference object. The camera images the calibrated reference object, and then calculates the camera's internal and external parameters through digital image processing and subsequent spatial arithmetic operations. Production line calibration methods include Tsai calibration method, Zhang Zhengyou calibration method, etc. Since the calibrated reference object has a high accuracy, the camera's external parameters calibrated by this method are also highly accurate, which is suitable for application scenarios with high precision requirements. However, the reference object required needs to maintain a high degree of accuracy, and is difficult to manufacture and maintain, which limits its scope of use.

[0051] The self-calibration method uses the constraints of camera motion to capture the surrounding scene and extract some parallel or orthogonal information, and then obtains the camera parameters through digital image processing and related geometric calculations. Self-calibration methods include vanishing point calibration, camera-odometer hand-eye calibration, etc. The self-calibration method does not require special calibration objects and has high flexibility, but it relies on the surrounding scene information, the calibration result has a large error, and is not suitable for high-precision application scenarios.

[0052] Active visual calibration is to obtain the known motion information of the camera and use the motion information to calibrate the camera. This method does not require a calibration object, but only requires the camera to perform special movements and use the motion characteristics to calculate the camera parameters. The advantage of the active visual calibration method is that the algorithm is simple and a linear solution can often be obtained, so it has high robustness. However, the disadvantage is that it is costly and requires high experimental conditions. It is not suitable for scenes where the motion parameters are unknown or uncontrollable.

[0053] In response to the above problems, the present application proposes a camera extrinsic calibration method and device. In this method, the extrinsic parameters of the camera to be calibrated are calibrated according to the extrinsic parameters of the anchor camera and the relative pose of the camera to be calibrated and the anchor camera at the same timestamp. The relative pose of the camera to be calibrated and the anchor camera at the same timestamp can avoid the error accumulation problem, and because the extrinsic parameters of the anchor camera are known and accurate, the method of the present application can calibrate the extrinsic parameters of the camera to be calibrated more accurately.

[0054] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0055] To facilitate the explanation of the plan, some information should be clarified first.

[0056] In this application, a camera is designated as an anchor camera, and the external parameters of the anchor camera are known and accurate. The cameras other than the anchor camera are cameras to be calibrated, and the initial external parameters of the cameras to be calibrated are known, and their rotation error is ≤3°.

[0057] In this application, the world coordinate system is defined as w, and the vehicle coordinate system is defined as b. At the jth frame (the concept of frame is described below), the coordinate system of the anchor camera is (c1, fj ), the coordinate system of the i-th camera is (c i , f j ), the vehicle coordinate system is b, f j .

[0058] In this application, the electronic device is used as the execution subject to execute the camera extrinsic calibration method of the following embodiment. Specifically, the execution subject can be a hardware device of the electronic device, or a software application that implements the following embodiment in the electronic device, or a computer-readable storage medium installed with the software application that implements the following embodiment, or a code that implements the software application of the following embodiment.

[0059] Figure 1 FIG. 1 is a flow chart of a camera extrinsic calibration method provided by an embodiment of the present application. Figure 1 As shown, with the electronic device as the execution subject, the method of this embodiment may include the following steps:

[0060] S101, acquiring image data during vehicle movement.

[0061] The image data includes image data collected by the camera to be calibrated and image data collected by the anchor camera.

[0062] In this embodiment, the vehicle movement process may be a process in which the vehicle moves along a preset trajectory. Figure 2 , Figure 3 As shown in the figure, the vehicle can drive in the order of straight, uphill, downhill, straight, and then drive in an S shape on flat ground to complete the preset motion trajectory. During the vehicle movement, the camera to be calibrated and the anchor camera continuously record image data, and the wheel odometer data is continuously recorded through the sensor.

[0063] Optionally, the electronic device may select image data of the vehicle in a non-stationary state based on the wheel odometer data.

[0064] S102: Determine, based on the image data, image data frames acquired by the camera to be calibrated and the anchor camera at the same timestamp.

[0065] In this embodiment, based on the image data of the vehicle in a non-stationary state, the image data collected by the camera to be calibrated and the anchor camera at the same timestamp are selected as the image data frame, that is, the image data frame includes the image data collected by the camera to be calibrated and the anchor camera, and the camera collection timestamp is the same timestamp. This step is the key to ensure the accuracy of subsequent analysis, because only the image data at the same moment can accurately reflect the relative position relationship between the camera to be calibrated and the anchor camera.

[0066] It should be understood that the electronic device can filter out multiple frames of image data from the image data.

[0067] Optionally, after determining the image data frame, the electronic device may also use a scale-invariant feature transform (SIFT) method to extract and match feature points of multiple image data in the image data frame. After the matching is completed, only the successfully matched feature point information, matching information, and wheel odometer data corresponding to the image data frame timestamp may be retained for subsequent data processing.

[0068] Among them, the successfully matched feature points include their specific positions in the image and the feature vectors that describe the feature points. Matching information includes matching pairs, matching confidence, and geometric constraints. Wheel odometer data includes the vehicle's position information, speed information, and corresponding timestamps. Feature points and matching information are the basis for image matching and stitching, and are used to explain the correspondence between images. Wheel odometer data provides information about the movement of vehicles in physical space, which facilitates understanding the relative position relationship between images and building maps.

[0069] S103: Determine the relative position and posture of the camera to be calibrated and the anchor camera corresponding to the image data frame.

[0070] In this embodiment, the electronic device obtains the initial pose of the camera to be calibrated and the initial pose of the anchor camera in the world coordinate system according to the image data frame, and determines the relative pose of the camera to be calibrated and the anchor camera when the image data frame is collected according to the initial pose of the camera to be calibrated and the initial pose of the anchor camera. The relative pose includes the rotation and translation relationship between the camera to be calibrated and the anchor camera, which is an important basis for calibrating the external parameters of the camera to be calibrated.

[0071] S104: Calibrate the external parameters of the camera to be calibrated according to the relative posture and the external parameters of the anchor camera.

[0072] In this embodiment, the external parameters of the anchor camera are known and accurate. The electronic device uses a professional calibration algorithm to calculate and process based on the determined relative position and the known external parameter data of the anchor camera, so as to accurately calibrate the external parameters of the camera to be calibrated.

[0073] In the camera extrinsic calibration method provided in this embodiment, the electronic device collects image data of the vehicle in the process of moving along the preset trajectory, so that the collected image features are richer, the feature distribution and feature constraints are more balanced, and the algorithm robustness is better. According to the relative posture of the camera to be calibrated and the anchor camera, and the known extrinsic parameters of the anchor camera, the extrinsic parameters of the camera to be calibrated are calibrated, without the need for pre-calibration of reference objects, low requirements for the calibration scene, easy operation, and high calibration accuracy, the accuracy can be controlled within 0.5° (rotation) and 5cm (translation).

[0074] In some embodiments, the specific implementation of step S103 includes the following steps:

[0075] S201, determining a vehicle posture corresponding to an image data frame.

[0076] In this embodiment, the electronic device can determine the vehicle posture corresponding to the image data frame based on the wheel odometer data integration. Specifically, the electronic device can determine the position of the vehicle when the image data frame is collected based on the timestamp corresponding to the image data frame, and then determine the speed, acceleration and posture information of the vehicle when the image data frame is collected in combination with the wheel odometer data, such as pitch angle, yaw angle, roll angle, etc. Based on the above data, the electronic device can calculate the vehicle posture when the image data frame is collected through the vehicle kinematic model.

[0077] S202: Determine the position and posture of the camera to be calibrated according to the vehicle position and posture and the initial external parameters of the camera to be calibrated.

[0078] The initial external parameters of the camera to be calibrated usually include the position coordinates and attitude angle of the camera relative to a fixed point on the vehicle. Therefore, after obtaining the vehicle's posture, the electronic device can use the known initial external parameters of the camera to be calibrated to calculate the posture of the camera to be calibrated in the world coordinate system corresponding to the image data frame.

[0079] S203: Determine the position and posture of the anchor camera according to the vehicle position and posture and the external parameters of the anchor camera.

[0080] Similar to step S202, the external parameters of the anchor camera are known and accurate. After obtaining the vehicle posture, the electronic device can use the known external parameters of the anchor camera to calculate the posture of the anchor camera corresponding to the image data frame in the world coordinate system. Since the external parameters of the anchor camera are relatively accurate, the calculated anchor camera posture has a high accuracy.

[0081] S204: Determine the relative pose of the camera to be calibrated and the anchor camera according to the pose of the camera to be calibrated and the pose of the anchor camera.

[0082] By comparing the pose of the camera to be calibrated with the pose of the anchor camera, the electronic device can calculate the relative position and direction difference between the camera to be calibrated and the anchor camera, thereby determining the relative pose of the camera to be calibrated and the anchor camera.

[0083] Optionally, the specific implementation of step S204 includes the following steps:

[0084] S2041. Perform rotational averaging on the pose of the camera to be calibrated and the pose of the anchor camera, respectively, to obtain a first optimized pose of the camera to be calibrated and a first optimized pose of the anchor camera.

[0085] Specifically, the camera pose includes position information and rotation information. The electronic device can use a rotation averaging algorithm to perform rotation averaging processing on the pose of the camera to be calibrated and the pose of the anchor camera to obtain the first optimized pose of the camera to be calibrated and the first optimized pose of the anchor camera.

[0086] After rotational averaging processing, the posture deviation caused by measurement error or noise can be reduced, and a more stable and accurate posture can be obtained.

[0087] S2042, using the bundle adjustment method, respectively optimize the first optimized pose of the camera to be calibrated and the first optimized pose of the anchor camera to obtain the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera.

[0088] Specifically, the electronic device initializes the bundle adjustment method according to the first optimized posture of the camera to be calibrated and the first optimized posture of the anchor camera, and the feature points in the image data frame.

[0089] The bundle adjustment method is an iterative optimization algorithm that optimizes the camera pose and the position of feature points by minimizing the reprojection error. The entire optimization process can be completed using tools such as the ceres solver or general graph optimization (g2o). The electronic device calculates the reprojection error according to formula (1).

[0090]

[0091] Among them, e phase1 is the reprojection error;

[0092] represents the position of the i-th camera at the j-th frame;

[0093] P k is the kth map point;

[0094] represents the projection function of the i-th camera, which can project a point in the camera coordinate system onto the camera image;

[0095] u l represents the lth feature point.

[0096] The electronic device performs the iterative steps of the bundle adjustment method, continuously updating the poses of the camera to be calibrated and the anchor camera, as well as the positions of the feature points. In each iteration, the electronic device calculates the reprojection error of the feature points in the current pose and adjusts the pose and the position of the feature points according to the error gradient. The iterative process continues until it reaches a preset convergence condition, such as the reprojection error is less than a threshold or the number of iterations reaches an upper limit.

[0097] After optimization by the bundle adjustment method, the electronic device can determine the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera.

[0098] S2043: Determine the relative pose of the camera to be calibrated and the anchor camera according to the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera.

[0099] Specifically, the electronic device calculates the relative pose of the camera to be calibrated and the anchor camera according to the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera.

[0100] The rotation matrix part of the relative pose It can be obtained by formula (2).

[0101]

[0102] in, The rotation matrix part representing the position of the anchor camera in the world coordinates in the jth frame;

[0103] The rotation matrix part representing the position of the i-th camera in the j-th frame in world coordinates;

[0104] The log(.) function is used to convert the rotation matrix into a rotation vector;

[0105] The exp(.) function can convert a rotation vector into a rotation matrix.

[0106] The translation vector part of the relative pose It can be obtained by formula (3).

[0107]

[0108] in, Represents the translation vector part of the anchor camera's pose in world coordinates in the jth frame;

[0109] Represents the translation vector portion of the pose of the i-th camera in the j-th frame in world coordinates.

[0110] In this embodiment, the electronic device optimizes the posture of the camera to be calibrated and the posture of the anchor camera through the rotation averaging algorithm and the bundle adjustment method to obtain more accurate posture information, so that the relative posture of the camera to be calibrated and the anchor camera determined by the electronic device is more accurate, and the accuracy of the external parameter calibration of the camera to be calibrated is also improved.

[0111] In some embodiments, the specific implementation of step S104 includes the following steps:

[0112] S301. Determine a feature point map according to an image data frame, where the feature point map includes a plurality of map points.

[0113] Among them, the feature point map uses geometric features to represent the environment, such as points, lines, and surfaces, and stores posture information at the same time, which has the advantages of small data storage and calculation amount. In this embodiment, the feature point map can only use points to represent the environment.

[0114] Specifically, the electronic device can use an image processing algorithm, such as SIFT, to extract key feature points in the image data frame. Feature points are usually points with significant local characteristics in the image, such as corner points, edge points or spots.

[0115] The electronic device matches the feature points in different image data frames through feature matching algorithms, such as brute force matching and triangulation algorithms, and establishes corresponding relationships between the feature points. Based on these corresponding relationships, the electronic device can construct a feature point map, which contains multiple map points, each of which corresponds to an actual physical point in the scene and is visible in multiple image data frames.

[0116] S302: Optimize the relative pose of the camera to be calibrated and the anchor camera according to the feature point map to obtain an optimized relative pose.

[0117] Specifically, the electronic device iteratively adjusts the relative pose parameters by minimizing the reprojection error between the map points in the feature point map and the image feature points. After multiple iterations of optimization, the electronic device outputs the optimized relative pose, which can more accurately reflect the actual spatial relationship between the camera to be calibrated and the anchor camera.

[0118] S303: Calibrate the external parameters of the camera to be calibrated according to the optimized relative pose and the external parameters of the anchor camera.

[0119] Optionally, the electronic device uses the optimized relative pose and the extrinsic parameters of the anchor camera to calculate the extrinsic parameters of the camera to be calibrated through matrix operations. Specifically, the electronic device can multiply the extrinsic parameter matrix of the anchor camera with the optimized relative pose matrix to obtain the extrinsic parameter matrix of the camera to be calibrated.

[0120] To ensure the accuracy of the calibration results, the electronic device may also verify and adjust the external parameters through other verification means, such as 3D reconstruction. If the verification result shows a large error, the electronic device may return to step S302 to re-optimize the posture, or adjust other parameters in the calibration process.

[0121] The method of this embodiment uses a feature point map to optimize the relative posture of the camera to be calibrated and the anchor camera, thereby improving the accuracy of the relative posture and facilitating more accurate calibration of the external parameters of the camera to be calibrated.

[0122] Optionally, a specific implementation of step S302 includes the following steps:

[0123] S401. Divide the plurality of map points into edge points and plane points according to the curvatures of the plurality of map points.

[0124] Specifically, the curvature of the map point in the feature point map is calculated according to formula (4).

[0125]

[0126] Among them, P k Represents any one of multiple map points;

[0127] Cloud k P k Other map points within the preset range;

[0128] Curva k P k The curvature.

[0129] After calculating the curvature of each map point, the curvature can be higher than Curva high The map points with curvature lower than Curva are marked as edge points. low The map points are marked as plane points. high and Curva low The value can be set as required.

[0130] S402: constructing point-line residuals and point-surface residuals according to the sub-image, edge points and plane points of the camera to be calibrated in the image data frame.

[0131] The sub-image of the camera to be calibrated includes map points within a preset range of the camera to be calibrated. It should be understood that the sub-image of the anchor camera includes map points within a preset range of the anchor camera.

[0132] Specifically, the electronic device first divides the image data frame into sub-images for each camera.

[0133] definition is the sub-image of the i-th camera in the j-th frame; define is the sub-image of the anchor camera in the jth frame. The sub-images in the same frame are matched with each other, and the point-line residual and point-surface residual between the sub-images are constructed according to formula (5) and formula (6).

[0134]

[0135] in, represents the relative pose of the i-th camera and the anchor camera;

[0136] for The mth edge point in ;

[0137] and for Middle distance The two nearest edge points;

[0138] for The nth plane point in ;

[0139] and for Middle distance The 3 nearest plane points.

[0140] S403: Optimize the relative pose of the camera to be calibrated and the anchor camera according to the point-line residual and the point-surface residual.

[0141] Specifically, the relative poses can be balanced by pose graph optimization Let the residual e in formula (7) phase2 Minimize, and then combine with formula (8) to determine the external parameters of each camera to be calibrated.

[0142] The pose graph optimization method can be implemented through tools such as ceres or g2o.

[0143] e phase2 =e pl +e ps (7)

[0144]

[0145] in, is the external parameter of the anchor camera; is the external parameter of the i-th camera.

[0146] In this embodiment, in the process of pairwise matching between sub-images, the distance residuals between feature points are not used, but point-line residuals and point-surface residuals are used, which improves the efficiency of extrinsic calibration of the camera to be calibrated.

[0147] Optionally, a specific implementation of step S302 includes the following steps:

[0148] S3021. Optimize the feature point map using the bundle adjustment method to obtain an optimized feature point map.

[0149] Specifically, the electronic device can construct an optimization model of the bundle adjustment method based on the correspondence between the map points in the feature point map and the image feature points. The optimization model includes a method for measuring the reprojection error of the feature points on the image, and corresponding constraints. The optimization model is solved using an iterative optimization algorithm. In each iteration, the electronic device calculates the reprojection error and adjusts the position of the feature points. The iteration stops when the preset convergence condition is reached, and an optimized feature point map is obtained.

[0150] S3022. Optimize the relative posture of the camera to be calibrated and the anchor camera according to the optimized feature point map.

[0151] Specifically, the electronic device iteratively adjusts the relative pose parameters by minimizing the reprojection error between the map points and the image feature points in the optimized feature point map. After multiple iterations of optimization, the electronic device outputs the optimized relative pose, which can more accurately reflect the actual spatial relationship between the camera to be calibrated and the anchor camera.

[0152] In this embodiment, the feature point map is optimized by the bundle adjustment method, and the relative pose of the camera to be calibrated and the anchor camera in the world coordinate system is optimized according to the feature point map to improve the accuracy of the relative pose. By dividing the pose of each camera into sub-graphs, matching the sub-graphs in pairs, and using the pose graph to optimize and balance the relative pose between each camera to be calibrated and the anchor camera. Compared with the existing solution, the accuracy of the extrinsic parameters of the camera to be calibrated obtained by the method of this embodiment is higher, and the error accumulation and drift problems can be avoided.

[0153] Figure 4 FIG. 1 shows a schematic diagram of the structure of a camera extrinsic calibration device provided in an embodiment of the present application. Figure 4 As shown, the camera extrinsic calibration device 10 of this embodiment is used to implement the operation of the electronic device corresponding to any of the above method embodiments. The camera extrinsic calibration device 10 of this embodiment includes:

[0154] An acquisition module 11 is used to acquire image data during the movement of the vehicle, wherein the image data includes image data acquired by the camera to be calibrated and image data acquired by the anchor camera;

[0155] A determination module 12 is used to determine, based on the image data, image data frames acquired by the camera to be calibrated and the anchor camera at the same timestamp;

[0156] The determination module 12 is also used to determine the relative pose of the camera to be calibrated and the anchor camera corresponding to the image data frame;

[0157] The calibration module 13 is used to calibrate the external parameters of the camera to be calibrated according to the relative posture and the external parameters of the anchor camera.

[0158] The camera extrinsic parameter calibration device 10 provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principle and technical effects can be found in the above method embodiment, and this embodiment will not be repeated here.

[0159] Figure 5 FIG. 1 shows a hardware structure diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 20 is used to implement the operations corresponding to the electronic device in any of the above method embodiments. The electronic device 20 of this embodiment may include: a memory 21, a processor 22 and a communication interface 24.

[0160] The memory 21 is used to store computer programs. The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk.

[0161] The processor 22 is used to execute the computer program stored in the memory to implement the method in the above embodiment. For details, please refer to the relevant description in the above method embodiment. The processor 22 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0162] Optionally, the memory 21 may be independent or integrated with the processor 22 .

[0163] When the memory 21 is a device independent of the processor 22, the electronic device 20 may further include a bus 23. The bus 23 is used to connect the memory 21 and the processor 22. The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0164] The communication interface 24 may be connected to the processor 22 via the bus 23. The processor 22 may control the communication interface 24 to implement the functions of receiving and sending signals.

[0165] The electronic device 20 provided in this embodiment can be used to execute the above-mentioned camera extrinsic parameter calibration method, and its implementation method and technical effect are similar, which will not be repeated in this embodiment.

[0166] The present application also provides a computer-readable storage medium, in which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, it is used to implement the methods provided in the various embodiments described above.

[0167] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a computer-readable storage medium is coupled to a processor so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the computer-readable storage medium can also exist in a communication device as discrete components.

[0168] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0169] The present application also provides a computer program product, which includes a computer program / instruction, and the computer program / instruction is stored in a computer-readable storage medium. At least one processor of the device can read the computer program / instruction from the computer-readable storage medium, and at least one processor executes the computer program / instruction so that the device implements the methods provided in the various embodiments described above.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0171] Among them, each module can be physically separated, for example, installed in different locations of a device, or installed on different devices, or distributed on multiple network units, or distributed on multiple processors. Each module can also be integrated together, for example, installed in the same device, or integrated in a set of codes. Each module can exist in the form of hardware, or can also exist in the form of software, or can also be implemented in the form of software plus hardware. The present application can select some or all of the modules according to actual needs to achieve the purpose of the present embodiment.

[0172] It should be understood that, although the various steps in the flowcharts in the above-described embodiments are sequentially displayed according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily to be carried out sequentially, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A camera extrinsic calibration method, characterized in that: The method comprises: Acquire image data during the vehicle movement process, wherein the image data includes image data acquired by the camera to be calibrated and image data acquired by the anchor camera; Determining, according to the image data, image data frames acquired by the camera to be calibrated and the anchor camera at the same timestamp; Determining the relative position and posture of the camera to be calibrated and the anchor camera corresponding to the image data frame; The external parameters of the camera to be calibrated are calibrated according to the relative posture and the external parameters of the anchor camera.

2. The method according to claim 1, characterized in that Determining the relative position and posture of the camera to be calibrated and the anchor camera corresponding to the image data frame includes: Determining a vehicle posture corresponding to the image data frame; Determining the posture of the camera to be calibrated according to the posture of the vehicle and the initial external parameters of the camera to be calibrated; Determining the position and posture of the anchor camera according to the vehicle position and the external parameters of the anchor camera; According to the pose of the camera to be calibrated and the pose of the anchor camera, the relative pose of the camera to be calibrated and the anchor camera is determined.

3. The method according to claim 2, characterized in that Determining the relative pose of the camera to be calibrated and the anchor camera according to the pose of the camera to be calibrated and the pose of the anchor camera, comprising: Performing rotational averaging on the pose of the camera to be calibrated and the pose of the anchor camera respectively to obtain a first optimized pose of the camera to be calibrated and a first optimized pose of the anchor camera; The first optimized pose of the camera to be calibrated and the first optimized pose of the anchor camera are optimized respectively by using a bundle adjustment method to obtain a second optimized pose of the camera to be calibrated and a second optimized pose of the anchor camera; The relative pose of the camera to be calibrated and the anchor camera is determined according to the second optimized pose of the camera to be calibrated and the second optimized pose of the anchor camera.

4. The method according to claim 1, characterized in that: Calibrating the external parameters of the camera to be calibrated according to the relative posture and the external parameters of the anchor camera includes: Determine a feature point map according to the image data frame, wherein the feature point map includes a plurality of map points; According to the feature point map, optimizing the relative posture of the camera to be calibrated and the anchor camera to obtain an optimized relative posture; The external parameters of the camera to be calibrated are calibrated according to the optimized relative pose and the external parameters of the anchor camera.

5. The method according to claim 4, characterized in that According to the feature point map, the relative position of the camera to be calibrated and the anchor camera is optimized, including: Dividing the plurality of map points into edge points and plane points according to the curvatures of the plurality of map points; Constructing point-line residuals and point-surface residuals according to the sub-image of the camera to be calibrated in the image data frame, the edge points and the plane points, wherein the sub-image of the camera to be calibrated includes map points within a preset range of the camera to be calibrated; The relative position of the camera to be calibrated and the anchor camera is optimized according to the point-line residual and the point-plane residual.

6. The method according to claim 4, characterized in that According to the feature point map, the relative position of the camera to be calibrated and the anchor camera is optimized, including: The feature point map is optimized by using a bundle adjustment method to obtain an optimized feature point map; According to the optimized feature point map, the relative posture of the camera to be calibrated and the anchor camera is optimized.

7. A camera extrinsic calibration device, characterized in that: The device comprises: An acquisition module, used to acquire image data during the movement of the vehicle, wherein the image data includes image data acquired by the camera to be calibrated and image data acquired by the anchor camera; A determination module, configured to determine, based on the image data, image data frames acquired by the camera to be calibrated and the anchor camera at the same timestamp; The determination module is further used to determine the relative position and posture of the camera to be calibrated and the anchor camera corresponding to the image data frame; The calibration module is used to calibrate the external parameters of the camera to be calibrated according to the relative posture and the external parameters of the anchor camera.

8. An electronic device, characterized in that: The device comprises: a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to implement the camera extrinsic parameter calibration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the camera extrinsic parameter calibration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the camera extrinsic parameter calibration method according to any one of claims 1 to 6 is implemented.

Images