Camera external parameter calibration method, device and equipment

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a camera external parameter calibration method, device and equipment. The camera extrinsic parameter calibration method comprises the steps of determining initial extrinsic parameters of a camera, constructing 2D-3D matching feature point pairs in 2D pixel points in a 2D image and 3D points in a 3D point cloud for edge lines of a static target in a target area based on each initial extrinsic parameter in an initial extrinsic parameter set, and determining final extrinsic parameters based on the 2D-3D matching feature point pairs, the 2D image is acquired by the camera; and determining a re-projection error of projecting 3D feature points in the 2D-3D matching feature point pair to the 2D image based on the final external parameter, when the re-projection error meets a preset standard, taking the final external parameter as a calibration external parameter of the camera, otherwise, taking the final external parameter as an initial external parameter and adding the initial external parameter into the initial external parameter set, and repeating the steps until the calibration external parameters are obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a camera extrinsic calibration method, device and equipment. Background Art

[0002] Intelligent transportation systems often use multi-sensor data fusion perception solutions to achieve accurate perception around the clock and in all weather conditions. For example, the fusion of 2D images obtained by cameras and 3D point clouds obtained by lidar is used to improve the detection accuracy of three-dimensional objects.

[0003] The parameter calibration of multiple sensors ensures that the data from different sensors can be accurately fused in a unified coordinate framework. Sensor parameters are usually divided into intrinsic parameters and extrinsic parameters. Intrinsic parameters are used to describe the mapping relationship between three-dimensional world information and two-dimensional image information. They are usually obtained during the production process and can be used directly in conventional applications without the need for users to recalibrate. External parameters are used to describe the mapping relationship between coordinate systems of different sensors. Since each sensor has an independent coordinate system, and the sensor will undergo slight position or angle changes during use due to changes in the external environment (such as temperature, vibration, collision, etc.), this change will introduce errors, so external parameter calibration is required to ensure that all sensor data can be unified into a common reference coordinate system, and error correction can be achieved through recalibration.

[0004] Traditional camera extrinsic calibration methods usually require the use of specific calibration plates or objects with known geometric features. This is cumbersome and costly to maintain in practical applications, especially when the camera is installed in an inaccessible location or in an environment without suitable calibration objects. Traditional extrinsic calibration methods involve complex mathematical operations, such as nonlinear optimization, which may result in long calculation time and high consumption of computing resources, and the results of the optimization process depend on the choice of initial parameters. Therefore, traditional extrinsic calibration methods are difficult to automate, which limits the application of camera-radar fusion perception solutions in large-scale or rapidly changing environments. Summary of the invention

[0005] The disclosed embodiment proposes a camera extrinsic calibration scheme based on static environment target feature matching, which avoids the problem that existing camera extrinsic calibration schemes are difficult to automate.

[0006] A first aspect of an embodiment of the present disclosure provides a camera extrinsic parameter calibration method, including:

[0007] Determine an initial camera extrinsic parameter, construct a 2D-3D matching feature point pair for an edge line of a stationary object in a target area from 2D pixel points in a 2D image and 3D points in a 3D point cloud based on each of the initial extrinsic parameters in the initial extrinsic parameter set, and determine a final extrinsic parameter based on the 2D-3D matching feature point pair, wherein the 2D image is acquired by the camera;

[0008] Determine a reprojection error of projecting the 3D feature points in the 2D-3D matching feature point pair onto the 2D image based on the final extrinsic parameter; when the reprojection error meets a preset standard, use the final extrinsic parameter as a calibration extrinsic parameter of the camera; otherwise, use the final extrinsic parameter as an initial extrinsic parameter and add it to the initial extrinsic parameter set; repeat the above steps until the calibration extrinsic parameter is obtained.

[0009] Preferably, determining the initial external parameters of the camera includes:

[0010] In the 3D point cloud, randomly setting an initial position for the camera, and determining an initial external parameter of the camera corresponding to the initial position;

[0011] The roll angle of the camera is set to zero, the camera is rotated clockwise along the x-axis by a first angle, and is rotated clockwise along the y-axis by a second angle, the roll angle, the first angle, and the second angle constitute the posture of the camera, and the initial external parameters of the camera corresponding to the posture are determined, wherein the first angle belongs to a preset yaw angle set, and the second angle belongs to a preset pitch angle set.

[0012] Preferably, constructing 2D-3D matching feature point pairs from 2D pixel points in the 2D image and 3D points in the 3D point cloud for edge lines of stationary objects in the target area based on each of the initial extrinsic parameters in the initial extrinsic parameter set comprises:

[0013] Based on the initial external parameters, mapping the 3D points corresponding to the target area to the 2D image in the 3D point cloud to form a rendered image;

[0014] Line features of edge lines of a stationary object in the target area are respectively identified in the 2D image and the rendered image, and the 2D-3D matching feature point pairs are determined based on the line features.

[0015] Preferably, mapping the 3D points corresponding to the target area in the 3D point cloud to the 2D image comprises:

[0016] Obtaining foreground points in the 3D point cloud according to preset rules and weighted fitting the foreground points to generate aggregated 3D points;

[0017] The aggregated 3D points are mapped to the 2D image based on the initial extrinsic parameters to form the rendered image.

[0018] Preferably, before acquiring the foreground point in the 3D point cloud according to a preset rule, the method further includes:

[0019] Background points in the 3D point cloud are removed, wherein the 3D point cloud is composed of background points and foreground points, and the background point refers to a 3D point whose distance from any adjacent 3D point exceeds a preset threshold.

[0020] Preferably, the step of respectively identifying line features of edge lines of the stationary object in the target area in the 2D image and the rendered image, and determining the 2D-3D matching feature point pairs based on the line features comprises:

[0021] Recognize a stationary object in the target area in the 2D image and the rendered image respectively, and extract a first line feature in the 2D image and a second line feature in the rendered image for an edge line of the selected stationary object;

[0022] Feature point matching is performed on the first line feature and the second line feature until a preset number of 2D-3D matching feature point pairs are obtained.

[0023] Preferably, the determining of the final external parameter based on the 2D-3D matching feature point pair comprises:

[0024] Based on the 2D-3D matching feature point pairs, a PnP algorithm is used to determine the final extrinsic parameters of the camera.

[0025] Preferably, determining a reprojection error of projecting a 3D feature point in the 2D-3D matching feature point pair onto the 2D image based on the final extrinsic parameter comprises:

[0026] In the 2D-3D matching feature point pairs, determining the 2D feature points corresponding to the 3D feature points;

[0027] Projecting the 3D feature points onto the 2D image based on the final extrinsic parameters to obtain projection points;

[0028] The distance between the projection point and the 2D feature point is used as a reprojection error of the 3D feature point projected onto the 2D image based on the final extrinsic parameter.

[0029] A second aspect of an embodiment of the present disclosure provides a camera extrinsic calibration device, including:

[0030] a determination module, configured to determine an initial camera extrinsic parameter, construct a 2D-3D matching feature point pair for an edge line of a stationary object in a target area from 2D pixel points in a 2D image and 3D points in a 3D point cloud based on each initial extrinsic parameter in an initial extrinsic parameter set, and determine a final extrinsic parameter based on the 2D-3D matching feature point pair, wherein the 2D image is acquired by the camera;

[0031] A calibration module is used to determine the reprojection error of projecting the 3D feature points in the 2D-3D matching feature point pairs onto the 2D image based on the final extrinsic parameter. When the reprojection error meets a preset standard, the final extrinsic parameter is used as the calibration extrinsic parameter of the camera. Otherwise, the final extrinsic parameter is used as the initial extrinsic parameter and added to the initial extrinsic parameter set. The above steps are repeated until the calibration extrinsic parameter is obtained.

[0032] A third aspect of an embodiment of the present disclosure provides a camera extrinsic calibration device, including a memory and a processor:

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement the method described in the first aspect of the present disclosure when executing the computer program.

[0035] In summary, the camera extrinsic calibration methods, devices and equipment provided by the embodiments of the present disclosure realize automatic extrinsic calibration of roadside cameras and point cloud maps through a calibration strategy based on static environment target feature matching. Among them, the method of generating RGB images based on view rendering realizes the rendering of 3D laser point clouds to RGB images, and each pixel of the rendered image retains 3D information, thereby reducing the modal difference between the image and the point cloud, and improving the robustness and accuracy of the calibration results; at the same time, by introducing a method for automatically estimating the initial extrinsic parameters using only a rough estimate of the camera position, the dependence of the calibration process on the initial extrinsic parameters is significantly reduced; finally, by minimizing the reprojection error of the line features extracted from the rendered RGB image and the original camera image, the iterative refinement of the calibration parameters is achieved, avoiding complex nonlinear optimization and significantly improving the accuracy of roadside camera and drone map calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0037] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;

[0038] Figure 2 It is an automatic external parameter calibration algorithm framework of roadside images and point cloud maps based on static environment target feature matching as shown in some embodiments of the present disclosure;

[0039] Figure 3 is a flow chart of a camera extrinsic calibration method according to some embodiments of the present disclosure;

[0040] Figure 4 is a schematic diagram of the installation position of the roadside camera in the present disclosure;

[0041] Figure 5 is a 2D rendered image with 3D information as shown in some embodiments of the present disclosure;

[0042] Figure 6 is an example of superimposing line features extracted from a rendered image onto line features extracted from an original image in one embodiment of the present disclosure;

[0043] Figure 7 is an example of feature point matching between a rendered image and an original image in one embodiment of the present disclosure;

[0044] Figure 8 is a schematic diagram of a data acquisition system based on a drone point cloud map and a roadside camera in one embodiment of the present disclosure;

[0045] Fig. 9 In one embodiment of the present disclosure, the calibration result of the camera is displayed;

[0046] Fig.10 is a schematic diagram of a camera extrinsic calibration device according to some embodiments of the present disclosure;

[0047] Fig.11 Schematic diagram of a camera extrinsic calibration device shown in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.

[0049] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.

[0050] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.

[0051] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.

[0052] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.

[0053] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1 The system shown includes an external parameter calibration server connected to at least one image sensor and at least one distance sensor data, and the external parameter calibration server performs external parameter calibration on the image sensor based on 2D image data of the target area acquired by the image sensor and 3D point cloud data of the target area acquired by the distance sensor.

[0054] In the field of vehicle-road collaboration in intelligent transportation systems, the image sensor is a camera deployed on the roadside. The roadside camera can be a variety of monocular, binocular cameras or monocular, binocular cameras, used to obtain 2D image data of the target area (I∈R H×W×3 , where H and W are the height and width of the image respectively), the distance sensor is a laser radar, which is used to obtain 3D point cloud data of the target area (also known as point cloud map, P∈R N×3 , where N is the number of point cloud points in the point cloud map), the target area contains stationary targets, and the stationary targets have clear edge lines (such as trees, lane lines, etc.).

[0055] The external parameter calibration server automatically performs external parameter calibration of the roadside camera and the point cloud map based on the edge line of the stationary target in the target area, that is, automatically determines the calibration matrix M=[RT] between the roadside camera coordinates and the point cloud map coordinates, where R is the rotation matrix and T is the translation matrix. The external parameter calibration server can be any of a stand-alone, cluster or distributed server.

[0056] External parameters are used to describe the mapping relationship between coordinate systems of different sensors. Since each sensor has an independent coordinate system, and the sensor will undergo slight position or angle changes during use due to changes in the external environment (such as temperature, vibration, collision, etc.), this change will introduce errors. Therefore, external parameter calibration is required when multi-sensor data is fused to ensure that all sensor data can be unified into a common reference coordinate system, and error correction can be achieved through recalibration.

[0057] Traditional camera extrinsic calibration methods usually require the use of specific calibration plates or objects with known geometric features. This is cumbersome and costly to maintain in practical applications, especially when the camera is installed in an inaccessible location or in an environment without suitable calibration objects. Traditional extrinsic calibration methods involve complex mathematical operations, such as nonlinear optimization, which may result in long calculation time and high consumption of computing resources, and the results of the optimization process depend on the choice of initial parameters. Therefore, traditional extrinsic calibration methods are difficult to automate, which limits the application of camera-radar fusion perception solutions in large-scale or rapidly changing environments.

[0058] In order to solve the problem that existing camera extrinsic calibration solutions are difficult to automate, the disclosed embodiment proposes an automatic extrinsic calibration solution for roadside cameras and point cloud maps based on static environment target feature matching. Figure 2 This is an automatic external parameter calibration algorithm framework for roadside images and point cloud maps based on static environment target feature matching as shown in some embodiments of the present disclosure. Figure 2 As shown, the automatic extrinsic parameter calibration algorithm framework constructs a view rendering strategy from 3D point cloud map to 2D image, and realizes the rendering of 3D point cloud map to 2D image; uses the rough position of the camera in the point cloud map to automatically estimate the initial extrinsic parameters between the camera and the 3D point cloud map; introduces the SAM segmentation model to extract the line features of static environment targets in the rendered image and the original image, and realizes the iterative refinement of the calibration matrix M by minimizing the reprojection error of the line features.

[0059] Figure 3 is a flow chart of a camera extrinsic calibration method according to some embodiments of the present disclosure. In some embodiments, the camera extrinsic calibration method is Figure 1 The external parameter calibration server shown is executed, and the camera external parameter calibration method includes the following steps:

[0060] S310, determining the initial external parameters of the camera, based on each of the initial external parameters in the initial external parameter set, constructing 2D-3D matching feature point pairs for the edge lines of the stationary objects in the target area in the 2D pixel points in the 2D image and the 3D points in the 3D point cloud, and determining the final external parameters based on the 2D-3D matching feature point pairs, wherein the 2D image is acquired by the camera.

[0061] Specifically, the initial external parameters of the roadside camera are estimated first. The camera external parameters depend on the camera pose, which is composed of the camera's roll angle roll, pitch angle pitch and yaw angle yaw. Figure 4 This is a schematic diagram of the installation location of roadside cameras in the field of intelligent transportation. Figure 4 As shown, the roadside camera has a zero roll, a very small pitch, and eight uniformly sampled yaws.

[0062] In some embodiments of the present disclosure, the method for determining the initial external parameters of the roadside camera is as follows:

[0063] (a) Randomly determine the camera’s position (x, y, z) in the point cloud map.

[0064] (b) Based on the random position (x, y, z), the initial external parameters are defined as:

[0065]

[0066] (c) Determine yaw = [45°, 90°, 135°, 180°, 225°, 270°, 315°, 360°].

[0067] (d) Determine pitch = [5°, 10°].

[0068] (e) Determine roll = [0°].

[0069] (f) Rotate the y-axis clockwise by the yaw value and the x-axis clockwise by the pitch value to obtain the initial external parameter combinations under different combinations.

[0070] Then, the rendering of the 3D point cloud map to a 2D image is realized based on the initial external parameter combination.

[0071] The target point (x u ,y i ,z u ) and pixels in the image coordinate system (u u ,v i) one-to-one correspondence. When the target is imaged in the camera coordinate system, only the target points that fall on the camera coordinate system ray can be projected onto the image. However, due to the sparsity of the roadside laser point cloud, many 3D point cloud points will not intersect with any ray in the camera coordinate system, resulting in a missing correspondence between the 3D point cloud and the 2D image. In order to complete the correspondence between the 3D point cloud and the 2D image, this disclosure proposes a view rendering strategy, which is as follows:

[0072] a. Initialize 2D rendering image.

[0073] According to the size of the original roadside image, initialize the 2D rendering image I′∈R H×W×3 For a given 2D rendered image pixel (u i ,v i ), we only need to infer the corresponding ray r in the camera coordinate system i 3D points on the point cloud map (X i ,Y i ,Z i ).

[0074] b. Remove redundant background points in the point cloud map and extract foreground points in the point cloud map.

[0075] Due to the sparsity and uneven sampling of point cloud data, multiple 3D points may correspond to one image pixel, so it is necessary to remove redundant background points in the point cloud map.

[0076] Since adjacent 3D points in the point cloud map must be adjacent in the 2D pixel space, we only need to consider i ,v i ) is extracted from the N adjacent pixels and is represented as The present invention introduces a simple distance filter to filter the point cloud points:

[0077]

[0078] Where dis(·) represents the Euclidean distance between two points. is an empirical threshold. If the distance between a point and any adjacent point exceeds the set threshold, then The value of is 0, the 3D point is a background point, otherwise The value of is 1, and the 3D point is the foreground point. i represents the i-th pixel in the 2D rendered image, j represents the j-th adjacent pixel of the i-th pixel; N represents that the i-th pixel has n adjacent pixels; k represents the k-th adjacent pixel of the i-th pixel, k∈[0,1,…N-1], and k≠j, such as when j=0, the value of k is [1,2,…,N-1]; when j=1, the value of k is [0,2,…,N-1].

[0079] Since the algorithm only needs to calculate the Euclidean distance between points, the computational burden is small and it is suitable for real-time applications.

[0080] Each ray r in the camera coordinate system i There are multiple 3D points on the image that correspond to a pixel on the 2D image. Although the foreground points have been obtained after filtering the background redundant points, However, the foreground point needs to be fitted to obtain the unique 3D point corresponding to the pixel point of the 2D rendered image.

[0081] c. Fit the foreground point to obtain the unique 3D point corresponding to the pixel point of the 2D rendered image.

[0082] The local foreground point of the point cloud map can be approximated as a plane, and the plane fitting function ax can be used i +by i +cz i +d=0Fit the foreground point obtained by filtering and put the ray r on the camera coordinate system i The intersection point with the plane (denoted as L i =(X i ,Y i ,Z i )) as the only 3D point. Among them, the intersection point L i The value (denoted as v(L i )) is obtained by weighting the RGB values ​​of all foreground points. The weighting formula is as follows:

[0083]

[0084] Where dis(·) represents the Euclidean distance between two points. is an empirical threshold. is the RGB value of the jth foreground point, represents the weight, is the foreground point, i represents the i-th pixel in the 2D rendered image, j represents the j-th adjacent pixel of the i-th pixel; N represents that the i-th pixel has n adjacent pixels; k represents the k-th adjacent pixel of the i-th pixel, k∈[0,1,…N-1], and k≠j, such as j=0, the value of k is [1,2,…,N-1]; when j=1, the value of k is [0,2,…,N-1].

[0085] d. Get a rendered image with 2D-3D correspondences.

[0086] By using the initial external parameter matrix M0, the 3D points in the point cloud map are projected onto the 2D image to generate a 2D rendered image with 3D information, such as Figure 5 The specific calculation formula is as follows:

[0087] (X i ,Y i ,Z i )=M0K -1 [u i ,v i ,1] T

[0088] Among them, (X i ,Y i ,Z i ) is a 3D point in the point cloud map, (u i ,v i ) is the 2D pixel point in the 2D rendered image, K is the camera intrinsic parameter, and M0 is the initial extrinsic parameter between the point cloud map and the camera.

[0089] Finally, feature point matching is performed based on the rendered image and the original image, and the final calibration extrinsic parameters of the camera are estimated.

[0090] Since there are a large number of dynamic targets (such as vehicles, pedestrians, etc.) in the traffic environment, the movement of these dynamic targets will introduce unstable or even erroneous feature points in the image sequence. If these feature points are used for feature matching, the matching accuracy will be greatly reduced. Static environmental features, such as lane lines, trees, etc., are fixed in their positions in the environment, and the corresponding extracted feature points are also fixed, and unstable feature points will not be introduced due to dynamic changes. Therefore, selecting static targets in the environment (lane lines, trees, etc.) for feature point extraction helps to improve the stability and reliability of image matching.

[0091] Based on the view rendering strategy and the initial external parameters, the 2D-3D correspondence can be established while generating the rendered view. Therefore, the 2D line features of the static target can be extracted from the rendered RGB image to obtain the corresponding 3D line features. At the same time, the image matching algorithm SuperGlue is introduced to match the feature points of these rendered images with the original images obtained from the roadside camera until a sufficient number of 2D-3D matching feature points are obtained. Finally, the PnP algorithm is used to estimate the final calibration parameters of the camera.

[0092] Specifically, it includes:

[0093] (1) Rendering image generation.

[0094] Based on view rendering strategy and initial external parameter combination Get the rendered image I′ corresponding to the point cloud map h .

[0095] (2) Introduce the image segmentation model SAM to extract the line features F of the original image I and the rendered image I′ h Line feature F′ h .

[0096] Figure 6 This is an example of superimposing line features extracted from a rendered image onto line features extracted from an original image in one embodiment of the present disclosure.

[0097] (3) Introduce the image matching algorithm SuperGlue to perform line feature F and line feature F′ h Match the feature points and obtain the line feature point pairs.

[0098] Rendered images have a built-in 2D-3D correspondence relationship. Therefore, by extracting the 2D line features of static objects from the rendered RGB images, we can obtain the corresponding 3D line features.

[0099] The image matching algorithm SuperGlue is introduced to match the line feature F extracted from the rendered image with the line feature F′ obtained from the original image from the roadside camera. h Perform feature point matching until a sufficient number of 2D-3D matching feature point pairs are obtained (u n ,v n ),(X n ,Y n ,Z n )), where n represents the number of feature points.

[0100] Figure 7 is an example of feature point matching between a rendered image and an original image in one embodiment of the present disclosure.

[0101] (4) The PnP algorithm is introduced to estimate the final calibration parameters of the camera.

[0102] Based on the acquired 2D-3D matching feature point pairs ((u n ,v n ),(X n ,Y n ,Z n )) Use the PnP algorithm to obtain the camera's calibration external parameter combination

[0103] S320, determining a reprojection error of projecting the 3D feature points in the 2D-3D matching feature point pair onto the 2D image based on the final extrinsic parameter; when the reprojection error meets a preset standard, using the final extrinsic parameter as a calibration extrinsic parameter of the camera; otherwise, using the final extrinsic parameter as an initial extrinsic parameter and adding it to the initial extrinsic parameter set, and repeating the above steps until the calibration extrinsic parameter is obtained.

[0104] Specifically, based on the feature point pairs obtained in step (3) ((u n ,v n ),(X n ,Y n ,Z n )) and obtain the 3D point (X n ,Y n ,Z n ) Using the calibration external parameter combination Project it into the 2D image to obtain the corresponding feature points in the 2D image (u n ′,v n ′). The specific calculation formula is as follows:

[0105]

[0106] After that, calculate the minimum (u n ,v n ) and (u n ′,v n ′) between:

[0107] reprojectionerror=min(dis((u n ,v n ),(u n ′,v n ′)))≤δ

[0108] Where dis(·) represents the Euclidean distance between two points, δ is the reprojection error threshold, which is an empirical estimate, and K is the camera intrinsic parameter. is the calibration external parameter. If the reprojection error obtained at this time is the smallest and within the error range, then the external parameter matrix at this time is considered That is the optimal external parameter calibration matrix M. Otherwise, append the external parameter matrix at this time to the initial external parameter combination In, repeat the above steps until the extrinsic calibration matrix M that meets the conditions is obtained.

[0109] Image-to-point cloud registration aims to align 2D images with 3D point clouds. However, since the prior point cloud and image of the roadside are often acquired at different times and in different coordinate systems, the calibration operation using traditional calibration plates is cumbersome and has high maintenance costs. Although learning-based algorithms have the potential to reduce the dependence on initial parameters through scene understanding, it is difficult to obtain a diverse and sufficient training dataset in the roadside environment. Therefore, for roadside cameras, this paper proposes a new method for automatic extrinsic calibration of roadside cameras and point cloud maps based on static environment target feature matching. First, a view rendering strategy is proposed to achieve the rendering of point cloud maps to RGB images with the help of the RGB attributes of point cloud maps, reduce the modal difference between images and point clouds, and thus improve the robustness and accuracy of the calibration results. Secondly, a method is introduced to automatically estimate the initial extrinsic parameters using only a rough estimate of the camera position, which significantly reduces the dependence of the calibration process on the initial extrinsic parameters. Finally, the iterative refinement of the calibration parameters is achieved by minimizing the reprojection error of line features extracted from the rendered RGB image and the original camera image, which significantly improves the accuracy of roadside camera and drone map calibration. Therefore, the extrinsic parameter calibration scheme for roadside cameras and point cloud maps proposed in the present disclosure does not rely on the selection of initial extrinsic parameters, reduces computational consumption, and can be performed automatically throughout the process.

[0110] One embodiment of the present disclosure is Figure 3 The method described in S310-S320 was verified. Figure 8 This is a schematic diagram of the data acquisition system based on UAV point cloud maps and roadside cameras in the verification environment. Figure 8 As shown in the figure, the point cloud map was obtained using a DJI-M300 drone equipped with a Zenmuse L1 module, next to a camera mounted on a roadside smart pole to collect image data. Figure 8 The data acquisition system shown collects data from 34 regions, each region provides a pair of point cloud-image data sets, and all point cloud data are unified to the global coordinate system. Fig. 9 The calibration visualization results of some cameras are given. The black represents the 3D point cloud line features of static environmental targets such as lane lines, road signs, trees, and road poles in the point cloud map. The external parameter calibration matrix M calculated above is projected onto the image obtained by the roadside camera. As you can see, Fig. 9The 3D point cloud line features of the static environment targets are projected onto the 2D line features of the image, which correspond one-to-one to the real lane lines, road signs, trees, poles and other static targets, which shows the accuracy of the external parameter calibration matrix M.

[0111] Fig.10 is a schematic diagram of a camera extrinsic calibration device according to some embodiments of the present disclosure. Fig.10 As shown, the camera extrinsic calibration device 1000 includes a determination module 1010 and a calibration module 1020. In some embodiments of the present disclosure, the camera extrinsic calibration function can be performed by Figure 1 The external parameter calibration server is executed as shown.

[0112] A determination module 1010 is used to determine an initial camera external parameter, construct a 2D-3D matching feature point pair for an edge line of a stationary object in a target area from 2D pixel points in a 2D image and 3D points in a 3D point cloud based on each initial external parameter in an initial external parameter set, and determine a final external parameter based on the 2D-3D matching feature point pair, wherein the 2D image is acquired by the camera;

[0113] The calibration module 1020 is used to determine the reprojection error of projecting the 3D feature points in the 2D-3D matching feature point pairs onto the 2D image based on the final extrinsic parameter. When the reprojection error meets a preset standard, the final extrinsic parameter is used as the calibration extrinsic parameter of the camera. Otherwise, the final extrinsic parameter is used as the initial extrinsic parameter and added to the initial extrinsic parameter set. The above steps are repeated until the calibration extrinsic parameter is obtained.

[0114] An embodiment of the present disclosure provides a camera extrinsic calibration device. Fig.11 As shown, the camera extrinsic calibration device 1100 includes a memory 1120 and a processor 1110, wherein the memory 1120 is used to store a computer program; and the processor 1110 is used to implement when executing the computer program. Figure 3 The method described in S310-S320.

[0115] In summary, the camera extrinsic parameter calibration method, device and equipment provided by the embodiments of the present disclosure realize automatic extrinsic parameter calibration of roadside cameras and point cloud maps through a calibration strategy based on static environment target feature matching, wherein a method for generating RGB images based on view rendering realizes the rendering of 3D laser point cloud to RGB image, and each pixel of the rendered image retains 3D information, thereby reducing the modal difference between the image and the point cloud and improving the robustness and accuracy of the calibration result; at the same time, by introducing a method for automatically estimating the initial extrinsic parameters using only a rough estimate of the camera position, the dependence of the calibration process on the initial extrinsic parameters is significantly reduced; finally, by minimizing the reprojection error of the line features extracted from the rendered RGB image and the camera original image, the iterative refinement of the calibration parameters is achieved, thereby avoiding complex nonlinear optimization and significantly improving the accuracy of roadside camera and drone map calibration.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.

[0117] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0118] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0119] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.

Claims

1. A camera extrinsic calibration method, characterized in that: include: Determine an initial camera extrinsic parameter, construct a 2D-3D matching feature point pair for an edge line of a stationary object in a target area from 2D pixel points in a 2D image and 3D points in a 3D point cloud based on each of the initial extrinsic parameters in the initial extrinsic parameter set, and determine a final extrinsic parameter based on the 2D-3D matching feature point pair, wherein the 2D image is acquired by the camera; Determine a reprojection error of projecting the 3D feature points in the 2D-3D matching feature point pair onto the 2D image based on the final extrinsic parameter; when the reprojection error meets a preset standard, use the final extrinsic parameter as a calibration extrinsic parameter of the camera; otherwise, use the final extrinsic parameter as an initial extrinsic parameter and add it to the initial extrinsic parameter set; repeat the above steps until the calibration extrinsic parameter is obtained.

2. The method according to claim 1, characterized in that: Determining the initial external parameters of the camera includes: In the 3D point cloud, randomly setting an initial position for the camera, and determining an initial external parameter of the camera corresponding to the initial position; The roll angle of the camera is set to zero, the camera is rotated clockwise along the x-axis by a first angle, and is rotated clockwise along the y-axis by a second angle, the roll angle, the first angle, and the second angle constitute the posture of the camera, and the initial external parameters of the camera corresponding to the posture are determined, wherein the first angle belongs to a preset yaw angle set, and the second angle belongs to a preset pitch angle set.

3. The method according to claim 1, characterized in that The step of constructing a 2D-3D matching feature point pair from a 2D pixel point in a 2D image and a 3D point in a 3D point cloud for an edge line of a stationary object in a target area based on each of the initial external parameters in the initial external parameter set comprises: Based on the initial external parameters, mapping the 3D points corresponding to the target area to the 2D image in the 3D point cloud to form a rendered image; Line features of edge lines of a stationary object in the target area are respectively identified in the 2D image and the rendered image, and the 2D-3D matching feature point pairs are determined based on the line features.

4. The method according to claim 3, characterized in that The mapping of the 3D points corresponding to the target area to the 2D image in the 3D point cloud comprises: Obtaining foreground points in the 3D point cloud according to preset rules and weighted fitting the foreground points to generate aggregated 3D points; The aggregated 3D points are mapped to the 2D image based on the initial extrinsic parameters to form the rendered image.

5. The method according to claim 4, characterized in that Before obtaining the foreground point in the 3D point cloud according to a preset rule, the method further includes: Background points in the 3D point cloud are removed, wherein the 3D point cloud is composed of background points and foreground points, and the background point refers to a 3D point whose distance from any adjacent 3D point exceeds a preset threshold.

6. The method according to claim 3, characterized in that: The step of respectively identifying line features of edge lines of the stationary object in the target area in the 2D image and the rendered image, and determining the 2D-3D matching feature point pairs based on the line features comprises: Recognize a stationary object in the target area in the 2D image and the rendered image respectively, and extract a first line feature in the 2D image and a second line feature in the rendered image for an edge line of the selected stationary object; Feature point matching is performed on the first line feature and the second line feature until a preset number of 2D-3D matching feature point pairs are obtained.

7. The method according to claim 1, characterized in that The determining of the final external parameter based on the 2D-3D matching feature point pair comprises: Based on the 2D-3D matching feature point pairs, a PnP algorithm is used to determine the final extrinsic parameters of the camera.

8. The method according to claim 1, characterized in that The determining of a reprojection error of projecting the 3D feature point in the 2D-3D matching feature point pair onto the 2D image based on the final extrinsic parameter comprises: In the 2D-3D matching feature point pairs, determining the 2D feature points corresponding to the 3D feature points; Projecting the 3D feature points onto the 2D image based on the final extrinsic parameters to obtain projection points; The distance between the projection point and the 2D feature point is used as a reprojection error of the 3D feature point projected onto the 2D image based on the final extrinsic parameter.

9. A camera extrinsic calibration device, characterized in that: include: a determination module, configured to determine an initial camera extrinsic parameter, construct a 2D-3D matching feature point pair for an edge line of a stationary object in a target area from 2D pixel points in a 2D image and 3D points in a 3D point cloud based on each initial extrinsic parameter in an initial extrinsic parameter set, and determine a final extrinsic parameter based on the 2D-3D matching feature point pair, wherein the 2D image is acquired by the camera; A calibration module is used to determine the reprojection error of projecting the 3D feature points in the 2D-3D matching feature point pairs onto the 2D image based on the final extrinsic parameter. When the reprojection error meets a preset standard, the final extrinsic parameter is used as the calibration extrinsic parameter of the camera. Otherwise, the final extrinsic parameter is used as the initial extrinsic parameter and added to the initial extrinsic parameter set. The above steps are repeated until the calibration extrinsic parameter is obtained.

10. A camera extrinsic calibration device, characterized in that: Including memory and processor: The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1-8 when executing the computer program.

Images