Camera parameter calculation method and device, storage medium and electronic device

Abstract

The invention provides a camera parameter calculation method and device, a storage medium and an electronic device.The method comprises the steps that in the moving process of a vehicle, a two-dimensional feature track, collected by a camera, of a same world road sign is obtained, and a local plane of a track scene where the world road sign is located is constructed based on radar, the camera and the radar are arranged on the vehicle; constructing an observation error function between the camera and the radar by using the two-dimensional feature trajectory and the local plane, the observation error function being used for calculating an error value between an actual position and a theoretical position of a feature point collected by the camera; and calculating an external parameter between the camera and the radar between the camera and an internal parameter of the camera by using the observation error function. According to the embodiment of the invention, the method solves a technical problem that the calibration of the camera parameters must depend on a specific calibration object in the prior art, and improves the robustness of the calibration of the camera parameters.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method and device for calculating camera parameters, a storage medium, and an electronic device. Background Art

[0002] In the relevant technology, in the scenario of autonomous driving, in order to obtain the surrounding environment information as much as possible and the accuracy and robustness of the information, the vehicle usually uses cameras with different fields of view to obtain high-resolution surrounding environment information, and uses laser radar to directly measure the depth of objects around the vehicle as a supplement to visual information. Obtaining the accurate mutual position relationship and internal parameters of multiple sensors is the basis for the vehicle to accurately and consistently perceive the surrounding environment information.

[0003] In the related technology, the solution to the joint calibration problem of multiple sensors in the current autonomous driving scenario is to park the vehicle in a scene with specially designed calibration objects to unify the external parameters of multiple sensors into the same coordinate system. This not only requires the calibration objects to have features that are easy for multiple sensors to identify, but also consumes a lot of time and manpower. If the sensor is offset during driving, it needs to be recalibrated regularly.

[0004] With respect to the above-mentioned problems existing in the related technologies, no efficient and accurate solutions have been found yet. Summary of the invention

[0005] The present invention provides a camera parameter calculation method and device, a storage medium, and an electronic device to solve the technical problems in related technologies.

[0006] According to one embodiment of the present invention, a method for calculating camera parameters is provided, comprising: obtaining a two-dimensional feature trajectory of the same world landmark captured by a camera during movement of a vehicle, and constructing a local plane of the trajectory scene where the world landmark is located based on a radar, wherein the camera and the radar are arranged on the vehicle; constructing an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate the error value between the actual position and the theoretical position of the feature point captured by the camera; and calculating the external parameters between the camera and the radar and the internal parameters of the camera using the observation error function.

[0007] Optionally, obtaining the two-dimensional feature trajectory of the same world landmark captured by the camera includes: obtaining multiple frames of images captured by the camera; selecting a world landmark and locating the world landmark in the multiple frames of images; and extracting the pixel coordinates of the world landmark from the multiple frames of images respectively to obtain the two-dimensional feature trajectory.

[0008] Optionally, constructing a local plane of the trajectory scene where the world landmark is located based on radar includes: acquiring multiple groups of three-dimensional point clouds of the world landmark collected by radar; aligning the multiple groups of three-dimensional point clouds, and filtering out dynamic obstacles in the multiple groups of three-dimensional point clouds to obtain a laser point cloud map; and reconstructing the ground grid and three-dimensional structure according to the laser point cloud map to obtain a local plane of the trajectory scene where the world landmark is located.

[0009] Optionally, using the two-dimensional feature trajectory and the local plane to construct an observation error function between the camera and the radar includes: extracting the first pixel coordinates of the world landmark in the first image frame and the second pixel coordinates in the second image frame from the two-dimensional feature trajectory; calculating the three-dimensional coordinates of the first pixel coordinates in three-dimensional space based on the local plane; inversely projecting the three-dimensional coordinates to the second image frame to obtain transformed coordinates; and using the transformed coordinates and the second pixel coordinates to construct the observation error function between the camera and the radar.

[0010] Optionally, using the three-dimensional coordinates and the second pixel coordinates to construct an observation error function between the camera and the radar includes: using the following formula to construct an observation error function between the camera and the radar: in, Π i represents the projection function of the 3D map point centered on the camera onto the i-th camera image, T i L ∈SE(3) represents the pose of the i-th camera coordinate system in the radar coordinate system L, represents the position of the radar coordinate system L in the world coordinate system W at time t, (n W ,p W ) represents the parameterization of the plane's normal vector and position in the map coordinate system, is the first pixel coordinate, is the second pixel coordinate.

[0011] Optionally, calculating the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane includes: reverse projecting the first pixel coordinates to the local plane to obtain the intersection of the projection lines on the local plane; and determining the coordinates of the intersection as the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space.

[0012] Optionally, using the observation error function to calculate the external parameters between the camera and the radar and the internal parameters of the camera includes: for each camera, using the observation error function to construct an observation error sum function for all pixels of two frames of images; constructing an error loss function of the observation error sum function, and calculating the preliminary internal parameters of the camera and the preliminary external parameters between the camera and the radar based on the error loss function; constructing an association relationship between the two-dimensional feature trajectory and the local plane; and optimizing the preliminary internal parameters and the preliminary external parameters based on the association relationship to obtain target internal parameters and target external parameters.

[0013] Optionally, constructing an association relationship between the two-dimensional feature trajectory and the local plane includes: for each feature point in the two-dimensional feature trajectory, calculating the intersection position of the reverse projection of the feature point to the local plane; identifying a set of feature objects in the trajectory scene where the local plane is located, and arranging the set of feature objects in descending order according to size to obtain a feature object sequence; selecting feature objects in the feature object sequence in order, and establishing matching relationships between the intersection positions of the feature objects and the feature points in turn to obtain an association relationship.

[0014] Optionally, the preliminary internal parameters and the preliminary external parameters are optimized based on the association relationship to obtain target internal parameters and target external parameters, including: for each camera, optimizing the association relationship of a single camera to obtain a first optimization relationship; for every two cameras, performing association optimization on the association relationship between the two cameras to obtain a second optimization relationship; and using the first optimization relationship and the second optimization relationship to optimize the preliminary internal parameters and the preliminary external parameters to obtain target internal parameters and target external parameters.

[0015] Optionally, optimizing the association relationship of a single camera to obtain a first optimization relationship includes: constructing an external parameter iterator from a single camera to the radar; repeatedly executing the external parameter iterator according to the following steps until the external parameters converge: calculating the back projection ray of each feature point of the two-dimensional feature trajectory of the camera to the local plane, and calculating the normal vector of the local plane; calculating the angle between the back projection ray and the normal vector; calculating the association score of the feature point based on the angle; filtering feature points whose association scores are less than a preset value in the two-dimensional feature trajectory, and using a random sampling consensus algorithm to eliminate discrete feature points in the two-dimensional feature trajectory; calculating the external parameters corresponding to the optimized two-dimensional feature trajectory; judging whether the external parameters converge; and outputting the first optimization relationship after execution is completed.

[0016] Optionally, the association relationship between the two cameras is optimized to obtain a second optimization relationship, including: using nearest neighbor search to find different feature points in the two-dimensional feature trajectories of multiple cameras corresponding to the same three-dimensional map point to obtain the trajectory association relationship of the multiple cameras; using the neighborhood plane of the two-dimensional feature trajectories of the multiple cameras to perform template matching to obtain multiple image blocks, converting the multiple image blocks into multiple grayscale images, equalizing the pixel intensity histograms of the multiple grayscale images, and associating by finding the centroid of the mutual correlation values ​​to obtain the pixel correspondence relationship of the multiple cameras; outputting the trajectory association relationship and the pixel correspondence relationship as the second optimization relationship.

[0017] Optionally, the first optimization relationship and the second optimization relationship are used to optimize the preliminary internal parameters and the preliminary external parameters to obtain target internal parameters and target external parameters, including: using the first optimization relationship and the second optimization relationship to eliminate weakly correlated points and discrete points in the association relationship to obtain a third optimization relationship; normalizing the sum of camera residuals of the preliminary internal parameters and the preliminary external parameters of multiple cameras respectively to obtain normalized internal parameters and normalized external parameters; using the observation error function and the third optimization relationship to optimize the normalized internal parameters and the normalized external parameters to obtain target internal parameters and target external parameters.

[0018] According to another embodiment of the present invention, a device for calculating camera parameters is provided, including: a processing module, used for obtaining a two-dimensional feature trajectory of the same world landmark captured by a camera during movement of a vehicle, and constructing a local plane of the trajectory scene where the world landmark is located based on a radar, wherein the camera and the radar are arranged on the vehicle; a construction module, used for constructing an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate the error value between the actual position and the theoretical position of the feature point captured by the camera; and a calculation module, used for calculating the external parameters between the camera and the radar and the internal parameters of the camera using the observation error function.

[0019] Optionally, the processing module includes: a first acquisition unit, used to acquire multiple frames of images captured by a camera; a positioning unit, used to select world landmarks and locate the world landmarks in the multiple frames of images; and an extraction unit, used to extract the pixel coordinates of the world landmarks from the multiple frames of images respectively to obtain a two-dimensional feature trajectory.

[0020] Optionally, the processing module includes: a second acquisition unit, used to acquire multiple groups of three-dimensional point clouds of the world landmarks collected by radar; a registration unit, used to register the multiple groups of three-dimensional point clouds and filter out dynamic obstacles in the multiple groups of three-dimensional point clouds to obtain a laser point cloud map; and a reconstruction unit, used to reconstruct the ground grid and three-dimensional structure according to the laser point cloud map to obtain a local plane of the trajectory scene where the world landmarks are located.

[0021] Optionally, the construction module includes: an extraction unit, used to extract the first pixel coordinates of the world landmark in the first image frame and the second pixel coordinates in the second image frame from the two-dimensional feature trajectory; a calculation unit, used to calculate the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane; a processing unit, used to inversely project the three-dimensional coordinates to the second image frame to obtain transformed coordinates; and a construction unit, used to construct an observation error function between the camera and the radar using the transformed coordinates and the second pixel coordinates.

[0022] Optionally, the construction unit includes: a construction subunit, configured to construct an observation error function between the camera and the radar using the following formula: in, Π i represents the projection function of the 3D map point centered on the camera onto the i-th camera image, T i L ∈SE(3) represents the pose of the i-th camera coordinate system in the radar coordinate system L, represents the position of the radar coordinate system L in the world coordinate system W at time t, (n W ,p W ) represents the parameterization of the plane's normal vector and position in the map coordinate system, is the first pixel coordinate, is the second pixel coordinate.

[0023] Optionally, the calculation unit includes: a processing subunit, used to reversely project the first pixel coordinates onto the local plane to obtain the intersection of the projection lines on the local plane; and a determination subunit, used to determine the coordinates of the intersection as the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space.

[0024] Optionally, the calculation module includes: a first construction unit, used to construct, for each camera, the observation error sum function of all pixel points of two frames of images using the observation error function; a second construction unit, used to construct an error loss function of the observation error sum function, and calculate the preliminary internal parameters of the camera and the preliminary external parameters between the camera and the radar based on the error loss function; a third construction unit, used to construct the association relationship between the two-dimensional feature trajectory and the local plane; an optimization unit, used to optimize the preliminary internal parameters and the preliminary external parameters based on the association relationship to obtain target internal parameters and target external parameters.

[0025] Optionally, the third construction unit includes: a calculation subunit, used to calculate, for each feature point in the two-dimensional feature trajectory, the intersection position of the reverse projection of the feature point to the local plane; a sorting subunit, used to identify a set of feature objects in the trajectory scene where the local plane is located, and arrange the set of feature objects in descending order according to size to obtain a feature object sequence; a construction subunit, used to select feature objects in the feature object sequence in order, and establish matching relationships between the intersection positions of the feature objects and the feature points in turn to obtain association relationships.

[0026] Optionally, the optimization unit includes: a first optimization subunit, used to optimize the association relationship of a single camera for each camera to obtain a first optimization relationship; for every two cameras, perform association optimization on the association relationship between the two cameras to obtain a second optimization relationship; and a second optimization subunit, used to optimize the preliminary internal parameters and the preliminary external parameters using the first optimization relationship and the second optimization relationship to obtain target internal parameters and target external parameters.

[0027] Optionally, the first optimization subunit is also used to: construct an external parameter iterator from a single camera to the radar; repeatedly execute the external parameter iterator according to the following steps until the external parameters converge: calculate the back projection ray of each feature point of the two-dimensional feature trajectory of the camera to the local plane, and calculate the normal vector of the local plane; calculate the angle between the back projection ray and the normal vector; calculate the association score of the feature point based on the angle; filter the feature points whose association scores are less than a preset value in the two-dimensional feature trajectory, and use a random sampling consensus algorithm to eliminate discrete feature points in the two-dimensional feature trajectory; calculate the external parameters corresponding to the optimized two-dimensional feature trajectory; determine whether the external parameters converge; after execution is completed, output the first optimization relationship.

[0028] Optionally, the first optimization subunit is also used to: use nearest neighbor search to find different feature points in the two-dimensional feature trajectories of multiple cameras corresponding to the same three-dimensional map point, and obtain the trajectory association relationship of multiple cameras; use the neighborhood plane of the two-dimensional feature trajectories of multiple cameras to perform template matching to obtain multiple image blocks, convert the multiple image blocks into multiple grayscale images, equalize the pixel intensity histograms of the multiple grayscale images, and associate them by finding the centroid of the mutual correlation values ​​to obtain the pixel correspondence relationship of multiple cameras; output the trajectory association relationship and the pixel correspondence relationship as a second optimization relationship.

[0029] Optionally, the second optimization subunit is also used to: use the first optimization relationship and the second optimization relationship to eliminate weakly correlated points and discrete points in the association relationship to obtain a third optimization relationship; normalize the sum of camera residuals of the preliminary internal parameters and the preliminary external parameters of multiple cameras respectively to obtain normalized internal parameters and normalized external parameters; use the observation error function and the third optimization relationship to optimize the normalized internal parameters and normalized external parameters to obtain target internal parameters and target external parameters.

[0030] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.

[0031] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0032] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.

[0033] Beneficial effects of the present invention:

[0034] 1. The external and internal parameters of the vehicle camera can be jointly calibrated by relying only on the corresponding relationship between the image features of the camera frames and the relatively weak assumption that the image features are in the local plane, without relying on the cross-modal environment features, thus ensuring robustness;

[0035] 2. The association between image feature points and 3D map points is jointly optimized by single camera and multiple cameras, which improves the accuracy of the correspondence between pixels and point clouds, thereby improving the accuracy of camera intrinsic and extrinsic parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0037] Figure 1 is a hardware structure block diagram of a car according to an embodiment of the present invention;

[0038] Figure 2 is a flow chart of a method for calculating camera parameters according to an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of the deployment of vehicle sensors in an embodiment of the present invention;

[0040] Figure 4 is a corresponding relationship diagram of the observation error function in an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of association of corresponding image blocks across cameras in an embodiment of the present invention;

[0042] Figure 6 is an overall flow chart of an embodiment of the present invention;

[0043] Figure 7 4 is a structural block diagram of a camera parameter calculation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] Example 1

[0047] The method embodiment provided in the first embodiment of the present application can be executed in a car, a processor, a controller or a similar processing device. Taking running on a car as an example, Figure 1 1 is a hardware structure diagram of a car according to an embodiment of the present invention. Figure 1 As shown, a car may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the above-mentioned automobile may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above-mentioned automobile. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0048] The memory 104 can be used to store automobile programs, for example, software programs and modules of application software, such as an automobile program corresponding to a method for calculating camera parameters of an automobile in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the automobile program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the automobile via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0049] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the car. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0050] In this embodiment, a method for calculating camera parameters is provided. Figure 2 is a flow chart of a method for calculating camera parameters according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0051] Step S202, when the vehicle is moving, obtaining a two-dimensional feature trajectory of the same world landmark collected by a camera, and constructing a local plane of the trajectory scene where the world landmark is located based on a radar, wherein the camera and the radar are arranged on the vehicle;

[0052] Optionally, the camera and the radar are fixedly mounted on the vehicle, or may be movably mounted on the vehicle, such as the camera's mounting position on the vehicle is movable, the radar's mounting position on the vehicle is movable, or both the camera and the radar's mounting positions on the vehicle are movable.

[0053] Optionally, the local plane includes a ground grid in three-dimensional coordinates and a three-dimensional structure of points on the ground. The local plane includes multiple laser three-dimensional points, and the laser three-dimensional points include depth information. The two-dimensional feature trajectory includes pixel coordinates of multiple feature points of the world landmark in the observed camera.

[0054] The radar in this embodiment can be a laser radar or other types of radars with a field of view close to that of the camera. The vehicle-mounted multi-sensor system in this embodiment consists of N cameras, a laser radar and an IMU (attitude sensor). Figure 3 It is a schematic diagram of the deployment of vehicle sensors in an embodiment of the present invention. The large field of view sector is the field of view of the camera, the small field of view sector is the front grille laser radar, and the irregular ellipse is the field of view of the corner radar.

[0055] Step S204, constructing an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate an error value between an actual position and a theoretical position of a feature point collected by the camera;

[0056] Step S206: using the observation error function to calculate the external parameters between the camera and the radar and the internal parameters of the camera.

[0057] Through the above steps, the vehicle obtains the two-dimensional feature trajectory of the same world landmark collected by the camera during movement, and constructs a local plane of the trajectory scene where the world landmark is located based on the radar, wherein the camera and the radar are arranged on the vehicle; the observation error function between the camera and the radar is constructed by using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate the error value between the actual position and the theoretical position of the feature point collected by the camera; the observation error function is used to calculate the external parameters between the camera and the radar and the internal parameters of the camera, relying only on the image feature correspondence between camera frames, and using the high-quality 3D scale information provided by the radar to constrain the position estimation between multiple sensors, thereby realizing a method for jointly calculating and calibrating the external parameters and internal parameters of vehicle-mounted multiple sensors without relying on cross-modal environmental features, solving the technical problem that the related technology must rely on specific calibration objects when calibrating camera parameters, and improving the robustness when calibrating camera parameters.

[0058] In one implementation of the present embodiment, obtaining a two-dimensional feature trajectory of the same world landmark captured by a camera includes: obtaining multiple frames of images captured by the camera; selecting a world landmark and locating the world landmark in the multiple frames of images; and extracting the pixel coordinates of the world landmark from the multiple frames of images to obtain a two-dimensional feature trajectory.

[0059] The vehicle travels a certain track, and each camera uses a monocular SFM (Structure From Motion), such as ORB-SLAM, to extract high-quality 2D image feature point trajectories (pixel coordinates of feature points in the observed camera) during the motion process, without using the spatial information of the feature points and the camera trajectory information.

[0060] In one implementation of the present embodiment, constructing a local plane of the trajectory scene where the world landmark is located based on radar includes: acquiring multiple groups of three-dimensional point clouds of the world landmark collected by radar; aligning the multiple groups of three-dimensional point clouds and filtering out dynamic obstacles in the multiple groups of three-dimensional point clouds to obtain a laser point cloud map; and reconstructing the ground grid and the three-dimensional structure according to the laser point cloud map to obtain a local plane of the trajectory scene where the world landmark is located.

[0061] The radar scans at a lower frequency to obtain the 3D point cloud information of the same track of the vehicle. The 3D point cloud obtained is registered through the ICP (Iterative Closest Point) algorithm to filter out dynamic obstacles and establish a laser point cloud map of this section of the vehicle track. The vehicle posture is initialized by the longitude and latitude and angle observations of the vehicle provided by the IMU. Based on the established laser point cloud map, the Poisson surface is used to reconstruct the mesh information of the ground, and SuMa (Surf-based Mapping) is used to complete the 3D structure reconstruction of the points on the ground. The ground mesh and the 3D structure together represent the local plane in this section of the track scene.

[0062] In this embodiment, using the two-dimensional feature trajectory and the local plane to construct the observation error function between the camera and the radar includes: extracting the first pixel coordinates of the world landmark in the first image frame and the second pixel coordinates in the second image frame from the two-dimensional feature trajectory; calculating the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane; inversely projecting the three-dimensional coordinates to the second image frame to obtain converted coordinates; and using the converted coordinates and the second pixel coordinates to construct the observation error function between the camera and the radar.

[0063] In one example, using the three-dimensional coordinates and the second pixel coordinates to construct an observation error function between the camera and the radar includes: using the following formula to construct an observation error function between the camera and the radar: in, Π i represents the projection function of the 3D map point centered on the camera onto the i-th camera image, T i L ∈SE(3) represents the pose of the i-th camera coordinate system in the radar coordinate system L, represents the position of the radar coordinate system L in the world coordinate system W at time t, (n W ,p W ) represents the parameterization of the plane's normal vector and position in the map coordinate system, is the first pixel coordinate, is the second pixel coordinate.

[0064] In one example, calculating the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane includes: inversely projecting the first pixel coordinates to the local plane to obtain the intersection of the projection lines on the local plane; and determining the coordinates of the intersection as the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space.

[0065] The loss function (1) is defined based on feature matching in the two camera views and the laser point cloud map. is a pixel of the camera in the i-th frame at time t1, is a pixel of the jth frame of the same camera at time t2. and is the observation of the same 3D map point from different camera perspectives, then and The inverse projection transformation should intersect the local plane in the space where the point lies. The distance between the actual observed three-dimensional space point and the theoretical observation point (Formula 2) reflects the observation error of the point (i.e., reprojection error). Minimizing the observation error can better and more accurately describe the correspondence between the pixel point and the three-dimensional space point.

[0066]

[0067] Among them, i T represents the projection function of the 3D map point centered on the camera onto the i-th camera image, which characterizes the internal parameters of the camera. i L ∈SE(3) represents the pose of the i-th camera coordinate system in the radar coordinate system L. represents the position of the radar coordinate system L in the world coordinate system W at time t. (n W ,p W ) represents the parameterization of the plane's normal vector and position in the map coordinate system.

[0068] As shown in formula (1), the pixel point passes Transformations and planes (n W ,p W ) can establish a mapping relationship between image pixels and laser point cloud map points. The position of the radar relative to the world coordinate system is known, and what needs to be solved are the external parameters of N cameras relative to the radar and the internal parameters of the cameras The e in formula (3) refers to the sum of the reprojection errors of all map points (points collected by the radar) observed in the camera's i-th frame and j-th frame corresponding to all pixel points (points in the image frame collected by the camera). Subsequently, the initial intrinsic and extrinsic parameters of the camera can be obtained by constructing the least squares method to minimize the reprojection error through Gauss-Newton or Levenberg-Marquardt.

[0069] Figure 4 is a corresponding relationship diagram of the observation error function in the embodiment of the present invention, 3D point x wIt is obtained by intersecting the inverse projection of the pixel point with the local plane in the radar point cloud map. The loss function describes the difference between the features transferred from another frame to the current frame and the features independently observed in the current frame.

[0070] In this embodiment, using the observation error function to calculate the external parameters between the camera and the radar and the internal parameters of the camera includes: for each camera, using the observation error function to construct an observation error sum function for all pixels of two frames of images; constructing an error loss function of the observation error sum function, and calculating the preliminary internal parameters of the camera and the preliminary external parameters between the camera and the radar based on the error loss function; constructing an association relationship between the two-dimensional feature trajectory and the local plane; and optimizing the preliminary internal parameters and the preliminary external parameters based on the association relationship to obtain target internal parameters and target external parameters.

[0071] In one example, constructing an association relationship between the two-dimensional feature trajectory and the local plane includes: for each feature point in the two-dimensional feature trajectory, calculating the intersection position of the reverse projection of the feature point to the local plane; identifying a set of feature objects in the trajectory scene where the local plane is located, and arranging the set of feature objects in descending order according to size to obtain a feature object sequence; selecting feature objects in the feature object sequence in order, and establishing matching relationships between the intersection positions of the feature objects and the feature points in turn to obtain an association relationship.

[0072] The two-dimensional feature trajectory of this embodiment is the pixel trajectory of the feature point extracted by the camera, and the association relationship is to establish the relationship between the pixel trajectory and the laser three-dimensional map point plane, first by associating the pixel points with the plane, and then iteratively optimizing to the three-dimensional map point on the local plane.

[0073] Since the normal vector of the feature plane must be obtained in the loss function to obtain the position of the corresponding local plane, it is necessary to obtain the intersection of the image pixel point with the feature plane after reverse projection. The intersection position is calculated by the external parameters from the camera to the radar and the internal parameters of the camera. The accuracy of the association between the trajectory and the map point depends on the initial estimation of the above two and the element size of the plane surface.

[0074] In a trajectory scene, there is information of multiple scales (sizes) of planes. Using this information, the calibration parameters (pixels) and intersection positions are matched from coarse to fine (for example, the road surface and the wall may be a large-sized grid, and the leaves on the tree can be relatively small-sized planes). By first matching large features to small features in the environment, a set of trajectories related to feature scales from large to small are gradually constructed.

[0075] In one example, the preliminary internal parameters and the preliminary external parameters are optimized based on the association relationship to obtain target internal parameters and target external parameters, including: for each camera, optimizing the association relationship of a single camera to obtain a first optimization relationship; for every two cameras, performing association optimization on the association relationship between the two cameras to obtain a second optimization relationship; and optimizing the preliminary internal parameters and the preliminary external parameters using the first optimization relationship and the second optimization relationship to obtain target internal parameters and target external parameters.

[0076] On the one hand, based on the optimization of a single camera, the association relationship of the single camera is optimized to obtain a first optimization relationship, including: constructing an external parameter iterator from the single camera to the radar; repeatedly executing the external parameter iterator according to the following steps until the external parameters converge: calculating the back projection ray of each feature point of the two-dimensional feature trajectory of the camera to the local plane, and calculating the normal vector of the local plane; calculating the angle between the back projection ray and the normal vector; calculating the association score of the feature point based on the angle; filtering feature points with association scores less than a preset value in the two-dimensional feature trajectory, and using a random sampling consensus algorithm to eliminate discrete feature points in the two-dimensional feature trajectory; calculating the external parameters corresponding to the optimized two-dimensional feature trajectory; judging whether the external parameters converge; after the execution is completed, outputting the first optimization relationship.

[0077] Each single camera to radar extrinsic parameter iterator is constructed to improve the initial estimate, remove outlier tracks, and estimate feature point locations. During each iteration, the track is associated to the map, the track-to-map association score is calculated, low association score tracks are removed, and the random sampling consensus (RANSAC) algorithm is used to query the inliers and corresponding extrinsic parameters of the track based on the root mean square error (RMSE). Iterations are repeated and new tracks are added to the inliers until the extrinsic parameters converge.

[0078] In one embodiment, when calculating the track-to-map association score, two scores are constructed to evaluate the quality of the pixel track-to-map association. Let i, j be the feature points observed in the same track (since there are N cameras, there are N track observations), x i and n i The intersection of image feature points i, j with the ground grid reconstructed by laser and the normal vector of the corresponding grid after reverse projection, then we have

[0079]

[0080] If the normal angle between the backprojection ray and the intersecting map grid plane is small, the correlation score s θ Is larger.

[0081]

[0082] If the distance between the intersection point between the back-projected ray and the intersecting map grid plane and the laser 3D point cloud is small, then s d Larger. θ and σ d Can be adjusted to accommodate smaller or larger spreads in statistics.

[0083] On the other hand, association optimization is performed based on multiple cameras, and the association relationship between the two cameras is optimized to obtain a second optimization relationship, including: using nearest neighbor search to find different feature points in the two-dimensional feature trajectories of multiple cameras corresponding to the same three-dimensional map point, to obtain the trajectory association relationship of the multiple cameras; using the neighborhood plane of the two-dimensional feature trajectories of the multiple cameras to perform template matching to obtain multiple image blocks, converting the multiple image blocks into multiple grayscale images, equalizing the pixel intensity histograms of the multiple grayscale images, and associating by finding the centroid of the mutual correlation values ​​to obtain the pixel correspondence relationship of the multiple cameras; outputting the trajectory association relationship and the pixel correspondence relationship as the second optimization relationship.

[0084] The multi-camera correspondence is composed of multi-camera trajectory association and image block matching refinement. The figure is a schematic diagram of the cross-camera corresponding image block association in an embodiment of the present invention. The feature trajectories observed by the two cameras are associated through the radar map, and an approximate plane homography is used to map one viewpoint to another viewpoint, and a cross-correlation function is used to find a more accurate sub-pixel match. Multi-camera feature matching does not require a shared field of view and is completed by establishing a list of pixel correspondences for different camera images. The nearest neighbor search algorithm is used to find the pixel points corresponding to the same three-dimensional map point for the i-th and j-th cameras.

[0085] Image space block matching refinement is based on the assumption that the local area of ​​the map feature point is a plane, and the neighborhood of the corresponding pixel point is also a pixel plane:

[0086]

[0087] Block the image (S i,t1 Representative is the center of the circle, and the neighborhood is S i,t1 pixel plane) and S j,t2 (S j,t2 Representative is the center of the circle, and the neighborhood is S j,t2 The pixel plane of the image is converted into a grayscale image, its pixel intensity histogram is equalized, and the centroid of the cross-correlation value is found for association to obtain a more accurate pixel correspondence.

[0088] Figure 5It is a schematic diagram of the association of corresponding image blocks across cameras in an embodiment of the present invention, in which the feature trajectories observed by two cameras are associated through a radar map, one viewpoint is mapped to another viewpoint using an approximate planar homography, and a cross-correlation function is used to find a more accurate sub-pixel match.

[0089] In an implementation based on the above example, the first optimization relationship and the second optimization relationship are used to optimize the preliminary internal parameters and the preliminary external parameters to obtain target internal parameters and target external parameters, including: using the first optimization relationship and the second optimization relationship to eliminate weakly correlated points and discrete points in the association relationship to obtain a third optimization relationship; normalizing the sum of camera residuals of the preliminary internal parameters and the preliminary external parameters of multiple cameras respectively to obtain normalized internal parameters and normalized external parameters; using the observation error function and the third optimization relationship to optimize the normalized internal parameters and the normalized external parameters to obtain target internal parameters and target external parameters.

[0090] Through the correspondence between single cameras and multiple cameras, the error can be minimized based on the existing multi-sensor constraint relationship and the geometric function in the above formula (3) to eliminate weakly correlated camera associations and discrete feature points. Since the camera positions and field of view are different, the number of features observed by the camera is also different, so it is necessary to normalize the sum of the camera residuals in the system. The Ceres solver can be used to calculate the constructed loss function and perform nonlinear least squares optimization to optimize the camera-to-laser external parameters and camera internal parameters to obtain the final target internal parameters and target external parameters.

[0091] This embodiment proposes a joint calibration method for the external and internal parameters of multi-sensor (camera and radar) that does not rely on specific calibration objects and cross-modal environmental features. Assume that the vehicle-mounted multi-sensor system consists of N cameras, a radar and an IMU (attitude sensor). Figure 6 It is an overall flow chart of an embodiment of the present invention, including:

[0092] First, we extract high-quality 2D feature trajectories for each camera, a list of pixels of the same world landmark points collected in multiple frames, i.e., the pixel coordinates of the feature points in the observed camera;

[0093] The radar continuously collects and scans to provide a 3D point cloud in the environment. The radar's motion trajectory is obtained using the ICP through the scale information provided by the IMU. The 3D point cloud is spliced ​​according to the radar's trajectory to complete the 3D reconstruction. At the same time, dynamic obstacles are removed and the ground is segmented. The ground grid is constructed using Poisson surface reconstruction, and SuMa is used to complete the 3D structure reconstruction of the points on the ground. The ground grid and the structural surface together describe the local planar structure of the scene;

[0094] Construct a geometric loss function (reprojection error): Given the observation of a feature point in two cameras at different times and the laser 3D reconstruction plane corresponding to the feature point, find the reprojection error of the feature point in the two camera observations;

[0095] Trajectory and map point association: The association between camera feature points and laser 3D planes is performed from coarse to fine using ray tracing. As the parameters to be estimated gradually converge, the laser 3D plane involved in the association gradually becomes smaller.

[0096] Single camera optimization: Remove abnormal trajectories and false associations with low association scores. Use random sampling consensus (RANSAC) to query the inliers and corresponding extrinsic parameters of the trajectory based on the root mean square error (RMSE). Repeat the iteration and add new trajectories to the inliers until the extrinsic parameters converge;

[0097] Multi-camera correspondence acquisition: Multi-camera correspondence consists of two parts: cross-camera trajectory association and image space block matching refinement. Cross-camera trajectory association is composed of a list of pixel correspondences consisting of camera image pairs from different cameras. Image space block matching performs template matching on the neighborhood planes of map points observed by different cameras and finds the mutually correlated centroids to complete sub-pixel precision alignment;

[0098] Joint optimization: After establishing the constraint relationship between multiple sensors, the constructed geometric loss function is used to perform least squares optimization to find the minimum value. Since the position and field of view of the camera are different, the number of features observed by the camera is also different, so the sum of the camera residuals in the system needs to be normalized. The Ceres solver is used to calculate the constructed loss function, and nonlinear least squares optimization is performed to optimize the camera-to-laser external parameters and the camera internal parameters.

[0099] By adopting the scheme of this embodiment, both the external parameters of the camera and the radar and the internal parameters of the camera can be calibrated. A joint calibration method for the external and internal parameters of vehicle-mounted multi-sensors can be calibrated without relying on cross-modal environmental features. This method does not rely on the prior information of the field attractions in different images. It constrains the posture transformation between cameras by minimizing the pure geometric loss function between the image feature pairs through the correspondence between images and different camera images. At the same time, the high-quality 3D scale information provided by the radar is used to constrain the position estimation between multiple sensors, thereby obtaining accurate multi-sensor external and internal parameters, relying only on the image feature correspondence between camera frames and the relatively weak assumption that the image features are in the local plane. An optimization problem is constructed to minimize its geometric loss function, which describes the correspondence between the corresponding image features and the local plane (surface or mesh) reconstructed by the radar scan under the same point view, eliminating the structural estimation in the optimization problem and improving the accuracy of the camera's internal and external parameters.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0101] Example 2

[0102] In this embodiment, a device for calculating camera parameters is also provided, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0103] Figure 7 is a structural block diagram of a camera parameter calculation device according to an embodiment of the present invention, such as Figure 7 As shown, the device comprises:

[0104] The processing module 70 is used to obtain the two-dimensional feature trajectory of the same world landmark collected by the camera during the movement of the vehicle, and to construct a local plane of the trajectory scene where the world landmark is located based on the radar, wherein the camera and the radar are arranged on the vehicle;

[0105] A construction module 72, configured to construct an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate an error value between an actual position and a theoretical position of a feature point collected by the camera;

[0106] The calculation module 74 is used to calculate the external parameters between the camera and the radar and the internal parameters of the camera by using the observation error function.

[0107] Optionally, the processing module includes: a first acquisition unit, used to acquire multiple frames of images captured by a camera; a positioning unit, used to select world landmarks and locate the world landmarks in the multiple frames of images; and an extraction unit, used to extract the pixel coordinates of the world landmarks from the multiple frames of images respectively to obtain a two-dimensional feature trajectory.

[0108] Optionally, the processing module includes: a second acquisition unit, used to acquire multiple groups of three-dimensional point clouds of the world landmarks collected by radar; a registration unit, used to register the multiple groups of three-dimensional point clouds and filter out dynamic obstacles in the multiple groups of three-dimensional point clouds to obtain a laser point cloud map; and a reconstruction unit, used to reconstruct the ground grid and three-dimensional structure according to the laser point cloud map to obtain a local plane of the trajectory scene where the world landmarks are located.

[0109] Optionally, the construction module includes: an extraction unit, used to extract the first pixel coordinates of the world landmark in the first image frame and the second pixel coordinates in the second image frame from the two-dimensional feature trajectory; a calculation unit, used to calculate the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane; a processing unit, used to inversely project the three-dimensional coordinates to the second image frame to obtain transformed coordinates; and a construction unit, used to construct an observation error function between the camera and the radar using the transformed coordinates and the second pixel coordinates.

[0110] Optionally, the construction unit includes: a construction subunit, configured to construct an observation error function between the camera and the radar using the following formula: in, Π i represents the projection function of the 3D map point centered on the camera onto the i-th camera image, T i L ∈SE(3) represents the pose of the i-th camera coordinate system in the radar coordinate system L, represents the position of the radar coordinate system L in the world coordinate system W at time t, (n W ,p W ) represents the parameterization of the plane's normal vector and position in the map coordinate system, is the first pixel coordinate, is the second pixel coordinate.

[0111] Optionally, the calculation unit includes: a processing subunit, used to reversely project the first pixel coordinates onto the local plane to obtain the intersection of the projection lines on the local plane; and a determination subunit, used to determine the coordinates of the intersection as the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space.

[0112] Optionally, the calculation module includes: a first construction unit, used to construct, for each camera, the observation error sum function of all pixel points of two frames of images using the observation error function; a second construction unit, used to construct an error loss function of the observation error sum function, and calculate the preliminary internal parameters of the camera and the preliminary external parameters between the camera and the radar based on the error loss function; a third construction unit, used to construct the association relationship between the two-dimensional feature trajectory and the local plane; an optimization unit, used to optimize the preliminary internal parameters and the preliminary external parameters based on the association relationship to obtain target internal parameters and target external parameters.

[0113] Optionally, the third construction unit includes: a calculation subunit, used to calculate, for each feature point in the two-dimensional feature trajectory, the intersection position of the reverse projection of the feature point to the local plane; a sorting subunit, used to identify a set of feature objects in the trajectory scene where the local plane is located, and arrange the set of feature objects in descending order according to size to obtain a feature object sequence; a construction subunit, used to select feature objects in the feature object sequence in order, and establish matching relationships between the intersection positions of the feature objects and the feature points in turn to obtain association relationships.

[0114] Optionally, the optimization unit includes: a first optimization subunit, used to optimize the association relationship of a single camera for each camera to obtain a first optimization relationship; for every two cameras, perform association optimization on the association relationship between the two cameras to obtain a second optimization relationship; and a second optimization subunit, used to optimize the preliminary internal parameters and the preliminary external parameters using the first optimization relationship and the second optimization relationship to obtain target internal parameters and target external parameters.

[0115] Optionally, the first optimization subunit is also used to: construct an external parameter iterator from a single camera to the radar; repeatedly execute the external parameter iterator according to the following steps until the external parameters converge: calculate the back projection ray of each feature point of the two-dimensional feature trajectory of the camera to the local plane, and calculate the normal vector of the local plane; calculate the angle between the back projection ray and the normal vector; calculate the association score of the feature point based on the angle; filter the feature points whose association scores are less than a preset value in the two-dimensional feature trajectory, and use a random sampling consensus algorithm to eliminate discrete feature points in the two-dimensional feature trajectory; calculate the external parameters corresponding to the optimized two-dimensional feature trajectory; determine whether the external parameters converge; after execution is completed, output the first optimization relationship.

[0116] Optionally, the first optimization subunit is also used to: use nearest neighbor search to find different feature points in the two-dimensional feature trajectories of multiple cameras corresponding to the same three-dimensional map point, and obtain the trajectory association relationship of multiple cameras; use the neighborhood plane of the two-dimensional feature trajectories of multiple cameras to perform template matching to obtain multiple image blocks, convert the multiple image blocks into multiple grayscale images, equalize the pixel intensity histograms of the multiple grayscale images, and associate them by finding the centroid of the mutual correlation values ​​to obtain the pixel correspondence relationship of multiple cameras; output the trajectory association relationship and the pixel correspondence relationship as a second optimization relationship.

[0117] Optionally, the second optimization subunit is also used to: use the first optimization relationship and the second optimization relationship to eliminate weakly correlated points and discrete points in the association relationship to obtain a third optimization relationship; normalize the sum of camera residuals of the preliminary internal parameters and the preliminary external parameters of multiple cameras respectively to obtain normalized internal parameters and normalized external parameters; use the observation error function and the third optimization relationship to optimize the normalized internal parameters and normalized external parameters to obtain target internal parameters and target external parameters.

[0118] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0119] Example 3

[0120] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0121] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0122] S1, when the vehicle is moving, obtaining a two-dimensional feature trajectory of the same world landmark captured by a camera, and constructing a local plane of the trajectory scene where the world landmark is located based on a radar, wherein the camera and the radar are arranged on the vehicle;

[0123] S2, constructing an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate an error value between an actual position and a theoretical position of a feature point collected by the camera;

[0124] S3, using the observation error function to calculate the external parameters between the camera and the radar and the internal parameters of the camera.

[0125] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store computer programs.

[0126] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0127] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0128] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0129] S1, when the vehicle is moving, obtaining a two-dimensional feature trajectory of the same world landmark captured by a camera, and constructing a local plane of the trajectory scene where the world landmark is located based on a radar, wherein the camera and the radar are arranged on the vehicle;

[0130] S2, constructing an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate an error value between an actual position and a theoretical position of a feature point collected by the camera;

[0131] S3, using the observation error function to calculate the external parameters between the camera and the radar and the internal parameters of the camera.

[0132] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0135] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described in the text are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0136] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for calculating camera parameters, characterized in that: include: When the vehicle is moving, a two-dimensional feature trajectory of the same world landmark captured by a camera is obtained, and a local plane of the trajectory scene where the world landmark is located is constructed based on a radar, wherein the camera and the radar are arranged on the vehicle; The two-dimensional feature trajectory and the local plane are used to construct an observation error function between the camera and the radar, wherein the observation error function is used to calculate the error value between the actual position and the theoretical position of the feature point collected by the camera; The observation error function is used to calculate the external parameters between the camera and the radar and the internal parameters of the camera.

2. The method according to claim 1, characterized in that Obtaining the two-dimensional feature trajectory of the same world landmark collected by the camera includes: Get multiple frames of images captured by the camera; Selecting a world landmark, and locating the world landmark in the multiple frames of images; The pixel coordinates of the world landmarks are respectively extracted from the multiple frames of images to obtain a two-dimensional feature trajectory.

3. The method according to claim 1, characterized in that The local plane of the trajectory scene where the world landmark is located is constructed based on radar, including: Acquire multiple groups of three-dimensional point clouds of the world landmarks collected by radar; Registering the multiple groups of three-dimensional point clouds and filtering out dynamic obstacles in the multiple groups of three-dimensional point clouds to obtain a laser point cloud map; The ground grid and the three-dimensional structure are reconstructed according to the laser point cloud map to obtain the local plane of the trajectory scene where the world landmark is located.

4. The method according to claim 1, characterized in that Using the two-dimensional feature trajectory and the local plane to construct an observation error function between the camera and the radar includes: Extracting a first pixel coordinate of the world landmark in the first image frame and a second pixel coordinate of the world landmark in the second image frame from the two-dimensional feature trajectory; Calculate the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane; Inversely projecting the three-dimensional coordinates onto the second image frame to obtain transformed coordinates; An observation error function between the camera and the radar is constructed using the converted coordinates and the second pixel coordinates.

5. The method according to claim 4, characterized in that Using the three-dimensional coordinates and the second pixel coordinates to construct an observation error function between the camera and the radar includes: The following formula is used to construct the observation error function between the camera and the radar: in, Π i represents the projection function of the 3D map point centered on the camera onto the i-th camera image, T i L ∈SE(3) represents the pose of the i-th camera coordinate system in the radar coordinate system L, represents the position of the radar coordinate system L in the world coordinate system W at time t, (n W ,p W ) represents the parameterization of the plane's normal vector and position in the map coordinate system, is the first pixel coordinate, is the second pixel coordinate.

6. The method according to claim 4, characterized in that Calculating the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space based on the local plane includes: Inversely projecting the first pixel coordinate onto the local plane to obtain an intersection point of the projection line on the local plane; The coordinates of the intersection point are determined as the three-dimensional coordinates of the first pixel coordinates in the three-dimensional space.

7. The method according to claim 1, characterized in that Using the observation error function to calculate the external parameters between the camera and the radar and the internal parameters of the camera includes: For each camera, the observation error function is used to construct the observation error sum function of all pixel points of two frames of images; Constructing an error loss function of the observation error and function, and calculating preliminary intrinsic parameters of the camera and preliminary extrinsic parameters between the camera and the radar based on the error loss function; Constructing an association relationship between the two-dimensional feature trajectory and the local plane; The preliminary internal parameters and the preliminary external parameters are optimized based on the association relationship to obtain target internal parameters and target external parameters.

8. The method according to claim 7, characterized in that Constructing the association relationship between the two-dimensional feature trajectory and the local plane includes: For each feature point in the two-dimensional feature trajectory, calculating the intersection position of the reverse projection of the feature point onto the local plane; Identify a set of feature objects in the trajectory scene where the local plane is located, and arrange the set of feature objects in descending order according to size to obtain a feature object sequence; The feature objects in the feature object sequence are selected in order, and matching relationships between intersection positions where the feature objects are located and feature points are established in turn to obtain an association relationship.

9. The method according to claim 7, characterized in that: Optimizing the preliminary internal parameters and the preliminary external parameters based on the association relationship to obtain target internal parameters and target external parameters includes: For each camera, the association relationship of the single camera is optimized to obtain a first optimization relationship; for every two cameras, the association relationship of the two cameras is optimized to obtain a second optimization relationship; The preliminary internal parameters and the preliminary external parameters are optimized using the first optimization relationship and the second optimization relationship to obtain target internal parameters and target external parameters.

10. The method according to claim 9, characterized in that Optimizing the association relationship of the single camera to obtain a first optimization relationship includes: Construct an external parameter iterator from a single camera to the radar; Repeat the extrinsic parameter iterator according to the following steps until the extrinsic parameters converge: calculate the inverse projection ray of each feature point of the two-dimensional feature trajectory of the camera to the local plane, and calculate the normal vector of the local plane; calculate the angle between the inverse projection ray and the normal vector; calculate the association score of the feature point based on the angle; filter the feature points whose association scores are less than a preset value in the two-dimensional feature trajectory, and use a random sampling consensus algorithm to eliminate discrete feature points in the two-dimensional feature trajectory; calculate the extrinsic parameters corresponding to the optimized two-dimensional feature trajectory; and determine whether the extrinsic parameters converge; After the execution is completed, the first optimization relationship is output.

11. The method according to claim 9, characterized in that Performing association optimization on the association relationship between the two cameras to obtain a second optimization relationship includes: The nearest neighbor search is used to find different feature points in the two-dimensional feature trajectories of multiple cameras corresponding to the same three-dimensional map point, and the trajectory correlation relationship of multiple cameras is obtained; Using the neighborhood plane of the two-dimensional feature trajectories of multiple cameras to perform template matching to obtain multiple image blocks, converting the multiple image blocks into multiple grayscale images, equalizing the pixel intensity histograms of the multiple grayscale images, and associating them by finding the centroid of the cross-correlation values ​​to obtain the pixel correspondence of multiple cameras; The trajectory association relationship and the pixel correspondence relationship are output as a second optimization relationship.

12. The method according to claim 9, characterized in that The first optimization relationship and the second optimization relationship are used to optimize the preliminary internal parameters and the preliminary external parameters to obtain target internal parameters and target external parameters, including: The first optimization relationship and the second optimization relationship are used to eliminate weakly correlated points and discrete points in the association relationship to obtain a third optimization relationship; Normalizing the camera residual sums of the preliminary internal parameters and the preliminary external parameters of multiple cameras respectively to obtain normalized internal parameters and normalized external parameters; The normalized internal parameters and the normalized external parameters are optimized using the observation error function and the third optimization relationship to obtain target internal parameters and target external parameters.

13. A camera parameter calculation device, characterized in that: include: A processing module, used for obtaining a two-dimensional feature trajectory of the same world landmark captured by a camera during the movement of the vehicle, and constructing a local plane of the trajectory scene where the world landmark is located based on a radar, wherein the camera and the radar are arranged on the vehicle; A construction module, used to construct an observation error function between the camera and the radar using the two-dimensional feature trajectory and the local plane, wherein the observation error function is used to calculate an error value between an actual position and a theoretical position of a feature point collected by the camera; A calculation module is used to calculate the external parameters between the camera and the radar and the internal parameters of the camera by using the observation error function.

14. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 12 when executed.

15. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 12.

Images