Camera posture calibration method and related device

Through movable lidar scanning and three-dimensional reconstruction technology, flexible and convenient calibration of camera position is achieved, solving the problem of relying on wheel limiters and target prediction and measurement in the prior art, and improving calibration flexibility and accuracy sex.

CN118314219BInactive Publication Date: 2024-12-03BEIJING JIDU TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410508606.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-25
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing camera position calibration methods rely on wheel limiters, which leads to the lack of flexibility in the calibration process and makes it difficult to accurately calibrate in different scenarios.

Method used

The calibration scene is scanned by movable lidar, and the position of the vehicle and camera is determined by three-dimensional reconstruction of point cloud data, avoiding the use of wheel limiters and target prediction measurements, and achieving flexible position calibration.

Benefits of technology

No complex calibration equipment and operations are required, which improves the flexibility and convenience of camera position calibration and is suitable for a variety of calibration scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118314219B_ABST
    Figure CN118314219B_ABST
Patent Text Reader

Abstract

The present application proposes a camera pose calibration method and related devices. The method may include: obtaining a three-dimensional reconstructed point cloud of the calibration scene based on multi-frame point cloud data obtained by scanning the calibration scene with a laser radar; determining the first pose of the vehicle in the three-dimensional reconstructed coordinate system based on the vehicle point cloud in the three-dimensional reconstructed point cloud; determining the second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system based on the position of the first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and the position of the second feature point on the target surface in the target image captured by the camera to be calibrated; determining the target pose of the camera to be calibrated relative to the vehicle based on the first pose of the vehicle in the three-dimensional reconstructed coordinate system and the second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system. The technical solution provided by the present application can solve the problem of low flexibility of the camera pose calibration method in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of sensor calibration technology, specifically, to the field of sensor extrinsic parameter calibration technology, and more specifically, to a camera posture calibration method and related devices. Background Art

[0002] The sensors of intelligent driving vehicles are the prerequisite for the normal operation of the intelligent driving system, which makes the parameter calibration of the sensors an essential task, which affects the accuracy of the intelligent driving system's judgment of the surrounding traffic conditions.

[0003] The parameter calibration of vehicle cameras is part of the vehicle sensor calibration work. Usually, the intrinsic parameters of vehicle cameras are provided by the manufacturer and are known. Therefore, the camera parameter calibration work mainly involves the calibration of camera extrinsic parameters, that is, the calibration of camera pose (i.e. position and attitude).

[0004] The most commonly used method for calibrating the pose of a vehicle camera is static calibration with a target in the calibration room. In static calibration with a target, the vehicle needs to be parked at a specified position using a wheel stopper to complete the calibration process. This shows that this calibration method has high requirements for the calibration equipment and calibration site, is highly dependent on the wheel stopper, and lacks flexibility. Summary of the invention

[0005] Based on the above-mentioned defects and shortcomings of the prior art, the present application proposes a camera pose calibration method, device, electronic device and computer program product, which can solve the problem of low flexibility of the camera pose calibration method in the prior art.

[0006] According to a first aspect of an embodiment of the present application, a camera pose calibration method is provided, the method comprising:

[0007] According to the multi-frame point cloud data obtained by scanning the calibration scene by the mobile laser radar, a three-dimensional reconstructed point cloud of the calibration scene is obtained; wherein the calibration scene includes a vehicle and a target, and the vehicle is provided with a camera to be calibrated; and the target includes a target surface;

[0008] Determine the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud; wherein the three-dimensional reconstructed coordinate system is the coordinate system of the three-dimensional reconstructed point cloud;

[0009] Determine a second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to a position of a first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and a position of a second feature point on the target surface in the target image acquired by the camera to be calibrated;

[0010] According to a first pose of the vehicle in the three-dimensional reconstructed coordinate system and a second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system, a target pose of the camera to be calibrated relative to the vehicle is determined.

[0011] According to a second aspect of an embodiment of the present application, a camera posture calibration device is provided, the device comprising:

[0012] A point cloud reconstruction module is used to obtain a three-dimensional reconstructed point cloud of the calibration scene according to multiple frames of point cloud data obtained by scanning the calibration scene by a mobile laser radar; wherein the calibration scene includes a vehicle and a target, and the vehicle is provided with a camera to be calibrated; and the target includes a target surface;

[0013] A first pose determination module, used to determine the first pose of the vehicle in a three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud; wherein the three-dimensional reconstructed coordinate system is the coordinate system of the three-dimensional reconstructed point cloud;

[0014] A second posture determination module is used to determine a second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to the position of the first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and the position of the second feature point on the target surface in the target image acquired by the camera to be calibrated;

[0015] The third posture determination module is used to determine the target posture of the camera to be calibrated relative to the vehicle according to the first posture of the vehicle in the three-dimensional reconstructed coordinate system and the second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system.

[0016] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: a memory and a processor;

[0017] The memory is connected to the processor and is used to store programs;

[0018] The processor is used to implement the camera pose calibration method as described in the first aspect by running the program in the memory.

[0019] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the camera pose calibration method as described in the first aspect is implemented.

[0020] According to a fifth aspect of the embodiments of the present application, a computer program product or a computer program is provided, wherein the computer program product includes the computer program, and when the processor of a computer device executes the computer program, the steps in the camera pose calibration method as described in the first aspect are implemented. Optionally, the computer program can be stored in a readable storage medium or in the cloud of the computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0021] In the technical solution provided in the present application, the calibration scene can be scanned based on a movable laser radar, and the three-dimensional reconstructed point cloud of the calibration scene can be obtained according to the point cloud data obtained by scanning, and the camera pose calibration can be performed based on the three-dimensional reconstructed point cloud. In the process of camera pose calibration based on the three-dimensional reconstructed point cloud, the pose of the vehicle in the three-dimensional reconstructed coordinate system (that is, the world coordinate system) of the three-dimensional reconstructed point cloud can be determined according to the vehicle point cloud; the pose of the camera in the three-dimensional reconstructed coordinate system can be obtained according to the position of some feature points on the target in the image and the position in the three-dimensional reconstructed coordinate system. It can be seen that in the present application, it is not necessary to use a wheel limiter to determine the pose of the vehicle in the world coordinate system, so the parking position of the vehicle in the calibration scene can be unfixed; there is no need to pre-measure the position of the target in the calibration room as a true value to obtain the pose of the target in the vehicle body coordinate system, so the placement position of the target in the calibration scene can also be unfixed, and it is only necessary to enable the camera to be calibrated to capture the target image. In summary, the camera pose calibration technology provided in this application does not need to rely on complex calibration equipment and calibration operations, making the camera pose calibration more flexible and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a camera pose calibration method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of a target surface provided in an embodiment of the present application;

[0025] Figure 3 A block diagram of a camera pose calibration device provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] Application Overview

[0029] The most commonly used method for the pose calibration of vehicle cameras is static target calibration in the calibration room. In static target calibration, the vehicle needs to be parked at the specified position using wheel stoppers to complete the calibration process. The position of the target in the calibration room is measured in advance and used as the true value. The position of the target relative to the wheel stopper is also known, so the position of the target relative to the vehicle body coordinate system is also known. Based on the position mapping relationship between the feature points on the target in the world coordinate system and the pixel coordinate system (or image coordinate system), the pose of the target relative to the camera coordinate system can be obtained. Finally, combined with the known pose of the target relative to the vehicle body coordinate system, the pose calibration of the vehicle camera relative to the vehicle body coordinate system can be obtained.

[0030] However, this camera pose calibration method has some limitations on the calibration equipment and relies on wheel limiters to locate the vehicle position. Otherwise, the calibration process is difficult to carry out and lacks flexibility.

[0031] In order to solve the aforementioned technical problems, the present application provides a camera posture calibration technology, through which the use of wheel limiters can be eliminated, the purpose of flexible parking of vehicles and flexible placement of targets can be achieved, and the flexibility and convenience of camera posture calibration can be improved.

[0032] Exemplary Methods

[0033] The embodiment of the present application provides a camera pose calibration method, which is used to calibrate the pose of a camera on a vehicle. The camera described here can specifically be an off-vehicle camera, through which the environmental information around the vehicle can be used to achieve intelligent driving, such as path planning in intelligent driving. The pose described here refers to the general term for the position and posture of the camera in the body coordinate system of the vehicle, that is, the external parameters of the camera (external parameters for short).

[0034] The method can be implemented by electronic devices, such as vehicle-mounted computers, laptop computers, desktop computers, servers, etc. In the case where the electronic device is a vehicle-mounted computer, the vehicle-mounted computer can also use the calculated camera pose information to update the pose information; in the case where the electronic device is not a vehicle-mounted computer, the electronic device can establish a communication connection with the vehicle, and then send the calculated camera pose information to the vehicle based on the communication connection, so that the vehicle updates the pose information using the new camera pose information.

[0035] The method is described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0036] like Figure 1 As shown, the method may include the following steps 101 to 104:

[0037] Step 101: Obtain a three-dimensional reconstructed point cloud of the calibration scene based on multiple frames of point cloud data obtained by scanning the calibration scene with a movable activated radar.

[0038] The calibration scene mentioned here refers to a scene used to calibrate the position and posture of a camera, and the calibration scene may include at least: a vehicle and a target.

[0039] The vehicle is equipped with a camera with a posture to be calibrated (referred to as the camera to be calibrated). During the camera posture calibration process, the vehicle can be parked in the calibration scene. The parking position of the vehicle can be different each time for calibration and does not need to be fixed. When the vehicle is parked, the steering wheel can be returned to the center, the wheels can be straightened, the doors, trunk, front hood and other openable parts can be closed, and the lights can be turned off to avoid affecting the data collection of the laser radar on the calibration scene. The ground in the calibration scene is preferably a flat ground, which is conducive to reducing the calibration error.

[0040] The aforementioned target can also be called a calibration plate, which is used to calibrate the camera pose. Its main role in camera pose calibration is to provide a reference object that can be accurately measured in both the world coordinate system and the pixel coordinate system, so that the camera pose can be accurately calculated. The number of targets is at least one, and the specific number of targets and their placement in the calibration scene can at least meet the following conditions: the image captured by the camera to be calibrated includes at least one image of the target. The target generally includes a target surface, the size and shape of the target surface are known, and the shape of the target surface is generally rectangular, preferably square.

[0041] The movable laser radar described in the embodiment of the present application can be a handheld laser radar or other laser radars capable of adjusting position so as to flexibly scan the calibration scene. In addition, the movable laser radar can specifically be a multi-line laser radar, such as a multi-line mechanical laser radar, a solid-state multi-line laser radar, etc.

[0042] When scanning the calibration scene using a mobile laser radar, the vehicle and the target (especially the target surface) in the calibration scene can be scanned. When scanning the vehicle, the entire vehicle body except the chassis can be scanned, including the front, rear, left, right and roof of the vehicle, that is, the entire vehicle body contour. When scanning the target, at least the target surface of each target can be scanned, preferably the entire target is scanned.

[0043] In order to facilitate the judgment of whether the point cloud data obtained by the laser radar scan meets the calibration requirements, that is, whether the point cloud data of the vehicle and the target is obtained, the point cloud data collected by the laser radar can be sent to the visualization tool, and the point cloud data can be viewed through the visualization tool. If the electronic device implementing this case has a point cloud data display function, the point cloud data collected by the laser radar can be sent to the electronic device; if the electronic device does not have a point cloud data display function, it can be viewed through other visualization tools with a point cloud data display function.

[0044] Preferably, during the scanning process of the calibration scene, the starting position of the movable laser radar can be the ground where the vehicle is parked, and the laser radar coordinates are parallel to the ground to facilitate subsequent calculation and processing.

[0045] The data collected by the laser radar is point cloud data, which is in frames. A frame of point cloud data is a set of data mainly containing point clouds within a period of time. The specific content and amount of data in the frame can be determined by the hardware performance of the laser radar and the data transmission method. A frame of point cloud data can be recorded as P k , represents the point cloud of the kth frame. Point cloud P k The points contained in can be defined as p i ={x,y,z,t},p i Represents the i-th point in a frame of point cloud, x, y, z are p i The three-dimensional coordinates in the laser radar coordinate system, t is p i By scanning the calibration scene, multi-frame point cloud data can be obtained, and the multi-frame point cloud data includes the point cloud data of the vehicle in the calibration scene and the point cloud data of the target.

[0046] The point cloud data obtained by scanning with a movable laser radar can be transmitted to an electronic device for executing the method, and can be transmitted by wired or wireless transmission methods. After obtaining the multi-frame point cloud data obtained by scanning the calibration scene, the electronic device can match the point clouds between frames based on a three-dimensional reconstruction algorithm (such as a continuous time nearest neighbor iterative matching algorithm (i.e., CT-ICP)), and process the multi-frame point cloud data to obtain a three-dimensional reconstructed point cloud. In the three-dimensional reconstructed point cloud, the coordinate position of each point cloud in the coordinate system of the three-dimensional reconstructed point cloud is known.

[0047] Since the multi-frame point cloud data includes the point cloud data of the vehicle and the point cloud data of the target in the calibration scene, the three-dimensional reconstructed point cloud also includes the point cloud data of the vehicle and the point cloud data of the target in the calibration scene. Among them, the coordinate system of the three-dimensional reconstructed point cloud is called the three-dimensional reconstruction coordinate system, which is equivalent to the world coordinate system. The three-dimensional reconstructed coordinate system can be a laser radar coordinate system parallel to the ground where the vehicle is parked. For example, when the starting position of the movable laser radar is the ground where the vehicle is parked and the laser radar coordinate system is parallel to the ground at this time, the laser radar coordinate system at this time can be used as the three-dimensional reconstruction coordinate system. When performing three-dimensional reconstruction, it can be based on the laser radar coordinate system at this time. It can be understood that as long as the laser radar coordinate system is parallel to the ground where the vehicle is parked, it can be used as a three-dimensional reconstruction coordinate system.

[0048] Step 102: Determine the first position of the vehicle in the three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud.

[0049] In an embodiment of the present application, based on the vehicle point cloud data included in the three-dimensional reconstructed point cloud, the first pose of the vehicle in the three-dimensional reconstructed coordinate system, that is, the pose of the vehicle body coordinate system in the three-dimensional reconstructed coordinate system, can be determined.

[0050] Step 103: Determine a second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to the position of the first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and the position of the second feature point on the target surface in the target image captured by the camera to be calibrated.

[0051] The first feature point is a target surface feature point that can be obtained based on the three-dimensional reconstructed point cloud, and the position of the first feature point in the three-dimensional reconstructed coordinate system (i.e., the first position) can be obtained directly or indirectly calculated. For example, the first feature point can be a target surface feature point directly identified by the three-dimensional reconstructed point cloud, such as a corner point of the entire target surface, such as Figure 2 Among the corner points E, F, G, and H, at least one of them can be selected for use, and the position of such feature points in the three-dimensional reconstruction coordinate system can be directly obtained; for another example, the first feature point can also be a target surface feature point obtained by indirect calculation through the three-dimensional reconstruction point cloud, such as the center of mass of the target surface. Specifically, the coordinates of the center of mass of the target surface can be calculated through the coordinates of the four corner points of the target surface. Among them, the center of mass of the target surface can also be understood as the center of the target surface. Since the thickness of the target surface is relatively thin, the thickness point cloud corresponding to the target surface point cloud data can be ignored, or only the point cloud data corresponding to the target surface plane can be considered. It can be understood that the first feature point can also be other target surface feature points that can be obtained based on the three-dimensional reconstruction point cloud, which will not be listed here.

[0052] Among them, the second feature point is a target surface feature point included in the target image captured by the camera, such as a corner point of the entire target surface, a target surface center point, etc. In the target, a mark for calibration is generally provided on the target, and the mark includes feature points that are easily detected and identified in the image and are used for camera pose calibration. Therefore, the second feature point can also be a feature point of the mark.

[0053] The markings on the target surface may include, but are not limited to, at least one of the following: a calibration pattern (such as a QR code pattern, a checkerboard pattern light), a calibration character (such as text, a symbol light).

[0054] Figure 2 The diagram shows a two-dimensional code pattern 201 used as a calibration pattern. The two-dimensional code pattern 201 includes a plurality of corner points that are easily detected and identified, such as the four corner points A, B, C, and D of the entire pattern, and some corner points inside the pattern, such as corner point a and corner point b. In the camera calibration process, the corner points of the entire pattern that are easier to identify can be used as the second feature points, such as Figure 2 The four corner points A, B, C, and D are shown.

[0055] Optionally, the type of the calibration pattern may be Apritag, which is a two-dimensional code pattern. By scanning the two-dimensional code pattern of the Apritag type, relevant information of the two-dimensional code pattern may be obtained, such as ID information and size information corresponding to the two-dimensional code pattern. For two-dimensional code patterns of the Apritag type, different patterns correspond to different IDs.

[0056] The first feature point and the second feature point may be the same feature point on the target surface, or different feature points on the target surface, or feature points that are partially the same and partially different on the target surface. For example, the first feature point and the second feature point are both corner points of the entire target surface, that is, the two are the same; for another example, the first feature point is the centroid of the target surface, and the second feature point is the corner point of the entire calibration pattern, that is, the two are different; for another example, the first feature point includes the corner points of the entire target surface and the centroid of the target surface, and the second feature point includes the corner points of the entire target surface and the corner points of the entire calibration pattern, that is, the two are partially the same and partially different.

[0057] The aforementioned camera is the camera to be calibrated. The target image described here is a frame of image captured by the camera, and the target image includes at least one target. In the case where the target image includes multiple targets, one of the targets can be selected for use, which can be randomly selected or selected according to preset rules. For the method of selecting according to preset rules, for example, in the case where the target image includes multiple targets, the identification number (i.e., ID) of the calibration pattern in each target can be obtained, and then the target where the calibration pattern corresponding to the smallest identification number is located is determined as the target target for calibrating the current camera posture, and then the position of the first feature point on the target target in the three-dimensional reconstructed coordinate system and the position of the second feature point in the target image are determined. The posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system. It can be understood that in the case where the target image includes multiple targets, at least two of them can also be selected for use.

[0058] Step 104: Determine a target pose of the camera to be calibrated relative to the vehicle according to the first pose of the vehicle in the three-dimensional reconstructed coordinate system and the second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system.

[0059] In an embodiment of the present application, based on the first pose of the vehicle in the three-dimensional reconstructed coordinate system and the second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system, the target pose of the camera to be calibrated relative to the vehicle can be obtained, that is, the pose of the camera to be calibrated in the vehicle body coordinate system.

[0060] Among them, the target pose of the camera to be calibrated in the vehicle coordinate system can be obtained according to the following formula (1):

[0061]

[0062] In formula (1), T k represents the pose of the camera to be calibrated in the 3D reconstructed coordinate system (i.e., the second pose), represents the position and posture of the vehicle in the 3D reconstructed coordinate system (i.e., the first position), Represents the target pose of the camera in the vehicle coordinate system.

[0063] In the embodiment of the present application, the calibration scene can be scanned based on the movable laser radar, and the three-dimensional reconstructed point cloud of the calibration scene can be obtained according to the point cloud data obtained by the scan, and the camera pose calibration can be performed based on the three-dimensional reconstructed point cloud. In the process of camera pose calibration based on the three-dimensional reconstructed point cloud, the pose of the vehicle in the three-dimensional reconstructed coordinate system (that is, the world coordinate system) of the three-dimensional reconstructed point cloud can be determined according to the vehicle point cloud; the pose of the camera in the three-dimensional reconstructed coordinate system can be obtained according to the position of some feature points on the target in the image and the position in the three-dimensional reconstructed coordinate system. It can be seen that the technical solution provided in the embodiment of the present application does not require the use of wheel limiters to determine the pose of the vehicle in the world coordinate system, so the parking position of the vehicle in the calibration scene can be unfixed; there is no need to pre-measure the position of the target in the calibration room as a true value to obtain the pose of the target in the vehicle body coordinate system, so the placement position of the target in the calibration scene can also be unfixed, and it is only necessary to enable the camera to be calibrated to capture the target image. In summary, the camera pose calibration method provided in this application does not need to rely on complex calibration equipment and calibration operations, making the camera pose calibration more flexible and convenient.

[0064] It should be noted that, for the aforementioned steps, those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the steps described, and some steps can be performed in other orders or simultaneously. For example, for step 102 and step 103, step 102 can be performed first and then step 103, or step 103 can be performed first and then step 102, or step 102 and step 103 can be performed simultaneously.

[0065] In some optional embodiments, step 101: obtaining a three-dimensional reconstructed point cloud of the calibration scene according to multiple frames of point cloud data obtained by scanning the calibration scene by the mobile laser radar may include steps A1 to A4 as described below:

[0066] Step A1: Set the laser radar coordinate system corresponding to the target frame point cloud as the three-dimensional reconstruction coordinate system.

[0067] The target frame point cloud described here is one of the frames in the multi-frame point cloud.

[0068] Optionally, the laser radar coordinate system of the target frame point cloud is parallel to the ground where the vehicle is parked. If the movable activated radar is located on the ground when collecting the first frame point cloud, and the laser radar coordinate system is parallel to the ground, then the target frame point cloud can be preferably the first frame point cloud of the multi-frame point cloud data.

[0069] Step A2: Determine the position transformation relationship between two adjacent frame point clouds.

[0070] The position transformation relationship described here may include: the position transformation relationship from the next frame of point cloud data to the previous frame of point cloud data, and / or, the position transformation relationship from the previous frame of point cloud data to the next frame of point cloud data. Through this position transformation relationship, two adjacent frames of point cloud can be converted to the laser radar coordinate system corresponding to the same point cloud. For example, in the case where the position transformation relationship is the position transformation relationship from the next frame of point cloud data to the previous frame of point cloud data, the next frame of point cloud data can be converted to the laser radar coordinate system corresponding to the previous frame of point cloud data. Similarly, in the case where the position transformation relationship is the position transformation relationship from the previous frame of point cloud data to the next frame of point cloud data, the previous frame of point cloud data can be converted to the laser radar coordinate system corresponding to the next frame of point cloud data.

[0071] In the case where the target frame point cloud is the first frame point cloud in the multiple frames of point cloud data, the position transformation relationship described here may be: a position transformation relationship from a subsequent frame point cloud data to a previous frame point cloud data.

[0072] In the case where the target frame point cloud is the last frame point cloud in the multiple frames of point cloud data, the position transformation relationship mentioned here may include: a position transformation relationship from a previous frame point cloud data to a next frame point cloud data.

[0073] When the target frame point cloud is an intermediate frame point cloud in the multi-frame point cloud data (i.e., the point cloud between the first frame point cloud and the last frame point cloud), for the point cloud before the target frame point cloud, the position transformation relationship described here may include: the position transformation relationship from the previous frame point cloud data to the next frame point cloud data; for the point cloud after the target frame point cloud, the position transformation relationship described here may include: the position transformation relationship from the next frame point cloud data to the previous frame point cloud data.

[0074] The position transformation relationship described here can specifically be: a rotation transformation matrix and a translation transformation matrix between two adjacent frames of point clouds.

[0075] Step A3: According to the position transformation relationship between two adjacent frames of point cloud data, other frame point clouds in the multi-frame point cloud data except the target frame point cloud are converted into a three-dimensional reconstructed coordinate system.

[0076] The purpose of this step is to unify the coordinate systems of all point clouds based on the coordinate system corresponding to the target frame point cloud, and convert other frame point clouds to the coordinate system corresponding to the target frame point cloud.

[0077] Step A4: Perform voxel filtering on the point cloud in the three-dimensional reconstructed coordinate system to obtain a three-dimensional reconstructed point cloud.

[0078] Voxel filtering is a technique used for processing three-dimensional point cloud data. The main goal of this processing method is to reduce the complexity and size of the data while maintaining key geometric features. In voxel filtering, the three-dimensional space is divided into a group of voxels of equal size. The multiple points included in each voxel are processed as a single point, usually the center point or average point of all the points in the voxel is selected as the single point. This can greatly reduce the number of data points while maintaining the approximate shape and features of the original point cloud. Optionally, the voxel size can be set to 0.05*0.05*0.05 cubic meters, or it can be set to other sizes according to actual needs.

[0079] In the embodiment of the present application, through three-dimensional reconstruction modeling, all frame point cloud data can be converted to the same coordinate system, thereby obtaining a three-dimensional reconstructed point cloud of coordinate system 1 for subsequent processing operations. In addition, this can also reduce the number of data points, making subsequent data processing and analysis more efficient, and can also reduce the resources required to store data.

[0080] In order to better understand the aforementioned implementation, an example is provided below for illustration.

[0081] Assume that the number of frames of the multi-frame point cloud data is N (N is an integer greater than or equal to 2), and assume that the laser radar coordinate system at the time of collecting the first frame point cloud P1 is the three-dimensional reconstruction coordinate system C0. When collecting the first frame point cloud P1, the movable laser radar is placed on the ground, and the laser radar coordinate system is parallel to the ground, so the xy plane of the three-dimensional reconstruction coordinate system C0 is parallel to the ground, and the z axis is perpendicular to the ground.

[0082] The CT-ICP in the 3D reconstruction algorithm matches the point clouds between frames to perform 3D reconstruction modeling. The specific process can be described as follows:

[0083] The first frame point cloud P1 is matched with the second frame point cloud P2 by CT-ICP, and the position transformation relationship of P2 relative to P1 is obtained as follows: in Represents the rotation transformation matrix of P2 relative to P1, a matrix of size 3x3, Represents the translation transformation vector of P2 relative to P1, which is a 3x1 matrix. The position transformation relationship of the third frame point cloud P3 relative to the second frame point cloud P2 is calculated as follows: represents the rotation transformation matrix of P3 relative to P2, represents the translation transformation vector of P3 relative to P2. By analogy, the position transformation relationship between frames is calculated in sequence, and we get:

[0084] After that, change P2 to P NThe position transformation relationship obtained by the previous calculation is converted to the three-dimensional reconstruction coordinate system C0, where the conversion formula is as follows:

[0085]

[0086] In formula (2), P ′ k Represents a frame of point cloud data after conversion, P k Indicates P2 to P N A frame of point cloud data in and Indicates P k To P k-1 transformation relationship.

[0087] It should be noted that the above formula (2) represents the conversion of the point cloud data of the next frame to the laser radar coordinate system of the previous frame point cloud. If the previous frame point cloud is not the first frame point cloud, it is necessary to further convert the previous frame point cloud and the point cloud converted to the previous frame point cloud coordinate system based on the position transformation relationship between adjacent frames, until all point clouds are converted to the laser radar coordinates of the first frame point cloud, that is, converted to the three-dimensional reconstructed coordinate system.

[0088] In the P2 to P N After being converted to the three-dimensional reconstructed coordinate system C0 through the position transformation relationship, the point cloud data in the three-dimensional reconstructed coordinate system C0 can be subjected to voxel filtering processing to obtain a three-dimensional reconstructed point cloud P, that is, a three-dimensional reconstructed point cloud of the calibration scene.

[0089] In some optional embodiments, step 102: determining the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud may include steps B1 to B3 as follows:

[0090] Step B1: Based on the vehicle point cloud, construct the target object in the 3D reconstructed point cloud.

[0091] The positional relationship between the target object and the vehicle is known, the relationship between the target object and the body coordinate system of the vehicle is known, and the correlation between the preset direction of the target object and the vehicle is known.

[0092] Optionally, the target object may include: a three-dimensional shape constructed based on the vehicle or a body coordinate system of the vehicle. The three-dimensional shape constructed based on the vehicle may specifically be a rectangular enclosing box constructed based on the vehicle outline, and the rectangular enclosing box encloses the vehicle.

[0093] In the case where the target object includes a rectangular bounding box surrounding the vehicle point cloud, the positional relationship between the target object and the vehicle can be: the positional relationship between the rectangular bounding box and the vehicle outline; the relationship between the target object and the vehicle's body coordinate system can be: the position and posture of the rectangular bounding box in the vehicle body coordinate system (which can be obtained by measurement); the relative relationship between the preset direction of the target object and the vehicle can be: the direction of the rectangular bounding box is the same as the direction of the front of the vehicle (not limited to this, it can also be the same as the direction of the rear of the vehicle, etc., which can be set according to actual needs).

[0094] When the target object includes a body coordinate system, the positional relationship between the target object and the vehicle can be: the positional relationship between the origin and three coordinate axes of the body coordinate system and the vehicle. For example, the origin of the body coordinate system can be set to the center of the rear axle of the vehicle; the positive direction of the x-axis is the direction of the front of the vehicle and is perpendicular to the rear axle of the vehicle; the positive direction of the y-axis points to the left side of the vehicle and is parallel to the rear axle of the vehicle; the z-axis is vertically perpendicular to the rear axle of the vehicle and points to the roof direction; the relationship between the target object and the body coordinate system of the vehicle can be: the target object is the body coordinate system of the vehicle; the relative relationship between the preset direction of the target object and the vehicle can be: the relationship between the directions of the three coordinate axes of the body coordinate system and the vehicle.

[0095] Step B2: Determine a third posture of the target object in the three-dimensional reconstructed coordinate system according to the coordinate information of the target object in the three-dimensional reconstructed coordinate system and the relative relationship between the preset direction of the target object and the vehicle.

[0096] Among them, according to the coordinate information of the target object in the three-dimensional reconstructed coordinate system, the position of the target object in the three-dimensional reconstructed coordinate system can be determined; according to the relative relationship between the preset direction of the target object and the vehicle, the posture of the target object in the three-dimensional reconstructed coordinate system can be determined. The position and posture of the target object in the three-dimensional reconstructed coordinate system are the pose of the target object in the three-dimensional reconstructed coordinate system, which is the aforementioned third pose.

[0097] Step B3: Obtain the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the third pose of the target object in the three-dimensional reconstructed coordinate system and the relationship between the target object and the body coordinate system of the vehicle.

[0098] When the target object includes a rectangular bounding box, the position and posture of the vehicle in the three-dimensional reconstructed coordinate system, that is, the aforementioned first position, can be obtained based on the position and posture of the rectangular bounding box in the three-dimensional reconstructed coordinate system and the position and posture of the rectangular bounding box in the vehicle body coordinate system.

[0099] When the target object is the body coordinate system of a vehicle, the position and posture of the body coordinate system in the three-dimensional reconstructed coordinate system is the position and posture of the vehicle in the three-dimensional reconstructed coordinate system.

[0100] Optionally, in the case where the target object is a vehicle body coordinate system, step B1: constructing the target object in the three-dimensional reconstructed point cloud according to the vehicle point cloud may include steps B101 to B103, as described below:

[0101] Step B101: Identify the wheel point cloud in the vehicle point cloud.

[0102] The wheel point clouds described here include two front wheel point clouds and two rear wheel point clouds.

[0103] Step B102: determining the front axle center of the vehicle point cloud based on the two front wheel point clouds, and determining the rear axle center of the vehicle point cloud based on the two rear wheel point clouds.

[0104] The outer side circle equation of each wheel point cloud can be extracted, and then a first line segment with the center of the outer side circle equation of the two front wheel point clouds as the endpoint is determined, and the center point position of the first line segment is used as the front axle center; and a second line segment with the center of the outer side circle equation of the two rear wheel point clouds as the endpoint is determined, and the center point position of the second line segment is used as the rear axle center.

[0105] Step B103: Take the rear axle center as the origin, the direction from the rear axle center to the front axle center as the positive direction of the first coordinate axis, the direction parallel to the first line segment and pointing to the center of the left rear wheel as the positive direction of the second coordinate axis, and the direction perpendicular to the first line segment and pointing to the roof as the positive direction of the third coordinate axis, and construct a vehicle body coordinate system in the three-dimensional reconstructed point cloud.

[0106] The first coordinate axis mentioned here may be an x-axis, the second coordinate axis may be a y-axis, and the third coordinate axis may be a z-axis.

[0107] The wheels mentioned here refer to wheels that are in contact with the ground for driving the vehicle, rather than spare wheels.

[0108] Accordingly, step B2: determining the third posture of the target object in the three-dimensional reconstructed coordinate system according to the coordinate information of the target object in the three-dimensional reconstructed coordinate system and the relative relationship between the preset direction of the target object and the vehicle may include steps B201 and B202, as described below:

[0109] Step B201: determining the coordinates of the origin of the vehicle body coordinate system in the three-dimensional reconstructed coordinate system as the position of the vehicle body coordinate system in the three-dimensional reconstructed coordinate system.

[0110] Step B202: Obtaining the posture of the vehicle body coordinate system in the three-dimensional reconstructed coordinate system according to the angular relationship between the three coordinate axes of the vehicle body coordinate system and the three-dimensional reconstructed coordinate system.

[0111] Accordingly, step B3: obtaining the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the third pose of the target object in the three-dimensional reconstructed coordinate system and the relationship between the target object and the body coordinate system of the vehicle may include:

[0112] The position and posture of the vehicle body coordinate system in the three-dimensional reconstructed coordinate system are determined as the first posture of the vehicle in the three-dimensional reconstructed coordinate system.

[0113] Optionally, when the target object is a rectangular bounding box, the vehicle outline may be determined based on the vehicle point cloud, and then a rectangular bounding box surrounding the vehicle point cloud may be constructed based on the vehicle outline, so that the vehicle point cloud is completely contained in the rectangular bounding box. The rectangular bounding box is a parallelepiped, and each face is a rectangle.

[0114] Accordingly, step B2: determining the third posture of the target object in the three-dimensional reconstructed coordinate system according to the coordinate information of the target object in the three-dimensional reconstructed coordinate system and the relative relationship between the preset direction of the target object and the vehicle may include:

[0115] Step B211: Determine the position of the target feature point of the rectangular bounding box in the three-dimensional reconstructed coordinate system according to the coordinate information of the rectangular bounding box in the three-dimensional reconstructed coordinate system.

[0116] The position of the target feature point is used to represent the position of the rectangular bounding box in the three-dimensional reconstruction coordinate system.

[0117] The position of the target feature point described here is used to represent the position of the rectangular bounding box in the three-dimensional reconstructed coordinate system. The target feature point can be a point on the surface or inside the rectangular bounding box, for example, it can be the geometric center point of the rectangular bounding box or a corner point of the rectangular bounding box.

[0118] Among them, the position of the target feature point in the three-dimensional reconstruction coordinate system can be expressed as P bbox ={x, y, z}.

[0119] The height, width, and length of the rectangular bounding box can be expressed as h, w, and l, respectively. The sizes of h, w, and l of the rectangular bounding box corresponding to the same vehicle model are the same. Therefore, different vehicles of the same model can use the same parameters (i.e., h, w, 1) to construct the rectangular bounding box when calibrating the camera pose.

[0120] The parameters of the rectangular bounding box can be expressed as {pose, h, w, l}, where pose represents the position of the rectangular bounding box in the 3D reconstruction coordinate system, pose = {P bbox , D bbox}, D bbox Represents the pose of the rectangular bounding box in the 3D reconstructed coordinate system.

[0121] Step B212: Determine the posture of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system according to the relative relationship between the preset direction of the rectangular enclosing frame and the vehicle.

[0122] In the embodiment of the present application, the direction of the rectangular bounding box can be preset. For example, the direction of the rectangular bounding box can be set to the direction of the vehicle head. The direction of the rectangular bounding box is also the direction of the target feature point. Based on the preset direction, the posture D of the rectangular bounding box in the three-dimensional reconstructed coordinates can be determined according to the vehicle point cloud. bbox ={roll, pitch, yaw}, roll, pitch, yaw are Euler angles, and the three degrees of freedom describe the direction of the rectangular bounding box relative to the direction of the 3D reconstructed coordinate system. So far, the pose of the rectangular bounding box in the 3D reconstructed coordinate system can be obtained. bbox , D bbox}.

[0123] Among them, the position and posture of the rectangular bounding box in the three-dimensional reconstruction coordinate system are the position and posture of the rectangular bounding box in the three-dimensional reconstruction coordinate system.

[0124] Among them, the position and posture of the rectangular bounding box in the vehicle body coordinate system can be measured in advance. Since the appearance dimensions of vehicles of the same model are fixed and consistent, the position and posture of the rectangular bounding box in the vehicle body coordinate system is also fixed. The position and posture of the rectangular bounding box in the vehicle body coordinate system can be expressed as: is a rotation matrix of size 3*3, is a translation vector of size 3*1, It can be obtained through actual measurement of the vehicle model or through the vehicle design drawings, and can be used directly in the calibration process without the need for measurement during the calibration process.

[0125] Accordingly, step B3: obtaining the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the third pose of the target object in the three-dimensional reconstructed coordinate system and the relationship between the target object and the body coordinate system of the vehicle may include:

[0126] According to the position and posture of the rectangular bounding box in the three-dimensional reconstructed coordinate system, and the position and posture relationship between the rectangular bounding box and the body coordinate system of the vehicle, the first position and posture of the vehicle in the three-dimensional reconstructed coordinate system are obtained.

[0127] Finally, through the parameters pose of the rectangular bounding box and the known Then the position and posture of the vehicle in the 3D reconstructed coordinate system (i.e. the first position) can be calculated as:

[0128]

[0129] Among them, R(D bbox ) is the Euler angle D bbox The calculated rotation matrix is ​​3*3 in size.

[0130] In some optional embodiments, step 103: determining a second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to the position of the first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and the position of the second feature point on the target surface in the target image captured by the camera to be calibrated, may include step C1 and step C2:

[0131] Step C1: Determine the second position of the second feature point in the three-dimensional reconstructed coordinate system according to the positional relationship between the first feature point and the second feature point in the target surface and the first position of the first feature point in the three-dimensional reconstructed coordinate system.

[0132] Among them, the positional relationship between the first feature point and the second feature point in the target surface is known. After obtaining the first position of the first feature point in the three-dimensional reconstructed coordinate system, the position of the second feature point in the three-dimensional reconstructed coordinate system can be obtained according to the positional relationship between the two types of feature points. Optionally, the positional relationship between the first feature point and the second feature point in the target surface can be the distance between the first feature point and the second feature point, and the distance can be measured in advance. When the position of the first feature point in the three-dimensional reconstructed coordinate is known, the position of the second feature point in the three-dimensional reconstructed coordinate system can be calculated according to the distance.

[0133] Optionally, the first feature point may be a corner point of the entire target surface, and the second feature point may be the entire corner point of the calibration pattern on the target surface. If the corner point of the entire target surface coincides with the corner point of the entire calibration pattern, the first feature point is the same as the second feature point. It is understandable that the first feature point may also be the centroid of the target surface, and the second feature point may also be a corner point inside the calibration pattern. The combination may be used according to actual needs.

[0134] Step C2: determining a second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to the second position of the second feature point in the three-dimensional reconstructed coordinate system and the third position in the target image.

[0135] The set of the second feature points in a frame of target image can be expressed as c k ={kp1, kp2, ..., kp M}, M represents the number of second feature points in a frame of target image, which is generally greater than or equal to 3, kp = {u p , v p} represents the coordinates of the second feature point in the pixel coordinate system. When there are multiple cameras to be calibrated, the set of second feature points can be recorded as: C = {c k, k = 1, 2, ..., N I}.

[0136] Optionally, c k The position of the second feature point in the 3D reconstruction coordinate system C0 is represented by c′ k ={kp′1,kp′2,...,kp′ M}, kp′={x p ,y p , z p} is the three-dimensional spatial position of the second feature point in the three-dimensional reconstruction coordinate system C0.

[0137] After obtaining the second position of the second feature point in the three-dimensional reconstructed coordinate system, the third position of the second feature point in the target image can be combined with the camera imaging principle model, that is, the mathematical model of mapping points in the three-dimensional world coordinate system to points in the two-dimensional image coordinate system, to obtain the pose (i.e., the second pose) of the camera coordinate system relative to the three-dimensional reconstructed coordinate system. The third position here can be a position in the pixel coordinate system.

[0138] Optionally, the second pose of the camera coordinate system of the camera to be calibrated relative to the three-dimensional reconstructed coordinate system can be determined by the following formula (1):

[0139]

[0140] Formula (4) is the camera model formula. is the intrinsic parameter matrix of the camera, where the parameter f x 、f y 、c x 、c y Known, The rotation transformation matrix from the 3D reconstruction coordinate system C0 to the camera coordinate system is 3*3 in size. is the translation transformation vector from the 3D reconstruction coordinate system C0 to the camera coordinate system, with a size of 3*1, P w is the coordinate position of the second feature point in the three-dimensional reconstruction coordinate system C0; The u and v in are the coordinate positions of the second feature point in the pixel coordinate system.

[0141] According to formula (4) and the position coordinates of multiple second feature points in pixel coordinates and the position coordinates in the three-dimensional reconstruction coordinate system, multiple sets of constraint equations can be established to solve and obtain and That is, the position of the camera coordinate system relative to the 3D reconstruction coordinate system. Among them, the number of the second feature points can be solved and For example, at least three second feature points are required to solve and Then the number of the second feature points is at least three. The more the number of the second feature points is, the more conducive it is to solving a more accurate second pose.

[0142] When there are multiple cameras to be calibrated, the pose set of the cameras to be calibrated in the 3D reconstructed coordinate system can be expressed as T cam ={T k , k = 1, 2, ..., N I}, where N I Indicates the number of cameras, Represents the pose of the kth camera to be calibrated in the 3D reconstructed coordinate system.

[0143] Before step 103, the method may further include step D1 and step D2, as described below:

[0144] Step D1: Identify the target point cloud in the 3D reconstructed point cloud.

[0145] Step D2: Determine the first position of the first feature point on the target surface in the three-dimensional reconstructed coordinate system according to the target point cloud.

[0146] The first feature point represents the target, and the position of the first feature point in the three-dimensional reconstructed coordinate system can be expressed as the position of the target in the three-dimensional reconstructed coordinate system.

[0147] Optionally, some feature points corresponding to the target surface are generally used as the first feature points. Therefore, when identifying the target point cloud, only the target surface point cloud of the target can be identified, and the target surface features can be specifically identified. For example, when the target surface is a square, a point cloud with a square shape and a size that matches the size of the target surface can be determined as the target surface point cloud.

[0148] It is understandable that when identifying the target point cloud, the target may also be identified in combination with the overall features of the target. For example, in some embodiments, the target may include: a target surface and a target rod vertically connected to the target surface, and the target rod is placed vertically on the ground. In this embodiment, the target point cloud can be identified in the following manner.

[0149] Accordingly, step D1: identifying the target point cloud in the three-dimensional reconstructed point cloud may include steps D11 and D12, as described below:

[0150] Step D11: Identify the plane features and straight line segment features in the 3D reconstructed point cloud.

[0151] Step D12: The plane features and straight line segment features that match the target features are determined as the target point cloud.

[0152] The target features described herein may include: a plane feature and a straight line feature, the plane feature and the straight line feature are perpendicular to the ground, and a vertical distance from the center of mass of the plane feature to the straight line where the straight line feature is located is less than a preset value.

[0153] Usually, a vertical target is mainly composed of a target surface and a pole perpendicular to the ground. Therefore, a plane extraction algorithm and a straight line segment extraction algorithm can be used to extract the target point cloud in the three-dimensional reconstructed point cloud P.

[0154] The plane feature set extracted by the plane extraction algorithm can be expressed as: S plan ={s k , k = 1, 2, ..., N plan}, a total of N plan Plane features, where x, y, z are the three-dimensional coordinates of the center of mass of the plane, is the plane feature s k is the normal vector of the plane, and s is the area of ​​the plane.

[0155] The set of straight line segments extracted by the straight line segment extraction algorithm is S line = {l k , k = 1, 2, ..., N line}, a total of N line line segments, where and are the 3D coordinates of the two endpoints of the line segment in the 3D reconstructed coordinate system.

[0156] Next, determine S plan and S line The plane and straight line segments belonging to the target in the target are obtained to obtain the target feature set S target , the judgment method is as follows:

[0157] Filter line The difference between the length of the midline segment and the actual target column length is less than the first threshold, and the line segment is perpendicular to the ground. The criterion for being perpendicular to the ground is Line segment vector The angle between the z-axis and the coordinate system C0 is less than the first degree threshold (which can be set to 5°). The first threshold and the first degree threshold can be set according to actual needs. The smaller the values ​​of the two, the more consistent the screening results are with the target characteristics. The screened set is recorded as A′ line .

[0158] Filter plan The specific method of screening the plane perpendicular to the ground is: the normal vector of each plane feature The angle between the z-axis of the three-dimensional reconstruction coordinate system C0 is within the preset angle range, and the difference between the area s of the plane feature and the actual area of ​​the target surface is less than the second threshold. The preset angle range can be set according to actual needs, for example, greater than or equal to 85 degrees and less than or equal to 95 degrees. The value of the second threshold can also be set according to actual needs. The smaller the value, the more consistent the screening result with the target feature. The screened set is recorded as S' plan .

[0159] Determine the target features and compare S′ one by one line and S′ plan The straight line segments and plane features in S′ plan In The centroid position x, y, z and S′ line middle Plane features and straight line segments whose vertical distance to the straight line is less than a preset value (such as 10 cm) are classified as a target corresponding to the target point cloud and stored in the target feature set S target middle.

[0160] So far, the target feature set S is obtained target , where each The center of mass position x, y, z and the plane feature normal vector in That is, the position and posture of each target in the three-dimensional reconstructed coordinate system C0.

[0161] It can be understood that the above-mentioned target point cloud recognition method is applicable to a target including a target surface and a target rod vertically connected to the target surface. For other types of targets, corresponding recognition methods can be designed according to the target characteristics.

[0162] Optionally, the ID of the calibration pattern in each target is known, and the position of the target corresponding to each ID in the three-dimensional reconstructed coordinate system C0 can be determined through the actual target placement position of the target relative to the vehicle body coordinate system.

[0163] The above is a description of the camera pose calibration method provided in the embodiment of the present application.

[0164] To sum up, in the technical solution provided in the present application, the parking position of the vehicle in the calibration scene may not be fixed, and the placement position of the target in the calibration scene may not be fixed. There is no need to rely on complex calibration equipment and calibration operations, making the calibration of the camera pose more flexible and convenient.

[0165] Exemplary Devices

[0166] Correspondingly, an embodiment of the present application also provides a camera posture calibration device, which is applied to electronic devices such as vehicle-mounted computers, laptop computers, desktop computers, servers, etc.

[0167] like Figure 3 As shown, the device may include:

[0168] The point cloud reconstruction module 301 is used to obtain a three-dimensional reconstructed point cloud of the calibration scene based on multiple frames of point cloud data obtained by scanning the calibration scene with a movable laser radar.

[0169] The calibration scene includes a vehicle and a target, the vehicle is provided with a camera to be calibrated; and the target includes a target surface.

[0170] The first pose determination module 302 is used to determine the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud.

[0171] The three-dimensional reconstructed coordinate system is the coordinate system of the three-dimensional reconstructed point cloud.

[0172] The second posture determination module 303 is used to determine the second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to the position of the first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and the position of the second feature point on the target surface in the target image captured by the camera to be calibrated.

[0173] The third posture determination module 304 is used to determine the target posture of the camera to be calibrated relative to the vehicle according to the first posture of the vehicle in the 3D reconstructed coordinate system and the second posture of the camera to be calibrated in the 3D reconstructed coordinate system.

[0174] Optionally, the point cloud reconstruction module 301 may include:

[0175] A setting unit is used to set the laser radar coordinate system corresponding to the target frame point cloud to the three-dimensional reconstruction coordinate system.

[0176] The target frame point cloud is one of the multiple frame point clouds.

[0177] The first determining unit is used to determine the position transformation relationship between two adjacent frames of point clouds.

[0178] The conversion unit is used to convert other frame point clouds in the multi-frame point cloud data except the target frame point cloud into the three-dimensional reconstructed coordinate system according to the position transformation relationship between two adjacent frame point clouds.

[0179] The filtering unit is used to perform voxel filtering on the point cloud in the three-dimensional reconstruction coordinate system to obtain the three-dimensional reconstructed point cloud.

[0180] Optionally, the first posture determination module 302 may include:

[0181] The object construction unit is used to construct a target object in the three-dimensional reconstructed point cloud according to the vehicle point cloud.

[0182] Among them, the positional relationship between the target object and the vehicle is known, the relationship between the target object and the body coordinate system of the vehicle is known, and the relative relationship between the preset direction of the target object and the vehicle is known. The target object includes: a rectangular bounding box surrounding the vehicle point cloud or the body coordinate system of the vehicle.

[0183] The second determining unit is used to determine a third posture of the target object in the three-dimensional reconstructed coordinate system according to the coordinate information of the target object in the three-dimensional reconstructed coordinate system and the relative relationship between the preset direction of the target object and the vehicle.

[0184] The third determining unit is used to obtain the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the third pose of the target object in the three-dimensional reconstructed coordinate system and the relationship between the target object and the body coordinate system of the vehicle.

[0185] Optionally, the third posture of the target object in the three-dimensional reconstructed coordinate system includes: the position and posture of the target object in the three-dimensional reconstructed coordinate system.

[0186] Accordingly, the second determining unit may be specifically configured to:

[0187] Determine the position of the target feature point of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system according to the coordinate information of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system; wherein the position of the target feature point is used to represent the position of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system;

[0188] The posture of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system is determined according to the relative relationship between the preset direction of the rectangular enclosing frame and the vehicle.

[0189] The third determining unit may be specifically configured to:

[0190] The first pose of the vehicle in the three-dimensional reconstructed coordinate system is obtained according to the position and pose of the rectangular bounding box in the three-dimensional reconstructed coordinate system and the pose relationship between the rectangular bounding box and the body coordinate system of the vehicle.

[0191] Optionally, the target object is a body coordinate system of the vehicle, and the origin of the body coordinate system is the center of the rear axle.

[0192] Optionally, the second posture determination module 304 may include

[0193] The fourth determination unit is used to determine the second position of the second feature point in the three-dimensional reconstructed coordinate system according to the positional relationship between the first feature point and the second feature point in the target surface and the first position of the first feature point in the three-dimensional reconstructed coordinate system.

[0194] A fifth determining unit is used to determine a second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to a second position of the second feature point in the three-dimensional reconstructed coordinate system and a third position in the target image.

[0195] Optionally, the device may further include:

[0196] The point cloud recognition unit is used to recognize the target point cloud in the three-dimensional reconstructed point cloud.

[0197] The fourth determination unit is used to determine the position of the first feature point on the target surface in the three-dimensional reconstructed coordinate system according to the target point cloud.

[0198] Optionally, in the case where the target includes a target rod and a target surface vertically connected to the target rod, the point cloud recognition unit may be specifically used for:

[0199] Identifying plane features and straight line segment features in the three-dimensional reconstructed point cloud;

[0200] The plane features and straight line segment features that match the target features are determined as the target point cloud; the target features include: a plane feature and a straight line segment feature, the plane feature and the straight line segment feature are perpendicular to the ground, and the vertical distance from the center of mass of the plane feature to the straight line where the straight line segment feature is located is less than a preset value.

[0201] The camera pose calibration device provided in the embodiment of the present application belongs to the same application concept as the camera pose calibration method provided in the above embodiment of the present application, and can execute the camera pose calibration method provided in any of the above embodiments of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the camera pose calibration method provided in the above embodiment of the present application, which will not be repeated here.

[0202] Exemplary Electronic Devices

[0203] The present application also provides an electronic device, such as Figure 4 As shown, the electronic device includes: a memory 400 and a processor 410 .

[0204] The memory 400 is connected to the processor 410 and is used to store programs.

[0205] The processor 410 is used to implement the camera pose calibration method in the above embodiment by running the program stored in the memory 400.

[0206] Specifically, the electronic device may further include: a communication interface 420 , an input device 430 , an output device 440 and a bus 450 .

[0207] The processor 410, the memory 400, the communication interface 420, the input device 430 and the output device 440 are connected to each other via a bus.

[0208] Bus 450 may include a pathway for transferring information between the various components of the computer system.

[0209] Processor 410 may be a general purpose processor, such as a general purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0210] The processor 410 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0211] The memory 400 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 400 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0212] The input device 430 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0213] Output device 440 may include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0214] The communication interface 420 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0215] The processor 410 executes the program stored in the memory 400 and calls other devices, which can be used to implement the various steps of the camera pose calibration method provided in the above embodiments of the present application.

[0216] Exemplary computer program products and storage media

[0217] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the camera pose calibration method described in the embodiment of the present application.

[0218] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0219] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0220] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the camera pose calibration method described in the embodiment of the present application.

[0221] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0222] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0223] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0224] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0225] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0226] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0228] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0229] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0230] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A camera pose calibration method, characterized in that: The method comprises: According to the multi-frame point cloud data obtained by scanning the calibration scene by the mobile laser radar, a three-dimensional reconstructed point cloud of the calibration scene is obtained; wherein the calibration scene includes a vehicle and a target, and the vehicle is provided with a camera to be calibrated; and the target includes a target surface; Determine the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud; wherein the three-dimensional reconstructed coordinate system is the coordinate system of the three-dimensional reconstructed point cloud; Determine a second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to a position of a first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and a position of a second feature point on the target surface in the target image acquired by the camera to be calibrated; According to a first pose of the vehicle in the three-dimensional reconstructed coordinate system and a second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system, a target pose of the camera to be calibrated relative to the vehicle is determined.

2. The camera pose calibration method according to claim 1, characterized in that: The step of obtaining a three-dimensional reconstructed point cloud of the calibration scene based on the multiple frames of point cloud data obtained by scanning the calibration scene with the movable laser radar comprises: Setting the laser radar coordinate system corresponding to the target frame point cloud as the three-dimensional reconstruction coordinate system; wherein the target frame point cloud is one of the multiple frame point clouds; Determine the position transformation relationship between two adjacent frames of point clouds; According to the position transformation relationship between two adjacent frame point clouds, other frame point clouds except the target frame point cloud in the multi-frame point cloud data are converted to the three-dimensional reconstructed coordinate system; The point cloud in the three-dimensional reconstruction coordinate system is subjected to voxel filtering processing to obtain the three-dimensional reconstructed point cloud.

3. The camera pose calibration method according to claim 1, characterized in that: Determining the first pose of the vehicle in the three-dimensional reconstructed coordinate system according to the vehicle point cloud in the three-dimensional reconstructed point cloud includes: According to the vehicle point cloud, a target object is constructed in the three-dimensional reconstructed point cloud; wherein the positional relationship between the target object and the vehicle is known, the relationship between the target object and the body coordinate system of the vehicle is known, and the relative relationship between the preset direction of the target object and the vehicle is known, and the target object includes: a rectangular bounding box surrounding the vehicle point cloud or the body coordinate system of the vehicle; Determining a third posture of the target object in the three-dimensional reconstructed coordinate system according to the coordinate information of the target object in the three-dimensional reconstructed coordinate system and the relative relationship between the preset direction of the target object and the vehicle; The first pose of the vehicle in the three-dimensional reconstructed coordinate system is obtained according to the third pose of the target object in the three-dimensional reconstructed coordinate system and the relationship between the target object and the body coordinate system of the vehicle.

4. The camera pose calibration method according to claim 3, characterized in that: The third posture of the target object in the three-dimensional reconstructed coordinate system includes: the position and posture of the target object in the three-dimensional reconstructed coordinate system; In a case where the target object is a rectangular bounding box surrounding the vehicle point cloud, determining a third pose of the target object in the three-dimensional reconstructed coordinate system according to coordinate information of the target object in the three-dimensional reconstructed coordinate system and a relative relationship between a preset direction of the target object and the vehicle includes: Determine the position of the target feature point of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system according to the coordinate information of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system; wherein the position of the target feature point is used to represent the position of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system; Determining the posture of the rectangular enclosing frame in the three-dimensional reconstructed coordinate system according to a relative relationship between a preset direction of the rectangular enclosing frame and the vehicle; The obtaining, according to the third posture of the target object in the three-dimensional reconstructed coordinate system and the relationship between the target object and the body coordinate system of the vehicle, the first posture of the vehicle in the three-dimensional reconstructed coordinate system comprises: The first pose of the vehicle in the three-dimensional reconstructed coordinate system is obtained according to the position and pose of the rectangular bounding box in the three-dimensional reconstructed coordinate system and the pose relationship between the rectangular bounding box and the body coordinate system of the vehicle.

5. The camera pose calibration method according to claim 3, characterized in that: The target object is a body coordinate system of the vehicle, and the origin of the body coordinate system is the center of the rear axle.

6. The camera pose calibration method according to claim 1, characterized in that: The method of determining a second posture of the camera to be calibrated in the three-dimensional reconstructed coordinate system according to a position of a first feature point on the target surface in the three-dimensional reconstructed point cloud in the three-dimensional reconstructed coordinate system and a position of a second feature point on the target surface in the target image acquired by the camera to be calibrated comprises: Determine a second position of the second feature point in the three-dimensional reconstructed coordinate system according to a positional relationship between the first feature point and the second feature point in the target surface and a first position of the first feature point in the three-dimensional reconstructed coordinate system; A second position of the camera to be calibrated in the three-dimensional reconstructed coordinate system is determined according to a second position of the second feature point in the three-dimensional reconstructed coordinate system and a third position in the target image.

7. The camera pose calibration method according to claim 1, characterized in that: Before determining the second pose of the camera to be calibrated in the three-dimensional reconstructed coordinate system, the method further includes: Identifying a target point cloud in the three-dimensional reconstructed point cloud; According to the target point cloud, the position of the first feature point on the target surface in the three-dimensional reconstructed coordinate system is determined.

8. The camera pose calibration method according to claim 7, characterized in that: In the case where the target includes a target rod and a target surface connected to the target rod, the identifying the target point cloud in the three-dimensional reconstructed point cloud includes: Identifying plane features and straight line segment features in the three-dimensional reconstructed point cloud; The plane features and straight line segment features that match the target features are determined as the target point cloud; wherein the target features include: a plane feature and a straight line segment feature, the plane feature and the straight line segment feature are perpendicular to the ground, and the vertical distance from the center of mass of the plane feature to the straight line where the straight line segment feature is located is less than a preset value.

9. A vehicle, characterized in that: The vehicle is used to implement the camera pose calibration method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the camera pose calibration method according to any one of claims 1 to 8 by running the program in the memory.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the camera pose calibration method according to any one of claims 1 to 8 is implemented.