Calibration method, transport equipment and computer program product

Abstract

The embodiment of the invention relates to a calibration method, transportation equipment and a computer program product, and the method comprises the steps: obtaining target point cloud data collected by a laser radar, and obtaining a target image obtained by carrying out the image collection of an external parameter calibration board through a camera device; detecting a circular image contained in the target image to obtain at least one circular hole area contained in the target image; performing circle center positioning on each circular hole area to determine a first circle center coordinate corresponding to each circular hole in a camera device coordinate system; according to a fitting circle algorithm, fitting the target point cloud data to obtain a second circle center coordinate corresponding to each circular hole in the radar coordinate system; and determining external parameters between the laser radar and the camera device according to the first circle center coordinates and the second circle center coordinates corresponding to the circular holes. According to the calibration method, the transportation equipment and the computer program product, the external parameters between the laser radar and the camera device on the transportation equipment can be calibrated more conveniently.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and particularly relates to a calibration method, a transportation device, and a computer program product. Background Art

[0002] The combined use of radar and camera has shown broad application prospects in various fields. This combination utilizes the long-range detection ability of the radar and the high-resolution imaging characteristics of the camera to achieve precise tracking, positioning, and identification of targets. Since the radar and the camera are two different sensors, they are usually set at different positions on the transportation device, resulting in different coordinate systems corresponding to the radar and the camera respectively. Therefore, it is necessary to calibrate the radar and the camera so that the data collected by the two can be converted. Therefore, how to more accurately and conveniently calibrate the radar and the camera on the transportation device has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] Embodiments of the present application disclose a calibration method, a transportation device, and a computer program product, which can more accurately and conveniently calibrate the external parameters between the lidar and the imaging device on the transportation device.

[0004] In a first aspect, embodiments of the present application disclose a calibration method, which is applied to a transportation device, and the transportation device is provided with a lidar and an imaging device; the method includes:

[0005] Obtaining target point cloud data collected by the lidar, and obtaining a target image that matches the target point cloud data by using the imaging device to perform image acquisition on an external parameter calibration board; the target point cloud data is point cloud data corresponding to the external parameter calibration board; the external parameter calibration board includes at least one round hole;

[0006] Detecting circular images included in the target image to obtain at least one round hole area included in the target image;

[0007] Performing center positioning on each of the round hole areas to determine first center coordinates corresponding to each of the round holes in the imaging device coordinate system; the imaging device coordinate system is a two-dimensional coordinate system with the optical center position of the imaging device as the origin;

[0008] According to the fitting circle algorithm, fitting the target point cloud data to obtain second center coordinates corresponding to each of the round holes in the radar coordinate system; the radar coordinate system is a three-dimensional coordinate system with the lidar as the origin;

[0009] Determining the external parameters between the lidar and the imaging device according to the first center coordinates and the second center coordinates corresponding to each of the round holes.

[0010] As an alternative implementation, obtaining the target point cloud data collected by the lidar includes: obtaining the historical point cloud data collected by the lidar at at least one historical acquisition moment respectively; fusing the historical point cloud data collected at the at least one historical acquisition moment respectively with the current frame point cloud data collected by the lidar at the current moment to obtain the target point cloud data.

[0011] As an alternative implementation, the transportation device is further provided with an odometer and a gyroscope; the fusing the historical point cloud data collected at the at least one historical acquisition moment respectively with the current frame point cloud data collected by the lidar at the current moment to obtain the target point cloud data includes: determining the pose data corresponding to the transportation device at the at least one historical acquisition moment and the current moment respectively according to the position data collected by the odometer and the attitude data collected by the gyroscope; mapping the historical point cloud data collected at each historical acquisition moment to the current moment according to the pose data corresponding to each historical acquisition moment and the pose data corresponding to the current moment, and fusing the mapped historical point cloud data with the current frame point cloud data collected at the current moment to obtain the target point cloud data.

[0012] As an alternative implementation, the at least one historical acquisition moment includes M historical acquisition moments, where M is an integer greater than 1; the step of mapping the historical point cloud data collected at each of the historical acquisition moments to the current moment according to the pose data corresponding to each of the historical acquisition moments and the pose data corresponding to the current moment, and fusing the mapped historical point cloud data with the current frame point cloud data collected at the current moment to obtain target point cloud data includes: mapping the fused point cloud data corresponding to the Nth historical acquisition moment to the (N + 1)th historical acquisition moment according to the pose data corresponding to the Nth historical acquisition moment and the pose data corresponding to the (N + 1)th historical acquisition moment to obtain the mapped historical point cloud data corresponding to the Nth historical acquisition moment; N is an integer greater than 0 and less than M, wherein the fused point cloud data corresponding to the first historical acquisition moment is the historical point cloud data collected at the first historical acquisition moment; fusing the mapped historical point cloud data corresponding to the Nth historical acquisition moment with the historical point cloud data collected at the (N + 1)th historical acquisition moment to obtain the fused point cloud data corresponding to the (N + 1)th historical acquisition moment; mapping the fused point cloud data corresponding to the Mth historical acquisition moment to the current moment according to the pose data corresponding to the Mth historical acquisition moment and the pose data corresponding to the current moment to obtain the mapped historical point cloud data corresponding to the Mth historical acquisition moment; and fusing the mapped historical point cloud data corresponding to the Mth historical acquisition moment with the current frame point cloud data collected at the current moment to obtain target point cloud data.

[0013] As an alternative implementation, the step of mapping the fused point cloud data corresponding to the Nth historical acquisition moment to the (N + 1)th historical acquisition moment according to the pose data corresponding to the Nth historical acquisition moment and the pose data corresponding to the (N + 1)th historical acquisition moment to obtain the mapped historical point cloud data corresponding to the Nth historical acquisition moment includes: performing an inverse transformation on the pose data corresponding to the Nth historical acquisition moment to obtain the inverse pose data corresponding to the Nth historical acquisition moment; and performing a dot product of the inverse pose data corresponding to the Nth historical acquisition moment, the fused point cloud data corresponding to the Nth historical acquisition moment, and the pose data corresponding to the (N + 1)th historical acquisition moment to obtain the mapped historical point cloud data corresponding to the Nth historical acquisition moment.

[0014] As an alternative implementation, mapping the historical point cloud data collected at each of the historical acquisition times to the current time according to the pose data corresponding to each of the historical acquisition times and the pose data corresponding to the current time, and fusing the mapped historical point cloud data with the current frame point cloud data collected at the current time to obtain target point cloud data includes: performing an inverse transformation on the pose data corresponding to each of the historical acquisition times to obtain the inverse pose data corresponding to each of the historical acquisition times; performing a dot product on the inverse pose data corresponding to each of the historical acquisition times, the historical point cloud data collected at each of the historical acquisition times, and the pose data corresponding to the current time to obtain the mapped historical point cloud data corresponding to each of the historical acquisition times; and fusing the mapped historical point cloud data corresponding to each of the historical acquisition times with the current frame point cloud data collected at the current time to obtain target point cloud data.

[0015] As an alternative implementation, performing centroid localization on each of the circular hole regions to determine the first centroid coordinates corresponding to each of the circular holes in the camera device coordinate system includes: performing centroid localization on each of the circular hole regions to obtain the third centroid coordinates corresponding to each of the circular hole regions in the pixel coordinate system; the pixel coordinate system is used to represent the coordinate system of each pixel position in the image; and based on the internal parameters corresponding to the camera device, transferring the third centroid coordinates corresponding to each of the circular hole regions to the camera device coordinate system to obtain the first centroid coordinates corresponding to each of the circular holes.

[0016] As an alternative implementation, before transferring the third centroid coordinates corresponding to each of the circular hole regions to the camera device coordinate system based on the internal parameters corresponding to the camera device to obtain the first centroid coordinates corresponding to each of the circular holes, the method further includes: acquiring an image of the internal parameter calibration board by the camera device to obtain a first image; extracting features of the internal parameter calibration board included in the first image to obtain a plurality of feature points corresponding to the internal parameter calibration board; and calculating the internal parameters corresponding to the camera device according to the plurality of feature points.

[0017] As an alternative implementation, detecting the circular images included in the target image to obtain at least one circular hole region included in the target image includes: detecting the circular images included in the image region corresponding to the external parameter calibration board to obtain at least one circular hole region included in the external parameter calibration board.

[0018] As an alternative implementation, before detecting the circular images included in the target image to obtain at least one circular hole region included in the target image, the method further includes: identifying an external parameter calibration board included in the target image to determine the position information of the external parameter calibration board in the target image; and performing image segmentation on the external parameter calibration board included in the target image according to the position information to obtain the image region corresponding to the external parameter calibration board.

[0019] As an alternative implementation, the detecting the circular images included in the target image to obtain at least one circular hole region included in the target image includes: detecting the circular regions included in the target image according to an edge detection algorithm to obtain at least one circular hole region in the target image; and / or, the positioning the center of each of the circular hole regions to determine the first center coordinates corresponding to each of the circular holes in the camera device coordinate system includes: positioning the center of each of the circular hole regions based on a Hough circle detection algorithm to determine the first center coordinates corresponding to each of the circular holes in the camera device coordinate system.

[0020] As an alternative implementation, the fitting the target point cloud data according to a fitting circle algorithm to obtain the second center coordinates corresponding to each of the circular holes in the radar coordinate system includes: performing plane fitting on the target point cloud data to obtain a first fitting plane corresponding to the external parameter calibration board; determining a plurality of edge point clouds according to the curvatures of the point clouds included in the first fitting plane; and fitting each of the edge point clouds according to the fitting circle algorithm to determine the second center coordinates corresponding to each of the circular holes in the radar coordinate system.

[0021] As an alternative implementation, the performing plane fitting on the target point cloud data to obtain a first fitting plane corresponding to the external parameter calibration board includes: determining the road surface normal vector corresponding to the transportation device; screening each of the point clouds included in the target point cloud data according to the road surface normal vector to obtain a plurality of candidate point clouds corresponding to the external parameter calibration board; and performing plane fitting on the plurality of candidate point clouds to obtain a first fitting plane corresponding to the external parameter calibration board.

[0022] As an alternative implementation, screening each point cloud included in the target point cloud data according to the road surface normal vector to obtain multiple candidate point clouds corresponding to the external parameter calibration board includes: obtaining K neighboring point clouds of a first point cloud from the target point cloud data, where the first point cloud is any point cloud in the target point cloud data, and K is an integer greater than 1; the neighboring point clouds are point clouds in the target point cloud data whose distance from the first point cloud is less than a first distance threshold; using the least squares method to perform plane fitting on the first point cloud and the K neighboring point clouds to obtain a second fitting plane, and determining the plane normal vector corresponding to the second fitting plane; if the angle between the road surface normal vector and the plane normal vector is greater than a preset angle threshold, then taking the first point cloud as a candidate point cloud corresponding to the external parameter calibration board.

[0023] As an alternative implementation, performing plane fitting on the multiple candidate point clouds to obtain a first fitting plane corresponding to the external parameter calibration board includes: arbitrarily obtaining P target candidate point clouds from the multiple candidate point clouds, where P is a positive integer greater than 2 and less than S, and S is the number of candidate point clouds; performing plane fitting on the P target candidate point clouds to obtain a third fitting plane; calculating the first distance between each candidate point cloud and the third fitting plane; determining the number of point clouds of candidate point clouds whose first distance is less than a second distance threshold; repeating the step of arbitrarily obtaining P target candidate point clouds from the multiple candidate point clouds Q times, respectively obtaining Q third fitting planes, and in each of the third fitting planes, the number of point clouds of candidate point clouds whose first distance from each third fitting plane is less than the second distance threshold, where Q is a positive integer; taking the third fitting plane with the largest number of point clouds among the point cloud numbers corresponding to the Q third fitting planes as the first fitting plane corresponding to the external parameter calibration board.

[0024] As an alternative implementation, before determining multiple edge point clouds according to the curvatures of the point clouds included in the first fitting plane, the method further includes: rotating the first fitting plane so that the rotated first fitting plane is perpendicular to the z-axis of the radar coordinate system, and determining the rotation transformation matrix corresponding to the rotated first fitting plane; determining multiple edge point clouds according to the curvatures of the point clouds included in the first fitting plane includes: determining multiple edge point clouds according to the curvatures of the point clouds included in the rotated first fitting plane.

[0025] As an alternative implementation, according to the fitting circle algorithm, fitting each of the edge point clouds to determine the second center coordinates corresponding to each of the circular holes in the radar coordinate system, includes: according to the fitting circle algorithm, performing circle fitting on the multiple edge point clouds to obtain the target edge point clouds forming each of the circular holes, and determining the third center coordinates corresponding to each of the circular holes according to the target edge point clouds of each of the circular holes; according to the rotation transformation matrix, performing reverse rotation transformation on each of the third center coordinates to obtain the second center coordinates corresponding to each of the circular holes in the radar coordinate system.

[0026] As an alternative implementation, the centering of each of the circular hole regions to determine the first center coordinates corresponding to each of the circular holes includes: identifying the first center coordinates of the first circular hole region; the first circular hole region is any one of the circular hole regions in the external parameter calibration board; according to the physical position relationship between the first circular hole region and other circular hole regions in the external parameter calibration board, calculating to obtain the first center coordinates corresponding to each of the other circular hole regions.

[0027] In a second aspect, an embodiment of the present application discloses a transportation device, including a memory and a processor, where a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is caused to implement the method described in any one of the above embodiments.

[0028] In a third aspect, an embodiment of the present application discloses a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0029] In a fourth aspect, an embodiment of the present application discloses a computer program product, including a computer program, and when the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0030] The calibration method, transportation device, and computer program product disclosed in the embodiments of the present application acquire target point cloud data collected by a lidar and acquire a target image that matches the target point cloud data obtained by an imaging device capturing an image of an external parameter calibration board; detect circular images included in the target image to obtain at least one circular hole area included in the target image; perform center positioning on each circular hole area to determine the first center coordinates corresponding to each circular hole; perform fitting on the target point cloud data according to the fitting circle algorithm to obtain the second center coordinates corresponding to each circular hole; determine the external parameters between the lidar and the imaging device according to the first center coordinates and the second center coordinates corresponding to each circular hole. In the embodiments of the present application, the transportation device detects circular images included in the target image captured by the imaging device, performs center positioning, determines the first center coordinates corresponding to at least one circular hole on the external parameter calibration board, and performs fitting on the target point cloud data collected by the lidar according to the fitting circle algorithm to obtain the second center coordinates corresponding to at least one circular hole on the external parameter calibration board. According to the first center coordinates and the second center coordinates corresponding to at least one circular hole, the mapping relationship between the radar coordinate system and the imaging device coordinate system can be better reflected to obtain more accurate external parameters between the imaging device and the lidar; and through center positioning and the fitting circle algorithm, the center coordinates of at least one circular hole on the external parameter calibration board in different coordinate systems can be obtained, without setting special markers on the external parameter calibration board and performing special identification on the special markers to obtain the center coordinates (for example, identifying QR code markers through the autoware toolbox, etc.). Therefore, the manufacturing cost of the external parameter calibration board can be reduced, and there is no need to rely on manual point selection by the operator, which can simplify the external parameter calibration process between the imaging device and the lidar and can more accurately and conveniently calibrate the external parameters between the lidar and the imaging device on the transportation device. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0032] Figure 1 It is an application scenario diagram of a transportation device in an embodiment;

[0033] Figure 2 It is a flowchart of a calibration method in an embodiment;

[0034] Figure 3 It is a schematic diagram of an external parameter calibration board in an embodiment;

[0035] Figure 4Flow chart of the step of obtaining target point cloud data collected by a lidar in an embodiment;

[0036] Figure 5 Flow chart of the step of fusing historical point cloud data collected at at least one historical acquisition moment with the current frame point cloud data collected by the lidar at the current moment to obtain target point cloud data in an embodiment;

[0037] Figure 6 Schematic diagram of the process of obtaining target point cloud data based on historical point cloud data corresponding to M historical acquisition moments and current frame point cloud data corresponding to the current moment in an embodiment;

[0038] Figure 7 Schematic diagram of the process of obtaining target point cloud data based on historical point cloud data corresponding to M historical acquisition moments and current frame point cloud data corresponding to the current moment in another embodiment;

[0039] Figure 8 Flow chart of the calibration method in an embodiment;

[0040] Figure 9 Flow chart of the step of fitting the target point cloud data according to the fitting circle algorithm to obtain the second center coordinates corresponding to each round hole in an embodiment;

[0041] Figure 10 Flow chart of the step of performing plane fitting on the target point cloud data to obtain the first fitting plane corresponding to the external parameter calibration board in an embodiment;

[0042] Figure 11 Flow chart of the step of fitting the target point cloud data according to the fitting circle algorithm to obtain the second center coordinates corresponding to each round hole in an embodiment;

[0043] Figure 12 Structure block diagram of a transportation device in an embodiment. Specific implementation manners

[0044] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0045] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0046] It can be understood that the terms "first", "second", etc. used in the present application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of the present application, the first center coordinate can be referred to as the second center coordinate, and similarly, the second center coordinate can be referred to as the first center coordinate. Both the first center coordinate and the second center coordinate are the center coordinates of the same circular hole, but they are coordinates in different coordinate systems.

[0047] In the related art, radar-camera calibration schemes are mainly divided into two categories: target-based calibration methods and scene-based calibration methods. Among them, the target-based calibration method relies on a specific calibration object (such as a calibration board) or manually set feature points. The Autoware calibration tool algorithm is a typical representative of this category, which requires the user to manually select multiple sets of feature corresponding points between 2D images and 3D point clouds on the interface. By solving the Perspective-n-Point (PNP) problem, the external parameter transformation matrix between the radar and the camera can be obtained. Further, the external parameter matrix can be fine-tuned using the reprojection optimization method to improve the calibration accuracy. However, there are two major problems when this method is applied in an actual production environment: one is that manually selecting feature points is time-consuming and laborious, reducing work efficiency; the other is that the calibration accuracy largely depends on the point selection level of the calibration operator, and the uncertainty introduced by human factors is relatively large.

[0048] The scene-based calibration method does not require a specific calibration object, but uses the features in the natural scene obtained by two sensors for calibration. The algorithm proposed by Yuan et al. is a representative of this category, which obtains accurate external parameters by aligning the line features in the image and the point cloud and minimizing the reprojection error. This method can achieve pixel-level calibration accuracy and does not require additional devices such as calibration boards. However, it also has some deficiencies: after extracting the edge features, a large number of optimization iteration operations are required to obtain better calibration accuracy, which results in low real-time performance and is difficult to meet the requirements of online calibration.

[0049] The embodiments of the present application disclose a calibration method, a transportation device, and a computer program product, which can more accurately and conveniently calibrate the external parameters between the lidar and the imaging device on the transportation device.

[0050] Figure 1 This is an application scenario diagram of the calibration method in an embodiment. As Figure 1 shown, the calibration method adopted in this application is applied to the transportation device 100, and the transportation device 100 is provided with a lidar 110 and an imaging device 120. Among them, the transportation device 100 is a movable device for transporting goods or instruments, and the transportation device 100 includes, but is not limited to, forklifts, mobile support vehicles, etc.

[0051] The lidar 110 is used to collect the point cloud data around the transportation device 100, and the lidar 110 may include, but is not limited to, pulsed lidar, continuous wave lidar, hybrid lidar, etc. The imaging device 120 is used to collect target images, and the imaging device 120 may include, but is not limited to, infrared cameras, color cameras, optical cameras, grayscale cameras, etc.

[0052] Taking the transportation device 100 as a forklift as an example, the forklift may include a main body torso, a fork, and driving wheels. The fork is used to insert and pick up goods, and the forklift moves through the driving wheels. Among them, the lidar 110 and the imaging device 120 may be respectively installed on the sliding rails of the fork, and the lidar 110 and the imaging device 120 may be arranged side by side. Optionally, the arrangement manner of the lidar 110 and the imaging device 120 may be arranged vertically or horizontally, and the installation positions of the lidar 110 and the imaging device 120 may be set according to the actual situation, which is not limited herein. For example, the imaging device 120 may be arranged side by side directly below the lidar 110, etc.

[0053] In some embodiments, an external parameter calibration board 130 may be provided in front of the transportation device 100. The external parameter calibration board 130 includes at least one round hole. Among them, the radius sizes of the respective round holes and the positional relationships between the respective round holes are not limited herein. In order to obtain more accurate external parameters, the external parameter calibration board 130 cannot be parallel to the shooting angle of the imaging device 120. If the external parameter calibration board 130 is parallel to the shooting angle of the imaging device 120, the imaging device 120 on the transportation device 100 will not be able to collect all the round holes of the external parameter calibration board 130, resulting in the inability to obtain the first center coordinates corresponding to some round holes, which affects the calculation of the external parameters. Specifically, the external parameter calibration board 130 may be placed perpendicular to the road surface, and this road surface is the road surface where the transportation device is located.

[0054] In some embodiments, the transportation device 100 controls the lidar 110 to collect target point cloud data, and controls the imaging device 120 to perform image acquisition on the external parameter calibration board 130 to obtain a target image that matches the target point cloud data. The transportation device 100 detects the circular images included in the target image to obtain at least one circular hole area included in the target image; locates the center of each circular hole area to determine the first center coordinates corresponding to each circular hole. The transportation device 100 fits the target point cloud data according to the fitting circle algorithm to obtain the second center coordinates corresponding to each circular hole. The transportation device 100 then determines the external parameters between the lidar 110 and the imaging device 120 according to the first center coordinates and the second center coordinates corresponding to each circular hole.

[0055] In some embodiments, an odometer 140 may be provided on the transportation device 100. The odometer 140 can be used to obtain the position data of the transportation device 100, and the odometer 140 may include, but is not limited to, a wheel odometer, etc. A gyroscope 150 may be provided on the transportation device 100. The gyroscope 150 can be used to obtain the attitude data of the transportation device 100, and the gyroscope 150 may include, but is not limited to, a heading gyroscope, etc. Taking the transportation device 100 as a forklift as an example, exemplarily, the odometer 140 and the gyroscope 150 may be provided on the driving wheels to obtain the position data and attitude data of the forklift.

[0056] As Figure 2 shown, in one embodiment, a calibration method is provided, which can be applied to the above-mentioned transportation device. The method may include the following steps 210 to 240.

[0057] Step 210, obtain the target point cloud data collected by the lidar, and obtain the target image that matches the target point cloud data by performing image acquisition on the external parameter calibration board by the imaging device.

[0058] The transportation device can control the lidar to perform data acquisition on the surrounding scene of the transportation device to obtain target point cloud data. The target point cloud data can be a set of multiple point clouds, and the target point cloud data includes the point cloud data corresponding to the external parameter calibration board. Further, the lidar can perform data acquisition on the external parameter calibration board provided in front of the transportation device to obtain target point cloud data. For example, the lidar can emit laser pulses around the transportation device; when the laser pulses encounter the external parameter calibration board, they will be reflected back; the receiver of the lidar will capture these reflected laser pulses. These laser pulses can carry key information such as the distance, shape, and position of the external parameter calibration board. Therefore, by emitting and receiving multiple laser pulses, the lidar can construct the external parameter calibration board provided in front of the transportation device to obtain target point cloud data.

[0059] Among them, the target point cloud data may include the three-dimensional coordinates (X, Y, Z) of each point cloud, and may also include other attributes such as reflection intensity, color, etc., jointly describing the three-dimensional shape and structure of a transportation device and the surrounding scene.

[0060] In some embodiments, the target point cloud data collected by the lidar can be said to be based on the radar coordinate system corresponding to the lidar, and this radar coordinate system refers to a three-dimensional coordinate system with the lidar as the origin.

[0061] The transportation device can control the imaging device to collect an image of the external parameter calibration board arranged in front of the transportation device, obtaining an image containing the external parameter calibration board. Among them, the external parameter calibration board can refer to a calibration board used to determine the external parameters (relative position and relative attitude) between the imaging device and the lidar.

[0062] Compared with the traditional scene-based calibration method, the calibration method of this application avoids a large number of iterative operations after extracting edge features in the natural scene. By setting the external parameter calibration board, the lidar and the imaging device can collect the target point cloud data and the target image containing the external parameter calibration board, and by detecting the center coordinates of the circular holes on the external parameter calibration board, obtain the first center coordinates and the second center coordinates corresponding to each circular hole respectively, and thereby determine the external parameters between the lidar and the imaging device, reducing the operation of repeatedly obtaining coordinate pairs, simplifying the calibration process, accelerating the speed of the calibration process, making the calibration process of the transportation device more direct and efficient to meet the online calibration requirements, so that when the positions of the lidar or the imaging device change, re-calibration can be carried out in real time and quickly to ensure the accuracy and real-time nature of the external parameters.

[0063] In some embodiments, the transportation device can obtain the target image collected by the imaging device that matches the target point cloud data. Among them, the target image that matches the target point cloud data can refer to the target image whose acquisition time is the same as or closest to the acquisition time of the target point cloud data.

[0064] In some embodiments, if both the transportation device and the external parameter calibration board are in a stationary state, then all relevant devices (devices) are stationary, and the collected data (whether it is point cloud data or image data) will not change with time, that is, the data collected at any time is the same data. Therefore, there is no need for additional synchronization operations. Therefore, the transportation device can control the lidar to directly collect the target point cloud data, and control the imaging device to collect an image of the external parameter calibration board to directly obtain the target image.

[0065] In some embodiments, if the transportation device and / or the external parameter calibration board move during the process of the lidar collecting point cloud data and the imaging device collecting images, since the collection frequencies of the lidar and the imaging device are usually different, it is necessary to synchronize the lidar and the imaging device to ensure that the target point cloud data and the target image are synchronously collected data.

[0066] In some embodiments, the transportation device can trigger the lidar and the imaging device to synchronously collect data from the external parameter calibration board through external synchronization signals, such as PPS (Pulse Per Second) signals, GPIO (General Purpose Input Output) signals, etc., so as to obtain the target point cloud data collected by the lidar and the target image collected by the imaging device and matching the target point cloud data. Specifically, the transportation device can send external synchronization signals to the lidar and the imaging device respectively at the same moment, and the external synchronization signal is used to instruct the lidar and the imaging device to synchronously collect data. The transportation device takes the point cloud data fed back by the lidar in response to the external synchronization signal as the target point cloud data, and takes the image fed back by the imaging device in response to the external synchronization signal as the target image.

[0067] In some embodiments, the lidar and the imaging device continuously collect data, and record information such as the collected timestamps during the data collection process. The transportation device aligns the point cloud data collected by the lidar with the images collected by the imaging device through the timestamps.

[0068] In some specific embodiments, the transportation device can perform timestamp alignment to achieve synchronization according to the collection frequencies of the lidar and the imaging device. Specifically, the transportation device can calculate the common multiple duration between the collection periods of the lidar and the imaging device, determine the same collection moment of the lidar and the imaging device according to the common multiple duration, and take the point cloud data collected by the lidar at the same collection moment as the target point cloud data, and take the image collected by the imaging device at the same collection moment as the target image.

[0069] For example, assume that the collection frequency of the lidar is 10 Hz (collecting point cloud data once every 0.1 second), and the collection frequency of the imaging device is 20 Hz (collecting one frame of image every 0.05 second). The collection frequency of the imaging device is twice that of the lidar. So actually, within every two collection periods of the lidar (i.e., 0.2 seconds), there will be a collection moment when the imaging device and the lidar are simultaneous. The transportation device can take the point cloud data collected by the lidar at 0.2 seconds as the target point cloud data, and take the image collected by the imaging device at 0.2 seconds as the target image.

[0070] In the embodiments of the present application, since the synchronously acquired data can more accurately reflect the relative position and attitude between the lidar and the imaging device, the transportation device realizes the synchronous acquisition of the lidar and the imaging device in various ways, so that the target point cloud data acquired by the lidar and the target image acquired by the imaging device are acquired at the same acquisition moment, and the external parameters between the lidar and the imaging device can be calculated more accurately.

[0071] Step 220: Detect the circular images included in the target image to obtain at least one circular hole area included in the target image.

[0072] The circular images included in the target image can be the circular holes on the external parameter calibration board within the field of view of the imaging device. The first center coordinate refers to the coordinate of the center of the circular hole on the external parameter calibration board in the imaging device coordinate system. The imaging device coordinate system is a two-dimensional coordinate system with the optical center position of the imaging device as the origin, and the optical center is the focal position of the imaging device.

[0073] In some embodiments, the transportation device identifies the external parameter calibration board included in the target image, determines the position information of the external parameter calibration board in the target image; according to the position information, performs image segmentation on the external parameter calibration board included in the target image to obtain the image area corresponding to the external parameter calibration board; detects the circular images included in the image area corresponding to the external parameter calibration board to obtain at least one circular hole area included in the external parameter calibration board. Compared with directly detecting circles in the target image, first determining the image area corresponding to the external parameter calibration board and then detecting circles in the image area corresponding to the external parameter calibration board can reduce the image area for detection, so as to reduce the calculation amount of the transportation device, and can more accurately detect the circular hole area included on the external parameter calibration board, avoiding the recognition interference of circular areas outside the external parameter calibration board in the target image.

[0074] Step 230: Locate the centers of each circular hole area to determine the first center coordinates corresponding to each circular hole in the imaging device coordinate system.

[0075] In some embodiments, the transportation device can detect the circular images included in the target image according to the edge detection algorithm to obtain at least one circular hole area in the target image; based on the Hough circle detection algorithm, locate the centers of each circular hole area to determine the first center coordinates corresponding to each circular hole.

[0076] Among them, the edge detection algorithms that the transportation device can adopt include but are not limited to the Canny operator, Roberts operator, Prewitt operator, etc. The Hough circle detection algorithm is an algorithm based on gradient information, which can detect circular features by finding peaks in the parameter space, search for pixel points that satisfy the circular features, and determine the position and radius of the center according to these pixel points.

[0077] Specifically, the transportation device can calculate the gradient direction and gradient intensity of each edge point in the first circular hole area, and based on the gradient direction and gradient intensity of each edge point in the first circular hole area, traverse all edge points to calculate the candidate center coordinates corresponding to each edge point in the first circular hole area; based on the matching degree of the distance from the edge point to the candidate center, vote on each candidate center coordinate. Among them, the higher the matching degree of the center coordinate, the more votes it obtains. The transportation device can use the candidate center coordinate with the most votes as the first center coordinate of the circular hole corresponding to the first circular hole area. The first circular hole area is any circular hole area.

[0078] By adopting the edge detection algorithm, the circular hole area in the target image can be accurately identified, providing clear and accurate edge information for subsequent Hough circle detection; moreover, by using the Hough circle detection algorithm to locate the center coordinates of each circular hole area, the first center coordinates of each circular hole can be determined more accurately, improving the detection accuracy.

[0079] In some embodiments, the transportation device can input the target image into a circular detection model, detect the circular images contained in the target image through the circular detection model, and perform image segmentation on the circular images contained in the target image to obtain at least one circular hole area contained in the target image; based on the Hough circle detection algorithm, perform center positioning on each circular hole area to obtain the first center coordinates corresponding to each circular hole.

[0080] Exemplarily, the circular detection model can be trained according to multiple sample images, and the multiple sample images can respectively contain sample circular areas with different sizes, shapes, quantities, illumination intensities, illumination angles, and background complexities. The transportation device inputs the multiple sample images into the pre-trained circular detection model respectively, detects the circular images in each sample image through the pre-trained circular detection model, and performs image segmentation on the circular areas contained in each sample image to obtain multiple predicted circles in each sample image. According to the sample circular areas and the corresponding predicted circles in each sample image, the pre-trained circular detection model is optimized until the loss function value corresponding to the pre-trained circular detection model meets the preset conditions, then the trained circular detection model is obtained.

[0081] By training with sample images containing different circular areas, the circular detection model can learn the features of various circles, thus more accurately identifying the circular hole area in the target image; moreover, once the circular detection model is trained, it can quickly detect the circular hole area in the target image without complex manual operations or time-consuming calculation processes, which can improve the efficiency.

[0082] In some embodiments, the transportation device can also identify the first center coordinates of the first circular hole area; then, according to the physical position relationship between the first circular hole area and other circular hole areas in the external parameter calibration board, calculate the corresponding first center coordinates of other circular hole areas respectively. Wherein, the first circular hole area is any circular hole area in the external parameter calibration board; the physical position relationship may include the actual distance ratio, actual distance length, etc. between each circular hole area on the external parameter calibration board.

[0083] Specifically, assume that there are 4 circular holes of the same size on the external parameter calibration board, and the 4 circular holes are arranged in a 2×2 form on the external parameter calibration board, and the 4 centers form a square on the external parameter calibration board (as Figure 3 shown), with a side length of c. If the transportation device detects that the first center coordinates of the first circular hole in the first row are (a, b), then the transportation device can calculate that the first center coordinates of the second circular hole in the first row are (a + c, b), the first center coordinates of the first circular hole in the second row are (a, b - c), and the first center coordinates of the second circular hole in the second row are (a + c, b - c).

[0084] In the embodiments of the present application, through the above various methods, the transportation device can obtain the first center coordinates of each circular hole of the external parameter calibration board from the target image more quickly and accurately, without the operator manually selecting points, which can simplify the external parameter calibration process between the camera device and the lidar.

[0085] Step 240, according to the circle fitting algorithm, fit the target point cloud data to obtain the second center coordinates corresponding to each circular hole in the radar coordinate system.

[0086] The circle fitting algorithm refers to an algorithm that fits the distribution of multiple specified points into the best circle. The circle fitting algorithms that the transportation device can adopt include but are not limited to the least squares method, the RANSAC (Random Sample Consensus) algorithm, etc.

[0087] The second center coordinates are the center coordinates of each circular hole on the external parameter calibration board in the radar coordinate system. Wherein, the second center coordinates can be three-dimensional coordinates.

[0088] In some embodiments, the transportation device can adopt an iterative optimization algorithm (such as the non-linear least squares method) to directly fit at least one circular surface in the three-dimensional space, and by continuously adjusting the parameters of each circle (such as the center coordinates and radius), to minimize the sum of the distances or the sum of the squares of the distances from each point cloud in the target point cloud data to the circle; take the center coordinates of at least one fitted circle with a sum value less than the threshold as the second center coordinates corresponding to at least one circular hole respectively.

[0089] In some embodiments, the transportation device may regard the round hole as the projection of a two-dimensional circle in a plane of a three-dimensional space, and fit the circle through this plane to determine the second center coordinates of the round hole.

[0090] Step 250: Determine the extrinsic parameters between the lidar and the camera device according to the first center coordinates and the second center coordinates respectively corresponding to each round hole.

[0091] The extrinsic parameters between the lidar and the camera device may include the transformation matrix between the lidar and the camera device, and the transformation matrix may include a rotation matrix and / or a translation matrix.

[0092] In some embodiments, the transportation device may match at least one first center coordinate with at least one second center coordinate according to the principle of singularity transformation to obtain at least one set of matched center coordinate pairs, and determine the center coordinate pairs corresponding to each round hole; wherein, a set of center coordinate pairs includes a first center coordinate and a second center coordinate. Specifically, the transportation device converts the first center coordinates corresponding to each round hole into three-dimensional coordinates; randomly selects an initial transformation matrix (rotation matrix and translation matrix) as the starting point for iteration, and in each iteration, according to the current transformation matrix, determines the center correspondence relationship among at least one first center coordinate and at least one second center coordinate, that is, finds the matched center coordinate pairs; a set of center coordinate pairs includes a first center coordinate and a second center coordinate.

[0093] In some embodiments, the transportation device may determine the extrinsic parameters between the lidar and the camera device based on the PnP (Perspective-n-Point) algorithm according to the center correspondence relationship corresponding to each round hole. The PnP algorithm can be used in computer vision to calculate the rotation matrix R and the translation vector t between the lidar and the camera device in the correspondence relationship from the three-dimensional coordinates of multiple points in the lidar to the two-dimensional coordinates in the camera device, that is, the extrinsic parameters between the lidar and the camera device.

[0094] Optionally, the PnP algorithm may include the DLT (Direct Linear Transform) algorithm, the P3P algorithm, the EPnP (Efficient PnP) algorithm, the BA (Bundle Adjustment) algorithm, etc. Specifically, the transportation device may construct an augmented matrix [R|t] according to the first center coordinates and the second center coordinates respectively corresponding to at least one round hole and the intrinsic matrix of the camera device, where R is the rotation matrix and t is the translation vector; and then obtain the rotation matrix R and the translation vector t between the lidar and the camera device by solving the augmented matrix.

[0095] Specifically, assume that the transportation device obtains the first center coordinates Q corresponding to i circular holes i =[u i ,v i T and the second center coordinates P i =[X i ,Y i ,Z i T , then according to the DLT (Direct Linear Transformation) method, the linear relationship between the camera coordinates and the radar coordinates is s i Q i =K(RP i +t), where K is the internal parameter of the camera, s i is the depth of the first center coordinate (i.e., the distance from the first center coordinate to the optical center of the camera), R is the rotation matrix, and t is the translation vector. To simplify the calculation, this linear relationship can be transformed into a homogeneous form Then expand the internal parameter matrix K and the rotation matrix R to obtain Equation (1); then simplify it to obtain Equation (2).

[0096]

[0097] For each point pair (Q i ,P i ), substitute it into the above Equation (2), and the transportation device can obtain i equations about R and t. Combine these equations to form an overdetermined system of equations (because the number of equations is usually greater than the number of unknowns), eliminate s i , and use the least squares method or other optimization methods to solve this overdetermined system of equations to obtain the rotation matrix R and the translation vector t between the lidar and the camera.

[0098] Compared with the traditional autoware toolbox calibration algorithm, the calibration method of the present application does not require manual point selection. Through circular image detection, center calibration, and fitting circle algorithm, the first center coordinates of each circular hole on the external parameter calibration board in the camera coordinate system and the second center coordinates in the radar coordinate system can be obtained respectively, avoiding the experience and skill level of the calibration operator in point selection, reducing the introduction of human errors, thereby improving the calibration accuracy, accelerating the process of determining the external parameters, improving the automation level of external parameter calibration in practical applications, and thus improving the production efficiency of its practical applications.

[0099] ​​In some embodiments, the transportation device may also obtain the lidar to collect the same external parameter calibration board multiple times to obtain the target point cloud data collected by the lidar each time; at the moment when the lidar collects the target point cloud data each time, obtain the target image collected by the imaging device for the same external parameter calibration board each time. The transportation device determines each circular hole on the external parameter calibration board according to the target point cloud data and the target image respectively corresponding to the multiple collections, and obtains a set of center coordinate pairs corresponding to each collection. A set of center coordinate pairs includes the first center coordinate and the second center coordinate corresponding to the same circular hole, and obtains multiple sets of center coordinate pairs corresponding to each collection; then determines the external parameters between the lidar and the imaging device according to the multiple sets of center coordinate pairs respectively corresponding to the multiple collections. By increasing the number of center coordinate pairs through multiple collections, the accuracy of the external parameters can be improved. Among them, using the same external parameter calibration board each time the lidar of the transportation device collects the target point cloud data can further improve the robustness.

[0100] Specifically, assume that there is only one circular hole on the target external parameter calibration board. The lidar collects point cloud data of the target external parameter calibration board at multiple times, and the transportation device respectively obtains the target point cloud data corresponding to each of these times, and according to the fitting circle algorithm, obtains the second center coordinates corresponding to the circular hole at each time. The imaging device collects images of the target external parameter calibration board at multiple times, and the transportation device respectively obtains the target images corresponding to each time, and obtains the first center coordinates corresponding to the circular hole at each time by performing center image detection and center positioning on each frame of the target image. The transportation device takes the first center coordinate and the second center coordinate corresponding to the circular hole at the same time as a set of center coordinate pairs, obtains the center coordinate pairs corresponding to each time respectively, and then determines the external parameters between the lidar and the imaging device based on the PnP algorithm according to the center coordinate pairs corresponding to each time.

[0101] It should be noted that in order to make the external parameters between the lidar and the imaging device more persuasive, there should be at least four sets of center pairs to avoid the external parameter calculation error caused by incorrect center coordinates or the external parameter calculation error caused by accidental center coordinates.

[0102] In the embodiment of the present application, the transportation device detects the circular images included in the target images collected by the imaging device, performs center positioning, determines the first center coordinates corresponding to at least one circular hole on the extrinsic calibration board, and fits the target point cloud data collected by the lidar through the circle fitting algorithm to obtain the second center coordinates corresponding to at least one circular hole on the extrinsic calibration board. According to the first center coordinates and the second center coordinates corresponding to at least one circular hole, the mapping relationship between the lidar coordinate system and the imaging device coordinate system can be better reflected, so as to obtain more accurate extrinsic parameters between the imaging device and the lidar. And through center positioning and the circle fitting algorithm, the center coordinates of at least one circular hole on the extrinsic calibration board in different coordinate systems can be obtained, without setting special markers (such as QR code markers, barcode markers, etc.) on the extrinsic calibration board and performing special recognition on the special markers to obtain the center coordinates (such as recognizing QR code markers through the autoware toolbox, etc.). Therefore, the manufacturing cost of the extrinsic calibration board can be reduced, and there is no need to rely on manual point selection by the operator, which can simplify the extrinsic calibration process between the imaging device and the lidar, and can more accurately and conveniently calibrate the extrinsic parameters between the lidar and the imaging device on the transportation device.

[0103] In some embodiments, as Figure 4 shown, the step of obtaining the target point cloud data collected by the lidar further includes steps 402 to 404.

[0104] Step 402, obtaining the historical point cloud data collected by the lidar at at least one historical acquisition moment respectively.

[0105] Since the point cloud included in the point cloud data collected by the lidar at one moment may be relatively sparse, in order to ensure that there is enough point cloud for subsequent extrinsic calibration calculation, the transportation device can fuse the historical point cloud data collected by the lidar at at least one historical acquisition moment into the current moment to increase the number of point cloud in the target point cloud data collected at the current moment.

[0106] The historical acquisition moment refers to the moment when the lidar collects point cloud data before the current moment. The historical acquisition moment is related to the acquisition frequency of the lidar. For example, if the acquisition frequency of the lidar is to collect data once every 0.2 seconds and the current moment is the 0.8th second, the historical acquisition moments can be the 0.2nd second, the 0.4th second, and the 0.6th second.

[0107] In some embodiments, the lidar on the transportation device can continuously collect point cloud data and store each piece of collected point cloud data in a database or a memory. The historical point cloud data collected at the historical collection moment obtained in step 402 can be the point cloud data collected at any moment stored in the database or the memory. For example, if the point cloud data collected by the lidar at 6 moments is stored in the database or the memory, and the transportation device needs to obtain the point cloud data collected by the lidar at 4 historical collection moments, it can arbitrarily select 4 moments from the 6 moments as the historical collection moments.

[0108] Step 404: Fuse the historical point cloud data collected at at least one historical collection moment with the current frame point cloud data collected by the lidar at the current moment to obtain target point cloud data.

[0109] In some embodiments, the transportation device can select a moment as the historical collection moment according to the number of point clouds contained in the point cloud data collected at each moment stored in the database or the memory, and fuse the historical point cloud data collected by the lidar at that moment with the current frame point cloud data collected at the current moment to obtain target point cloud data. For example, the transportation device can select a moment with the largest number of point clouds contained in the point cloud data collected at each moment as the historical collection moment for fusion with the current moment.

[0110] In some embodiments, the transportation device can successively fuse the historical point cloud data collected at the historical collection moment closest to the current moment with the current frame point cloud data collected at the current moment until the number of point clouds in the fused point cloud data is greater than the target number threshold, and then use the fused point cloud data as the target point cloud data. Specifically, the transportation device can fuse the historical point cloud data collected at the Y-frame historical collection moment into the current frame point cloud data; for example, Y can be a positive integer greater than 1 such as 5, 10, etc.

[0111] For example, assume that the lidar collects point cloud data once per second, the number of point clouds contained in the point cloud data collected at each collection moment is 20, the target number threshold is 50, and the current moment is the 8th second. Then the transportation device fuses the historical point cloud data at the 7th second with the current frame point cloud data collected at the 8th second of the current moment, and the number of point clouds in the obtained fused point cloud data is 40 < 50, so it is necessary to fuse the historical point cloud data at the 6th second with the fused point cloud data. The number of point clouds in the fused point cloud data obtained again is 60 > 50, so the point cloud data obtained by fusing the historical point cloud data at the 6th second, the historical point cloud data at the 7th second, and the current frame point cloud data at the 8th second is used as the target point cloud data. The target point cloud data obtained by fusion has a sufficient number of point clouds for subsequent calculation of the center coordinates and external parameters, which can reduce contingency and improve the accuracy of the calculation.

[0112] In some embodiments, the transportation device is further provided with an odometer and a gyroscope. Among them, the odometer is used to obtain the position data of the transportation device, and the gyroscope is used to obtain the attitude data of the transportation device. As Figure 5 shown, the step of fusing the historical point cloud data collected at at least one historical acquisition moment with the current frame point cloud data collected by the lidar at the current moment to obtain the target point cloud data may further include steps 502 to 504.

[0113] Step 502: Determine the pose data corresponding to the transportation device at at least one historical acquisition moment and the current moment respectively according to the position data collected by the odometer and the attitude data collected by the gyroscope.

[0114] The pose data refers to the position data and attitude data of the transportation device. Among them, the position data may include the position coordinates of the transportation device in the world coordinate system (such as the x-axis and y-axis of the world coordinate system), etc.; the attitude data may include the pitch angle, yaw angle, roll angle, etc. of the transportation device. The odometer and the gyroscope rely on sensors (such as rotary encoders, optoelectronic encoders, etc.) installed on the driving wheels of the transportation device to obtain the pose data. The driving wheels may be components that drive the movement of the transportation device, such as driving wheels or driven wheels.

[0115] In some embodiments, the transportation device can calculate the displacement of the driving wheel at the current moment through the position coordinates collected by the odometer and the yaw angle collected by the gyroscope; combined with the geometric dimensions of the driving wheel, the transportation device can further deduce the overall moving distance and direction change of the transportation device through the odometer and the gyroscope, so as to estimate the position data and attitude data of the transportation device at at least one historical acquisition moment and the current moment.

[0116] Step 504: Map the historical point cloud data collected at each historical acquisition moment to the current moment according to the pose data corresponding to each historical acquisition moment and the pose data corresponding to the current moment, and fuse the mapped historical point cloud data with the current frame point cloud data collected at the current moment to obtain the target point cloud data.

[0117] In some embodiments, the at least one historical acquisition moment may include M historical acquisition moments, where M is an integer greater than 1. The number of historical acquisition moments is not limited here, and the transportation device can determine it according to the number of point clouds included in the current frame point cloud data collected by the lidar at the current moment. For example, if the number of point clouds included in the current frame point cloud data is larger, the number of historical acquisition moments can be smaller; if the number of point clouds included in the current frame point cloud data is smaller, the number of historical acquisition moments can be larger.

[0118] In some embodiments, the transportation device may fuse the historical point cloud data collected by the lidar at M historical acquisition times before the current time with the current frame point cloud data collected at the current time, and use the fused point cloud data as the target point cloud data. Assume that the lidar collects point cloud data at T frame times in total, and the current time is the T-th frame time collected by the lidar, where T is an integer greater than M. Then the current frame point cloud data is the point cloud data collected by the lidar at the T-th frame time, and the times from the 1st frame time to the (T - 1)-th frame time are all historical acquisition times, and the point cloud data collected at the times from the 1st frame time to the (T - 1)-th frame time are all historical point cloud data. The transportation device may then select M frame times from the T - 1 frame times between the 1st frame time and the (T - 1)-th frame time of the lidar, and fuse the historical point cloud data collected at these M frame times respectively with the current frame point cloud data collected at the T-th frame time to obtain the target point cloud data. Specifically, the transportation device may fuse the point cloud data collected by the lidar at the (T - M + 1)-th frame time, the (T - M + 2)-th frame time, ……, the (T - 1)-th frame time between the 1st frame time and the (T - 1)-th frame time with the current frame point cloud data collected at the T-th frame time to obtain the target point cloud data.

[0119] In some specific embodiments, the transportation device may map the fused point cloud data corresponding to the N-th historical acquisition time to the (N + 1)-th historical acquisition time according to the pose data corresponding to the N-th historical acquisition time and the pose data corresponding to the (N + 1)-th historical acquisition time, to obtain the mapped historical point cloud data corresponding to the N-th historical acquisition time; N is an integer greater than 0 and less than M, where the fused point cloud data corresponding to the 1st historical acquisition time is the historical point cloud data collected at the 1st historical acquisition time; fuse the mapped historical point cloud data corresponding to the N-th historical acquisition time with the historical point cloud data collected at the (N + 1)-th historical acquisition time to obtain the fused point cloud data corresponding to the (N + 1)-th historical acquisition time; map the fused point cloud data corresponding to the M-th historical acquisition time to the current time according to the pose data corresponding to the M-th historical acquisition time and the pose data corresponding to the current time, to obtain the mapped historical point cloud data corresponding to the M-th historical acquisition time; fuse the mapped historical point cloud data corresponding to the M-th historical acquisition time with the current frame point cloud data collected at the current time to obtain the target point cloud data.

[0120] Specifically, Figure 6 FIG. is a schematic diagram of the process of obtaining the target point cloud data according to the historical point cloud data corresponding to M historical acquisition times respectively and the current frame point cloud data corresponding to the current time in one embodiment.

[0121] Such as Figure 6As shown, the transportation device maps the historical point cloud data 601 collected at the first historical acquisition moment to the second historical acquisition moment, obtaining the mapped historical point cloud data 606 corresponding to the first historical acquisition moment; fuses the historical point cloud data 602 at the second historical acquisition moment with the mapped historical point cloud data 606, obtaining the fused point cloud data 607 corresponding to the second historical acquisition moment. The transportation device then maps the fused point cloud data 607 at the second historical acquisition moment to the third historical acquisition moment, obtaining the mapped historical point cloud data 608 corresponding to the second historical acquisition moment; fuses the historical point cloud data 603 at the third historical acquisition moment with the mapped historical point cloud data 608, obtaining the fused point cloud data 609 corresponding to the third historical acquisition moment. By analogy, the transportation device can obtain the fused point cloud data 610 corresponding to the (M - 1)th one, map the fused point cloud data 610 corresponding to the (M - 1)th one to the Mth historical acquisition moment, obtaining the mapped historical point cloud data 611 corresponding to the (M - 1)th historical acquisition moment; fuses the historical point cloud data 604 at the Mth historical acquisition moment with the mapped historical point cloud data 611, thereby obtaining the fused point cloud data 612 corresponding to the Mth historical acquisition moment.

[0122] Then, the transportation device can map the fused point cloud data 612 corresponding to the Mth historical acquisition moment to the current moment, obtaining the mapped historical point cloud data 613 corresponding to the Mth historical acquisition moment, and fuses the current frame point cloud data 605 collected at the current moment with the mapped historical point cloud data 613, obtaining the target point cloud data 614.

[0123] In some embodiments, when the transportation device maps the fused point cloud data corresponding to the Nth historical acquisition moment to the (N + 1)th historical acquisition moment, it can perform an inverse transformation on the pose data corresponding to the Nth historical acquisition moment, obtaining the inverse pose data corresponding to the Nth historical acquisition moment; multiplies the inverse pose data corresponding to the Nth historical acquisition moment, the fused point cloud data corresponding to the Nth historical acquisition moment, and the pose data corresponding to the (N + 1)th historical acquisition moment, thereby obtaining the mapped historical point cloud data corresponding to the Nth historical acquisition moment.

[0124] In some specific embodiments, the transportation device can calculate the mapped historical point cloud data corresponding to the Nth historical acquisition moment according to formula (1).

[0125] X N+1 ′=P N -1 *P N+1 *X N Formula (1);

[0126] Wherein, X N+1′ represents the historical point cloud data corresponding to the Nth historical acquisition moment after being mapped to the (N + 1)th historical moment; P N -1 represents the inverse pose data corresponding to the Nth historical acquisition moment; X N represents the fused point cloud data corresponding to the Nth historical acquisition moment; P N+1 represents the pose data corresponding to the (N + 1)th historical acquisition moment.

[0127] In some other embodiments, the transportation device can also perform an inverse transformation on the pose data corresponding to each historical acquisition moment to obtain the inverse pose data corresponding to each historical acquisition moment; perform a dot product on the inverse pose data corresponding to each historical acquisition moment, the historical point cloud data collected at each historical acquisition moment, and the pose data corresponding to the current moment to obtain the mapped historical point cloud data corresponding to each historical acquisition moment; fuse the mapped historical point cloud data corresponding to each historical acquisition moment with the current frame point cloud data collected at the current moment to obtain the target point cloud data.

[0128] Specifically, Figure 7 is a schematic diagram of the process of obtaining the target point cloud data according to the historical point cloud data corresponding to M historical acquisition moments and the current frame point cloud data corresponding to the current moment in another embodiment. As Figure 7 shown, the transportation device maps the historical point cloud data 701 collected at the 1st historical acquisition moment to the current moment to obtain the mapped historical point cloud data 706 corresponding to the 1st historical acquisition moment; maps the historical point cloud data 702 collected at the 2nd historical acquisition moment to the current moment to obtain the mapped historical point cloud data 707 corresponding to the 2nd historical acquisition moment; maps the historical point cloud data 703 collected at the 3rd historical acquisition moment to the current moment to obtain the mapped historical point cloud data 708 corresponding to the 3rd historical acquisition moment; and so on; maps the historical point cloud data 704 collected at the Mth historical acquisition moment to the current moment to obtain the mapped historical point cloud data 709 corresponding to the Mth historical acquisition moment. The transportation device can fuse the mapped historical point cloud data (706, 707, 708,..., 709) corresponding to each historical acquisition moment with the current frame point cloud data 705 collected at the current moment to obtain the target point cloud data 710.

[0129] In the embodiments of the present application, the transportation device increases the number of point clouds in the target point cloud data by fusing the historical point cloud data collected at at least one historical acquisition moment, thereby improving the density of the point cloud data, providing richer data support for the calculation of the external parameter calibration, and improving the accuracy and reliability of the calibration. In addition, the transportation device also uses the pose data of each historical acquisition moment obtained by the odometer to map the historical point cloud data to the current moment, realizing the spatio-temporal alignment of the point cloud data during the process of point cloud data fusion, which can not only simplify the data processing flow, but also avoid the error accumulation caused by the time difference of the historical acquisition moments, and improve the efficiency and accuracy of data processing.

[0130] As Figure 8 shown, in one embodiment, a calibration method is provided, which can be applied to the above-mentioned transportation device. The method may include the following steps 802 to step 818.

[0131] Step 802, obtain an image of the internal parameter calibration board captured by the imaging device to obtain a first image.

[0132] The internal parameter refers to the internal parameter coefficient of the imaging device, which is used to reflect the mapping relationship between the environmental information and the image information, that is, the conversion relationship between the pixel coordinate system and the imaging device coordinate system. Obtaining the internal parameter accurately can obtain more accurate first center coordinates, and thus can obtain accurate external parameters. The pixel coordinate system is used to represent the coordinate system of each pixel position in the image.

[0133] The internal parameter calibration board can be an object with known shape and size, including a series of regular pattern elements, such as checkerboards, dot arrays, etc. The internal parameter calibration board can generate clear inner corner points or feature points in the first image, so as to facilitate the recognition of the pattern elements and subsequent processing and calculation.

[0134] It should be noted that the internal parameter calibration board and the external parameter calibration board are not the same calibration board.

[0135] Step 804, perform feature extraction on the internal parameter calibration board included in the first image to obtain multiple feature points corresponding to the internal parameter calibration board.

[0136] Taking the internal parameter calibration board as a checkerboard calibration board as an example, in some embodiments, the transportation device can use a corner detection algorithm to extract multiple inner corner points of the internal parameter calibration board in the first image as multiple feature points corresponding to the internal parameter calibration board. The inner corner points can refer to the intersection points in the checkerboard calibration board. The corner detection algorithms that can be used include but are not limited to Harris corner detection, Shi-Tomasi corner detection, etc.

[0137] Step 806, calculate the internal parameter corresponding to the imaging device according to the multiple feature points.

[0138] Taking the internal reference calibration board as a checkerboard calibration board as an example, in some specific embodiments, the transportation device can determine the pixel coordinates (u, v) of each feature point on the first image; then, according to the size and the number of grids of the checkerboard calibration board, a fixed world coordinate (x, y, z = 0) is assigned to each feature point, where z = 0 represents the checkerboard plane; the transportation device calculates the homography matrix H by the least squares method using the pixel coordinates and world coordinates of the feature points based on multiple frames of the first image. The homography matrix refers to the homography relationship existing between the checkerboard plane and the image plane. The transportation device calculates the internal parameter A of the imaging device through mathematical transformation and calculation using the homography matrix H of multiple frames of the first image and the size of the checkerboard calibration board.

[0139] Step 808, obtain the target point cloud data collected by the lidar.

[0140] Step 810, detect the circular images included in the target image to obtain at least one circular hole area included in the target image.

[0141] Step 812, perform centroid localization on each circular hole area to obtain the third centroid coordinates corresponding to each circular hole area in the pixel coordinate system.

[0142] For the relevant descriptions of steps 808 to 812, reference can be made to the relevant descriptions of steps 210 to 230 in the above embodiments, which will not be elaborated here.

[0143] Step 814, based on the internal parameter, transfer the third centroid coordinates corresponding to at least one circular hole area to the imaging device coordinate system to obtain the first centroid coordinates corresponding to each circular hole.

[0144] Step 816, fit the target point cloud data according to the fitting circle algorithm to obtain the second centroid coordinates corresponding to each circular hole.

[0145] Step 818, determine the external parameter between the lidar and the imaging device according to the first centroid coordinates and the second centroid coordinates corresponding to each circular hole.

[0146] For the relevant descriptions of steps 816 to 818, reference can be made to the relevant descriptions of steps 240 to 250 in the above embodiments, which will not be elaborated here.

[0147] In the embodiments of the present application, the transportation device determines the conversion relationship between the pixel coordinate system and the imaging device coordinate system by calculating the internal parameter of the imaging device, improving the accuracy of extracting the first centroid coordinates in the imaging device coordinate system from the target image, and thus more accurately determining the external parameter between the lidar and the imaging device.

[0148] Such as Figure 9As shown, the step of fitting the target point cloud data according to the fitting circle algorithm to obtain the second center coordinates corresponding to each circular hole may include steps 902 to 906.

[0149] Step 902: Perform plane fitting on the target point cloud data to obtain the first fitting plane corresponding to the external parameter calibration board.

[0150] The first fitting plane refers to the spatial plane where the external parameter calibration board is located.

[0151] In some embodiments, the transportation device may set the general form of the plane equation of the first fitting plane (such as Ax + By + Cz + D = 0); according to the target point cloud data, construct an error function, that is, calculate the sum of the squares of the distances from all point clouds in the target point cloud data to the first fitting plane; by minimizing the error function, solve the parameters A, B, C, and D of the plane equation, and moreover, after obtaining the first fitting plane, the transportation device can also calculate the distances from each point cloud on the external parameter calibration board to the first fitting plane to evaluate the fitting effect of the first fitting plane. If the fitting effect is not satisfied, the parameters of the plane equation can be recalculated until the fitting effect is satisfied.

[0152] In some embodiments, the external parameter calibration board may be placed perpendicular to the road surface, or the angle between the external parameter calibration board and the road surface is close to 90 degrees. Then, in the process of determining the first fitting plane corresponding to the external parameter calibration board, it is not necessary to consider the z-axis coordinates of all point clouds in the target point cloud data. The transportation device can determine the first fitting plane corresponding to the external parameter calibration board only through the x-axis coordinates and y-axis coordinates of each point cloud, which can reduce the complexity of plane fitting of the transportation device and improve the efficiency and accuracy of plane fitting.

[0153] In some embodiments, before performing plane fitting on the target point cloud data, the transportation device may perform band-pass filtering on the target point cloud data according to a preset range to filter out the point clouds that obviously do not belong to the first fitting plane where the external parameter calibration board is located (that is, the point clouds outside the preset range) in the target point cloud data, thereby reducing the number of point clouds to be fitted subsequently and reducing the calculation amount. The preset range includes the ranges of the point clouds in the x, y, and z dimensions respectively.

[0154] In some embodiments, as Figure 10 shown, the step of performing plane fitting on the target point cloud data to obtain the first fitting plane corresponding to the external parameter calibration board may include steps 1002 to 1006.

[0155] Step 1002: Determine the road surface normal vector corresponding to the transportation device.

[0156] The road surface normal vector refers to the normal vector of the road surface where the transportation device is located in the radar coordinate system.

[0157] If the road surface is a horizontal plane, the transport device can directly determine that the normal vector of the road surface is If the road surface is not a horizontal plane, the transport device can obtain the equation of the plane where the road surface is located through methods such as road surface fitting algorithm and SVD decomposition, so as to obtain the normal vector of the road surface

[0158] Step 1004: According to the normal vector of the road surface, screen each point cloud included in the target point cloud data to obtain multiple candidate point clouds corresponding to the external parameter calibration board.

[0159] Candidate point cloud refers to the point cloud in the target point cloud data that may be located in the spatial plane where the external parameter calibration board is located.

[0160] In some embodiments, the transport device can obtain K nearest neighbor point clouds of the first point cloud from the target point cloud data, where the first point cloud is any point cloud in the target point cloud data, and K is an integer greater than 1; the nearest neighbor point cloud is the point cloud in the target point cloud data whose distance from the first point cloud is less than the first distance threshold; use the least squares method to perform plane fitting on the first point cloud and the K nearest neighbor point clouds to obtain a second fitting plane, and determine the plane normal vector corresponding to the second fitting plane; if the angle between the road surface normal vector and the plane normal vector is greater than the preset angle threshold, then use the first point cloud as the candidate point cloud corresponding to the external parameter calibration board.

[0161] For example, assume that K is 3 and the road surface normal vector is Then the transport device performs plane fitting according to the first point cloud and the 3 nearest neighbor point clouds of the first point cloud to obtain the second fitting plane corresponding to the first point cloud, and determines that the plane normal vector of the second fitting plane corresponding to the first point cloud is If the calculated road surface normal vector and the plane normal vector is of the angle (T is the preset angle threshold), then the transport device can use the first point cloud as the candidate point cloud corresponding to the external parameter calibration board.

[0162] Step 1006: Perform plane fitting on multiple candidate point clouds to obtain a first fitting plane corresponding to the external parameter calibration board.

[0163] In some embodiments, P target candidate point clouds are arbitrarily obtained from multiple candidate point clouds, where P is a positive integer greater than 2 and less than S, and S is the number of candidate point clouds; plane fitting is performed on the P target candidate point clouds to obtain a third fitting plane; the first distance between each candidate point cloud and the third fitting plane is calculated; the number of point clouds of the candidate point clouds with the first distance less than the second distance threshold is determined; the step of arbitrarily obtaining P target candidate point clouds from multiple candidate point clouds is repeatedly executed Q times to respectively obtain Q third fitting planes, and in each third fitting plane, the number of point clouds of the candidate point clouds with the first distance less than the second distance threshold from each third fitting plane, where Q is a positive integer; among the numbers of point clouds corresponding to the Q third fitting planes respectively, the third fitting plane with the largest number of point clouds is used as the first fitting plane corresponding to the external parameter calibration board.

[0164] Among them, the largest number of point clouds means that the candidate point clouds with the first distance less than the second distance threshold from this third fitting plane are the most, indicating that the most candidate point clouds are closest to this third fitting plane. Then the transportation device can consider that this third fitting plane fits all candidate point clouds, and this third fitting plane is used as the first fitting plane corresponding to the external parameter calibration board.

[0165] Step 904, determine multiple edge point clouds according to the curvature of each point cloud included in the first fitting plane.

[0166] The edge point cloud refers to the point cloud corresponding to the edge contour of at least one round hole in the external parameter calibration board in the first fitting plane.

[0167] In some embodiments, before determining the curvature of each point cloud included in the first fitting plane, the transportation device can screen the point clouds included in the first fitting plane, and use the point clouds in the target point cloud data with the distance from the external parameter calibration board plane less than a specific threshold as the point clouds included in the first fitting plane.

[0168] In some embodiments, the transportation device can calculate the normal vector angle or distance change rate between any adjacent point clouds among each point cloud included in the first fitting plane to determine the curvature of each point cloud.

[0169] Optionally, the transportation device can determine the curvature threshold according to the overall curvature distribution of the point cloud data; use the point clouds with the curvature greater than this curvature threshold as the edge point clouds. It should be noted that the transportation device can adjust the curvature threshold according to the distribution of the edge point clouds. For example, if the transportation device detects that the edge point clouds are too sparse or too dense, it can adjust the curvature threshold or recalculate the curvature of the point cloud.

[0170] Step 906, perform fitting on each edge point cloud according to the fitting circle algorithm to determine the second center coordinates corresponding to each round hole.

[0171] The relevant descriptions of fitting each edge point cloud in step 906 and the relevant descriptions of fitting the target point cloud data in step 230 of the above embodiments will not be elaborated here.

[0172] In the embodiments of the present application, the transportation device performs plane fitting on the target point cloud data to obtain the first fitting plane corresponding to the external parameter calibration board, so as to reduce the error of subsequent circle fitting. Moreover, through the screening of the selected point cloud and the edge point cloud, the number of point clouds to be processed subsequently is effectively reduced, thereby reducing the computational complexity, improving the efficiency of determining the second center coordinates, and realizing the accurate extraction of the second center coordinates of each circular hole in the target point cloud data.

[0173] As Figure 11 shown, the step of performing fitting on the target point cloud data according to the fitting circle algorithm to obtain the second center coordinates corresponding to each circular hole may include steps 1102 to 1110.

[0174] Step 1102, perform plane fitting on the target point cloud data to obtain the first fitting plane corresponding to the external parameter calibration board.

[0175] For the relevant description of step 1102, reference may be made to the relevant description of step 902 in the above embodiments, which will not be elaborated here.

[0176] Step 1104, rotate the first fitting plane so that the rotated first fitting plane is perpendicular to the z-axis of the radar coordinate system, and determine the rotation transformation matrix corresponding to the rotated first fitting plane.

[0177] The radar coordinate system refers to a three-dimensional coordinate system with the lidar as the origin.

[0178] Rotating the first fitting plane to be perpendicular to the z-axis of the radar coordinate system can make the coordinates of all point clouds on the rotated first fitting plane the same or approximately the same in the z-axis. Then, in subsequent calculations, the change in the z-axis can be temporarily not considered, and the subsequent calculations of three-dimensional coordinates can be simplified to calculations of two-dimensional coordinates, thereby reducing the amount of calculation and improving the robustness of the extraction of the second center coordinates.

[0179] In some embodiments, the transportation device can also make the rotated first fitting plane perpendicular to the y-axis or x-axis of the radar coordinate system, so as not to consider the change in the y-axis or x-axis.

[0180] Step 1106, determine a plurality of edge point clouds according to the curvatures of the respective point clouds included in the rotated first fitting plane.

[0181] Step 1108, perform circle fitting on the plurality of edge point clouds according to the fitting circle algorithm to obtain the target edge point clouds constituting each circular hole, and determine the third center coordinates corresponding to each circular hole according to the target edge point clouds of each circular hole.

[0182] For the relevant descriptions of steps 1106 to 1108, reference may be made to the relevant descriptions of steps 904 to 906 in the above embodiments, which will not be elaborated here.

[0183] Step 1110: According to the rotation transformation matrix, perform reverse rotation transformation on each third center coordinate to obtain the second center coordinate corresponding to each round hole.

[0184] Since the first fitting plane is rotated for the convenience of calculation, after obtaining the third center coordinates of the rotated first fitting plane, performing reverse rotation transformation is to restore the center of the round hole from the rotated third center coordinates to the original second center coordinates, ensuring that the obtained second center coordinates are the center coordinates of the round holes in the target point cloud data collected by the lidar.

[0185] Specifically, the transportation device can rotate the first fitting plane corresponding to the external parameter calibration board so that the rotated first fitting plane is perpendicular to the z-axis of the radar coordinate system, and determine the rotation transformation matrix corresponding to the rotated first fitting plane; then use the boundary extraction algorithm based on curvature change in PCL (Point Cloud Library) to extract multiple edge point clouds in the rotated first fitting plane; the transportation device can use the Ransac 2D circle fitting algorithm to perform circle fitting on the multiple edge point clouds to determine the third center coordinates corresponding to each round hole; and according to the rotation transformation matrix, perform reverse rotation transformation on each third center coordinate obtained by fitting to obtain the second center coordinates corresponding to each round hole.

[0186] Among them, PCL is an open-source three-dimensional point cloud processing library dedicated to processing two-dimensional / three-dimensional images and point cloud data, and can be used for processing point cloud data, such as filtering, segmentation, feature extraction, etc.; and registration algorithms such as the ICP (Iterative Closest Point) algorithm in the PCL library can find the best matching relationship between two point clouds through an iterative optimization process, so as to determine the external parameters between the lidar and the camera device.

[0187] In the embodiment of the present application, the transportation device can perform plane fitting and rotation processing on the target point cloud data. After rotating the first fitting plane to be perpendicular to the z-axis of the radar coordinate system, the change of a certain axis (such as the z-axis) can be temporarily not considered, thereby reducing the calculation dimension and complexity, being able to obtain more accurate center coordinates, and accelerating the extraction speed of the center coordinates, and improving the data processing efficiency.

[0188] Figure 12 It is a structural block diagram of the transportation device in an embodiment. As Figure 12As shown, the transportation device 1200 may include one or more of the following components: a processor 1210, and a memory 1220 coupled to the processor 1210. The memory 1220 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above various embodiments when executed by the one or more processors 1210.

[0189] The processor 1210 may include one or more processing cores. The processor 1210 connects various parts within the entire transportation device 1200 through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 1220, and by calling data stored in the memory 1220, it performs various functions of the transportation device 1200 and processes data. Optionally, the processor 1210 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1210 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the displayed content; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor 1210 and may be implemented separately through a communication chip.

[0190] The memory 1220 may include random access memory (RAM), and may also include read-only memory. The memory 1220 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1220 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above various method embodiments, etc. The data storage area may also store data created during the use of the transportation device 1200.

[0191] Understandably, the transportation device 1200 may include more or fewer structural elements than those shown in the above structural block diagram. For example, it may include a power supply, input buttons, a camera, a speaker, a screen, an RF (Radio Frequency) circuit, a Wi-Fi (Wireless Fidelity) module, a Bluetooth module, sensors, etc., and no further limitation is provided here.

[0192] An embodiment of the present application discloses a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, it implements the methods described in the above embodiments.

[0193] An embodiment of the present application discloses a computer program product that includes a computer program. When the computer program is executed by a processor, it implements the methods described in the above embodiments.

[0194] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above methods. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), etc.

[0195] Any reference to memory, storage, database, or other media as used herein may include non-volatile and / or volatile memory. Suitable non-volatile memory may include ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which acts as an external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as Static RAM (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), and Direct Rambus DRAM (DRDRAM).

[0196] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures, or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0197] In various embodiments of the present application, it should be understood that the magnitude of the sequence numbers of the above processes does not necessarily mean the inevitable sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0198] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0199] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional unit in the embodiments of the present application may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0201] When the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc., specifically, the processor in the computer device) to execute some or all of the steps of the above methods in the various embodiments of the present application.

[0202] The above has introduced in detail a calibration method, a transportation device, and a computer program product disclosed in the embodiments of the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A calibration method, characterized in that: Applied to transportation equipment, the transportation equipment is provided with a laser radar and a camera device; the method comprises: Acquire the target point cloud data collected by the laser radar, and acquire the target image matching the target point cloud data obtained by the camera device performing image acquisition on the external parameter calibration plate; the target point cloud data is the point cloud data corresponding to the external parameter calibration plate; the external parameter calibration plate includes at least one circular hole; Detecting a circular image contained in the target image to obtain at least one circular hole region contained in the target image; Positioning the center of each circular hole area to determine the first center coordinates corresponding to each circular hole in the camera device coordinate system; the camera device coordinate system is a two-dimensional coordinate system with the optical center position of the camera device as the origin; According to the fitting circle algorithm, the target point cloud data is fitted to obtain the second circle center coordinates corresponding to each of the circular holes in the radar coordinate system; the radar coordinate system is a three-dimensional coordinate system with the laser radar as the origin; The external parameters between the laser radar and the camera device are determined according to the first center coordinates and the second center coordinates corresponding to each of the circular holes.

2. The method according to claim 1, characterized in that The step of acquiring the target point cloud data collected by the laser radar comprises: Acquire historical point cloud data collected by the laser radar at at least one historical collection moment; The historical point cloud data collected at the at least one historical collection moment is fused with the current frame point cloud data collected by the laser radar at the current moment to obtain the target point cloud data.

3. The method according to claim 2, characterized in that The transport equipment is also provided with an odometer and a gyroscope; the historical point cloud data collected at the at least one historical collection moment is fused with the current frame point cloud data collected by the laser radar at the current moment to obtain the target point cloud data, including: Determine, based on the position data collected by the odometer and the attitude data collected by the gyroscope, the attitude data corresponding to the transportation equipment at the at least one historical collection time and the current time; According to the posture data corresponding to each of the historical collection moments and the posture data corresponding to the current moment, the historical point cloud data collected at each of the historical collection moments are mapped to the current moment, and the mapped historical point cloud data are fused with the current frame point cloud data collected at the current moment to obtain the target point cloud data.

4. The method according to claim 3, characterized in that The at least one historical collection moment includes M historical collection moments, where M is an integer greater than 1; mapping the historical point cloud data collected at each of the historical collection moments to the current moment according to the posture data corresponding to each of the historical collection moments and the posture data corresponding to the current moment, and fusing the mapped historical point cloud data with the current frame point cloud data collected at the current moment to obtain target point cloud data, including: According to the posture data corresponding to the Nth historical collection moment and the posture data corresponding to the N+1th historical collection moment, the fused point cloud data corresponding to the Nth historical collection moment is mapped to the N+1th historical collection moment to obtain the mapped historical point cloud data corresponding to the Nth historical collection moment; N is an integer greater than 0 and less than M, wherein the fused point cloud data corresponding to the 1st historical collection moment is the historical point cloud data collected at the 1st historical collection moment; Fusing the mapped historical point cloud data corresponding to the Nth historical collection moment with the historical point cloud data collected at the N+1th historical collection moment to obtain fused point cloud data corresponding to the N+1th historical collection moment; According to the posture data corresponding to the Mth historical collection moment and the posture data corresponding to the current moment, mapping the fused point cloud data corresponding to the Mth historical collection moment to the current moment, to obtain the mapped historical point cloud data corresponding to the Mth historical collection moment; The mapped historical point cloud data corresponding to the Mth historical collection moment is fused with the current frame point cloud data collected at the current moment to obtain target point cloud data.

5. The method according to claim 4, characterized in that The method maps the fused point cloud data corresponding to the Nth historical collection moment to the N+1th historical collection moment according to the posture data corresponding to the Nth historical collection moment and the posture data corresponding to the N+1th historical collection moment to obtain the mapped historical point cloud data corresponding to the Nth historical collection moment, including: Performing an inverse transformation on the posture data corresponding to the Nth historical collection moment to obtain the inverse posture data corresponding to the Nth historical collection moment; The inverse posture data corresponding to the Nth historical collection moment, the fused point cloud data corresponding to the Nth historical collection moment, and the posture data corresponding to the N+1th historical collection moment are point-multiplied to obtain the mapped historical point cloud data corresponding to the Nth historical collection moment.

6. The method according to claim 3, characterized in that According to the posture data corresponding to each of the historical collection moments and the posture data corresponding to the current moment, the historical point cloud data collected at each of the historical collection moments is mapped to the current moment, and the mapped historical point cloud data is fused with the current frame point cloud data collected at the current moment to obtain the target point cloud data, including: Performing an inverse transformation on the posture data corresponding to each of the historical collection moments to obtain inverse posture data corresponding to each of the historical collection moments; Performing point multiplication on the inverse posture data corresponding to each of the historical collection moments, the historical point cloud data collected at each of the historical collection moments, and the posture data corresponding to the current moment to obtain the mapped historical point cloud data corresponding to each of the historical collection moments; The mapped historical point cloud data corresponding to each of the historical collection moments are fused with the current frame point cloud data collected at the current moment to obtain target point cloud data.

7. The method according to claim 1, characterized in that The performing of center positioning on each of the circular hole regions to determine first center coordinates corresponding to each of the circular holes in the camera device coordinate system includes: Locating the center of each circular hole area to obtain third center coordinates corresponding to each circular hole area in a pixel coordinate system; the pixel coordinate system is used to represent the coordinate system of each pixel position in the image; Based on the internal parameters corresponding to the camera device, the third center coordinates corresponding to each of the circular hole areas are transferred to the coordinate system of the camera device to obtain the first center coordinates corresponding to each of the circular holes.

8. The method according to claim 7, characterized in that Before transferring the third center coordinates corresponding to each of the circular hole regions to the coordinate system of the camera device based on the internal parameters corresponding to the camera device to obtain the first center coordinates corresponding to each of the circular holes, the method further includes: Acquire a first image obtained by acquiring an image of the intrinsic reference calibration plate by the camera device; Performing feature extraction on the intrinsic reference calibration plate contained in the first image to obtain a plurality of feature points corresponding to the intrinsic reference calibration plate; An internal parameter corresponding to the camera device is calculated based on the multiple feature points.

9. The method according to claim 1, characterized in that: The detecting of the circular image contained in the target image to obtain at least one circular hole region contained in the target image includes: The circular image contained in the image area corresponding to the external parameter calibration plate is detected to obtain at least one circular hole area contained in the external parameter calibration plate.

10. The method according to claim 9, characterized in that Before detecting the circular image contained in the target image to obtain at least one circular hole area contained in the target image, the method further includes: Identifying an external parameter calibration plate contained in the target image, and determining position information of the external parameter calibration plate in the target image; According to the position information, image segmentation is performed on the extrinsic calibration plate contained in the target image to obtain an image area corresponding to the extrinsic calibration plate.

11. The method according to claim 1, characterized in that: The detecting of the circular image contained in the target image to obtain at least one circular hole region contained in the target image includes: According to an edge detection algorithm, the circular area contained in the target image is detected to obtain at least one circular hole area in the target image; and / or, The performing of center positioning on each of the circular hole regions to determine first center coordinates corresponding to each of the circular holes in the camera device coordinate system includes: Based on the Hough circle detection algorithm, the center of each circular hole area is located to determine the first center coordinates corresponding to each circular hole in the camera device coordinate system.

12. The method according to claim 1, characterized in that The step of fitting the target point cloud data according to the fitting circle algorithm to obtain the second circle center coordinates corresponding to each of the circular holes in the radar coordinate system includes: Performing plane fitting on the target point cloud data to obtain a first fitting plane corresponding to the external parameter calibration plate; Determining a plurality of edge point clouds according to the curvature of each point cloud included in the first fitting plane; According to the fitting circle algorithm, each edge point cloud is fitted to determine the second circle center coordinates corresponding to each circular hole in the radar coordinate system.

13. The method according to claim 12, characterized in that The performing plane fitting on the target point cloud data to obtain a first fitting plane corresponding to the external parameter calibration plate includes: Determining a road surface normal vector corresponding to the transportation equipment; According to the road surface normal vector, each point cloud included in the target point cloud data is screened to obtain a plurality of candidate point clouds corresponding to the extrinsic calibration plate; Perform plane fitting on the multiple candidate point clouds to obtain a first fitting plane corresponding to the extrinsic calibration plate.

14. The method according to claim 13, characterized in that The step of screening each point cloud included in the target point cloud data according to the road surface normal vector to obtain a plurality of candidate point clouds corresponding to the extrinsic calibration plate includes: Acquire K neighboring point clouds of a first point cloud from the target point cloud data, where the first point cloud is any point cloud in the target point cloud data, and K is an integer greater than 1; the neighboring point cloud is a point cloud in the target point cloud data whose distance from the first point cloud is less than a first distance threshold; Performing plane fitting on the first point cloud and the K neighboring point clouds using the least squares method to obtain a second fitting plane, and determining a plane normal vector corresponding to the second fitting plane; If the angle between the road surface normal vector and the plane normal vector is greater than a preset angle threshold, the first point cloud is used as a candidate point cloud corresponding to the extrinsic parameter calibration plate.

15. The method according to claim 13, characterized in that The performing plane fitting on the multiple candidate point clouds to obtain a first fitting plane corresponding to the extrinsic calibration plate includes: Arbitrarily obtain P target candidate point clouds from the multiple candidate point clouds, where P is a positive integer greater than 2 and less than S, and S is the number of the candidate point clouds; Performing plane fitting on the P target candidate point clouds to obtain a third fitting plane; Calculating a first distance between each of the candidate point clouds and the third fitting plane; Determine the number of point clouds of the candidate point clouds whose first distance is less than a second distance threshold; Repeat the step of arbitrarily acquiring P target candidate point clouds from the multiple candidate point clouds Q times, respectively obtaining Q third fitting planes, and the number of point clouds of the candidate point clouds whose first distance from each of the third fitting planes is less than the second distance threshold in each of the third fitting planes, where Q is a positive integer; Among the point clouds respectively corresponding to the Q third fitting planes, the third fitting plane with the largest number of point clouds is used as the first fitting plane corresponding to the extrinsic parameter calibration plate.

16. The method according to claim 12, characterized in that Before determining a plurality of edge point clouds according to the curvatures of the point clouds included in the first fitting plane, the method further includes: Rotate the first fitting plane so that the rotated first fitting plane is perpendicular to the z-axis of the radar coordinate system, and determine a rotation transformation matrix corresponding to the rotated first fitting plane; The step of determining a plurality of edge point clouds according to the curvature of each point cloud included in the first fitting plane comprises: A plurality of edge point clouds are determined according to the curvatures of the point clouds included in the rotated first fitting plane.

17. The method according to claim 16, characterized in that The step of fitting each edge point cloud according to the fitting circle algorithm to determine the second circle center coordinates corresponding to each circular hole in the radar coordinate system includes: According to the circle fitting algorithm, circle fitting is performed on each of the edge point clouds to obtain target edge point clouds constituting each of the circular holes, and the third circle center coordinates corresponding to each of the circular holes are determined according to the target edge point clouds of each of the circular holes; According to the rotation transformation matrix, each of the third circle center coordinates is subjected to a reverse rotation transformation to obtain the second circle center coordinates corresponding to each of the circular holes in the radar coordinate system.

18. The method according to claim 1, characterized in that The positioning of the center of each circular hole region to determine the first center coordinates corresponding to each circular hole includes: Identify the first center coordinates of a first circular hole area; the first circular hole area is any circular hole area in the external parameter calibration plate; According to the physical position relationship between the first circular hole area and other circular hole areas in the external parameter calibration plate, the first circle center coordinates respectively corresponding to the other circular hole areas are calculated.

19. A transport device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 18.

20. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images