Sensor calibration method and device, calibration board and vehicle

Abstract

The invention discloses a sensor calibration method and device, a calibration board and a vehicle. A recognizable pattern fixed relative to a calibration plate in position and a circular hole are arranged on the calibration plate, image data and radar point cloud data of the calibration plate are acquired through a sensor, first coordinates of angular points of the recognizable pattern in the image data are acquired, and a theoretical position of the calibration plate is acquired according to the radar point cloud data. And determining a second coordinate of the angular point of the recognizable pattern in the point cloud data according to the theoretical position of the calibration plate, and determining external parameters of the image sensor and the radar according to the first coordinate and the second coordinate. Therefore, detection features can be provided for the image sensor and the radar at the same time through one calibration plate, and the accuracy of sensor calibration is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor calibration, and in particular to a sensor calibration method, a sensor calibration device, a calibration plate and a vehicle. Background Art

[0002] With the development of science and technology, many vehicles are equipped with various functions, such as automatic driving, assisted driving, automatic parking, panoramic imaging, etc. The realization of these functions depends on various sensors installed on the vehicle, such as image sensors and lidar. In order to improve the accuracy of various functions and the safety of the vehicle, it is necessary to calibrate the vehicle's sensors to obtain the external parameter information between each sensor. At present, a common way to calibrate the sensor is to detect the calibration plate through the sensor, and determine the external parameter information between each sensor according to the detection results of each sensor. However, the features of the calibration plate in the prior art are relatively simple and cannot meet the data requirements for multi-sensor calibration, resulting in large errors in the calibration results. Summary of the invention

[0003] In view of this, an object of an embodiment of the present invention is to provide a sensor calibration method, device, calibration plate and vehicle, which can provide detection features for both image sensors and radars through a calibration plate, thereby improving the accuracy of sensor calibration.

[0004] In a first aspect, an embodiment of the present invention provides a sensor calibration method, the method comprising:

[0005] Acquire detection data of a calibration plate through a sensor, wherein the calibration plate includes a recognizable pattern and a circular hole, and the relative positions of the recognizable pattern and the circular hole to the calibration plate are fixed, and the detection data includes image data and radar point cloud data;

[0006] Acquire first coordinates according to the identifiable pattern in the image data, where the first coordinates are coordinates of a corner point of the identifiable pattern in the image data;

[0007] Acquiring the theoretical position of the calibration plate according to the radar point cloud data;

[0008] Determining a second coordinate according to the theoretical position of the calibration plate and the relative positions of the identifiable pattern and the circular hole to the calibration plate, wherein the second coordinate is a three-dimensional coordinate of a corner point of the identifiable pattern in the radar point cloud data; and

[0009] The external parameters of the image sensor and the radar are determined according to the first coordinate and the second coordinate.

[0010] In some embodiments, obtaining the theoretical position of the calibration plate according to the radar point cloud data includes:

[0011] Determining intermediate point cloud data according to the size of the calibration plate and the radar point cloud data, wherein the intermediate point cloud data is used to characterize the point cloud data within the contour of the calibration plate;

[0012] Determining an offset according to the intermediate point cloud data; and

[0013] The theoretical position of the calibration plate is obtained according to the offset and the intermediate point cloud data.

[0014] In some embodiments, determining the intermediate point cloud data according to the size of the calibration plate and the radar point cloud data comprises:

[0015] Preprocessing the radar point cloud data to obtain first point cloud data; and

[0016] The intermediate point cloud data is determined according to the size of the calibration plate and the first point cloud data.

[0017] In some embodiments, preprocessing the radar point cloud data to obtain first point cloud data includes:

[0018] Processing the radar point cloud data according to the position information of the calibration plate to obtain point cloud data of an area of ​​interest;

[0019] Performing plane segmentation on the point cloud data of the region of interest to obtain first plane point cloud data, where the first plane point cloud data is the point cloud data of the plane where the calibration plate is located;

[0020] Projecting and transforming the first plane point cloud data to obtain second plane point cloud data, where the second plane point cloud data is point cloud data of a plane parallel to the plane where the calibration plate is located; and

[0021] The second plane point cloud data is rasterized and converted into a binary image to obtain the first point cloud data.

[0022] In some embodiments, determining the intermediate point cloud data according to the size of the calibration plate and the first point cloud data comprises:

[0023] Setting a search box according to the size of the calibration plate;

[0024] Searching the first point cloud data through the search box to obtain a target area, the target area being the area where the search box is located when the number of point clouds in the search box is the largest; and

[0025] The point cloud data within the target area is determined as the intermediate point cloud data.

[0026] In some embodiments, determining the offset according to the intermediate point cloud data comprises:

[0027] Determining a theoretical position of the circular hole in the intermediate point cloud data according to the position of the circular hole in the calibration plate;

[0028] Screening the circular holes according to the theoretical positions of the circular holes to obtain a first candidate circular hole;

[0029] Determine the second candidate circular holes corresponding to each first candidate circular hole; and

[0030] The offset is determined according to the first candidate circular hole and the second candidate circular hole.

[0031] In some embodiments, screening the circular holes according to the theoretical positions of the circular holes to obtain the first candidate circular holes comprises:

[0032] Acquire a predetermined number of sampling points at the edge of the theoretical position of the circular hole;

[0033] Obtain the difference in point cloud density around each sampling point; and

[0034] The circular hole whose difference value is less than or equal to a predetermined threshold is determined as the first candidate circular hole.

[0035] In some embodiments, the step of determining the second candidate circular holes corresponding to the first candidate circular holes is specifically as follows:

[0036] According to the shape of the circular hole, a search is performed around each of the first candidate circular holes to obtain an area with the least number of point clouds as the second candidate circular hole.

[0037] In some embodiments, determining the offset according to the first candidate circular hole and the second candidate circular hole comprises:

[0038] Determining a weight value of each first candidate circular hole according to the point cloud density in the first candidate circular hole;

[0039] Determining an initial offset according to the positions of the first candidate circular holes and the second candidate circular holes; and

[0040] The offset is determined according to the initial offset and the weight value.

[0041] In some embodiments, determining the second coordinate according to the theoretical position of the calibration plate and the relative positions of the identifiable pattern and the circular hole and the calibration plate comprises:

[0042] Determining a third coordinate according to the theoretical position of the calibration plate, the third coordinate being a two-dimensional coordinate of a corner point of a recognizable pattern obtained according to the radar point cloud data; and

[0043] The second coordinate is obtained according to the third coordinate.

[0044] In a second aspect, an embodiment of the present invention provides a sensor calibration device, the device comprising:

[0045] A detection data acquisition unit, used to acquire detection data of a calibration plate through a sensor, wherein the calibration plate includes a recognizable pattern and a circular hole, wherein the recognizable pattern and the circular hole have a fixed positional relationship, and wherein the detection data includes image data and radar point cloud data;

[0046] A first coordinate acquisition unit, configured to acquire first coordinates according to the identifiable pattern in the image data, wherein the first coordinates are coordinates of a corner point of the identifiable pattern in the image data;

[0047] A theoretical position acquisition unit, used for acquiring the theoretical position of the calibration plate according to the radar point cloud data;

[0048] A second coordinate acquisition unit, used to determine a second coordinate according to a theoretical position of the calibration plate and a relative position between the identifiable pattern and the circular hole and the calibration plate, wherein the second coordinate is a three-dimensional coordinate of a corner point of the identifiable pattern in the point cloud data; and

[0049] An external parameter calibration unit is used to determine the external parameters of the image sensor and the radar according to the first coordinate and the second coordinate.

[0050] In a third aspect, an embodiment of the present invention provides a calibration plate, the calibration plate comprising:

[0051] at least one recognizable pattern; and

[0052] at least one circular hole;

[0053] Wherein, the recognizable pattern and the circular hole have a fixed positional relationship.

[0054] In some embodiments, the calibration plate includes a plurality of regions, each region being configured as the identifiable pattern or circular hole.

[0055] In some embodiments, the calibration plate is square, and the calibration plate includes N*N square areas, and the N*N square areas are distributed in N rows and N columns, where N is an integer greater than 1.

[0056] In some embodiments, the value of N is 3, the three square areas in the first row of the calibration plate are, from left to right, a recognizable pattern, a circular hole, and a recognizable pattern, the three square areas in the second row are, from left to right, a circular hole, a recognizable pattern, and a circular hole, and the three square areas in the third row are, from left to right, a recognizable pattern, a circular hole, and a recognizable pattern;

[0057] Wherein, each of the identifiable patterns carries different information.

[0058] In a fourth aspect, an embodiment of the present invention provides a vehicle, the vehicle comprising:

[0059] An image sensor, used for acquiring image data;

[0060] Radar, used to acquire radar point cloud data; and

[0061] A memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0062] In a fifth aspect, an embodiment of the present invention provides a sensor calibration system, the system comprising:

[0063] a calibration plate, at least one identifiable pattern and at least one circular hole, wherein the identifiable pattern and the circular hole have a fixed positional relationship; and

[0064] A vehicle comprises an image sensor, a radar, a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect.

[0065] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0066] The technical solution of the embodiment of the present invention is to set a recognizable pattern and a circular hole whose relative position to the calibration plate is fixed on the calibration plate, obtain the image data and radar point cloud data of the calibration plate through the sensor, obtain the first coordinates of the corner point of the recognizable pattern in the image data, obtain the theoretical position of the calibration plate according to the radar point cloud data, determine the second coordinates of the corner point of the recognizable pattern in the point cloud data according to the theoretical position of the calibration plate, and determine the external parameters of the image sensor and the radar according to the first coordinate and the second coordinate. In this way, detection features can be provided for both the image sensor and the radar through a calibration plate, thereby improving the accuracy of sensor calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0068] Figure 1 is a schematic diagram of a sensor calibration system according to an embodiment of the present invention;

[0069] Figure 2 is a schematic diagram of a calibration plate according to an embodiment of the present invention;

[0070] Figure 3 is a schematic diagram of an image sensor detection calibration plate according to an embodiment of the present invention;

[0071] Figure 4 is a schematic diagram of a calibration plate according to another embodiment of the present invention;

[0072] Figure 5 is a circuit diagram of a vehicle according to an embodiment of the present invention;

[0073] Figure 6 is a flow chart of a sensor calibration method according to an embodiment of the present invention;

[0074] Figure 7 is a flow chart of obtaining a theoretical position of a calibration plate according to an embodiment of the present invention;

[0075] Figure 8 is a flow chart of determining intermediate point cloud data according to radar point cloud data according to an embodiment of the present invention;

[0076] Fig. 9 is a flow chart of preprocessing radar point cloud data according to an embodiment of the present invention;

[0077] Fig.10 is a flow chart of determining intermediate point cloud data from first point cloud data according to an embodiment of the present invention;

[0078] Fig.11 is a schematic diagram of point cloud density according to an embodiment of the present invention;

[0079] Fig.12 is a flow chart of determining an offset according to intermediate point cloud data according to an embodiment of the present invention;

[0080] Fig.13 is a flow chart of obtaining a first candidate circular hole according to an embodiment of the present invention;

[0081] Fig.14 is a flow chart of determining an offset according to a first candidate circular hole and a second candidate circular hole according to an embodiment of the present invention;

[0082] Fig.15 is a flow chart of obtaining the second coordinate according to an embodiment of the present invention;

[0083] Fig.16 Schematic diagram of a sensor calibration device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.

[0085] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0086] At the same time, it should be understood that in the following description, "circuit" refers to a conductive loop composed of at least one element or subcircuit through electrical connection or electromagnetic connection. When an element or circuit is said to be "connected to" another element or an element / circuit is said to be "connected between" two nodes, it can be directly coupled or connected to another element or there can be an intermediate element, and the connection between the elements can be physical, logical, or a combination thereof. On the contrary, when an element is said to be "directly coupled to" or "directly connected to" another element, it means that there is no intermediate element between the two.

[0087] Unless the context clearly requires otherwise, the words "include", "comprising" and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning "including but not limited to".

[0088] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0089] Vehicles equipped with functions such as autonomous driving, assisted driving, automatic parking, and panoramic imaging are equipped with multiple sensors, which are used to achieve the above functions through perception fusion of data detected by multiple sensors. The accuracy of perception fusion of multiple sensors is highly dependent on the external parameters between sensors. Therefore, in many scenarios, the external parameters between sensors need to be calibrated to improve the accuracy of perception fusion. For example, external parameter calibration is required before the vehicle leaves the factory or when the vehicle is damaged and the sensor needs to be replaced.

[0090] Therefore, the embodiments of the present invention provide a sensor calibration method, device, calibration board and vehicle to achieve external parameter calibration between sensors.

[0091] Figure 1 Schematic diagram of a sensor calibration system according to an embodiment of the present invention. Figure 1In the illustrated embodiment, the sensor calibration system includes at least one calibration plate and a vehicle 2. The vehicle 2 is provided with a variety of sensors, and at least one calibration plate is provided in a detectable area of ​​the sensor.

[0092] exist Figure 1 In the illustrated embodiment, the number of calibration plates is 5, namely 11, 12, 13, 14, and 15 in the figure.

[0093] Specifically, it is assumed that three image sensors are provided on the vehicle 2, Q1, Q2 and Q3 are the fields of view of the three image sensors respectively, and calibration plates of different distances are placed in the field of view of each camera, wherein calibration plate 11 is provided in the field of view Q1, calibration plate 13 is provided in the field of view Q2, calibration plate 15 is provided in the field of view Q3, and calibration plate 12 is provided in the common area of ​​the field of view Q1 and Q2, and calibration plate 13 is provided in the common area of ​​the field of view Q3 and Q3. That is, at least two calibration plates are provided in each field of view, and the two calibration plates are at different distances from the vehicle. On the one hand, calibration plates of different distances can improve the diversity of calibration points, and on the other hand, calibration plates at close distances occupy a larger proportion in the field of view of the camera, so that the calibration points cover the field of view of the camera.

[0094] It should be understood that Figure 1 The setting of the calibration plate in the sensor calibration system shown is only explained by taking the field of view of the image sensor as an example. In actual use, the position of the calibration plate can be set by considering the detection range of the image sensor and the radar at the same time.

[0095] At the same time, the present invention can build a calibration room based on the calibration plate, and can configure the number, placement and angle of the calibration plate according to the sensor of the mass-produced vehicle model. Among them, the arrangement of the calibration plate must meet the following principles: the field of view of each phase image sensor should contain multiple calibration plates to increase the diversity of calibration points, and the calibration points should cover the field of view of the image sensor as evenly as possible.

[0096] Figure 2 Schematic diagram of a calibration plate according to an embodiment of the present invention. Figure 2 In the illustrated embodiment, the calibration plate includes at least one identifiable pattern and at least one circular hole. The relative positions of the identifiable pattern and the circular hole to the calibration plate are fixed. That is, the relative positions of the identifiable pattern and the calibration plate are fixed, and the relative positions of the circular hole and the calibration plate are fixed.

[0097] In some embodiments, the calibration plate includes a plurality of regions, each region being configured as the identifiable pattern or circular hole.

[0098] In some embodiments, the calibration plate is square, and the calibration plate includes N*N square areas, and the N*N square areas are distributed in N rows and N columns, where N is an integer greater than 1.

[0099] In some embodiments, the calibration plate has a size of 1000 mm*1000 mm.

[0100] exist Figure 2 In the illustrated embodiment, N=3 is used as an example for explanation. That is to say, the three square areas in the first row of the calibration plate are, from left to right, recognizable pattern A1, circular hole A2, and recognizable pattern A3; the three square areas in the second row are, from left to right, circular hole A4, recognizable pattern A5, and circular hole A6; the three square areas in the third row are, from left to right, recognizable pattern A7, circular hole A8, and recognizable pattern A9. Among them, the information carried by each of the recognizable patterns is different. In the embodiment of the present invention, the recognizable patterns A1, A3, A5, A7, and A9 are respectively referred to as the first recognizable pattern, the second recognizable pattern, the third recognizable pattern, the fourth recognizable pattern, and the fifth recognizable pattern.

[0101] In some embodiments, the recognizable pattern can be implemented by ArUco code. ArUco code is an iconic visual code that uniquely identifies a code through different black and white patterns, and each ArUco code has a unique ID number. ArUco code is mainly used in image processing and computer vision for detection, recognition and positioning. It is often attached to the surface of objects. The camera's posture can be estimated by recognizing the ArUco code through the camera. The recognition of ArUco code is to identify the ID by matching the known pattern in the library. No decoding is required. After recognition, the ID number of the ArUco code is directly returned. The ArUco code has a predefined pattern library, which can directly generate code maps with different ID numbers. Thus, camera features can be provided by recognizable patterns, and the four corner points of each ArUco code can provide four image feature points. In addition, the information encoded by the ArUco code can be used to define the position of the ArUco code in the calibration plate.

[0102] In this embodiment, the circular holes are formed by drilling holes in some areas of the calibration plate, that is, the circular holes are through holes. Since the radar measures the target distance by transmitting radar signals and calculating their flight time, and senses the environment by scanning different angles to construct a three-dimensional point cloud, the circular holes can provide radar features to assist the radar in accurately locating the calibration plate.

[0103] exist Figure 2 In the illustrated embodiment, the calibration plate is divided into a 3*3 nine-square grid, and the circular holes are distributed in a cross shape in the nine-square grid. Multiple circular hole features can provide richer point cloud features and improve the point cloud detection accuracy.

[0104] The ArUco codes are distributed at the center and four corners of the square calibration plate. The ArUco codes at the corners can increase the coverage of the calibration points in the camera's field of view. Multiple ArUco codes can provide more abundant calibration points and reduce calibration errors.

[0105] Since most vehicles are equipped with image sensors in different horizontal orientations, image sensors in different orientations may have a common view area in the horizontal direction. If the calibration plate is placed in the common view area of ​​​​the two cameras, the ArUco codes in the four corners can ensure that both image sensors have the maximum number of calibration point pairs, thereby improving the utilization rate of the calibration plate.

[0106] Figure 3 Schematic diagram of an image sensor detection calibration plate according to an embodiment of the present invention. Figure 3 In the embodiment shown, Q4 is the field of view of one image sensor, and Q5 is the field of view of another image sensor. Figure 3 As shown, the field of view of the two image sensors intersect in the horizontal direction. If the calibration plate of the embodiment of the present invention is placed in the common viewing area, the two cameras can respectively detect two complete ArUco codes.

[0107] The embodiment of the present invention can provide detection features for both the image sensor and the radar by setting the calibration plate in a form that combines a recognizable pattern with a circular hole. Thus, it can provide a rich positioning feature for the sensor, fully utilize the point cloud features for positioning, and provide a sufficient number of calibration point pairs to improve the accuracy of external parameter calibration.

[0108] It should be noted that Figure 2 The calibration plate shown is only an example provided by the embodiment of the present invention. The embodiment of the present invention does not limit the specific implementation of the calibration plate, as long as the calibration plate includes both the identifiable pattern and the circular hole, and the relative positions of the identifiable pattern and the circular hole to the calibration plate are fixed. That is, the relative positions of the identifiable pattern and the calibration plate are fixed, and the relative positions of the circular hole and the calibration plate are fixed.

[0109] Figure 4 FIG. 1 is a schematic diagram of a calibration plate according to another embodiment of the present invention. Figure 4 In the illustrated embodiment, the calibration plate includes at least one identifiable pattern and at least one circular hole. The relative positions of the identifiable pattern and the circular hole to the calibration plate are fixed. That is, the relative positions of the identifiable pattern and the calibration plate are fixed, and the relative positions of the circular hole and the calibration plate are fixed.

[0110] In some embodiments, the calibration plate is square, and the calibration plate includes N*N square areas, and the N*N square areas are distributed in N rows and N columns, where N is an integer greater than 1.

[0111] In some embodiments, the calibration plate has a size of 1000 mm*1000 mm.

[0112] exist Figure 4 In the illustrated embodiment, N=2 is used as an example for explanation. The three square areas in the first row of the calibration plate are the recognizable pattern B1 and the circular hole B2 from left to right, and the three square areas in the second row are the circular hole B3 and the recognizable pattern B4 from left to right. The information carried by each of the recognizable patterns is different. In the embodiment of the present invention, the recognizable patterns B1 and B4 are respectively referred to as the sixth recognizable pattern and the seventh recognizable pattern.

[0113] Among them, the specific implementation method of the recognizable pattern and circular hole is the same as Figure 1 Similarly, the embodiments of the present invention are not described in detail here.

[0114] The embodiment of the present invention can provide detection features for both the image sensor and the radar by setting the calibration plate in a form that combines a recognizable pattern with a circular hole. Thus, it can provide a rich positioning feature for the sensor, fully utilize the point cloud features for positioning, and provide a sufficient number of calibration point pairs to improve the accuracy of external parameter calibration.

[0115] It should be understood that Figure 2 and Figure 4 The calibration plate shown is only an example provided by the embodiment of the present invention. The embodiment of the present invention does not limit the specific implementation of the calibration plate, as long as the calibration plate includes both the identifiable pattern and the circular hole, and the relative positions of the identifiable pattern and the circular hole to the calibration plate are fixed. That is, the relative positions of the identifiable pattern and the calibration plate are fixed, and the relative positions of the circular hole and the calibration plate are fixed.

[0116] For example, the recognizable pattern can also be realized by means of a QR code, a barcode, etc. For another example, the size and shape of the calibration plate can be set according to the actual situation, the size can be set to any size, and the shape can be set to a triangle, a rectangle, various regular polygons or other irregular shapes. For another example, the area where the recognizable pattern and the circular hole are located can also be various shapes. For another example, the shape of the circular hole can also be a triangular hole, a square hole, a rectangular hole, or other regular or irregular shaped holes.

[0117] Figure 5 is a circuit diagram of a vehicle according to an embodiment of the present invention. Figure 5 In the illustrated embodiment, the vehicle includes an image sensor 21 , a radar 22 , a processor 23 , and a memory 24 .

[0118] The image sensor 21 is a camera module installed on the vehicle and is used to obtain image data.

[0119] Radar 22 is used to obtain radar point cloud data. The radar 22 can be implemented by laser radar, millimeter wave radar, etc. Among them, the laser radar emits laser signals to the surroundings, then collects the reflected laser signals, and then obtains radar point cloud data through field data collection, combined navigation, point cloud solution, etc. The points describing the spatial coordinates can be represented by three-dimensional (x, y, z), and a collection of multiple three-dimensional points is called a point cloud. The millimeter wave radar includes a transmitter that can emit millimeter wave electromagnetic wave signals with a frequency of 30-300GHz. The millimeter wave signal returned after reflection by the target object is received by the radar antenna. According to the Doppler effect, there will be a frequency change between the return signal and the transmitted signal, that is, a Doppler frequency shift occurs. According to the frequency shift of the return signal and the transmitted signal, the target distance can be calculated. By processing the returned multiple distance data, a distance map of the object can be constructed, and then point cloud data can be obtained. The embodiment of the present invention is only described by taking the laser radar as an example.

[0120] The memory 24 is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the sensor calibration method according to the embodiment of the present invention.

[0121] Figure 6 FIG. 1 is a flow chart of a sensor calibration method according to an embodiment of the present invention. Figure 6 In the illustrated embodiment, the sensor calibration method comprises the following steps:

[0122] Step S100: Acquire detection data of the calibration plate through a sensor.

[0123] In this embodiment, the calibration plate includes a recognizable pattern and a circular hole, and the relative positions of the recognizable pattern and the circular hole to the calibration plate are fixed. That is, the relative positions of the recognizable pattern and the calibration plate are fixed, and the relative positions of the circular hole and the calibration plate are fixed. The calibration plate can be implemented as follows: Figure 2 or Figure 4 As shown, the embodiments of the present invention are not described in detail here.

[0124] The detection data includes image data and radar point cloud data. Specifically, the image data is acquired through an image sensor, and the radar point cloud data is acquired through a radar.

[0125] Step S200: Acquire a first coordinate according to a recognizable pattern in the image data.

[0126] In this embodiment, the first coordinates are the coordinates of the corner points of the identifiable pattern in the image data.

[0127] Specifically, after acquiring the image data, the image recognition technology is used to detect whether the image data includes a recognizable pattern. If the recognizable pattern is included, the recognizable pattern is extracted from the image data, and the coordinates of the four corner points of the recognizable pattern in the coordinate system are determined according to a predetermined coordinate system, that is, the first coordinates. The first coordinates are two-dimensional coordinates.

[0128] Furthermore, a mark corresponding to the recognizable pattern is identified, and a corresponding relationship between the mark and the first coordinate is recorded.

[0129] The vehicle pre-stores a logo of a recognizable pattern and the position of the recognizable pattern in the target plate, combined with Figure 2 , the position corresponding to the recognizable pattern A1 is the upper left corner, the position corresponding to the recognizable pattern A3 is the upper right corner, the position corresponding to the recognizable pattern A5 is the middle, the position corresponding to the recognizable pattern A7 is the lower left corner, and the position corresponding to the recognizable pattern A9 is the lower right corner. Therefore, after obtaining the mark corresponding to the recognizable pattern, the position of the recognizable pattern on the calibration plate can be determined.

[0130] Step S300: Acquire the theoretical position of the calibration plate according to the radar point cloud data.

[0131] In this embodiment, after the radar point cloud data is acquired by the laser radar, the theoretical position of the calibration plate is acquired according to the radar point cloud data.

[0132] Figure 7 FIG. 1 is a flow chart of obtaining the theoretical position of the calibration plate according to an embodiment of the present invention. Figure 7 In the illustrated embodiment, obtaining the theoretical position of the calibration plate according to the radar point cloud data comprises the following steps:

[0133] Step S310: determining intermediate point cloud data according to the size of the calibration plate and the radar point cloud data.

[0134] In this embodiment, the intermediate point cloud data is used to characterize the point cloud data within the contour of the calibration plate. Since part of the point cloud in the radar point cloud data belongs to the calibration plate and the other part is not related to the calibration plate, it is necessary to process the radar point cloud data to filter the radar point cloud data related to the calibration plate.

[0135] Specifically, Figure 8 1 is a flow chart of determining intermediate point cloud data according to radar point cloud data according to an embodiment of the present invention. Figure 8 In the illustrated embodiment, determining the intermediate point cloud data according to the size of the calibration plate and the radar point cloud data comprises the following steps:

[0136] Step S311: pre-process the radar point cloud data to obtain first point cloud data.

[0137] The process of preprocessing the radar point cloud data to obtain the first point cloud data is as follows: Fig. 9 As shown, the following steps are included:

[0138] Step S3111: Process the radar point cloud data according to the position information of the calibration plate to obtain point cloud data of an area of ​​interest.

[0139] In this embodiment, the radar point cloud data is taken as input, and the point cloud of the region of interest containing a certain calibration plate is segmented according to the prior information of the position of the calibration plate to obtain the point cloud data of the region of interest.

[0140] Step S3112: performing plane segmentation on the point cloud data of the region of interest to obtain first plane point cloud data.

[0141] In this embodiment, the point cloud data of the region of interest is plane segmented to obtain first plane point cloud data, and the first plane point cloud data is the point cloud of the calibration plate plane. That is, the first plane point cloud data is the point cloud data of the plane where the calibration plate is located. More specifically, the first plane point cloud data is the point cloud data of the plane where the calibration plate is located detected by the radar point cloud data.

[0142] The plane segmentation may be implemented based on various existing methods. For example, the plane point cloud that best matches the shape of the calibration plate is determined as the first plane point cloud data.

[0143] Step S3113: Projection transform the first plane point cloud data to obtain second plane point cloud data.

[0144] In this embodiment, since the radar signal is transmitted from a certain point and scanned within a certain angle range, that is, in a radial shape, the shape of the first plane point cloud data obtained will be deformed relative to the original calibration plate. Therefore, the first plane point cloud data is projected and transformed to obtain the second plane point cloud data. The second plane point cloud data is the point cloud data of the plane parallel to the plane where the calibration plate is located. In other words, the second plane point cloud data is the point cloud data on the plane parallel to the actual position of the calibration plate.

[0145] Specifically, when the calibration plate is identified based on the first plane point cloud data, since the calibration plate may not be perpendicular to the coordinate axis, how to find the four vertices of the square calibration plate is a problem to be solved. Since the calibration plate must be flat, three-dimensional data is difficult to process, so it is necessary to map the three-dimensional plane point cloud to a two-dimensional plane and discard one dimension.

[0146] The plane where the first plane point cloud data is located is recorded as the first plane, and the first plane is a three-dimensional plane. The three-dimensional coordinates of three points are randomly selected in the first plane point cloud data. The plane equation of the first plane can be calculated based on the plane coordinates of the three selected points, which is recorded as ax+by+cz+d=0. At this time, the normal vector of the first plane is (a, b, c). Assuming that (x0, y0, z0) is any point in space, and its projection coordinates on the plane are (x, y, z), then the vector composed of the two points (x0, y0, z0) and (x, y, z) is also the normal vector of the first plane, and should be parallel to (a, b, c), so we can get:

[0147]

[0148] From this we can get:

[0149] x=a*t+x0

[0150] y=b*t+y0

[0151] z=c*t+z0

[0152] Substituting the above x, y, and z values ​​into the plane equation of the first plane, we get:

[0153]

[0154] According to the value of t, the three-dimensional plane point cloud data can be converted into two-dimensional plane point cloud data.

[0155] The plane where the second plane point cloud data is located is recorded as the second plane, and the second plane is a two-dimensional plane. If the coordinates in the x-axis direction are to be discarded, then t is substituted into y=b*t+y0 and z=c*t+z0 to obtain (y, z), which is the projection of the first plane point cloud coordinates on the yz plane, which is the second plane point cloud data.

[0156] It should be noted that, in the above-mentioned first plane point cloud data and second plane point cloud data, the first plane point cloud data is the plane point cloud data where the calibration plate is located obtained by segmenting the radar point cloud data. At this time, the plane where the calibration plate is located is the plane where the calibration plate is located in the radar point cloud data. Since the first plane point cloud data has a certain deformation, the second plane point cloud data is obtained by transforming the first plane point cloud data, that is, the second plane is a plane parallel to the plane where the calibration plate is actually located.

[0157] Step S3114: rasterize the second plane point cloud data and convert it into a binary image to obtain the first point cloud data.

[0158] In this embodiment, the second plane point cloud data is rasterized and converted into a binary image to obtain the first point cloud data.

[0159] Specifically, according to the point cloud resolution and scene size, determine the grid length, number of grids and other parameters of the grid map. Project the point cloud data from the radar coordinate system to the grid map coordinate system to determine the coordinates of each point in the grid map. According to the grid coordinates of each point, divide it into the corresponding grid unit. Perform statistics on each grid, calculate the number of points in the grid, set a threshold, and set the grid with a number of points greater than the threshold to 1, otherwise it is set to 0. Weave the grid values ​​into a matrix to obtain a binary image, 1 indicates an obstacle, 0 indicates idle, and post-process the binary image, such as opening operation denoising. Save the binary image.

[0160] Step S312: determining intermediate point cloud data according to the size of the calibration plate and the first point cloud data.

[0161] Fig.10 FIG. 1 is a flow chart of determining intermediate point cloud data from first point cloud data according to an embodiment of the present invention. Fig.10 In the illustrated embodiment, determining the intermediate point cloud data according to the size of the calibration plate and the first point cloud data comprises the following steps:

[0162] Step S3121, setting a search box according to the size of the calibration plate.

[0163] In this embodiment, information such as the actual size and shape of the calibration plate is obtained, and a search box is set according to the information such as the size and shape.

[0164] The search box is obtained by calculation according to the actual size and shape of the calibration plate and the parameters of the radar (such as the distance between the radar and the calibration plate).

[0165] Step S3122: Search the first point cloud data through the search box to obtain a target area.

[0166] In this embodiment, the first point cloud data is searched through the search box, and the area where the search box is located when the number of point clouds in the search box is the largest is obtained, and the area is determined as the target area.

[0167] Step S3123: determine the point cloud data within the target area as the intermediate point cloud data.

[0168] In this embodiment, a square search box is set according to the prior size of the calibration plate. A two-dimensional brute force search is performed on the binary image. The cost function defined by the search is the number of point clouds contained in the square box, and the search goal is to obtain a square box containing the largest number of point clouds.

[0169] Step S320: determining an offset according to the intermediate point cloud data.

[0170] In this embodiment, the distribution of the laser radar point cloud on the calibration plate is often not uniform, but rather presents the characteristics of being dense in the middle and sparse at the top and bottom. Fig.11 As shown in the figure, the point cloud density of some circular holes is different. This is because when obtaining circular holes from radar point cloud data, the circles with the least point cloud are selected. In this way, when locating these circular holes according to the point cloud density, the circular holes will be biased towards the direction of sparse point cloud, resulting in a relatively large error. Therefore, the embodiment of the present invention corrects the circular hole positioning by determining the offset through the intermediate point cloud data.

[0171] Specifically, Fig.12 1 is a flow chart of determining the offset according to the intermediate point cloud data according to an embodiment of the present invention. Fig.12 In the illustrated embodiment, determining the offset according to the intermediate point cloud data comprises the following steps:

[0172] Step S321 : determining the theoretical position of the circular hole in the intermediate point cloud data according to the position of the circular hole in the calibration plate.

[0173] In this embodiment, since the circular hole in the calibration plate has a fixed position, the theoretical position of the circular hole can be obtained by inferring based on the intermediate point cloud data obtained above.

[0174] Step S322: Screen the circular holes according to the theoretical positions of the circular holes to obtain the first candidate circular holes.

[0175] Fig.13 FIG. 1 is a flowchart of obtaining the first candidate circular hole according to an embodiment of the present invention. Fig.13 In the illustrated embodiment, screening the circular holes according to the theoretical positions of the circular holes to obtain the first candidate circular holes comprises the following steps:

[0176] Step S3221: Acquire a predetermined number of sampling points at the edge of the theoretical position of the circular hole.

[0177] Step S3222: Obtain the difference in point cloud density around each sampling point.

[0178] Step S3223: determine the circular hole whose difference is less than or equal to the predetermined threshold as the first candidate circular hole.

[0179] For steps S3221-S3223, according to the theoretical position of each circular hole, a predetermined number of points are sampled on the edge of each circular hole, and the point cloud density near each sampling point is calculated respectively. The difference of the point cloud density around each sampling point is obtained, and the largest difference is obtained. If the largest difference is greater than a predetermined threshold, the corresponding circular hole is discarded. If the largest difference is less than or equal to the predetermined threshold, the circular hole is determined as the first candidate circular hole.

[0180] Step S323: Determine the second candidate circular holes corresponding to the first candidate circular holes.

[0181] In this embodiment, according to the shape of the circular hole, a region with the least number of point clouds is searched around each of the first candidate circular holes to obtain the region as the second candidate circular hole.

[0182] Specifically, a circular hole search is performed within a certain range around the first candidate circular hole to refine the circular hole position. The cost function defined by the search is the number of point clouds within the circular hole radius, and the search goal is to obtain a circle with the least number of point clouds. In this way, the second candidate circular hole corresponding to each first candidate circular hole can be obtained.

[0183] Step S324: determine the offset according to the first candidate circular hole and the second candidate circular hole.

[0184] Fig.14 is a flow chart of determining an offset according to a first candidate circular hole and a second candidate circular hole according to an embodiment of the present invention. Fig.14 In the illustrated embodiment, determining the offset according to the first candidate circular hole and the second candidate circular hole comprises the following steps:

[0185] Step S3241: determine the weight value of each first candidate circular hole according to the point cloud density in the first candidate circular hole.

[0186] In this embodiment, the weight value of each first candidate circular hole is determined according to the point cloud density in the first candidate circular hole, wherein the weight value corresponding to each circular hole is positively correlated with the point cloud density, that is, the greater the point cloud density, the greater the corresponding weight value.

[0187] In some embodiments, the point cloud density may be directly determined as the weight value, or the point cloud density may be normalized to obtain a corresponding weight value.

[0188] Step S3242: determine an initial offset according to the positions of the first candidate circular holes and the second candidate circular holes.

[0189] In this embodiment, determining the initial offset according to the positions of the first candidate circular holes and the second candidate circular holes can be achieved in various existing ways, for example, taking the difference between the coordinates of the center of the first candidate circular hole and the corresponding second candidate circular hole as the initial offset. Thus, the initial offset of each circular hole can be obtained.

[0190] Step S3243: determine the offset according to the initial offset and the weight value.

[0191] In this embodiment, the initial offset and the weight value are weighted and summed to obtain the offset.

[0192] That is to say, the embodiment of the present invention infers the positions of the four circular holes according to the obtained contour positions of the calibration plate, samples a predetermined number of points on the edge of each circular hole according to the position of each circular hole, and calculates the point cloud density near the sampling points respectively. If the difference in point cloud density of the sampling points is greater than the set threshold, the circular hole feature is discarded. According to the average density of the sampled point clouds of each circular hole, a weight value corresponding to the circular hole is assigned. Within a certain range around the retained circular hole, a circular hole search is performed to refine the circular hole position. The cost function defined by the search is the number of point clouds within the radius of the circular hole, and the goal of the search is to obtain the circle containing the least number of point clouds. The search obtains the offset of the refined circular hole position relative to the original position.

[0193] Step S330: Acquire the theoretical position of the calibration plate according to the offset and the intermediate point cloud data.

[0194] In this embodiment, the intermediate point cloud data is corrected according to the offset to obtain the theoretical position of the calibration plate.

[0195] Step S400: determining a second coordinate according to the theoretical position of the calibration plate and the relative position of the identifiable pattern and the circular hole to the calibration plate.

[0196] In this embodiment, the second coordinates are the coordinates of the corner points of the identifiable pattern in the radar point cloud data.

[0197] Fig.15 is a flow chart of obtaining the second coordinates according to an embodiment of the present invention. Fig.15 In the illustrated embodiment, determining the second coordinate according to the theoretical position of the calibration plate and the relative positions of the identifiable pattern and the circular hole to the calibration plate comprises the following steps:

[0198] Step S410: determining a third coordinate according to the theoretical position of the calibration plate.

[0199] In this embodiment, the third coordinate is the two-dimensional coordinate of the corner point of the identifiable pattern obtained according to the radar point cloud data. After the theoretical position of the calibration plate is obtained, since the position of the identifiable pattern in the calibration plate is fixed, the coordinates of the corner points of each identifiable pattern, that is, the third coordinate, can be inferred from the theoretical position of the calibration plate.

[0200] Step S420: Acquire the second coordinate according to the third coordinate.

[0201] In this embodiment, the second coordinates are the three-dimensional coordinates of the corner points of the identifiable pattern in the radar point cloud data. Since the third coordinates obtained above are converted two-dimensional coordinates, it is necessary to convert the two-dimensional third coordinates into three-dimensional second coordinates.

[0202] Specifically, in step S3113, the three-dimensional coordinates in the first plane point cloud data and the two-dimensional coordinates in the second plane point cloud data corresponding to each three-dimensional coordinate can be obtained. Based on this, the transformation matrix can be calculated by the corresponding three-dimensional coordinates and two-dimensional coordinates, and the third coordinate can be inversely transformed based on the transformation matrix to obtain the second coordinate.

[0203] It should be noted that the embodiment of the present invention does not limit the implementation method of the mutual transformation between the three-dimensional coordinates and the two-dimensional coordinates of the point cloud data, and it can be implemented based on various existing methods.

[0204] Step S500: determining external parameters of the image sensor and the radar according to the first coordinates and the second coordinates.

[0205] In this embodiment, the first coordinate and the second coordinate can be obtained through the above process, the first coordinate is the coordinate of the corner point of the recognizable pattern in the image data, and the second coordinate is the three-dimensional coordinate of the corner point of the recognizable pattern in the radar point cloud data. That is, the first coordinate is the coordinate of the corner point of the recognizable pattern in the first coordinate system, and the first coordinate system is the image coordinate system. The second coordinate is the coordinate of the corner point of the recognizable pattern in the second coordinate system, and the second coordinate system is the point cloud coordinate system. The first coordinate system is a plane coordinate system, and the second coordinate system is a three-dimensional coordinate system. Therefore, the external parameters of the image sensor and the radar can be determined according to the first coordinate and the second coordinate.

[0206] Determining the external parameters of the image sensor and the radar according to the first coordinate and the second coordinate can be implemented based on various existing methods, which are not limited in the embodiments of the present invention. For example, it can be implemented based on a PnP (Perspective-n-Point) algorithm.

[0207] The relationship between the first coordinate and the second coordinate in the embodiment of the present invention can be regarded as the corresponding relationship between the 3D second coordinate and its 2D first coordinate projection. Figure 2 The calibration plate shown is used as an example for explanation, each calibration plate includes five recognizable images, each recognizable image includes four corner points, and a total of 20 corner points. Therefore, when the radar and image sensor can simultaneously detect the entire area of ​​a calibration plate, 20 first coordinates and 20 second coordinates are obtained, and each first coordinate has a corresponding second coordinate, wherein in a set of corresponding first coordinates and second coordinates, the first coordinate can be regarded as a 2D projection of the second coordinate, and the purpose of calculating the external parameters in the embodiment of the present invention is to obtain the conversion relationship between the first coordinate and the second coordinate.

[0208] In a specific implementation, it can be implemented by direct linear transformation (DLT). The first coordinate is taken as the coordinate in the first coordinate system, denoted as [uv] T , the second coordinate is taken as the coordinate in the second coordinate system, recorded as [X w Y w Z w ] T Among them, [uv] T represents the transposed matrix of the matrix [uv], [X w Y w Z w ] T Represents the matrix [X w Y w Z w ] is the transposed matrix of [uv] T The homogeneous coordinates of are expressed as [uv 1] T ,[X w Y w Z w ] T The homogeneous coordinates of w Y w Z w 1] T The external parameter of the image sensor is denoted as K, where K is pre-acquired. The perspective projection model can be expressed as:

[0209]

[0210] The purpose of the solution is to obtain the values ​​of the rotation matrix R and the translation matrix t. R and t are the external parameters of the radar and image sensor.

[0211] Expanding the above formula yields:

[0212]

[0213] Converted into the form of a system of equations, we get:

[0214] Z c u c =f 11 X w +f 12 Y w +f 13 Z w +f 14

[0215] Z c u c =f 21 X w +f22 Y w +f 23 Z w +f 24

[0216] Z c =f 31 X w +f 32 Y w +f 33 Z w +f 34

[0217] Eliminate Z c We can get:

[0218] f 11 X w +f 12 Y w +f 13 Z w +f 14 -f 31 X w u c -f 32 Y w u c -f 33 Z w u c -f 34 u c =0

[0219] f 21 X w +f 22 Y w +f 23 Z w +f 24 -f 31 X w u c -f 32 Y w u c -f 33 Z w u c -f 34 u c =0

[0220] Each set of matching points (the first coordinate and the second coordinate corresponding to the same corner point) corresponds to two equations, with a total of 12 unknowns, and at least 6 sets of matching points are required. Suppose there are N sets of matching points, and the first coordinate of the i-th set of matching points is (u i ,v i ), the second coordinate is marked as (X i ,Y i ,Zi ), i = 1, 2, 3, ..., N; eliminate Z from the above c The two formulas obtained can be converted into: E*F=0.

[0221] Where E is:

[0222]

[0223] F can be expressed as:

[0224]

[0225] In this embodiment, each calibration plate includes five identifiable images, each identifiable image includes four corner points, and a total of 20 corner points, that is, N is greater than 6.

[0226] To obtain the least squares solution under the constraint of |F|=1, we can use SVD (singular value decomposition) to solve it. The last one of the V matrix is ​​the desired solution. Specifically, perform singular value decomposition on the matrix F:

[0227] F=UDV T

[0228] The vectors in the matrix U are orthogonal, and the vectors in U are called left singular vectors. Except for the diagonal, all other elements of the matrix D are 0, and the elements on the diagonal are called singular values. T (The transposed matrix of V) The vectors in the matrix are also orthogonal, and the vectors in V are called right singular vectors.

[0229] Because F = [KR Kt]

[0230] Therefore, the calculation formula of the rotation matrix R is:

[0231]

[0232] The calculation formula of the translation matrix t is:

[0233]

[0234] Thus, the rotation matrix R and translation matrix t, that is, the external parameters of the image sensor and radar, can be obtained.

[0235] It should be understood that the calculation method given above is only an example provided by the embodiment of the present invention. The embodiment of the present invention does not limit the specific calculation method, and it can be implemented based on various existing methods. For example, it can also be implemented by direct linear transformation (Direct Linear Transform, DLT), P3P algorithm, PST (Perspective Similar Triangle, perspective similar triangle method), etc.

[0236] The embodiment of the present invention sets a recognizable pattern and a circular hole whose relative position to the calibration plate is fixed on the calibration plate, obtains the image data and radar point cloud data of the calibration plate through the sensor, obtains the first coordinates of the corner point of the recognizable pattern in the image data, obtains the theoretical position of the calibration plate according to the radar point cloud data, determines the second coordinates of the corner point of the recognizable pattern in the point cloud data according to the theoretical position of the calibration plate, and determines the external parameters of the image sensor and the radar according to the first coordinates and the second coordinates. In this way, detection features can be provided for both the image sensor and the radar through one calibration plate, thereby improving the accuracy of sensor calibration.

[0237] It should be understood that the embodiment of the present invention is explained by calibrating the external parameters between the radar and the image sensor as an example, but the sensor calibration method of the embodiment of the present invention is also applicable to the external parameter calibration between different image sensors, and the specific implementation method is similar, and the embodiment of the present invention will not be repeated here.

[0238] Fig.16 Schematic diagram of a sensor calibration device according to an embodiment of the present invention. Fig.16 In the illustrated embodiment, the calibration device of the sensor includes a detection data acquisition unit 161, a first coordinate acquisition unit 162, a theoretical position acquisition unit 163, a second coordinate acquisition unit 164, and an external parameter calibration unit 165. The detection data acquisition unit 161 is used to acquire detection data of a calibration plate through a sensor, wherein the calibration plate includes a recognizable pattern and a circular hole, wherein the recognizable pattern and the circular hole have a fixed positional relationship, and the detection data includes image data and radar point cloud data. The first coordinate acquisition unit 162 is used to acquire a first coordinate according to the recognizable pattern in the image data, wherein the first coordinate is the coordinate of the corner point of the recognizable pattern in the image data. The theoretical position acquisition unit 163 is used to acquire the theoretical position of the calibration plate according to the radar point cloud data. The second coordinate acquisition unit 164 is used to determine a second coordinate according to the theoretical position of the calibration plate and the relative position of the recognizable pattern and the circular hole to the calibration plate, wherein the second coordinate is the three-dimensional coordinate of the corner point of the recognizable pattern in the point cloud data. The external parameter calibration unit 165 is used to determine the external parameters of the image sensor and the radar according to the first coordinate and the second coordinate.

[0239] In some embodiments, the theoretical position acquisition unit includes:

[0240] An intermediate point cloud data acquisition subunit, used to determine intermediate point cloud data according to the size of the calibration plate and the radar point cloud data, wherein the intermediate point cloud data is used to characterize the point cloud data within the contour of the calibration plate;

[0241] an offset acquisition subunit, configured to determine an offset according to the intermediate point cloud data; and

[0242] The theoretical position acquisition subunit is used to acquire the theoretical position of the calibration plate according to the offset and the intermediate point cloud data.

[0243] In some embodiments, the intermediate point cloud data acquisition subunit includes:

[0244] A first point cloud data acquisition module, configured to pre-process the radar point cloud data to acquire first point cloud data; and

[0245] The intermediate point cloud data acquisition module is used to determine the intermediate point cloud data according to the size of the calibration plate and the first point cloud data.

[0246] In some embodiments, the first point cloud data acquisition module includes:

[0247] An area of ​​interest point cloud data acquisition submodule, used to process the radar point cloud data according to the position information of the calibration plate to acquire area of ​​interest point cloud data;

[0248] A first plane point cloud data acquisition submodule is used to perform plane segmentation on the point cloud data of the region of interest to acquire first plane point cloud data, where the first plane point cloud data is the point cloud data of the plane where the calibration plate is located;

[0249] A second plane point cloud data acquisition submodule, configured to project and transform the first plane point cloud data to acquire second plane point cloud data, wherein the second plane point cloud data is point cloud data of a plane parallel to the plane where the calibration plate is located; and

[0250] The first point cloud data acquisition submodule is used to rasterize the second plane point cloud data and convert it into a binary image to obtain the first point cloud data.

[0251] In some embodiments, the intermediate point cloud data acquisition module includes:

[0252] A search box setting submodule, used to set the search box according to the size of the calibration plate;

[0253] a target area acquisition submodule, configured to search the first point cloud data through the search box to acquire a target area, wherein the target area is an area where the search box is located when the number of point clouds in the search box is the largest; and

[0254] The intermediate point cloud data acquisition submodule is used to determine the point cloud data in the target area as the intermediate point cloud data.

[0255] In some embodiments, the offset acquisition subunit includes:

[0256] A theoretical position acquisition module, used to determine the theoretical position of the circular hole in the intermediate point cloud data according to the position of the circular hole in the calibration plate;

[0257] A first candidate circular hole acquisition module, used for screening the circular holes according to the theoretical positions of the circular holes to acquire the first candidate circular holes;

[0258] A second candidate circular hole acquisition module is used to determine the second candidate circular holes corresponding to each first candidate circular hole; and

[0259] An offset determination module is used to determine the offset according to the first candidate circular hole and the second candidate circular hole.

[0260] In some embodiments, the first candidate circular hole acquisition module includes:

[0261] A sampling point acquisition submodule, used to acquire a predetermined number of sampling points at the edge of the theoretical position of the circular hole;

[0262] The density difference acquisition submodule is used to obtain the difference of the point cloud density around each sampling point; and

[0263] The first candidate circular hole determination submodule is used to determine the circular hole whose difference value is less than or equal to a predetermined threshold as the first candidate circular hole.

[0264] In some embodiments, the second candidate circular hole acquisition module is specifically used to:

[0265] According to the shape of the circular hole, a search is performed around each of the first candidate circular holes to obtain an area with the least number of point clouds as the second candidate circular hole.

[0266] In some embodiments, the offset determination module includes:

[0267] A weight value determination submodule, used to determine the weight value of each first candidate circular hole according to the point cloud density in the first candidate circular hole;

[0268] an initial offset determination submodule, configured to determine an initial offset according to positions of the first candidate circular holes and the second candidate circular holes; and

[0269] An offset determination submodule is used to determine the offset according to the initial offset and the weight value.

[0270] In some embodiments, the second coordinate acquisition unit includes:

[0271] a third coordinate determining subunit, configured to determine a third coordinate according to a theoretical position of the calibration plate, wherein the third coordinate is a two-dimensional coordinate of a corner point of a recognizable pattern obtained according to the radar point cloud data; and

[0272] The second coordinate acquisition subunit is used to acquire the second coordinate according to the third coordinate.

[0273] The embodiment of the present invention sets a recognizable pattern and a circular hole whose relative position to the calibration plate is fixed on the calibration plate, obtains the image data and radar point cloud data of the calibration plate through the sensor, obtains the first coordinates of the corner point of the recognizable pattern in the image data, obtains the theoretical position of the calibration plate according to the radar point cloud data, determines the second coordinates of the corner point of the recognizable pattern in the point cloud data according to the theoretical position of the calibration plate, and determines the external parameters of the image sensor and the radar according to the first coordinates and the second coordinates. In this way, detection features can be provided for both the image sensor and the radar through one calibration plate, thereby improving the accuracy of sensor calibration.

[0274] It should be noted that in Figure 5 In the embodiment, the processor 23 and the memory 24 are connected via a bus. The memory 24 is suitable for storing instructions or programs executable by the processor 23. The processor 23 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 23 executes the instructions stored in the memory 24, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller and the display device and the input / output (I / O) device. The input / output (I / O) device can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device is connected to the system via an input / output (I / O) controller.

[0275] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices (equipment) or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0276] The present invention is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0277] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0278] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0279] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A sensor calibration method, characterized in that: The method comprises: Acquire detection data of a calibration plate through a sensor, wherein the calibration plate includes a recognizable pattern and a circular hole, and the relative positions of the recognizable pattern and the circular hole to the calibration plate are fixed, and the detection data includes image data and radar point cloud data; Acquire first coordinates according to the identifiable pattern in the image data, where the first coordinates are coordinates of a corner point of the identifiable pattern in the image data; Acquiring the theoretical position of the calibration plate according to the radar point cloud data; Determining a second coordinate according to the theoretical position of the calibration plate and the relative positions of the identifiable pattern and the circular hole to the calibration plate, wherein the second coordinate is a three-dimensional coordinate of a corner point of the identifiable pattern in the radar point cloud data; and The external parameters of the image sensor and the radar are determined according to the first coordinate and the second coordinate.

2. The method according to claim 1, characterized in that The step of obtaining the theoretical position of the calibration plate according to the radar point cloud data comprises: Determining intermediate point cloud data according to the size of the calibration plate and the radar point cloud data, wherein the intermediate point cloud data is used to characterize the point cloud data within the contour of the calibration plate; Determining an offset according to the intermediate point cloud data; and The theoretical position of the calibration plate is obtained according to the offset and the intermediate point cloud data.

3. The method according to claim 2, characterized in that Determining the intermediate point cloud data according to the size of the calibration plate and the radar point cloud data comprises: Preprocessing the radar point cloud data to obtain first point cloud data; and The intermediate point cloud data is determined according to the size of the calibration plate and the first point cloud data.

4. The method according to claim 3, characterized in that The preprocessing of the radar point cloud data to obtain first point cloud data comprises: Processing the radar point cloud data according to the position information of the calibration plate to obtain point cloud data of an area of ​​interest; Performing plane segmentation on the point cloud data of the region of interest to obtain first plane point cloud data, where the first plane point cloud data is the point cloud data of the plane where the calibration plate is located; Projecting and transforming the first plane point cloud data to obtain second plane point cloud data, where the second plane point cloud data is point cloud data of a plane parallel to the plane where the calibration plate is located; and The second plane point cloud data is rasterized and converted into a binary image to obtain the first point cloud data.

5. The method according to claim 3, characterized in that: The determining of the intermediate point cloud data according to the size of the calibration plate and the first point cloud data comprises: Setting a search box according to the size of the calibration plate; Searching the first point cloud data through the search box to obtain a target area, the target area being the area where the search box is located when the number of point clouds in the search box is the largest; and The point cloud data within the target area is determined as the intermediate point cloud data.

6. The method according to claim 2, characterized in that Determining the offset according to the intermediate point cloud data comprises: Determining a theoretical position of the circular hole in the intermediate point cloud data according to the position of the circular hole in the calibration plate; Screening the circular holes according to the theoretical positions of the circular holes to obtain a first candidate circular hole; Determine the second candidate circular holes corresponding to each first candidate circular hole; and The offset is determined according to the first candidate circular hole and the second candidate circular hole.

7. The method according to claim 6, characterized in that The step of screening the circular holes according to the theoretical positions of the circular holes to obtain the first candidate circular holes comprises: Acquire a predetermined number of sampling points at the edge of the theoretical position of the circular hole; Obtain the difference in point cloud density around each sampling point; and The circular hole whose difference value is less than or equal to a predetermined threshold is determined as the first candidate circular hole.

8. The method according to claim 6, characterized in that The method of determining the second candidate circular holes corresponding to each first candidate circular hole is specifically as follows: According to the shape of the circular hole, a search is performed around each of the first candidate circular holes to obtain an area with the least number of point clouds as the second candidate circular hole.

9. The method according to claim 6, characterized in that The determining the offset according to the first candidate circular hole and the second candidate circular hole comprises: Determining a weight value of each first candidate circular hole according to the point cloud density in the first candidate circular hole; Determining an initial offset according to the positions of the first candidate circular holes and the second candidate circular holes; and The offset is determined according to the initial offset and the weight value.

10. The method according to claim 1, characterized in that Determining the second coordinate according to the theoretical position of the calibration plate and the relative positions of the identifiable pattern and the circular hole and the calibration plate comprises: Determining a third coordinate according to the theoretical position of the calibration plate, the third coordinate being a two-dimensional coordinate of a corner point of a recognizable pattern obtained according to the radar point cloud data; and The second coordinate is obtained according to the third coordinate.

11. A sensor calibration device, characterized in that: The device comprises: A detection data acquisition unit, used to acquire detection data of a calibration plate through a sensor, wherein the calibration plate includes a recognizable pattern and a circular hole, wherein the recognizable pattern and the circular hole have a fixed positional relationship, and wherein the detection data includes image data and radar point cloud data; A first coordinate acquisition unit, configured to acquire first coordinates according to the identifiable pattern in the image data, wherein the first coordinates are coordinates of a corner point of the identifiable pattern in the image data; A theoretical position acquisition unit, used for acquiring the theoretical position of the calibration plate according to the radar point cloud data; A second coordinate acquisition unit, used to determine a second coordinate according to a theoretical position of the calibration plate and a relative position between the identifiable pattern and the circular hole and the calibration plate, wherein the second coordinate is a three-dimensional coordinate of a corner point of the identifiable pattern in the point cloud data; and An external parameter calibration unit is used to determine the external parameters of the image sensor and the radar according to the first coordinate and the second coordinate.

12. A calibration plate, characterized in that: The calibration plate comprises: at least one recognizable pattern; and at least one circular hole; Wherein, the recognizable pattern and the circular hole have a fixed positional relationship.

13. The calibration plate according to claim 12, characterized in that: The calibration plate includes a plurality of regions, each of which is configured as the identifiable pattern or circular hole.

14. The calibration plate according to claim 13, characterized in that: The calibration plate is square and includes N*N square areas, the N*N square areas are distributed in N rows and N columns, and N is an integer greater than 1.

15. The calibration plate according to claim 14, characterized in that: The value of N is 3, the three square areas in the first row of the calibration plate are, from left to right, a recognizable pattern, a circular hole, and a recognizable pattern, the three square areas in the second row are, from left to right, a circular hole, a recognizable pattern, and a circular hole, and the three square areas in the third row are, from left to right, a recognizable pattern, a circular hole, and a recognizable pattern; Wherein, each of the identifiable patterns carries different information.

16. A vehicle, characterized in that: The vehicle comprises: An image sensor, used for acquiring image data; Radar, used to acquire radar point cloud data; and A memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 10.

17. A sensor calibration system, characterized in that: The system comprises: a calibration plate, at least one identifiable pattern and at least one circular hole, wherein the identifiable pattern and the circular hole have a fixed positional relationship; and A vehicle comprises an image sensor, a radar, a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 10.

18. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 10 when executed by a processor.

Images