External parameter calibration method, data processing device, storage medium and electronic device

Abstract

The present invention provides an external parameter calibration method, a data processing device, a storage medium, and an electronic device. The external parameter calibration method includes collecting original point cloud data reflected by a QR code calibration board through a lidar, wherein a QR code calibration array is deployed on the QR code calibration board; performing image conversion on the original point cloud data to generate a first image; respectively performing edge detection on the first image and a second image captured by a camera of the QR code calibration board to generate a first edge image and a second edge image; performing edge structure analysis on the first edge image and the second edge image, and screening to obtain at least one polygon image that meets a preset condition; encoding the polygon image based on the dot matrix coordinates of each polygon image to generate an encoding result; and determining an external parameter calibration result of the QR code calibration board based on the encoding. By adopting the above technical solution, the purpose of improving the joint external parameter calibration accuracy of the camera and the lidar can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of external parameter calibration of sensors, and more particularly, to an external parameter calibration method, a data processing device, a storage medium, and an electronic device. Background Art

[0002] Although more and more mass-produced vehicles are equipped with auxiliary driving functions with cameras as the main sensors, the number of traffic accidents caused by drivers after turning on the auxiliary driving functions is also increasing. In order to improve the safety redundancy of the entire vehicle perception system, lidar, as a high-precision ranging sensor, is also about to be mass-produced and installed in vehicles. This will inevitably lead to a difficult problem that needs to be solved urgently - the spatio-temporal calibration of lidar and camera.

[0003] During the process of autonomous driving or assisted driving, the vehicle must continuously perceive the surrounding environment in real time. Its tasks include object detection and classification, multi-object tracking, and scene understanding. In the fields of object detection and scene understanding, obstacle detection, lane line detection, pedestrian recognition, traffic sign recognition, traffic light recognition, etc. are the primary tasks that need to be solved. As a visual sensor, cameras have been maturely applied in the field of object detection. In 2012, the Alexnet convolutional neural network also achieved a new breakthrough in the field of image classification. A series of 2D object detection networks based on deep learning, such as RCNN, MS-CNN, YOLO, FastRCNN, DCNN, etc., have successively improved the detection accuracy and speed. However, cameras have some limitations. The data collected is a pixel array of RGB images without depth information (distance information). Binocular cameras can perform ranging, but the algorithm is not yet mature, and the measurement error is much larger than that of professional ranging devices such as lidar. The field of view angle of the camera itself is also limited, and the camera is also greatly affected by external conditions, such as the intensity of light. Lidar can accurately collect 3D point cloud data with distance information, is not affected by light, and can perceive distances of more than 150m. However, its angular resolution is far inferior to that of cameras. In complex scenarios, such as small objects and distant objects, cameras can clearly identify targets, but the number of target object points collected by lidar is relatively small and difficult to cluster, which is not conducive to using point cloud data for object detection and is easily affected by rain, snow, sand, dust, haze, etc.

[0004] As autonomous vehicles face increasing challenges in environmental perception tasks with a single sensor and need to implement more functions, multi-sensor fusion solutions are favored by more and more researchers. Multi-sensor fusion technology can solve the limitations of a single sensor in autonomous driving. Through multi-sensor data fusion, sensors can complement each other's strengths and weaknesses, improving the effectiveness of the entire perception system. LiDAR ranging has the characteristics of high accuracy and can provide 360-degree full-field information. Due to the particularities of each sensor, they are suitable as part of the same perception system. Multi-sensor deep fusion can map 3D point cloud data onto an image, that is, some pixels on the image also have depth information, which can help the perception system perform image-based segmentation or object detection. The basic principle of sensor information fusion technology is the same as the process of the human brain's comprehensive processing of information. Information is complemented at multiple levels and in multiple spaces among sensors, and optimized combination processing is carried out to finally generate a consistent interpretation of environmental information. The ultimate goal of information fusion is to derive more effective information through multi-level and multi-faceted combinations of information based on the separate observation information obtained by each sensor. Sensor fusion utilizes the advantages of multiple sensors operating in cooperation with each other and also comprehensively processes data from other information sources to improve the intelligence of the entire perception system. Multi-sensor joint calibration is the prerequisite for multi-sensor data fusion and is the core issue in research fields such as mobile robots and unmanned driving technology. However, there are still problems of low accuracy and low efficiency in current multi-sensor joint calibration. Summary of the Invention

[0005] The main object of the present invention is to provide an external parameter calibration method, a data processing device, a storage medium, and an electronic device to solve the problem of low accuracy in multi-sensor joint calibration in the prior art.

[0006] According to one aspect of an embodiment of the present invention, there is provided an external parameter calibration method, a data processing device, a storage medium, and an electronic device applied to a vehicle. The method includes: collecting original point cloud data reflected by a QR code calibration board through a LiDAR, wherein a QR code calibration array is deployed on the QR code calibration board; performing image conversion on the original point cloud data to generate a first image; respectively performing edge detection on the first image and a second image captured by a camera of the QR code calibration board to generate a first edge image and a second edge image; performing edge structure analysis on the first edge image and the second edge image, and screening to obtain at least one polygon image that meets a preset condition; encoding the polygon image based on the dot matrix coordinates of each polygon image to generate an encoding result; and determining an external parameter calibration result of the QR code calibration board based on the encoding result.

[0007] Optionally, perform image conversion on the original point cloud data to generate a first image, including: parsing the original point cloud data to obtain target point cloud data, where the target point cloud data at least includes: the coordinate position and reflectivity of the point cloud in space; performing voxelization partitioning on the target point cloud data to generate a three-dimensional voxel grid; projecting the three-dimensional voxel grid onto a two-dimensional plane grid to generate a first image.

[0008] Optionally, perform voxelization partitioning by marking the target point cloud data to a spatial rectangular coordinate system to generate a three-dimensional voxel grid, where the two-dimensional plane grid is any two-dimensional plane extracted from the spatial rectangular coordinate system.

[0009] Optionally, the above external parameter calibration method includes: using a fitting function to perform polygon fitting on polygons with a preset number of sides in the first edge image and the second edge image; determining the first dot matrix coordinates of polygons with a preset number of sides in the fitted first edge image, encoding the polygons with a preset number of sides in the first edge image according to the first dot matrix coordinates, serializing the vertices of the two-dimensional code array image obtained by filtering polygons with a preset number of sides in the first edge image and the encoding of the polygons with a preset number of sides in the first edge image to obtain a first vertex sequence; determining the second dot matrix coordinates of polygons with a preset number of sides in the fitted second edge image, encoding the polygons with a preset number of sides in the second edge image according to the second dot matrix coordinates, serializing the vertices of the two-dimensional code array image obtained by filtering polygons with a preset number of sides in the second edge image and the encoding of the polygons with a preset number of sides in the second edge image to obtain a second vertex sequence; determining the transformation matrix between the first vertex sequence and the second vertex sequence through a loss function, and determining the external parameter calibration result based on the transformation matrix.

[0010] Optionally, after encoding the polygons with a preset number of sides in the first edge image according to the first dot matrix coordinates, rotate the first edge image a preset number of times, and re-encode the polygons with a preset number of sides in the first edge image after each rotation; compare the encoding obtained by rotating the first edge image each time with the encoding in a preset encoding library. If the encoding of the polygons with a preset number of sides in the first edge image matches one of the encodings in the preset encoding library, determine that the ID of the encoding of the first edge image is the ID of the encoding in the preset encoding library that matches it.

[0011] Optionally, after encoding the polygons with a preset number of sides in the second edge image according to the second dot matrix coordinates, rotate the second edge image by a preset number of times, and re-encode the polygons with a preset number of sides in the second edge image after each rotation; compare the encoding obtained by rotating the second edge image each time with the encoding in the preset encoding library. If the encoding of the polygons with a preset number of sides in the second edge image matches one of the encodings in the preset encoding library, determine that the ID of the encoding of the second edge image is the ID of the encoding in the preset encoding library that matches it.

[0012] Optionally, calculate the average reflectivity of the point cloud in the voxel grid where the coordinate values on the W-axis and H-axis in the spatial rectangular coordinate system are the same, and perform binarization processing on the image obtained by projecting the three-dimensional voxel grid of the target point cloud data onto a two-dimensional plane based on the average value to obtain a first image, where the average value is used to characterize the attributes of the first image.

[0013] According to another aspect of the embodiments of the present invention, there is also provided a data processing device, including: an acquisition module that acquires original point cloud data reflected back via a QR code calibration board through a lidar, where a QR code calibration array is deployed on the QR code calibration board; a generation module configured to perform image conversion on the original point cloud data to generate a first image; a detection module configured to perform edge detection on the first image and the second image generated by the camera photographing the QR code calibration board respectively to generate a first edge image and a second edge image; an analysis module configured to perform edge structure analysis on the first edge image and the second edge image to screen and obtain at least one polygon image that meets a preset condition; an encoding module configured to encode the polygon image based on the dot matrix coordinates of each polygon image to generate an encoding result; and a determination module configured to determine the external parameter calibration result of the QR code calibration board based on the encoding result.

[0014] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and the computer program is configured to execute the above external parameter calibration method when running.

[0015] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the above processor executes the above external parameter calibration method through the computer program.

[0016] In an embodiment of the present invention, the original point cloud data reflected by the QR code calibration board is collected through a lidar. Among them, a QR code calibration array is deployed on the QR code calibration board. After the original point cloud data is converted into a first image through image conversion, edge detection is respectively performed on the first image and a second image generated by the camera photographing the QR code calibration board to generate a first edge image and a second edge image. After that, edge structure analysis is performed on the first edge image and the second edge image, and at least one polygon image that meets the preset conditions is screened out. Based on the dot matrix coordinates of each polygon image, the polygon image is encoded to generate an encoding result, and the external parameter calibration result of the QR code calibration board is determined based on the encoding. By using this method to perform joint external parameter calibration on the QR code array on the QR code calibration board, since there is more information that can be detected in the QR code array, the number of target object points collected by the lidar can be increased, and the registration accuracy of the corner points can be improved. Moreover, compared with the calibration of a single QR code calibration board, when using the QR code array for external parameter calibration, the external parameter estimation accuracy is further improved by increasing the constraints for optimizing the external parameter estimation, and the ambiguity existing in the calibration result can be reduced, that is, this method can improve the joint external parameter calibration accuracy of the camera and the lidar. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The specification drawings forming a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0018] Figure 1 is a hardware structure block diagram of a computer terminal according to an embodiment of the external parameter calibration method of the present invention;

[0019] Figure 2 is a schematic flowchart of a first embodiment of the external parameter calibration method of the present invention;

[0020] Figure 3 is a schematic diagram of an embodiment in which the external parameter calibration method of the present invention is applied to a vehicle;

[0021] Figure 4 is a schematic flowchart of a second embodiment of the external parameter calibration method according to an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a QR code standard array according to an embodiment of the present invention;

[0023] Figure 6 is a structure block diagram of a data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0027] Now, exemplary embodiments according to the present application will be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many different forms and should not be construed as being limited only to the embodiments set forth herein. It should be understood that these embodiments are provided so that the disclosure of the present application is thorough and complete, and the concept of these exemplary embodiments is fully conveyed to those of ordinary skill in the art. In the drawings, for clarity, the thickness of layers and regions may be exaggerated, and the same reference numerals are used to denote the same devices, and thus their description will be omitted.

[0028] The method embodiments provided by the embodiments of the present application can be executed on a computer terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structural block diagram of a computer terminal for an external parameter calibration method of the present application. As Figure 1 shown, the computer terminal may include one or more ( Figure 1Only one processor 102 is shown (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data. In one exemplary embodiment, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 The structure shown is only illustrative and does not limit the structure of the above computer terminal. For example, the computer terminal may further include more or fewer components than Figure 1 shown in, or have a different configuration with Figure 1 equivalent functions to those shown or more functions than Figure 1 shown.

[0029] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the data request processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer terminal. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, an external parameter calibration method is provided, which is applied to the above computer terminal. Figure 2 It is a schematic flowchart of the first embodiment of the external parameter calibration method according to the present invention, including the following steps:

[0032] Step S101, collect the original point cloud data reflected by the laser radar via the QR code calibration board, wherein a QR code calibration array is deployed on the QR code calibration board;

[0033] Step S102, performing image conversion on the original point cloud data to generate a first image;

[0034] Step S103, performing edge detection on the first image and the second image generated by photographing the two-dimensional code calibration plate with a camera, respectively, to generate a first edge image and a second edge image;

[0035] Step S104, performing edge structure analysis on the first edge image and the second edge image, and screening out at least one polygonal image that meets a preset condition;

[0036] Step S105, encoding the polygonal images based on the dot coordinates of each polygonal image to generate an encoding result;

[0037] Step S106, determining the external parameter calibration result of the two-dimensional code calibration plate based on the encoding result.

[0038] The above external parameter calibration method uses a laser radar to collect the original point cloud data reflected by the two-dimensional code calibration plate, wherein a two-dimensional code calibration array is deployed on the two-dimensional code calibration plate, and after the original point cloud data is converted into an image to generate a first image, edge detection is performed on the first image and the second image generated by the camera shooting the two-dimensional code calibration plate, respectively. After the first edge image and the second edge image are generated, the edge structure analysis is performed on the first edge image and the second edge image, and at least one polygonal image that meets the preset conditions is screened out, and the polygonal image is encoded based on the dot matrix coordinates of each polygonal image to generate an encoding result, and the external parameter calibration result of the two-dimensional code calibration plate is determined based on the encoding result. This method is used to perform joint external parameter calibration on the two-dimensional code array on the two-dimensional code calibration plate. Since the two-dimensional code array can detect more information, it can increase the target object points collected by the laser radar and improve the registration accuracy of the corner points. Moreover, compared with the calibration of a single QR code calibration plate, the use of a QR code array for extrinsic parameter calibration can further improve the accuracy of extrinsic parameter estimation by adding constraints on extrinsic parameter estimation optimization, which can reduce the ambiguity in the calibration results, that is, this method can improve the accuracy of joint extrinsic parameter calibration of cameras and lidars. Among them, the preset conditions can be whether the polygon is a quadrilateral, and whether the ratio of the area of ​​the polygon to the area of ​​the convex hull is greater than 0.8.

[0039] In an exemplary embodiment, a QR code calibration array is arranged on a flat wall perpendicular to the ground, wherein the QR code uses Tag36h11 in the visual reference library AprilTag, and other specifications may also be used, such as other standard libraries or custom tag libraries. In this embodiment, a calibration room site standard is also designed to match the above external parameter calibration method, such as Figure 3The figure shows the site design drawing of the calibration room. If the length, width, and height of the autonomous vehicle are L×W×H, then the internal dimensions of the calibration room are: (L + 16)×(W + 6)×(H + 2), and the QR code calibration arrays are evenly distributed on the surrounding walls for the joint external parameter calibration of the vehicle's panoramic camera and panoramic lidar.

[0040] In an exemplary embodiment, the original point cloud data is subjected to image conversion to generate a first image, including: parsing the original point cloud data to obtain target point cloud data, where the target point cloud data at least includes: the coordinate position and reflectivity of the point cloud in space. Specifically, the format of obtaining the target point cloud data by parsing the original point cloud data is x, y, z, i, where x, y, z represent the coordinate position of the point cloud in space, and i represents the reflectivity. The target point cloud data is voxelized and partitioned to generate a three-dimensional voxel grid. For the space where the input point cloud is located, D, W, H are used to represent its length, width, and height. Taking the lidar as the origin, the forward direction of the lidar is defined as the D axis, the left direction of the lidar is defined as the W axis, and the upward direction of the lidar is defined as the H axis. The length, width, and height of each small voxel are defined as VD, VW, VH. Assuming that D, W, H are integer multiples of VD, VW, VH respectively, and the multiples are a, b, c respectively, and the product of b and c is equal to the resolution of the camera used in the system. The three-dimensional voxel grid is projected onto a two-dimensional plane grid, and the first image is generated by dividing and compressing the three-dimensional voxel grid. Of course, in another embodiment, the depth information (x value) of the point cloud can also be directly removed, which is equivalent to tiling it onto a plane for imaging.

[0041] In an exemplary embodiment, voxelization and partitioning are performed by marking the target point cloud data to a spatial rectangular coordinate system to generate a three-dimensional voxel grid, where the two-dimensional plane grid is any two-dimensional plane extracted from the spatial rectangular coordinate system.

[0042] In an exemplary embodiment, the first image generated and the second image generated by the camera photographing the QR code calibration board are respectively subjected to edge detection using an edge detection operator, where the edge detection operator can be a Sobel operator, a Canny operator, or can be designed by oneself.

[0043] In an exemplary embodiment, the above external parameter calibration method includes: using a fitting function to perform polygon fitting on polygons with a preset number of sides in the first edge image and the second edge image, determining the first dot matrix coordinates of the polygons with the preset number of sides in the first edge image after fitting, encoding the polygons with the preset number of sides in the first edge image according to the first dot matrix coordinates, serializing the vertices of the two-dimensional code array image in the first edge image after filtering out the polygons with the preset number of sides and the encoding of the polygons with the preset number of sides in the first edge image to obtain a first vertex sequence. Determining the second dot matrix coordinates of the polygons with the preset number of sides in the second edge image after fitting, encoding the polygons with the preset number of sides in the second edge image according to the second dot matrix coordinates, serializing the vertices of the two-dimensional code array image in the second edge image after filtering out the polygons with the preset number of sides and the encoding of the polygons with the preset number of sides in the second edge image to obtain a second vertex sequence. Determining the transformation matrix between the first vertex sequence and the second vertex sequence through a loss function, and determining the external parameter calibration result based on the transformation matrix, where the transformation matrix is a single linear transformation matrix between the first vertex sequence and the second vertex sequence. In this embodiment, by transforming the problem of finding the single linear transformation matrix between the first vertex sequence and the second vertex sequence into a gradient optimization problem, the L1 loss function (minimum absolute deviation) of the vertex arrays after the first vertex sequence and the second vertex sequence are transformed can be minimized, and then the optimal solution of the single linear transformation matrix between the first vertex sequence and the second vertex sequence can be obtained. In addition to using the L1 loss function, other loss functions can also be used in the loss function of this embodiment, such as ordinary absolute value loss function, L2 loss function, log logarithmic loss function, square loss function, exponential loss function, logarithmic likelihood function, etc.

[0044] In an exemplary embodiment, after encoding the polygons with the preset number of sides in the first edge image according to the first dot matrix coordinates, the first edge image is rotated a preset number of times. Optionally, the first edge image is rotated 90° three times to obtain a total of four directions of encoding. After each rotation, the polygons with the preset number of sides in the first edge image are re-encoded, and the encoding obtained by rotating the first edge image each time is compared with the encoding in the preset encoding library. If the encoding of the polygons with the preset number of sides in the first edge image matches one of the encodings in the preset encoding library, the ID of the encoding of the first edge image is determined to be the ID of the encoding in the preset encoding library that matches it.

[0045] In an exemplary embodiment, after encoding polygons with a preset number of sides in the second edge image according to the second dot matrix coordinates, the second edge image is rotated a preset number of times. Optionally, the second edge image is rotated 90° three times to obtain encodings in four directions. After each rotation, the polygons with a preset number of sides in the second edge image are re-encoded; the encodings obtained by rotating the second edge image each time are compared with the encodings in the preset encoding library. If the encoding of the polygon with a preset number of sides in the second edge image matches one of the encodings in the preset encoding library, the ID of the encoding of the second edge image is determined to be the ID of the encoding in the preset encoding library that matches it.

[0046] In an exemplary embodiment, the average reflectivity of the point cloud in the voxel grid where the coordinate values on the W-axis and H-axis in the spatial rectangular coordinate system are the same is calculated. Based on the average value, binarization processing is performed on the image obtained by projecting the three-dimensional voxel grid of the target point cloud data onto a two-dimensional plane to obtain a first image. Here, the average value is used to characterize the attributes of the first image, and the attributes of the first image include the color of the first image. For example: if the average value is greater than or equal to 125, the average value is set to 255; if the average value is less than 125, the average value is set to 0, and the re-set average value represents the color of the generated plane grid. Among them, the binarization rule can be adaptively modified according to the type of two-dimensional code and the actual experimental effect.

[0047] In an exemplary embodiment, for the above binary edge image, the edge structure analysis ((Topological structural analysis of digitized binary images) is used to find polygons. The algorithm starts from a starting point, edits the pixels of the edge, and looks for edge points of the same type as the starting point. When the starting point is scanned, the polygon closed loop is formed, and the operation is repeated for the next starting point until all binary points are traversed. Exclude those with less than 4 sides. For those that meet the conditions, use the algorithm for finding the convex hull of a simple polygon (Finding the convex hull of a simple polygon) to calculate the convex hull of each polygon itself, and find the areas of the convex hull and its polygon. Compare the two areas. When the area of the polygon is larger than the convex hull, exclude the polygon. In this way, non-convex polygons and polygons that do not meet the condition that the ratio of the polygon area to the convex hull area is greater than 0.8 can be effectively excluded, and the remaining polygons that meet the conditions are retained. For the finally qualified polygons, use the Douglas-Peucker algorithm for quadrilateral approximation. It is also possible to use SVM or deep learning methods to detect quadrilaterals. Determine the dot matrix coordinates in the determined quadrilateral. Extract the average value of the pixels in the outermost circle of the dot matrix in the grayscale image as Value1, and then extract the average value of the pixels in the second outermost circle of the dot matrix as Value2. According to the design of the two-dimensional code library of the AprilTag itself, the gray values of all points in the outermost layer of the quadrilateral are black, while the gray values in the outer layer are a mixture of black and white. Therefore, in the same lighting environment, there is an obvious threshold boundary between Value1 and Value2. Determine the threshold as the average value of Value1 and Value2. Re-traverse all the pixel values of the dot matrix coordinates higher than the threshold and encode them as 0, and the part lower than the threshold is encoded as 1. In this way, starting from the first row, encoding is carried out until the entire dot matrix is encoded. Arranging the encoded results will obtain a string of binary codes. The length of the binary code is determined by the specific encoding method (there are three types: 36, 25, and 16. In this application, Tag36h11 is used, and the corresponding binary code length is 36). Each quadrilateral can obtain a string of binary codes, and this code represents the encoding of the two-dimensional code in this state. Further determine whether the encoding is reliable by matching it with a known encoding library. Since the observed encoding may be rotated, the obtained encoding should be rotated three times, each time by 90°, to obtain encodings in four directions, and then compare them with the encoding library one by one to calculate the Hamming Distance between the encodings (The Hamming Distance is a distance representing the similarity between two binary encodings. The Hamming Distance is the number of different bits between two long binary strings.When the Hamming distance between the observed code and a certain code in the known library is less than a given threshold (usually 2), the ID of the observed code is determined to be the ID in the matching code library, and the Hamming distance is recorded. If there is no matching code in the code library, it is determined that the observed code is incorrect, and the quadrilateral corresponding to the code is discarded. After screening and verification, the IDs of all individual QR codes have been determined, and thus the ID sequence of the QR code encoding array has also been obtained. In this embodiment, in addition to the Hamming distance, other similarity calculation methods can also be used, such as the Minkowski distance, etc.

[0048] As Figure 4 As shown in the flowchart of the second embodiment of the external parameter calibration method according to the present application. First, the lidar point cloud returned by the QR code calibration array is obtained. After voxelizing and partitioning the point cloud, the three-dimensional voxel grid (3D voxel grid) is projected onto a two-dimensional plane grid (2D plane) to convert it into a first image. At the same time, the second image generated by the camera returned by the QR code calibration array is obtained. The image after edge detection processing is binarized, and then the edges of the images (the first image and the second image) are detected. The edge structure of the binarized edge image is analyzed to filter out polygons with the number of sides greater than or equal to 4. Then, the Douglas-Peucker algorithm is used for quadrilateral fitting. The dot matrix coordinates are determined in the determined quadrilateral, the quadrilateral is encoded, and the rotation of the encoding is verified. The Hamming distance (abbreviation: Hamming distance) between the encodings is solved. When the Hamming distance between the encoding and the encoding in the known library is less than the given threshold, the ID of a single QR code is determined, and the ID sequence of the QR code encoding array is obtained. The transfer matrix between the camera and the lidar is solved through gradient optimization. When the value of the loss function is less than the given threshold, the optimal solution of the transfer matrix is obtained, that is, the external parameter calibration result is obtained.

[0049] According to another embodiment of the present application, a data processing device is also provided, such as Figure 6It is a structural block diagram of a data processing device according to the present application, including an acquisition module 10, a generation module 11, a detection module 12, an analysis module 13, an encoding module 14, and a determination module 15. The acquisition module 10 acquires the original point cloud data reflected by the QR code calibration board through a lidar. Among them, a QR code calibration array is deployed on the QR code calibration board. The generation module 11 is used to perform image conversion on the original point cloud data to generate a first image. The detection module 12 is used to perform edge detection on the first image and the second image generated by the camera photographing the QR code calibration board respectively to generate a first edge image and a second edge image. The analysis module 13 is used to perform edge structure analysis on the first edge image and the second edge image, and screen at least one polygon image that meets the preset conditions. The encoding module 14 is used to encode the polygon image based on the dot matrix coordinates of each polygon image to generate an encoding result. The determination module 15 is used to determine the external parameter calibration result of the QR code calibration board based on the encoding result.

[0050] Through the above device, the lidar acquires the original point cloud data reflected by the QR code calibration board. Among them, a QR code calibration array is deployed on the QR code calibration board. After performing image conversion on the original point cloud data to generate a first image, edge detection is performed on the first image and the second image generated by the camera photographing the QR code calibration board respectively to generate a first edge image and a second edge image. Then, edge structure analysis is performed on the first edge image and the second edge image, and at least one polygon image that meets the preset conditions is screened. Based on the dot matrix coordinates of each polygon image, the polygon image is encoded to generate an encoding result, and the external parameter calibration result of the QR code calibration board is determined based on the encoding result. By using this method to perform joint external parameter calibration on the QR code array on the QR code calibration board, since there is more information that can be detected in the QR code array, the number of target object points collected by the lidar can be increased, and the registration accuracy of the corner points can be improved. Moreover, compared with the calibration of a single QR code calibration board, when using the QR code array for external parameter calibration, the external parameter estimation accuracy is further improved by increasing the constraints for optimizing the external parameter estimation, and the ambiguity existing in the calibration result can be reduced, that is, this method can improve the joint external parameter calibration accuracy of the camera and the lidar.

[0051] According to another specific embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored program. Among them, the computer program is set to execute the above external parameter calibration method when running. Optionally, in this embodiment, the above storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc that can store program codes.

[0052] According to another specific embodiment of the present application, an electronic device is further provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, the processor executes the steps of the external parameter calibration method in the above embodiment through the computer program.

[0053] According to another specific embodiment of the embodiment of the present invention, the two-dimensional code calibration array deployed on the two-dimensional code calibration board is obtained by arranging multiple two-dimensional codes in an orderly manner. Compared with the traditional checkerboard calibration, the two-dimensional code has more information and can improve the detection and registration accuracy of corner points. Compared with the calibration of a single two-dimensional code calibration board, using the two-dimensional code array obtained by arranging multiple two-dimensional codes in an orderly manner for external parameter calibration increases the constraint equations for external parameter estimation optimization, thereby further improving the external parameter estimation accuracy and reducing the ambiguity in the calibration result.

[0054] For the sake of convenience of description, spatial relative terms such as "above", "over", "on the upper surface", "upper" etc. can be used here to describe the spatial position relationship between a device or feature shown in the figure and other devices or features. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation described in the figure for the device. For example, if the device in the drawing is inverted, the device described as "above" or "over" other devices or structures will be positioned "below" or "beneath" other devices or structures afterwards. Thus, the exemplary term "above" can include both the orientations of "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the corresponding explanations for the spatial relative descriptions used here are made.

[0055] In addition to the above, it should also be noted that the "one embodiment", "another embodiment", "embodiment" etc. mentioned in this specification refer to the specific features, structures or characteristics described in connection with this embodiment being included in at least one embodiment generally described in the present application. The same expression appearing in multiple places in the specification does not necessarily refer to the same embodiment. Further, when describing a specific feature, structure or characteristic in connection with any one embodiment, it is intended that the implementation of such feature, structure or characteristic in combination with other embodiments also falls within the scope of the present invention.

[0056] In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0057] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An external parameter calibration method applied to a vehicle, characterized in that Including: Collecting original point cloud data reflected back via a QR code calibration board by a lidar, wherein a QR code calibration array is deployed on the QR code calibration board; Converting the original point cloud data into an image to generate a first image; Performing edge detection on the first image and a second image generated by a camera photographing the QR code calibration board respectively to generate a first edge image and a second edge image; Performing edge structure analysis on the first edge image and the second edge image, and screening to obtain at least one polygon image that meets a preset condition; Encoding the polygon image based on the dot matrix coordinates of each polygon image to generate an encoding result; Determining the external parameter calibration result of the QR code calibration board based on the encoding result; Performing polygon fitting on the polygons with a preset number of sides in the first edge image and the second edge image by using a fitting function; determining the first dot matrix coordinates of the polygons with a preset number of sides in the first edge image after fitting, encoding the polygons with a preset number of sides in the first edge image according to the first dot matrix coordinates, and serializing the vertices of the QR code array image in the first edge image after filtering out the polygons with a preset number of sides and the encoding of the polygons with a preset number of sides in the first edge image to obtain a first vertex sequence; Determining the second dot matrix coordinates of the polygons with a preset number of sides in the second edge image after fitting, encoding the polygons with a preset number of sides in the second edge image according to the second dot matrix coordinates, and serializing the vertices of the QR code array image in the second edge image after filtering out the polygons with a preset number of sides and the encoding of the polygons with a preset number of sides in the second edge image to obtain a second vertex sequence; Determining a transformation matrix of the first vertex sequence and the second vertex sequence through a loss function, and determining the external parameter calibration result based on the transformation matrix.

2. The external parameter calibration method according to claim 1, wherein Converting the original point cloud data into an image to generate a first image includes: Parsing the original point cloud data to obtain target point cloud data, wherein the target point cloud data at least includes: the coordinate position and reflectivity of the point cloud in space; Performing voxelization partitioning on the target point cloud data to generate a three-dimensional voxel grid; Projecting the three-dimensional voxel grid onto a two-dimensional plane grid to generate the first image.

3. The external parameter calibration method according to claim 2, wherein Performing the voxelization partitioning to generate the three-dimensional voxel grid by labeling the target point cloud data to a spatial rectangular coordinate system, wherein the two-dimensional plane grid is any two-dimensional plane extracted from the spatial rectangular coordinate system.

4. The external parameter calibration method according to claim 1, wherein After encoding the polygons with a preset number of sides in the first edge image according to the first dot matrix coordinates, rotating the first edge image a preset number of times, and re-encoding the polygons with a preset number of sides in the first edge image after each rotation; Compare the code obtained by each rotation of the first edge image with the codes in the preset code library. If the code of the polygon in the first edge image that meets the preset number of sides matches one of the codes in the preset code library, determine that the ID of the first edge image code is the ID of the code in the preset code library that matches it.

5. The external parameter calibration method according to claim 1, wherein After encoding the polygon in the second edge image that meets the preset number of sides according to the second dot matrix coordinates, rotate the second edge image a preset number of times, and re-encode the polygon in the second edge image that meets the preset number of sides after each rotation; Compare the code obtained by each rotation of the second edge image with the codes in the preset code library. If the code of the polygon in the second edge image that meets the preset number of sides matches one of the codes in the preset code library, determine that the ID of the second edge image code is the ID of the code in the preset code library that matches it.

6. The external parameter calibration method according to claim 3, characterized in that Calculate the average reflectivity of the point cloud in the voxel grid where the coordinate values on the W-axis and H-axis in the spatial rectangular coordinate system are the same, and perform binarization processing on the image obtained by projecting the three-dimensional voxel grid of the target point cloud data onto a two-dimensional plane based on the average value to obtain the first image, where the average value is used to characterize the attributes of the first image.

7. A data processing device, characterized in that, It includes: An acquisition module that acquires the original point cloud data reflected by the lidar via the QR code calibration board, where a QR code calibration array is deployed on the QR code calibration board; A generation module for converting the original point cloud data into an image to generate a first image; A detection module for performing edge detection on the first image and the second image generated by the camera photographing the QR code calibration board respectively to generate a first edge image and a second edge image; An analysis module for performing edge structure analysis on the first edge image and the second edge image, and screening to obtain at least one polygon image that meets the preset conditions; An encoding module, configured to encode each of the polygon images based on the dot matrix coordinates of the polygon images to generate an encoding result; perform polygon fitting on the polygons with a preset number of sides in the first edge image and the second edge image by using a fitting function; determine the first dot matrix coordinates of the polygons with a preset number of sides in the first edge image after fitting, encode the polygons with a preset number of sides in the first edge image according to the first dot matrix coordinates, serialize the vertices of the two-dimensional code array image in the first edge image after filtering out the polygons with a preset number of sides and the encoding of the polygons with a preset number of sides in the first edge image to obtain a first vertex sequence; determine the second dot matrix coordinates of the polygons with a preset number of sides in the second edge image after fitting, encode the polygons with a preset number of sides in the second edge image according to the second dot matrix coordinates, serialize the vertices of the two-dimensional code array image in the second edge image after filtering out the polygons with a preset number of sides and the encoding of the polygons with a preset number of sides in the second edge image to obtain a second vertex sequence; A determination module, configured to determine the external parameter calibration result of the two-dimensional code calibration board based on the encoding result; determine a transformation matrix between the first vertex sequence and the second vertex sequence through a loss function, and determine the external parameter calibration result based on the transformation matrix.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when running, executes the external parameter calibration method described in any one of claims 1 to 6 above.

9. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the external parameter calibration method described in any one of claims 1 to 6 through the computer program.

Images