Calibration method, calibration board, storage medium, electronic equipment and program product

Abstract

The invention provides a calibration method, a calibration board, a storage medium, electronic equipment and a program product, and relates to the technical field of sensor calibration. The calibration method comprises the following steps: acquiring a target image of a calibration plate shot by a camera and scanning data of a laser radar on the calibration plate, wherein the calibration plate comprises a visual pattern used for extracting camera feature points and a scanning structure used for extracting laser radar feature points; acquiring a relative position relationship between the visual pattern and the scanning structure in the calibration plate; determining coordinates of the camera feature points in a camera coordinate system; coordinates of the laser radar feature points in a laser radar coordinate system are determined; and determining external parameter data of the camera and the laser radar based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system and the coordinates of the laser radar feature points in the laser radar coordinate system. According to the invention, joint calibration of the camera and the laser radar is realized, and the applicability and flexibility of calibration are improved.

Description

Technical Field

[0001] The present application relates to the technical field of sensor calibration, and in particular to a calibration method, a calibration board, a storage medium, an electronic device and a program product. Background Art

[0002] Camera and LiDAR joint calibration is a key technology in sensor calibration, which is used to achieve spatial alignment and data fusion between camera and LiDAR.

[0003] However, existing calibration methods, such as those based on natural features, projection, and deep learning, are widely used, but they have limitations in calibration accuracy, feature matching difficulty, and environmental adaptability. Moreover, many calibration plates are only optimized for a single sensor. These limitations hinder the application of existing calibration methods for joint calibration of cameras and lidars in dynamic scenes and high-precision scenarios. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a calibration method, a calibration board, a storage medium, an electronic device, and a program product.

[0005] In a first aspect, an embodiment of the present application provides a calibration method, including: obtaining a target image of a calibration plate taken by a camera and scanning data of the calibration plate by a laser radar, the calibration plate including a visual pattern for extracting camera feature points and a scanning structure for extracting laser radar feature points; obtaining the relative position relationship between the visual pattern and the scanning structure in the calibration plate; based on the target image, determining the coordinates of the camera feature points in a camera coordinate system; based on the scanning data, determining the coordinates of the laser radar feature points in the laser radar coordinate system; based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system, and the coordinates of the laser radar feature points in the laser radar coordinate system, determining the external parameter data of the camera and the laser radar.

[0006] In combination with the first aspect, in certain implementations of the first aspect, the visual pattern and the scanning structure are located in the same geometric plane in the calibration plate; based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system and the coordinates of the lidar feature points in the lidar coordinate system, the external parameter data of the camera and the lidar are determined, including: based on the coordinates of the lidar feature points in the lidar coordinate system and the relative position relationship, the coordinates of the camera feature points in the lidar coordinate system are determined; based on the coordinates of the camera feature points in the lidar coordinate system and the coordinates of the camera feature points in the camera coordinate system, the external parameter data of the camera and the lidar are determined.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the coordinates of the camera feature points in the laser radar coordinate system are determined based on the coordinates of the laser radar feature points in the laser radar coordinate system and the relative position relationship, including: determining a plane coordinate system based on the laser radar feature points, the plane coordinate system represents the coordinate system of the plane where the visual pattern and the scanning structure are located; determining the coordinates of the camera feature points in the plane coordinate system based on the relative position relationship; determining the coordinates of the camera feature points in the laser radar coordinate system based on the coordinates of the target laser radar feature points in the laser radar coordinate system and the coordinates of the camera feature points in the plane coordinate system, the target laser radar feature points represent feature points as the origin in the plane coordinate system.

[0008] In combination with the first aspect, in certain implementations of the first aspect, a plane coordinate system is determined based on the laser radar feature points, including: selecting any laser radar feature point as the origin; determining two basis vectors in the plane based on the coordinates of the laser radar feature points in any other two directions and the origin in the laser radar coordinate system; and determining the plane coordinate system based on the origin and the two basis vectors.

[0009] In combination with the first aspect, in some implementations of the first aspect, the scanning structure includes a hole structure, and / or the visual pattern includes a target color pattern.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the hole structure includes a circular hole, the lidar feature point includes the center of the circular hole; the target color pattern includes a target color circle, and the camera feature point includes the center of the target color circle.

[0011] In a second aspect, an embodiment of the present application provides a calibration plate, comprising: a substrate; a visual pattern disposed on the substrate; and a scanning structure disposed on the substrate and spaced apart from the visual pattern by a target interval.

[0012] In combination with the second aspect, in certain implementations of the second aspect, the visual pattern includes a target color pattern, which is distributed in an array on the surface of the substrate; and / or the scanning structure includes a hole structure, which is distributed along at least two directions on the substrate.

[0013] In a third aspect, an embodiment of the present application provides a calibration device, comprising: a first acquisition module, used to acquire a target image of a calibration plate taken by a camera and scanning data of the calibration plate by a laser radar, the calibration plate comprising a visual pattern for extracting camera feature points and a scanning structure for extracting laser radar feature points; a second acquisition module, used to determine the relative position relationship between the visual pattern and the scanning structure in the calibration plate; a first determination module, used to determine the coordinates of the camera feature points in the camera coordinate system based on the target image; a second determination module, used to determine the coordinates of the laser radar feature points in the laser radar coordinate system based on the scanning data; a third determination module, used to determine the external parameter data of the camera and the laser radar based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system, and the coordinates of the laser radar feature points in the laser radar coordinate system.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the calibration method described in the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; the processor is used to execute the calibration method described in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes instructions, and when the instructions are executed on an electronic device, the electronic device implements the calibration method described in the first aspect.

[0017] In this application, by integrating the visual patterns and scanning structures required by the camera and the lidar on the same calibration board, the problem that the traditional calibration board is only optimized for a single sensor is overcome, and the joint calibration of the camera and the lidar is realized, which improves the applicability and flexibility of the calibration. In addition, the calibration board provides a common reference frame for the camera and the lidar, allowing the feature points observed by the camera and the lidar to correspond on the same physical entity, simplifying the calibration process in dynamic scenes. Then, by determining the relative position relationship between the visual pattern and the scanning structure on the calibration board, it can adapt to different environmental changes and improve the calibration accuracy. Finally, through the relative position relationship, the coordinates of the camera feature points in the camera coordinate system, and the coordinates of the lidar feature points in the lidar coordinate system, the external parameter data of the camera and the lidar can be calculated, simplifying the calibration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 Shown is a flow chart of a calibration method provided in one embodiment of the present application.

[0020] Figure 2 The figure is a schematic diagram of a process for determining external parameter data of a camera and a lidar provided in an embodiment of the present application.

[0021] Figure 3 The figure is a schematic diagram of a process for determining the coordinates of camera feature points in a laser radar coordinate system provided by an embodiment of the present application.

[0022] Figure 4 Shown is a schematic diagram of the arrangement of the scanning structure and the visual pattern in the calibration plate provided in one embodiment of the present application.

[0023] Figure 5 Shown is a schematic structural diagram of a calibration plate provided in one embodiment of the present application.

[0024] Figure 6 Shown is a schematic diagram of the structure of a calibration device provided in one embodiment of the present application.

[0025] Figure 7 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Figure 1 FIG. 1 is a flow chart of a calibration method provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps.

[0028] Step S110, obtaining the target image of the calibration plate taken by the camera and the scanning data of the calibration plate by the laser radar.

[0029] The calibration plate includes a visual pattern for extracting camera feature points and a scanning structure for extracting lidar feature points.

[0030] Visual patterns and scanning structures are two different types of features in the calibration plate. Specifically, visual patterns refer to those patterns on the calibration plate that can be recognized and processed by the camera. These patterns can generate obvious camera feature points in the target image so that the camera can recognize them through image processing algorithms. Optionally, the camera feature point refers to the geometric center point of the visual pattern that can be recognized in the target image.

[0031] The scanned structures represent those structures on the calibration plate that can be identified and measured by the LiDAR. These structures can generate identifiable LiDAR feature points in the point cloud corresponding to the scanned data, so that the LiDAR can identify them through the point cloud processing algorithm. Optionally, the LiDAR feature point refers to the geometric center point of the scanned structure that can be identified in the point cloud.

[0032] In addition, when acquiring the target image of the calibration plate taken by the camera and the scanning data of the calibration plate by the laser radar, it is necessary to ensure that the calibration plate is completely within the field of view of the camera and the laser radar so that the camera and the laser radar can capture the complete features of the calibration plate. During the data acquisition process, ensure that the calibration plate is stable to prevent the position changes of the feature points caused by movement, which will affect the subsequent calibration accuracy. Finally, the collected target images and scanning data need to meet the requirements of algorithm processing, including a sufficient number of feature points, so that feature matching and parameter calculation can be performed effectively. Optionally, the scanning data of the calibration plate by the laser radar includes multiple frames of scanning data.

[0033] Step S120, obtaining the relative position relationship between the visual pattern and the scanning structure in the calibration plate.

[0034] Exemplarily, design parameters of the visual pattern and scanning structure in the calibration plate are determined, and based on the design parameters, the target spacing of the visual pattern and the scanning structure in the calibration plate is obtained, and then based on the target spacing, the relative position relationship of the visual pattern and the scanning structure in the calibration plate is obtained.

[0035] Step S130: determining the coordinates of the camera feature points in the camera coordinate system based on the target image.

[0036] Exemplarily, based on the target image, the pixel coordinates of the camera feature points in the target image are determined; based on the intrinsic parameter data of the camera, the pixel coordinates are converted to the camera coordinate system to obtain the coordinates of the camera feature points in the camera coordinate system.

[0037] Specifically, the image processing algorithm is first used to identify and locate the visual pattern in the target image, thereby determining the pixel coordinates of the camera feature points in the target image. Subsequently, the camera's intrinsic parameter data, including the camera's focal length, principal point position (usually the center of the image), and lens distortion parameters, are used to apply geometric transformation and correction algorithms to transform the pixel coordinates from the image coordinate system to the camera coordinate system, and the coordinates of the camera feature points in the camera coordinate system are obtained.

[0038] Step S140, based on the scanning data, determine the coordinates of the laser radar feature point in the laser radar coordinate system.

[0039] Exemplarily, based on the scanning data, the coordinates of each laser point in the laser radar coordinate system are determined; the laser radar feature points are extracted from the laser points, and then the coordinates of the laser radar feature points in the laser radar coordinate system are obtained.

[0040] Specifically, LiDAR measures the distance and shape of the surrounding environment by emitting a laser beam and receiving the reflected light, generating a scan data containing a large number of points. Through the scan data, the coordinates of each laser point in the LiDAR coordinate system can be determined. Then, point cloud processing algorithms such as clustering, edge detection, or specific shape matching are applied to identify the LiDAR feature points of the scanned structure. Once these LiDAR feature points are identified, their coordinates directly reflect their positions in the LiDAR coordinate system.

[0041] Step S150, determining the external parameter data of the camera and the laser radar based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system, and the coordinates of the laser radar feature points in the laser radar coordinate system.

[0042] Exemplarily, based on the relative position relationship and the coordinates of the lidar feature points in the lidar coordinate system, the coordinates of the lidar feature points in the camera coordinate system are determined; based on the coordinates of the lidar feature points in the camera coordinate system and the coordinates of the lidar feature points in the lidar coordinate system, the external parameter data of the camera and the lidar are determined.

[0043] Optionally, a spatial transformation algorithm, such as a PnP algorithm, is used to determine the extrinsic parameter data between the camera and the lidar, wherein the extrinsic parameter data includes a rotation matrix and / or a translation vector.

[0044] In this embodiment, by integrating the visual patterns and scanning structures required by the camera and the laser radar on the same calibration board, the problem that the traditional calibration board is only optimized for a single sensor is overcome, and the joint calibration of the camera and the laser radar is realized, which improves the applicability and flexibility of the calibration. In addition, the calibration board provides a common reference frame for the camera and the laser radar, allowing the feature points observed by the camera and the laser radar to correspond on the same physical entity, simplifying the calibration process in dynamic scenes. Then, by determining the relative position relationship between the visual pattern and the scanning structure on the calibration board, it can adapt to different environmental changes and improve the calibration accuracy. Finally, through the relative position relationship, the coordinates of the camera feature points in the camera coordinate system, and the coordinates of the laser radar feature points in the laser radar coordinate system, the external parameter data of the camera and the laser radar can be calculated, which simplifies the calibration process.

[0045] In step S150, a method for determining the external parameter data of the camera and the laser radar is briefly introduced. Figure 2 , another implementation method for determining the external parameter data of the camera and lidar is analyzed in detail.

[0046] Figure 2 The figure is a schematic diagram of a process for determining the external parameter data of a camera and a laser radar provided by an embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 2 The embodiment shown is described below in detail. Figure 2 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0047] First, in this embodiment, the visual pattern and the scanning structure are located in the same geometric plane in the calibration plate. In other words, although the visual pattern and the scanning structure serve different sensors (camera and lidar), they are physically coplanar. This design allows the camera and lidar to simultaneously observe and measure the corresponding feature points on the calibration plate, and ensures that the feature points observed by the two sensors are on the same plane in physical space, so that the spatial relationship (external parameters) between the two sensors can be accurately calculated. In addition, this coplanar design reduces the complexity and potential errors introduced by the feature points not being in the same geometric plane, further simplifying the calibration process.

[0048] like Figure 2 As shown, based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system and the coordinates of the lidar feature points in the lidar coordinate system, the external parameter data of the camera and the lidar are determined, including the following steps.

[0049] Step S210, based on the coordinates of the laser radar feature points in the laser radar coordinate system and the relative position relationship, determine the coordinates of the camera feature points in the laser radar coordinate system.

[0050] Specifically, since the camera feature points are all located in the same geometric plane of the calibration plate, a fixed depth value can be assumed (such as zero, or a known distance from the calibration plate to the camera), so as to transform the two-dimensional coordinates of the camera feature points in the camera coordinate system into three-dimensional space; then, search for a point close to the position of the converted camera feature point in the lidar scanning data, and the coordinate of this point in the lidar coordinate system is the coordinate of the corresponding camera feature point in the lidar coordinate system.

[0051] Step S220, determining the external parameter data of the camera and the laser radar based on the coordinates of the camera feature points in the laser radar coordinate system and the coordinates of the camera feature points in the camera coordinate system.

[0052] Optionally, after determining the coordinates of the camera feature points in the lidar coordinate system, the PnP algorithm is used to solve the external parameter data between the camera and the lidar based on the coordinates of the camera feature points in the lidar coordinate system and the coordinates of the camera feature points in the camera coordinate system.

[0053] In summary, it can be seen that, unlike the embodiment in step S150, Figure 2 In the embodiment, the coordinates of the camera feature points in the camera coordinate system are converted to the lidar coordinate system, and then the external parameter data of the camera and the lidar are solved. The reason is that the visual pattern and the scanning structure are located in the same geometric plane of the calibration plate. Therefore, this coplanarity can be used, combined with the known relative position relationship and the geometric model of the calibration plate, to convert the coordinates of the camera feature points from the camera coordinate system to the lidar coordinate system, which simplifies the calculation process and reduces error accumulation.

[0054] In this embodiment, the coordinates of the camera feature points in the laser radar coordinate system are first determined based on the coordinates of the laser radar feature points in the laser radar coordinate system and the relative positional relationship between the visual pattern and the scanning structure. This step utilizes the design of the calibration plate to simplify the conversion process from the camera coordinate system to the laser radar coordinate system. Subsequently, the rotation matrix and / or translation vector between the camera and the laser radar are calculated through a geometric algorithm using the coordinates of the camera feature points in the laser radar coordinate system and the coordinates in the camera coordinate system. This reduces the complexity of the calibration process, improves the accuracy and reliability of the calibration, and has important practical application value for robot navigation.

[0055] In step S210, an implementation is provided to determine the coordinates of the camera feature points in the laser radar coordinate system. Figure 3 The embodiment in the embodiment is extended to another implementation method.

[0056] Figure 3 FIG. 1 is a flow chart of determining the coordinates of a camera feature point in a laser radar coordinate system according to an embodiment of the present application. Figure 2 Based on the embodiment shown, Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 2 The differences and similarities of the illustrated embodiments are not described in detail.

[0057] like Figure 3 As shown, in this embodiment, based on the coordinates of the laser radar feature points in the laser radar coordinate system and the relative position relationship, the coordinates of the camera feature points in the laser radar coordinate system are determined, including the following steps.

[0058] Step S310, determining a plane coordinate system based on the laser radar feature points.

[0059] In some embodiments, any laser radar feature point is selected as the origin; based on the coordinates of the laser radar feature points in any other two directions and the origin in the laser radar coordinate system, two basis vectors in the plane are determined; based on the origin and the two basis vectors, the plane coordinate system is determined.

[0060] Specifically, after the origin is selected, two basis vectors are calculated based on the coordinates of the origin in the laser radar coordinate system. The two basis vectors point from the origin to the two laser radar feature points, respectively. 11 is the origin, and the laser radar feature points in the other two directions are recorded as P 12 , P 13 , then the two basis vectors can be expressed as μ = P 12 -P 11 ; v = P 13 -P 11 .

[0061] Furthermore, the two basis vectors in the plane are normalized, that is, the lengths of the two basis vectors are adjusted to 1 while keeping their directions unchanged. If the two calculated basis vectors are not orthogonal, the two basis vectors need to be corrected to ensure their orthogonality. Optionally, the correction is performed using the formula v=v-(v·μ)μ.

[0062] Since this scheme can select any LiDAR feature point to define the origin and basis vector, it can adapt to different calibration plate designs and LiDAR scanning data, providing higher flexibility.

[0063] Step S320: determining the coordinates of the camera feature points in the plane coordinate system based on the relative position relationship.

[0064] Exemplarily, the relative position relationship includes the distance and / or angle between them. Assuming that the distribution direction of the visual pattern in the calibration plate is parallel to the two basis vectors, the coordinates (α, β) of other camera feature points in the plane coordinate system can be determined based on the coordinates of the laser radar feature point as the origin.

[0065] Figure 4 FIG. 1 is a schematic diagram showing the arrangement of the scanning structure and the visual pattern in the calibration plate provided by an embodiment of the present application. Assuming that the scanning structure in the upper left corner is P 11 , P 11 The horizontal interval between the center of the circle and the boundary of the visual pattern in the upper left corner is 0.15, and the vertical interval between the center of the circle and the visual pattern in the upper left corner is 0.05 (not shown in the figure). In addition, the interval between two adjacent visual patterns (including horizontal interval and vertical interval) and the radius of each visual pattern are also known. For example, the interval in the horizontal direction is 0.06, the interval in the vertical direction is 0.045, and the radius is 0.08. Then, it can be determined that each camera feature point is close to P 11 The horizontal and vertical spacing of 11 is the origin of the plane coordinate system, so each camera feature point is 11 The horizontal interval and vertical interval of represent the x-coordinate and y-coordinate of the camera feature points in the plane coordinate system, respectively.

[0066] Step S330, based on the coordinates of the target laser radar feature point in the laser radar coordinate system and the coordinates of the camera feature point in the plane coordinate system, determine the coordinates of the camera feature point in the laser radar coordinate system.

[0067] The target laser radar feature point represents a feature point as the origin in the plane coordinate system. For example, the camera feature point has a coordinate P in the laser radar coordinate system. ci =P 11 +α i u+β i v, i represents the serial number of the camera feature point.

[0068] In this embodiment, the plane coordinate system is first determined based on the laser radar feature points, ensuring that the coordinate system of the plane where the visual pattern and the scanning structure are located is directly associated with the measurement data of the laser radar, thereby simplifying the conversion process between different sensor data. Next, the coordinates of the camera feature points in the plane coordinate system are determined using the design information of the calibration plate (i.e., the known relative position relationship). Finally, the position of the camera feature points is accurately converted to the laser radar coordinate system by combining the coordinates of the target laser radar feature points in the laser radar coordinate system and the coordinates of the camera feature points in the plane coordinate system. This method improves the flexibility and adaptability of the calibration process and enhances the accuracy and robustness of the calibration results. In this way, it is possible to ensure that the data of the camera and the laser radar are accurately aligned in the same coordinate system, providing a basis for subsequent spatial analysis and decision-making.

[0069] In some embodiments, the scanning structure in the calibration plate includes a hole structure, and / or the visual pattern includes a target color pattern.

[0070] Specifically, the hole structure provides clear three-dimensional features for the LiDAR, which appear as obvious depressions in the LiDAR point cloud data, facilitating accurate feature extraction and positioning of the LiDAR. At the same time, the target color pattern provides the camera with high-contrast visual features that are easily recognizable in the target image captured by the camera, helping the camera to accurately detect camera feature points. This design enables the calibration plate to meet the calibration requirements of both sensors (camera and LiDAR) at the same time, improving the compatibility and efficiency of the calibration process.

[0071] In some embodiments, the hole structure includes a circular hole, and the lidar feature point includes the center of the circular hole; the target color pattern includes a target color circle, and the camera feature point includes the center of the target color circle.

[0072] For camera calibration, using a target color circle to extract camera feature points has higher accuracy and robustness than traditional checkerboard corner points. Specifically, the circular feature points of the target color circle (e.g., the boundary points of the target color circle) can better adapt to the nonlinear distortion of the camera lens, thereby improving the accuracy of calibration. In addition, the target color circle can maintain stable performance even under poor lighting conditions. For example, in an environment with uneven lighting, blurred edges, or high image noise, the extraction accuracy of its circular feature points will not be greatly affected, and it can be stably detected even in low-contrast or high-reflection environments.

[0073] For LiDAR calibration, the center of the circular hole is used as the geometric center. Even if the circular hole is partially blocked, as long as there are enough contour points of the circular hole, the LiDAR can accurately fit the circular boundary of the circular hole and calculate the position of the center. In addition, the geometric characteristics of the circular hole are stable and do not depend on the reflection intensity or material of the surface. This design enables the calibration plate to stably extract features under different materials and lighting conditions. In addition, the position of the center of the circular hole is calculated by fitting the point cloud data of the LiDAR. This method can achieve sub-point cloud level accuracy, which is much higher than the traditional edge detection method. The symmetry and geometric constraints of the circle further enhance the anti-interference ability of the circular hole center positioning, so that the center position of the circular hole can still be accurately determined in the presence of noise and local errors.

[0074] In summary, the calibration plate used in the calibration method of this embodiment provides a high-precision and highly robust calibration solution for cameras and lidars through its unique design, especially in dealing with nonlinear distortion, illumination changes and noise interference.

[0075] Figure 5 FIG. 1 is a schematic diagram of the structure of a calibration board provided in an embodiment of the present application. Figure 5 As shown, the calibration plate includes: a substrate 510; a visual pattern 520 disposed on the substrate 510; and a scanning structure 530 disposed on the substrate 510 and spaced from the visual pattern 520 by a target interval.

[0076] Specifically, the substrate 510 is the base part of the calibration plate, providing a physical platform for the visual pattern 520 and the scanning structure 530. Exemplarily, the substrate 510 is a planar substrate. The visual pattern 520 contains camera feature points that can be recognized and extracted by the camera, and these camera feature points can be detected in the target image captured by the camera for camera calibration. The scanning structure 530 forms identifiable feature points in the point cloud data of the laser radar for laser radar calibration.

[0077] In addition, in this embodiment, the visual pattern 520 and the scanning structure 530 are spaced apart from the target, that is, the relative positional relationship between the visual pattern 520 and the scanning structure 530 is known.

[0078] It should be noted that the target interval also varies according to the arrangement of the visual patterns in the calibration plate. For example, Figure 5 The target interval between the scanning structure 530 in the upper left corner and the visual pattern c1 is d, and the target interval between the scanning structure 530 and the visual pattern c1 is d+d1, wherein d1 represents the interval between the visual pattern c1 and the visual pattern c2.

[0079] Figure 5The calibration plate shown can be used to achieve joint calibration between the camera and the LiDAR. Specifically, through the visual pattern and scanning structure in the calibration plate, as well as the target interval between them, it can be ensured that both sensors can accurately extract their respective feature points and convert them to a unified coordinate system, which can improve the calibration accuracy and efficiency of the camera and LiDAR.

[0080] Further, continue to refer to Figure 5 The visual pattern 520 includes a target color pattern, which is distributed in an array on the surface of the substrate 510; and / or the scanning structure 530 includes a hole structure, which is distributed on the substrate 510 along at least two directions.

[0081] Specifically, the target color pattern of the array distribution provides a set of clear, regular and easy-to-detect visual markers. This distribution method is conducive to the camera capturing and identifying camera feature points, thereby improving the accuracy and efficiency of calibration. At the same time, the array distribution also helps to reduce the impact of environmental changes or camera viewing angle changes, and enhances the robustness of the calibration process. For example, Figure 5 As shown, the target color patterns are distributed in a 9×9 array.

[0082] The hole structures are distributed along at least two directions on the substrate 510. This multi-directional distribution provides sufficient geometric constraints, so that the plane of the calibration plate can be defined by the center positions of these hole structures, thereby enhancing the accuracy and reliability of plane determination. Secondly, by selecting the center of any hole structure as the origin and using the centers of the other two non-collinear hole structures as reference points, two basis vectors in the plane can be determined. These two basis vectors define the direction of the plane coordinate system, thus simplifying subsequent coordinate transformation and calculation.

[0083] In some embodiments, in the horizontal direction, intervals between adjacent visual patterns 520 are all first values; and in the vertical direction, intervals between adjacent visual patterns 520 are all second values.

[0084] That is, in the horizontal direction and the vertical direction, the interval between two adjacent visual patterns 520 is fixed. This fixed interval setting enables the position of each camera feature point to be determined through simple geometric calculations, and makes it easier to identify and locate the camera feature points, simplifying the process of calculating the coordinates of the camera feature points in the plane coordinate system of the plane where the calibration plate is located.

[0085] In some embodiments, the visual patterns 520 distributed in the horizontal direction and / or the visual patterns 520 distributed in the vertical direction are parallel to the distribution direction of the scanning structures 530 on the substrate 510 .

[0086] Specifically, by ensuring that the distribution directions of the visual pattern 520 and the scanning structure 530 are parallel, the variables that need to be considered in the calibration process can be reduced and the computational complexity can be reduced. In addition, when the distribution directions of the visual pattern 520 and the scanning structure 530 are parallel, the relative position relationship between them is mainly defined by distance, rather than angles or other more complex geometric relationships. This means that on the calibration plate, the correspondence between the camera feature points and the lidar feature points is more direct and simple. Since the correspondence between the feature points is more intuitive, the required calculation steps and possible error sources are simplified when converting the camera feature points to the lidar coordinate system. In addition, this parallel design also reduces the uncertainty introduced by factors such as changes in viewing angle or tilt of the calibration plate, which helps to improve the robustness of the calibration process.

[0087] Combination of the above Figures 1 to 4 , describes in detail the calibration method embodiment of the present application, and the following is combined with Figure 6 , describes in detail the calibration device embodiment of the present application. It should be understood that the description of the calibration method embodiment corresponds to the description of the calibration device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0088] Figure 6 FIG. 1 is a schematic diagram of the structure of a calibration device provided in an embodiment of the present application. Figure 6 As shown, the calibration device 60 provided in the embodiment of the present application includes:

[0089] A first acquisition module 610 is used to acquire a target image of a calibration plate captured by a camera and scanning data of the calibration plate by a laser radar, wherein the calibration plate includes a visual pattern for extracting feature points of the camera and a scanning structure for extracting feature points of the laser radar;

[0090] A second acquisition module 620, used to determine the relative position relationship between the visual pattern and the scanning structure in the calibration plate;

[0091] A first determination module 630 is used to determine the coordinates of the camera feature points in the camera coordinate system based on the target image;

[0092] A second determination module 640 is used to determine the coordinates of the laser radar feature point in the laser radar coordinate system based on the scan data;

[0093] The third determination module 650 is used to determine the external parameter data of the camera and the laser radar based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system, and the coordinates of the laser radar feature points in the laser radar coordinate system.

[0094] In one embodiment of the present application, the visual pattern and the scanning structure are located in the same geometric plane in the calibration plate; the third determination module 650 is also used to determine the coordinates of the camera feature points in the lidar coordinate system based on the coordinates of the lidar feature points in the lidar coordinate system and the relative position relationship; based on the coordinates of the camera feature points in the lidar coordinate system and the coordinates of the camera feature points in the camera coordinate system, determine the external parameter data of the camera and the lidar.

[0095] In one embodiment of the present application, the third determination module 650 is also used to determine a plane coordinate system based on the lidar feature points, the plane coordinate system representing the coordinate system of the plane where the visual pattern and the scanning structure are located; determine the coordinates of the camera feature points in the plane coordinate system based on the relative position relationship; determine the coordinates of the camera feature points in the lidar coordinate system based on the coordinates of the target lidar feature points in the lidar coordinate system and the coordinates of the camera feature points in the plane coordinate system, the target lidar feature points representing the feature points as the origin in the plane coordinate system.

[0096] In one embodiment of the present application, the third determination module 650 is also used to select any lidar feature point as the origin; determine two basis vectors in the plane based on the coordinates of the lidar feature points in any other two directions and the origin in the lidar coordinate system; and determine the plane coordinate system based on the origin and the two basis vectors.

[0097] In an embodiment of the present application, the scanning structure includes a hole structure, and / or the visual pattern includes a target color pattern.

[0098] In one embodiment of the present application, the hole structure includes a circular hole, and the laser radar feature point includes the center of the circular hole; the target color pattern includes a target color circle, and the camera feature point includes the center of the target color circle.

[0099] Below, reference Figure 7 To describe an electronic device according to an embodiment of the present application. Figure 7 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0100] like Figure 7 As shown, the electronic device 70 includes one or more processors 701 and a memory 702 .

[0101] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 70 to perform desired functions.

[0102] The memory 702 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may run the program instructions to implement the calibration method of each embodiment of the present application described above and / or other desired functions. Various contents such as target images, scan data, coordinates of camera feature points in the camera coordinate system, and coordinates of laser radar feature points in the laser radar coordinate system may also be stored in the computer-readable storage medium.

[0103] In one example, the electronic device 70 may further include: an input device 703 and an output device 704, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0104] The input device 703 may include, for example, a keyboard, a mouse, etc.

[0105] The output device 704 can output various information to the outside, including target images, scan data, coordinates of camera feature points in the camera coordinate system, coordinates of laser radar feature points in the laser radar coordinate system, etc. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0106] Of course, to simplify, Figure 7 Only some of the components related to the present application in the electronic device 70 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 70 may also include any other appropriate components.

[0107] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the calibration method according to various embodiments of the present application described above in this specification.

[0108] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0109] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the calibration method according to various embodiments of the present application described above in this specification.

[0110] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0111] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0112] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0113] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0114] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0115] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A calibration method, characterized in that: include: Acquire a target image of a calibration plate captured by a camera and scan data of the calibration plate by a laser radar, wherein the calibration plate includes a visual pattern for extracting camera feature points and a scanning structure for extracting laser radar feature points; Acquiring a relative position relationship between the visual pattern and the scanning structure in the calibration plate; Based on the target image, determining the coordinates of the camera feature points in a camera coordinate system; Based on the scan data, determining the coordinates of the laser radar feature point in the laser radar coordinate system; Based on the relative position relationship, the coordinates of the camera feature points in the camera coordinate system and the coordinates of the laser radar feature points in the laser radar coordinate system, the external parameter data of the camera and the laser radar are determined.

2. The method according to claim 1, characterized in that The visual pattern and the scanning structure are located in the same geometric plane of the calibration plate; the external parameter data of the camera and the laser radar are determined based on the relative position relationship, the coordinates of the camera feature point in the camera coordinate system, and the coordinates of the laser radar feature point in the laser radar coordinate system, including: Based on the coordinates of the laser radar feature point in the laser radar coordinate system and the relative position relationship, determine the coordinates of the camera feature point in the laser radar coordinate system; Based on the coordinates of the camera feature points in the laser radar coordinate system and the coordinates of the camera feature points in the camera coordinate system, the external parameter data of the camera and the laser radar are determined.

3. The method according to claim 2, characterized in that The determining the coordinates of the camera feature point in the laser radar coordinate system based on the coordinates of the laser radar feature point in the laser radar coordinate system and the relative position relationship includes: Based on the laser radar feature points, determining a plane coordinate system, wherein the plane coordinate system represents a coordinate system of a plane where the visual pattern and the scanning structure are located; Based on the relative position relationship, determining the coordinates of the camera feature points in a plane coordinate system; Based on the coordinates of the target lidar feature point in the lidar coordinate system and the coordinates of the camera feature point in the plane coordinate system, the coordinates of the camera feature point in the lidar coordinate system are determined, and the target lidar feature point represents the feature point as the origin in the plane coordinate system.

4. The method according to claim 3, characterized in that The determining of a plane coordinate system based on the laser radar feature points includes: Select any laser radar feature point as the origin; Determine two basis vectors in the plane based on the laser radar feature points in any other two directions and the coordinates of the origin in the laser radar coordinate system; The plane coordinate system is determined based on the origin and the two basis vectors.

5. The method according to claim 1, characterized in that: The scanning structure comprises a hole structure, and / or The visual pattern includes a target color pattern.

6. The method according to claim 5, characterized in that The hole structure includes a circular hole, and the laser radar feature point includes the center of the circular hole; The target color pattern includes a target color circle, and the camera feature point includes the center of the target color circle.

7. A calibration plate, characterized in that: include: substrate; a visual pattern, disposed on the substrate; The scanning structure is disposed on the substrate and is spaced a target distance from the visual pattern.

8. The calibration plate according to claim 7, characterized in that: The visual pattern comprises a target color pattern, and the target color pattern is distributed in an array on the surface of the substrate; and / or The scanning structure includes a hole structure, and the hole structure is distributed along at least two directions on the substrate.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the calibration method described in any one of claims 1 to 6 above.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the calibration method described in any one of claims 1 to 6 above.

11. A computer program product, characterized in that The computer program product comprises instructions, which, when executed on an electronic device, enable the electronic device to implement the calibration method according to any one of claims 1 to 6.

Images