Joint calibration method and device for laser radar and camera

Abstract

The invention provides a joint calibration method and equipment for a laser radar and a camera, and relates to the technical field of sensor calibration, the equipment comprises a composite reference calibration plate and a data processor, the surface of the composite reference calibration plate is composed of a plurality of cubes, the plurality of cubes are non-uniformly distributed in the height direction, and the data processor is connected with the composite reference calibration plate. The data processor is used for receiving a point cloud which is sent by the laser radar and comprises the front surface of the composite reference calibration plate and an image which is sent by the camera and comprises the front surface of the composite reference calibration plate; and solving an extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point cloud and the image. The composite reference calibration plate can provide calibration features which are easy to extract and high in precision for the laser radar and the camera at the same time, the limitation that in the prior art, a single calibration object cannot meet the calibration requirements of the laser radar and the camera at the same time is overcome, and the richness and precision of the calibration features are improved.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, and in particular to a method and apparatus for the joint calibration of a lidar and a camera. Background Technology

[0002] In fields such as autonomous driving, robot navigation, and industrial inspection, multi-sensor fusion is an important means to improve the perception capability and robustness of a system. LiDAR and visible light cameras are commonly used complementary sensors. LiDAR provides accurate 3D geometric information but lacks texture and color information; cameras provide rich texture and color information but lack depth information and are susceptible to illumination effects. To achieve effective fusion of LiDAR point clouds and camera images, high-precision extrinsic parameter calibration is required, i.e., determining the spatial pose relationship between the two sensors.

[0003] Existing joint calibration methods for LiDAR and cameras typically rely on specific calibration objects, such as checkerboard patterns or stereo targets. However, while traditional checkerboard patterns primarily provide rich corner features for the camera, they are difficult to extract precise feature points from LiDAR point clouds, especially when the point cloud density is low or the target is far away. Although stereo targets can be identified in LiDAR point clouds, they may lack the rich texture features required by the camera or be sensitive to changes in lighting. Therefore, in existing technologies, a single calibration object cannot simultaneously meet the high-precision synchronous extraction requirements of calibration features for both LiDAR and the camera. Summary of the Invention

[0004] The purpose of this invention is to provide a joint calibration method and device for lidar and camera, so as to overcome the limitation of the prior art that a single calibration object is difficult to meet the calibration requirements of lidar and camera at the same time, and improve the richness and accuracy of calibration features.

[0005] In a first aspect, the present invention provides a joint calibration device for a lidar and a camera, comprising: a composite reference calibration board and a data processor; the surface of the composite reference calibration board is composed of multiple cubes, which are non-uniformly distributed in the height direction, and the front view corresponding to the multiple cubes is a black and white checkerboard pattern; the data processor is communicatively connected to the lidar and the camera to be calibrated; the data processor is used to receive a point cloud sent by the lidar containing the front view of the composite reference calibration board and an image sent by the camera containing the front view of the composite reference calibration board, so as to solve the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point cloud and the image; wherein the point cloud and the image are acquired at the same time.

[0006] In an optional implementation, the system further includes: an environmental sensor; the environmental sensor being connected to a data processor; the environmental sensor being integrated onto a composite reference calibration plate, or fixed within a specified distance range of the composite reference calibration plate; the environmental sensor being used to collect ambient light intensity data and ambient temperature data, and to send the ambient light intensity data and ambient temperature data to the data processor; the data processor being used to adjust the brightness of the image based on the ambient light intensity data to obtain an updated image, and then to adjust the point cloud and the updated image based on the ambient temperature data to obtain an updated point cloud and a second updated image.

[0007] In an optional implementation, the surface of the composite reference calibration plate is made of a matte material with stable reflectivity to lasers.

[0008] Secondly, the present invention provides a joint calibration method for a lidar and a camera, applicable to a joint calibration device for a lidar and a camera according to any of the foregoing embodiments, comprising: activating the lidar and camera to be calibrated, and controlling the lidar and camera to synchronously acquire data; wherein, the front of the composite reference calibration board is located within the common field of view of the lidar and the camera; receiving a point cloud containing the front of the composite reference calibration board sent by the lidar and an image containing the front of the composite reference calibration board sent by the camera; and solving the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point cloud and the image.

[0009] In an optional implementation, if an environmental sensor is integrated on the composite reference calibration board or fixed within a specified distance range, the method further includes the following steps before solving the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point cloud and the image: acquiring ambient light intensity data and ambient temperature data collected by the environmental sensor; adjusting the brightness of the image based on the ambient light intensity data to obtain an updated image; and adjusting the point cloud and the updated image based on the ambient temperature data to obtain an updated point cloud and a second-updated image.

[0010] In an optional implementation, the extrinsic parameter matrix from the LiDAR coordinate system to the camera coordinate system is solved based on point clouds and images, including: extracting the corner coordinates of the checkerboard pattern on the composite reference calibration board from the image using an image processing algorithm; determining the three-dimensional coordinates of the center of each checkerboard on the composite reference calibration board in the camera coordinate system based on the corner coordinates; identifying the point clusters corresponding to the top face of each cube on the composite reference calibration board from the point cloud using a point cloud processing algorithm; determining the three-dimensional coordinates of the center of the top face of each cube in the LiDAR coordinate system based on the point clusters; determining the three-dimensional coordinates of multiple physical points in space in the LiDAR coordinate system and in the camera coordinate system based on the prior structural data of the composite reference calibration board; and solving the extrinsic parameter matrix based on the three-dimensional coordinates of multiple physical points in the LiDAR coordinate system and in the camera coordinate system.

[0011] In an optional implementation, adjusting the brightness of the image based on ambient light intensity data to obtain an updated image includes: acquiring preset standard light intensity data; calculating the difference coefficient between the standard light intensity data and the ambient light intensity data; scaling the brightness of the image based on the difference coefficient to obtain the updated image.

[0012] In an optional implementation, adjusting the point cloud and the updated image based on ambient temperature data to obtain an updated point cloud and a second updated image includes: obtaining the correspondence between temperature and the physical deformation of the composite reference calibration plate; determining target physical deformation data based on the correspondence and ambient temperature data; and adjusting the point cloud and the updated image based on the target physical deformation data to obtain an updated point cloud and a second updated image.

[0013] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the joint calibration method of lidar and camera as described in any of the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the joint calibration method of lidar and camera as described in any of the foregoing embodiments.

[0015] This invention provides a joint calibration device for LiDAR and a camera. The device includes a composite reference calibration board and a data processor. The surface of the composite reference calibration board is composed of multiple cubes, which are non-uniformly distributed along the height direction. The front view of each cube corresponds to a black and white checkerboard pattern. The data processor receives point clouds from the LiDAR containing the front view of the composite reference calibration board and images from the camera containing the front view of the composite reference calibration board. Based on the point cloud and the image, it solves for the extrinsic parameter matrix from the LiDAR coordinate system to the camera coordinate system. The composite reference calibration board of this invention can simultaneously provide easily extractable and high-precision calibration features for both LiDAR and camera, overcoming the limitation of existing technologies where a single calibration object cannot simultaneously meet the calibration requirements of both LiDAR and camera, thus improving the richness and accuracy of the calibration features. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A front view of a composite reference calibration plate provided in an embodiment of the present invention; Figure 2 A side-view or top-view image of a composite reference calibration plate provided in an embodiment of the present invention; Figure 3 A top view of a composite reference calibration plate provided in an embodiment of the present invention; Figure 4 A side view of a composite reference calibration plate provided in an embodiment of the present invention; Figure 5 A flowchart illustrating a joint calibration method for a lidar and a camera, provided as an embodiment of the present invention; Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0021] Example 1 This invention provides a joint calibration device for a lidar and a camera, comprising: a composite reference calibration board 100 and a data processor.

[0022] Figure 1 This is a front view of a composite reference calibration plate provided in an embodiment of the present invention. Figure 2 This is a side-view or top-view image of a composite reference calibration plate provided in an embodiment of the present invention. Figure 3 This is a top view of a composite reference calibration plate provided in an embodiment of the present invention. Figure 4 A side view of a composite reference calibration plate provided in an embodiment of the present invention, with reference to... Figures 1 to 4As can be seen, in this embodiment of the invention, the surface of the composite reference calibration plate is composed of multiple cubes, which are non-uniformly distributed in the height direction, and the front view corresponding to the multiple cubes is a black and white checkerboard pattern.

[0023] In other words, the composite reference calibration plate is equipped with a cube array. This embodiment of the invention does not specifically limit the height distribution trend of the cube array; it can be high in the middle and low on both sides, low in the middle and high on both sides, or a staggered distribution. As described above, the composite reference calibration plate possesses both checkerboard and cube array features. The checkerboard feature refers to its surface using a classic black and white checkerboard pattern; the corner points of the checkerboard are commonly used high-precision feature points for camera calibration. The cube array feature refers to its composition of multiple "high-contrast" (i.e., height-difference) cubes, which form easily identifiable and locating clusters in the lidar point cloud.

[0024] The embodiments of the present invention do not specifically limit the size, spacing and layout of the checkerboard and cube arrays, as long as they can be clearly and accurately captured by the lidar and camera to be calibrated at different distances and angles.

[0025] The data processor communicates with the lidar and camera to be calibrated, and integrates a joint calibration algorithm for the lidar and camera.

[0026] The data processor is used to receive point clouds containing the front of the composite reference calibration plate sent by the lidar and images containing the front of the composite reference calibration plate sent by the camera, so as to solve the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point clouds and images; wherein the point clouds and images are acquired at the same time.

[0027] Before joint calibration, ensure that the relative positions of the lidar and camera to be calibrated are fixed and that both are communicatively connected to the data processor. The data processor can send control commands to the lidar and camera, and also receive the acquired data from both. The front of the composite reference calibration board should be within the common field of view of the lidar and camera to avoid obstructing calibration features. After joint calibration begins, the data processor controls the lidar and camera to synchronously acquire data from the composite reference calibration board. The lidar acquires a point cloud containing the front of the composite reference calibration board, and the camera acquires an image containing the front of the composite reference calibration board. Then, both send their acquired data to the data processor. That is, the point cloud and image received by the data processor have the same timestamp, thus effectively avoiding calibration errors caused by spatiotemporal misalignment.

[0028] After receiving the point cloud and image, the data processor processes the point cloud using a point cloud processing algorithm to extract the 3D coordinates of the calibration feature points in the LiDAR coordinate system. It then processes the image using an image processing algorithm to extract the 3D coordinates of the calibration feature points in the camera coordinate system. Finally, based on the coordinates of multiple calibration feature points in both coordinate systems, the extrinsic parameter matrix (including rotation and translation vectors) from the LiDAR coordinate system to the camera coordinate system can be solved. The calibration feature points represent the geometric reference points used to establish the spatial pose transformation relationship between the LiDAR and camera coordinate systems. This embodiment of the invention does not specifically limit the selection of calibration feature points; they can be corner points or the center point of the top surface of a cube. After obtaining the coordinates of multiple calibration feature points, this embodiment of the invention does not specifically limit the method for solving the extrinsic parameter matrix. Users can use any existing extrinsic parameter solution method to solve the above extrinsic parameter matrix, such as linear solutions or nonlinear optimization solutions.

[0029] This invention provides a joint calibration device for LiDAR and a camera. The device includes a composite reference calibration board and a data processor. The surface of the composite reference calibration board is composed of multiple cubes, which are non-uniformly distributed along the height direction. The front view of each cube is a black and white checkerboard pattern. The data processor receives point clouds from the LiDAR containing the front view of the composite reference calibration board and images from the camera containing the front view of the composite reference calibration board. Based on the point clouds and images, it calculates the extrinsic parameter matrix from the LiDAR coordinate system to the camera coordinate system. The composite reference calibration board in this invention can simultaneously provide easily extractable and high-precision calibration features for both LiDAR and camera, overcoming the limitation of existing technologies where a single calibration object cannot simultaneously meet the calibration requirements of both LiDAR and camera, thus improving the richness and accuracy of the calibration features.

[0030] Traditional static calibration methods are typically performed under specific conditions, and the calibration results are easily affected by factors such as ambient light and temperature changes. For example, changes in lighting can affect the camera's accuracy in recognizing features of the calibrated object; temperature changes may cause slight deformations in the calibrated object or the sensor itself, introducing calibration errors. In practical applications, environmental conditions are often dynamically changing, making it difficult to guarantee the accuracy of static calibration results under different environments. Therefore, in one optional implementation, the joint calibration device for the LiDAR and camera further includes: an environmental sensor 200; the environmental sensor is connected to a data processor, and the connection method can be wired (e.g., USB, serial port) or wireless (e.g., Bluetooth, Wi-Fi).

[0031] Environmental sensors are integrated onto the composite reference calibration plate or fixed within a specified distance range of the composite reference calibration plate to ensure that the data collected by the environmental sensors can accurately reflect the environmental conditions of the composite reference calibration plate.

[0032] In this embodiment of the invention, the environmental sensor includes a light intensity sensor and a temperature sensor. The environmental sensor is used to collect ambient light intensity data and ambient temperature data, and to send the ambient light intensity data and ambient temperature data to the data processor.

[0033] The data processor is used to adjust the brightness of the image based on ambient light intensity data to obtain an updated image. Then, it adjusts the point cloud and the updated image based on ambient temperature data to obtain an updated point cloud and a second updated image.

[0034] In other words, after receiving ambient light intensity and ambient temperature data, the data processor needs to dynamically compensate for the calibration environment using this data. Specifically, it adjusts the image brightness using the ambient light intensity data to avoid the image being too bright or too dark, which would affect the accuracy of coordinate extraction of calibration feature points. It is known that changes in ambient temperature can cause slight deformations in the composite reference calibration plate due to thermal expansion and contraction. Therefore, this embodiment of the invention further adjusts the point cloud and image based on ambient temperature data to compensate for these slight deformations of the composite reference calibration plate.

[0035] In other words, the joint calibration equipment with added environmental sensors can monitor changes in ambient light and temperature in real time. By feeding back the light intensity and temperature data to the data processor, the calibration data (point cloud and image) can be corrected. Finally, the data processor uses the corrected point cloud and image—that is, the updated point cloud and the second-updated image—to solve for the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system. The feedback from the environmental sensors makes the final calibration results more robust to environmental changes, solving the problem of traditional static calibration being susceptible to environmental fluctuations, and improving the calibration accuracy and reliability of the calibration results in practical application environments.

[0036] In one optional implementation, the surface of the composite reference calibration plate is made of a matte material with stable reflectivity to laser, in order to reduce the impact of specular reflection on the lidar point cloud. Stable reflectivity to laser means that the composite reference calibration plate's ability to reflect laser energy remains constant when facing laser irradiation, and does not fluctuate significantly with time or environmental conditions.

[0037] Example 2 This invention also provides a joint calibration method for lidar and camera. This method is applied to the joint calibration device for lidar and camera provided in Embodiment 1 above. The following is a detailed description of the joint calibration method for lidar and camera provided in this invention.

[0038] Figure 5 This is a flowchart of a joint calibration method for lidar and camera provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the method specifically includes the following steps: Step S102: Start the lidar and camera to be calibrated, and control the lidar and camera to collect data synchronously.

[0039] The front of the composite reference calibration board is located within the common field of view of the lidar and the camera.

[0040] Step S104: Receive the point cloud image of the front of the composite reference calibration board sent by the lidar and the image of the front of the composite reference calibration board sent by the camera.

[0041] Step S106: Based on the point cloud and image, solve for the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system.

[0042] Before calibrating the lidar and camera to be calibrated using the joint calibration device for lidar and camera provided in Embodiment 1 above, the composite reference calibration board should first be placed face up within the common field of view of the lidar and camera, and a connection should be established between the lidar and camera and the execution subject of the method (i.e., the data processor of the joint calibration device for lidar and camera). Next, the lidar and camera to be calibrated are started, and data within their respective fields of view is synchronously acquired under the control of the data processor. After data acquisition is completed, the data acquisition results are sent to the data processor. Then, based on the received point cloud and image, the data processor extracts the coordinates of the calibration feature points in the lidar coordinate system and the camera coordinate system through point cloud processing and image processing, respectively. The extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system can then be solved based on the coordinate pair. Embodiment 1 above has already described in detail the structure and calibration principle of the joint calibration device for lidar and camera, which can be referred to above for details, and will not be repeated here.

[0043] In an alternative implementation, if an environmental sensor is integrated on the composite reference calibration board or fixed within a specified distance range, the method further includes the following steps before solving the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on point clouds and images: Step S1051: Obtain ambient light intensity data and ambient temperature data collected by the environmental sensor.

[0044] Step S1052: Adjust the brightness of the image based on the ambient light intensity data to obtain the updated image.

[0045] Step S1053: Adjust the point cloud and the updated image based on the ambient temperature data to obtain the updated point cloud and the image after a second update.

[0046] The above embodiment also provides a detailed description of the method for calibrating the lidar and camera using a joint calibration device with added environmental sensors. Please refer to the above text for details, which will not be repeated here.

[0047] There are various methods for solving the extrinsic parameter matrix. This embodiment of the invention does not specifically limit this method; one method is described in detail below. In an optional embodiment, step S106 above, based on point cloud and image, solves the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system, specifically including the following steps: Step S1061: Use an image processing algorithm to extract the corner coordinates of the checkerboard pattern on the composite reference calibration board from the image.

[0048] Step S1062: Determine the three-dimensional coordinates of the center of each checkerboard grid on the composite reference calibration board in the camera coordinate system based on the corner coordinates.

[0049] The images captured by the camera include the above. Figure 1 In the front view of the image, the coordinates of each corner point can be accurately determined using existing image processing algorithms. In this embodiment of the invention, the corner point is also the intersection of the black and white checkerboard, which is also the vertex of each checkerboard square. In this embodiment, the center of the checkerboard is selected as the calibration feature point; therefore, the coordinates of the checkerboard center need to be further calculated based on the corner point coordinates. Knowing the positional relationship between the vertices and the center point of each checkerboard square, the coordinates of the checkerboard center point can be obtained by processing the corner point coordinates using relevant data formulas.

[0050] Step S1063: Use a point cloud processing algorithm to identify the point clusters corresponding to the top surface of each cube on the composite reference calibration plate from the point cloud.

[0051] Step S1064: Determine the three-dimensional coordinates of the top surface center of each cube in the lidar coordinate system based on the point cluster.

[0052] Specifically, after identifying the point clusters corresponding to the top surface of each cube using point cloud processing algorithms, the three-dimensional coordinates of the top surface center in the lidar coordinate system can be accurately extracted through fitting and other methods. The top surface center of the cube is also the center of the checkerboard.

[0053] Step S1065: Based on the prior structural data of the composite reference calibration plate, determine the three-dimensional coordinates of multiple physical points in space in the lidar coordinate system and the three-dimensional coordinates in the camera coordinate system.

[0054] Step S1066: Solve for the extrinsic parameter matrix based on the three-dimensional coordinates of multiple physical points in the lidar coordinate system and the three-dimensional coordinates in the camera coordinate system.

[0055] After step S1064 is completed, the 3D coordinates of the centers of each checkerboard grid in the camera coordinate system and the 3D coordinates of the centers of the top faces of each cube in the lidar coordinate system are obtained. To solve the extrinsic parameter matrix, it is necessary to match these two types of 3D coordinates, that is, to determine the coordinates of the same physical point in space in both coordinate systems. The above coordinate matching requires the use of the structural prior data of the composite reference calibration plate. For example, the first checkerboard grid in the upper left of the image corresponds to the first cube in the upper left of the point cloud. Therefore, the 3D coordinates of the center of the first checkerboard grid in the upper left are matched with the 3D coordinates of the center of the top face of the first cube in the upper left, forming a coordinate pair.

[0056] Users need to determine the number of physical points in step S1065 according to the actual extrinsic parameter solution method used. This embodiment of the invention does not specifically limit the number of physical points or the extrinsic parameter solution method. Theoretically, there should be at least 3 physical points (i.e., 3 coordinate pairs). However, in order to overcome noise and errors and improve accuracy, stability and robustness, as many high-quality coordinate pairs as possible are generally used to obtain an accurate and reliable extrinsic parameter matrix.

[0057] In an optional implementation, step S1052, which adjusts the brightness of the image based on ambient light intensity data to obtain an updated image, specifically includes the following: First, obtain the preset standard light intensity data.

[0058] Then, the difference coefficient between the standard light intensity data and the ambient light intensity data is calculated.

[0059] Finally, the brightness of the image is scaled based on the difference coefficient to obtain the updated image.

[0060] Specifically, the standard light intensity data is set by the user based on experience, and by default, images taken under this light intensity are the clearest. To adjust an image taken under the current ambient light intensity, first calculate the ratio of the standard light intensity data to the ambient light intensity data, and use the ratio as a light intensity difference coefficient. Next, use this difference coefficient as a scaling factor for image brightness. By multiplying the image brightness by the difference coefficient, you obtain the image with adjusted brightness.

[0061] Obviously, if the ambient light intensity is less than the standard light intensity, the difference coefficient is greater than 1. After adjusting the difference coefficient, the brightness of the updated image will be improved. Conversely, if the ambient light intensity is greater than the standard light intensity, the difference coefficient is less than 1. After adjusting the difference coefficient, the brightness of the updated image will be reduced.

[0062] In an optional implementation, step S1053, which adjusts the point cloud and the updated image based on ambient temperature data to obtain an updated point cloud and a second-updated image, specifically includes the following: First, the correspondence between temperature and the physical deformation of the composite reference calibration plate is obtained.

[0063] Then, based on the correspondence and ambient temperature data, the target physical deformation data are determined.

[0064] Finally, the point cloud and the updated image are adjusted based on the target physical deformation data to obtain the updated point cloud and the image after a second update.

[0065] The correspondence between temperature and the physical deformation of the composite reference calibration plate can be established by collecting a large amount of sample data and training a neural network model to fit the corresponding curve. Then, after acquiring ambient temperature data, the physical deformation of the composite reference calibration plate at that temperature can be determined by querying this correspondence curve, and this is recorded as the target physical deformation data. Finally, the target physical deformation data is used to correct each coordinate point in both the lidar coordinate system and the camera coordinate system, i.e., adjusting the point cloud and the updated brightness image, resulting in an updated point cloud and a second-updated image. Based on this, the data processor can solve for the extrinsic parameter matrix using the latest point cloud and image.

[0066] Example 3 See Figure 6 This invention provides an electronic device comprising: a processor 60, a memory 61, a bus 62, and a communication interface 63, wherein the processor 60, the communication interface 63, and the memory 61 are connected via the bus 62; the processor 60 is used to execute executable modules, such as computer programs, stored in the memory 61.

[0067] The memory 61 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 63 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0068] Bus 62 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0069] The memory 61 is used to store programs. After receiving an execution instruction, the processor 60 executes the program. The method executed by the apparatus defined by the process disclosed in any of the foregoing embodiments of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0070] Processor 60 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 60 or by instructions in software form. Processor 60 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 61. Processor 60 reads the information in memory 61 and, in conjunction with its hardware, completes the steps of the above method.

[0071] The computer program product of the joint calibration method of lidar and camera provided in the embodiments of the present invention includes a computer-readable storage medium storing non-volatile program code executable by a processor. The instructions included in the program code can be used to execute the method described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0072] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0073] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0075] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0076] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0077] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A joint calibration device for lidar and camera, characterized in that, include: Composite reference calibration board and data processor; The surface of the composite reference calibration plate is composed of multiple cubes, which are non-uniformly distributed in the height direction, and the front view of the multiple cubes is a black and white checkerboard pattern. The data processor is communicatively connected to the lidar and camera to be calibrated; The data processor is used to receive point clouds containing the front of the composite reference calibration plate sent by the lidar and images containing the front of the composite reference calibration plate sent by the camera, so as to solve the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point clouds and the images; wherein the point clouds and the images are acquired at the same time.

2. The joint calibration device for lidar and camera according to claim 1, characterized in that, Also includes: An environmental sensor; the environmental sensor is connected to the data processor; The environmental sensor is integrated into the composite reference calibration plate, or fixed within a specified distance range of the composite reference calibration plate; The environmental sensor is used to collect ambient light intensity data and ambient temperature data, and send the ambient light intensity data and ambient temperature data to the data processor; The data processor is used to adjust the brightness of the image based on the ambient light intensity data to obtain an updated image, and then adjust the point cloud and the updated image based on the ambient temperature data to obtain an updated point cloud and a second updated image.

3. The joint calibration device for lidar and camera according to claim 1, characterized in that, The surface of the composite reference calibration plate is made of a matte material with stable reflectivity to lasers.

4. A joint calibration method for lidar and camera, characterized in that, A joint calibration device for a lidar and a camera as described in any one of claims 1-3, comprising: The lidar and camera to be calibrated are activated, and the lidar and camera are controlled to acquire data synchronously; wherein, the front of the composite reference calibration board is located within the common field of view of the lidar and the camera; Receive the point cloud image containing the front of the composite reference calibration plate sent by the lidar and the image containing the front of the composite reference calibration plate sent by the camera; Based on the point cloud and the image, solve for the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system.

5. The joint calibration method for lidar and camera according to claim 4, characterized in that, If an environmental sensor is integrated on the composite reference calibration board or fixed within a specified distance range, then before solving the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system based on the point cloud and the image, the following steps are also included: Acquire ambient light intensity and ambient temperature data collected by environmental sensors; The brightness of the image is adjusted based on the ambient light intensity data to obtain an updated image; The point cloud and the updated image are adjusted based on the ambient temperature data to obtain the updated point cloud and the image after a second update.

6. The joint calibration method for lidar and camera according to claim 4, characterized in that, Based on the point cloud and the image, the extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system is solved, including: The corner coordinates of the checkerboard pattern on the composite reference calibration board are extracted from the image using image processing algorithms; Based on the corner coordinates, determine the three-dimensional coordinates of the center of each checkerboard grid on the composite reference calibration board in the camera coordinate system; The point cloud processing algorithm is used to identify the point clusters corresponding to the top surface of each cube on the composite reference calibration plate from the point cloud; The three-dimensional coordinates of the top surface center of each cube in the lidar coordinate system are determined based on the point clusters. Based on the prior structural data of the composite reference calibration plate, the three-dimensional coordinates of multiple physical points in space in the lidar coordinate system and the three-dimensional coordinates in the camera coordinate system are determined. The extrinsic parameter matrix is ​​solved based on the three-dimensional coordinates of the multiple physical points in the lidar coordinate system and the three-dimensional coordinates in the camera coordinate system.

7. The joint calibration method for lidar and camera according to claim 5, characterized in that, The brightness of the image is adjusted based on the ambient light intensity data to obtain an updated image, including: Obtain preset standard light intensity data; Calculate the difference coefficient between the standard light intensity data and the ambient light intensity data; The brightness of the image is scaled based on the difference coefficient to obtain the updated image.

8. The joint calibration method for lidar and camera according to claim 5, characterized in that, Adjusting the point cloud and the updated image based on the ambient temperature data yields an updated point cloud and a second-updated image, including: Obtain the correspondence between temperature and the physical deformation of the composite reference calibration plate; Based on the correspondence and the ambient temperature data, the target physical deformation data is determined; Based on the target physical deformation data, the point cloud and the updated image are adjusted respectively to obtain the updated point cloud and the second updated image.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the joint calibration method of the lidar and camera as described in any one of claims 3 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the joint calibration method of the lidar and camera as described in any one of claims 3 to 8.

Images