Calibration method and device between pan-tilt zoom camera and laser radar

Abstract

The present invention relates to the field of three-dimensional measurement and calibration, and provides a calibration method and device between a pan-tilt zoom camera and a laser radar, the method comprising: obtaining the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system; fitting the conversion relationship between the pan-tilt zoom camera magnification and distance based on the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system; determining the initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship; optimizing the initial conversion matrix to obtain the final conversion matrix. The present invention solves the problems of complex calibration process and large conversion matrix error in the prior art, and realizes efficient and accurate collaborative measurement of PTZ camera and laser radar.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional measurement and calibration, and in particular to a calibration method and device between a pan-tilt zoom camera and a laser radar. Background Art

[0002] In the field of automated measurement and monitoring, the combined use of LiDAR and PTZ (Pan-Tilt-Zoom) cameras is becoming more and more common. LiDAR plays an important role in three-dimensional space measurement with its high-precision and efficient ranging capabilities. PTZ cameras, with their flexible rotation and zoom functions, can easily capture and track targets at different positions and distances. The combination of the two provides powerful measurement and monitoring capabilities for a variety of application scenarios.

[0003] At present, the calibration technology between LiDAR and PTZ camera mainly adopts the following methods: One is to jointly determine the transformation matrix between the two by calculating the distance between the camera coordinate origin and the LiDAR coordinate origin on the X, Y, and Z axes when the PTZ camera is installed, as well as the horizontal rotation angle and vertical rotation angle between the camera coordinate system and the LiDAR coordinate system. This method is based on an idealized assumption that the origin position and rotation angle of the camera coordinate system can be accurately measured. However, in practical applications, these parameters are often difficult to obtain accurately, resulting in large deviations in the transformation matrix.

[0004] Another commonly used calibration method is to find two sets of common points and perform the best fit. This method uses the SDK interface of the PTZ camera to obtain the horizontal and vertical angles of the camera. If the distance parameter can be obtained, the spherical coordinates can be converted into Cartesian coordinates to obtain the three-dimensional coordinates of the target in the camera coordinate system. However, the difficulty of this method is that it cannot accurately obtain the distance of the target captured by the PTZ camera.

[0005] The above-mentioned prior art has obvious defects in practical applications. The first method cannot accurately know the origin position and rotation angle of the camera coordinate system, resulting in a large deviation in the final transformation matrix. Especially when the measurement target is a long-distance target, the impact of the angle error will be more significant, thereby affecting the accuracy of the measurement. Although the second method can theoretically obtain the transformation matrix through best fitting, it is difficult to implement in actual operation because it cannot obtain the exact distance of the target captured by the PTZ camera. Even if the distance parameters can be obtained, the final transformation matrix still has a large uncertainty due to the influence of measurement errors and human factors. Summary of the invention

[0006] The present invention provides a calibration method and device between a pan-tilt zoom camera and a laser radar, which solves the problems of complex calibration process and large conversion matrix error in the prior art, and enables the PTZ camera and the laser radar to perform collaborative measurement efficiently and accurately.

[0007] The present invention provides a calibration method between a pan-tilt zoom camera and a laser radar, comprising the following steps:

[0008] Get the position of the origin of the gimbal-zoom camera coordinate system in the lidar coordinate system;

[0009] Fitting the conversion relationship between the magnification and distance of the gimbal-zoom camera based on the position of the origin of the gimbal-zoom camera coordinate system in the laser radar coordinate system;

[0010] Determine an initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship;

[0011] The initial conversion matrix is ​​optimized to obtain a final conversion matrix.

[0012] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the obtaining of the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system specifically includes: setting the vertical angle of the gimbal zoom camera to 0°, and then setting the horizontal angles of the gimbal zoom camera to 0°, 90° and 180° respectively; at each horizontal angle, pointing the laser radar to the center of the gimbal zoom camera screen at the corresponding angle, and obtaining three laser radar coordinate points A, B and C; taking any point P on the surface of the gimbal zoom camera, and projecting P onto the line segment AC to obtain the projection point P'; continuously adjusting the position of the projection point P' so that the angle of ∠BP'A meets the preset conditions; and using the position of the projection point P' that meets the preset conditions as the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system.

[0013] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the conversion relationship between the magnification of the gimbal zoom camera and the distance is fitted based on the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system, specifically including: placing a target object at multiple positions at different distances from the gimbal zoom camera; using the laser radar to obtain the coordinates of the target object at multiple different distances; obtaining the distance of the target object relative to the gimbal zoom camera according to the coordinates of the target object at the multiple different distances and the coordinates of the origin of the gimbal zoom camera; in the picture of the gimbal zoom camera, enlarging multiple targets at different distances to the same size and recording the corresponding magnification values; fitting the relationship between the magnification and the distance into a quadratic equation to determine the relationship between the magnification and the distance.

[0014] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the initial conversion matrix between the laser radar and the gimbal zoom camera is determined based on the conversion relationship, specifically including: placing a spherical object at at least n positions around the device, and obtaining the three-dimensional coordinates of the spherical object in the laser radar coordinate system; the device includes a laser radar and a ball camera; obtaining a gimbal zoom value, converting the gimbal zoom value into Cartesian coordinates, and obtaining the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system; the gimbal zoom value includes a horizontal angle, a vertical angle and a magnification; and optimally fitting the three-dimensional coordinates of the spherical object in the laser radar coordinate system and the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system to obtain the initial conversion matrix between the laser radar and the gimbal zoom camera.

[0015] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the initial transformation matrix is ​​optimized to obtain a final transformation matrix, specifically including: determining that the error of the initial transformation matrix comes from the three-dimensional coordinate error of the gimbal zoom camera coordinate system; obtaining an objective function as the difference between the distance of each point in the laser radar coordinate system and the distance of each point in the gimbal zoom camera coordinate system; obtaining the scaling factor of each point in the three-dimensional coordinate system by minimizing the objective function, and multiplying the coordinates of the gimbal zoom camera coordinate system by their respective scaling factors; and again performing optimal fitting on the three-dimensional coordinates of the spherical object in the laser radar coordinate system and the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system to obtain the final transformation matrix.

[0016] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, converting the gimbal zoom value into Cartesian coordinates specifically includes: converting the gimbal zoom value into spherical coordinates, and then converting it into Cartesian coordinates through trigonometric functions.

[0017] The present invention also provides a calibration device between a pan-tilt zoom camera and a laser radar, comprising the following modules:

[0018] The PTZ zoom camera coordinate origin acquisition module is used to obtain the position of the origin of the PTZ zoom camera coordinate system in the laser radar coordinate system;

[0019] A fitting module, used for fitting the conversion relationship between the magnification and distance of the pan-tilt zoom camera based on the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system;

[0020] An initial conversion matrix determination module, used to determine the initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship;

[0021] The final conversion matrix determination module is used to optimize the initial conversion matrix to obtain the final conversion matrix.

[0022] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a calibration method between a pan-tilt zoom camera and a laser radar as described in any one of the above is implemented.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the calibration method between the pan-tilt zoom camera and the laser radar as described in any one of the above is implemented.

[0024] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a calibration method between a pan-tilt zoom camera and a laser radar as described in any one of the above.

[0025] The present invention provides a calibration method and device between a pan-tilt zoom camera and a laser radar, which has the following beneficial effects: by fitting the conversion relationship between the magnification and distance of the pan-tilt zoom camera, the three-dimensional coordinates of the target in the pan-tilt zoom camera coordinate system can be more accurately obtained, thereby improving the calibration accuracy between the laser radar and the pan-tilt zoom camera, automatically finding and measuring the target, realizing automatic measurement based on the laser radar and the camera, and improving the measurement efficiency. By optimizing the conversion matrix, the calibration error can be further reduced, and a more accurate conversion matrix can be obtained, providing strong support for subsequent measurement work. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 This is one of the flow charts of the calibration method between the pan-tilt zoom camera and the laser radar provided by the present invention.

[0028] Figure 2 It is a schematic diagram of the relationship between the magnification and distance of the fitted PTZ camera provided by the present invention.

[0029] Figure 3 This is a schematic diagram of the best fitting conversion matrix provided by the present invention.

[0030] Figure 4 It is a schematic diagram of converting spherical coordinates into Cartesian coordinates provided by the present invention.

[0031] Figure 5This is the second flow chart of the calibration method between the pan-tilt zoom camera and the laser radar provided by the present invention.

[0032] Figure 6 It is a structural schematic diagram of a calibration device between a pan-tilt zoom camera and a laser radar provided by the present invention.

[0033] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present invention provides a PTZ camera and laser radar calibration method based on linear distance conversion and weight scaling, aiming to solve the linkage problem between the laser radar and the rotatable camera. In some measurement scenarios, obtaining measurement data only by relying on the laser radar requires human cooperation, while the rotatability of the PTZ camera can easily view the measurement targets at various positions. By solving the calibration problem of the PTZ camera and the laser radar, automatic measurement based on the laser radar and the camera can be realized. Therefore, the present invention proposes a calibration method based on linear distance conversion and weight scaling, which makes the camera the "eyes" of the radar, and can automatically find and measure targets.

[0036] The existing calibration technology between LiDAR and PTZ camera is to jointly determine the transformation matrix between the camera coordinate system and the LiDAR coordinate system by calculating the distance between the camera coordinate origin and the LiDAR coordinate origin on the X, Y, and Z axes when the PTZ camera is installed, and the horizontal rotation angle and vertical rotation angle between the camera coordinate system and the LiDAR coordinate system. However, this method is too idealistic. Since it is impossible to know the origin of the camera coordinate system and the rotation angle of the coordinate system, the above information cannot be accurately measured, resulting in a large deviation in the final transformation matrix. Especially when the target is a long-distance target, the impact of the angle error will be greater.

[0037] Another method to calibrate two 3D coordinate systems is to find two sets of common points and perform the best fit. Generally speaking, the horizontal and vertical angles of a PTZ camera can be obtained more accurately through its SDK interface. Therefore, if the distance parameter can be obtained, it is easy to obtain the 3D coordinates of the target in the camera coordinate system by converting the spherical coordinates into Cartesian coordinates. However, the difficulty of this method lies in the inability to obtain the distance of the target captured by the PTZ camera.

[0038] Combine the following Figure 1-Figure 7 Embodiments of the present invention are described in detail.

[0039] The present invention mainly solves the problem of relationship calibration between laser radar and PTZ camera.

[0040] 1. Get the 3D coordinates of the PTZ camera. Calibration requires obtaining the 3D coordinates of the target object in two sets of coordinate systems. The 3D coordinates of the target in the laser radar can be obtained through the measurement software, while the coordinates of the target in the PTZ camera coordinate system can be obtained through conversion. The PTZ camera can obtain its horizontal rotation angle and vertical rotation angle through the SDK (Software Development Kit) interface, so the PTZ camera coordinate system can be regarded as a spherical coordinate system lacking distance. The spherical coordinate system can be converted into a Cartesian coordinate system through coordinate conversion. Therefore, only the approximate value of the distance between the target and the PTZ camera is needed to approximate the 3D coordinates of the target in the PTZ camera coordinate system. The distance can be obtained by converting the distance and magnification, so a set of magnification values ​​and a set of distance values ​​are needed to analyze the relationship between them. By enlarging the same object to the same size at n different distances, a set of n magnification values ​​can be obtained, and the n distances between the camera and the object can be obtained through the laser radar. Finally, by fitting the two sets of data, the relationship between magnification and distance is approximately obtained. After that, the PTZ camera can obtain the magnification through the SDK to obtain its approximate distance from the target.

[0041] 2. In the first step, there are influencing factors, such as 1) manually enlarging objects at different distances to the same size in the picture. This step cannot guarantee that the size of each distance is the same. 2) Because the center of the camera cannot be determined, there is a small error in the distance between the object and the center of the camera measured by the lidar. These factors will cause errors in the conversion of the relationship between magnification and distance. For the first problem, a point-to-line projection is used to approximate the position of the origin of the camera coordinate system in the lidar coordinate system. For the second problem, the coordinates of the PTZ camera are scaled by an optimization method so that the relative relationship between the coordinates of each point in the camera coordinate system is close to the coordinate group in the lidar coordinate system.

[0042] Through this calibration method, a calibration relationship with a small error between the camera and the LiDAR can be obtained at close range, so that the LiDAR and PTZ camera can assist each other in measuring the same object. This technology can be applied in many scenarios. For example, in the scenario of measuring points at the level of the entire aircraft, surveyors no longer need to work at high altitudes or hold a target ball for measurement, and the measurement speed is faster and the measurement accuracy is higher.

[0043] Figure 1It is one of the flow charts of the calibration method between the pan-tilt zoom camera and the laser radar provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0044] S110, obtaining the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system.

[0045] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the obtaining of the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system specifically includes: setting the vertical angle of the gimbal zoom camera to 0°, and then setting the horizontal angles of the gimbal zoom camera to 0°, 90° and 180° respectively; at each horizontal angle, pointing the laser radar to the center of the gimbal zoom camera screen at the corresponding angle, and obtaining three laser radar coordinate points A, B and C; taking any point P on the surface of the gimbal zoom camera, and projecting P onto the line segment AC to obtain the projection point P'; continuously adjusting the position of the projection point P' so that the angle of ∠BP'A meets the preset conditions; and using the position of the projection point P' that meets the preset conditions as the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system.

[0046] Specifically, the origin of the coordinate system in the PTZ camera is calculated.

[0047] The principle of calculating the origin in this step is that the rotation angle error of the camera itself is small enough to be ignored. First, set the vertical angles to 0°, then set the horizontal angles of the PTZ camera to 0°, 90°, and 180° respectively, point the lidar to the center of the camera screen at the corresponding angles, and obtain the coordinates of lidar_p0, lidar_p90, and lidar_p180. Represented by points A, B, and C respectively, theoretically the origin of the camera must be on the line connecting AC.

[0048] 1) Take any point P on the camera surface and project P onto AC. Assume that the projection point is P'.

[0049] 2) Calculate the angle of ∠BP'A. If it is greater than 90°, move the projection point P' toward point C. Otherwise, control it to move toward point A.

[0050] 3) Repeat step 2) until ∠BP'A is close to 90°.

[0051] S120, fitting the conversion relationship between the magnification and distance of the gimbal zoom camera based on the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system.

[0052] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the conversion relationship between the magnification of the gimbal zoom camera and the distance is fitted based on the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system, specifically including: placing a target object at multiple positions at different distances from the gimbal zoom camera; using the laser radar to obtain the coordinates of the target object at multiple different distances; obtaining the distance of the target object relative to the gimbal zoom camera according to the coordinates of the target object at the multiple different distances and the coordinates of the origin of the gimbal zoom camera; in the picture of the gimbal zoom camera, enlarging multiple targets at different distances to the same size and recording the corresponding magnification values; fitting the relationship between the magnification and the distance into a quadratic equation to determine the relationship between the magnification and the distance.

[0053] Specifically, the relationship between the PTZ camera magnification and distance is fitted.

[0054] like Figure 2 The figure shows the relationship between the magnification and distance of the PTZ camera. Figure 2 In the equation, L depends on the maximum distance between the ball camera and the target point; n means the denser the points are, the more accurate the fitting effect will be.

[0055] First, the target needs to be placed at n different distances from the camera. The position of the origin of the camera coordinate system in the laser radar coordinate system is known from the previous step, and then the laser radar is used to obtain the coordinates of the target at n different distances [X_1, Y_1, Z_1], ..., [X_n, Y_n, Z_n]. By subtracting the coordinates of the camera origin [X_c, Y_c, Z_c] from the n coordinates, the position and distance of the n targets relative to the camera can be obtained. On the other hand, the PTZ camera needs to enlarge the n targets at different distances to the same size in the picture, and then the SDK interface of the PTZ camera can also obtain the camera's magnification value separately. Finally, the relationship between magnification and distance can be obtained by fitting a quadratic equation.

[0056] S130. Determine an initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship.

[0057] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the initial conversion matrix between the laser radar and the gimbal zoom camera is determined based on the conversion relationship, specifically including: placing a spherical object at at least n positions around the device, and obtaining the three-dimensional coordinates of the spherical object in the laser radar coordinate system; the device includes a laser radar and a ball camera; obtaining a gimbal zoom value, converting the gimbal zoom value into Cartesian coordinates, and obtaining the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system; the gimbal zoom value includes a horizontal angle, a vertical angle and a magnification; and optimally fitting the three-dimensional coordinates of the spherical object in the laser radar coordinate system and the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system to obtain the initial conversion matrix between the laser radar and the gimbal zoom camera.

[0058] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, converting the gimbal zoom value into Cartesian coordinates specifically includes: converting the gimbal zoom value into spherical coordinates, and then converting it into Cartesian coordinates through trigonometric functions.

[0059] Specifically, the conversion matrix between the laser radar and the PTZ camera is obtained.

[0060] like Figure 3 The figure shows the schematic diagram of the best fit conversion matrix. The conversion relationship between the magnification and distance of the PTZ camera has been obtained. Next, two sets of three-dimensional coordinates are needed for the best fit. Place a spherical object at least 5 positions around the device, because a spherical object is a circle from all angles and will not produce distortion. The three-dimensional coordinates in the laser radar coordinate system can be measured by the measurement software Spatial Analysis or Polyworks.

[0061] The three-dimensional coordinates in the PTZ camera coordinate system are first obtained through the SDK interface to obtain the PTZ (horizontal angle, vertical angle, magnification) value, and then the PTZ value is converted into spherical coordinates, and finally converted into Cartesian coordinates through trigonometric functions, such as Figure 4 As shown, the coordinate transformation formula is as follows.

[0062] 1. Cartesian (x, y, z) ---> spherical coordinates (θ, Φ, r), use after calibration.

[0063] r = √x^ 2 +y^ 2 +z^ 2

[0064] θ = arctan(y / x)

[0065] Φ=arcsin(z / r)

[0066] 2. Spherical coordinates (θ, Φ, r) ---> Cartesian (x, y, z), used for calibration.

[0067] x=r·cos(Φ)·cos(θ)

[0068] y=r.cos(Φ)·sin(θ)

[0069] z=r.sin(Φ)

[0070] S140: Optimize the initial conversion matrix to obtain a final conversion matrix.

[0071] According to a calibration method between a gimbal zoom camera and a laser radar provided by the present invention, the initial transformation matrix is ​​optimized to obtain a final transformation matrix, specifically including: determining that the error of the initial transformation matrix comes from the three-dimensional coordinate error of the gimbal zoom camera coordinate system; obtaining an objective function as the difference between the distance of each point in the laser radar coordinate system and the distance of each point in the gimbal zoom camera coordinate system; obtaining the scaling factor of each point in the three-dimensional coordinate system by minimizing the objective function, and multiplying the coordinates of the gimbal zoom camera coordinate system by their respective scaling factors; and again performing optimal fitting on the three-dimensional coordinates of the spherical object in the laser radar coordinate system and the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system to obtain the final transformation matrix.

[0072] Specifically, the conversion matrix between the laser radar and the PTZ camera has been obtained. However, the calculation process of this matrix contains some human factors, which leads to a large error in the conversion matrix, so this matrix needs to be optimized. The main source of the conversion matrix error is the large error in the three-dimensional coordinates of the PTZ camera coordinate system, and the reason for this is that the conversion function of the magnification and the distance has a large error, that is, the main source of the error is the distance in the spherical coordinates. Therefore, it is necessary to appropriately scale the distance to obtain more accurate spherical coordinates. The values ​​of the three axes of the Cartesian coordinates are proportional to the distance, so it is sufficient to scale directly on the Cartesian coordinates. Assuming that there are n coordinates in total, the scaling factors of each coordinate are a_1,..., a_n, respectively, and the objective function is set to the difference between the distance of each point in the laser radar coordinate system and the distance of each point in the PTZ camera coordinate system. By minimizing this objective function, a_1,..., a_n can be obtained. The coordinates of the camera coordinate system are multiplied by their respective scaling factors to obtain the final coordinates, and finally the best fit is performed to obtain a more accurate conversion matrix.

[0073] like Figure 5The figure shows the second calibration flow chart between the camera and the LiDAR. The camera origin is calculated, and the relationship between the magnification and the distance is fitted; based on the camera SDK, a set of camera coordinates is obtained according to the fitted relationship, and then the scaling factor of the camera coordinates is obtained according to the obtained set of LiDAR coordinates, and a new set of camera coordinates is obtained according to the scaling factor, and then the best fit is performed to obtain the transformation matrix.

[0074] The calibration device between the gimbal zoom camera and the laser radar provided by the present invention is described below. The calibration device between the gimbal zoom camera and the laser radar described below and the calibration method between the gimbal zoom camera and the laser radar described above can be referenced to each other.

[0075] like Figure 6 The present invention shows a calibration device between a pan-tilt zoom camera and a laser radar, comprising:

[0076] The pan-tilt zoom camera coordinate origin acquisition module 610 is used to obtain the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system;

[0077] A fitting module 620, used for fitting the conversion relationship between the magnification of the pan-tilt zoom camera and the distance based on the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system;

[0078] An initial conversion matrix determination module 630 is used to determine an initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship;

[0079] The final conversion matrix determination module 640 is used to optimize the initial conversion matrix to obtain a final conversion matrix.

[0080] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the calibration method between the pan-tilt zoom camera and the laser radar, the method comprising: obtaining the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system; fitting the conversion relationship between the magnification and distance of the pan-tilt zoom camera based on the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system; determining the initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship; optimizing the initial conversion matrix to obtain the final conversion matrix.

[0081] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0082] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the calibration method between the gimbal zoom camera and the laser radar provided by the above methods, and the method includes: obtaining the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system; fitting the conversion relationship between the gimbal zoom camera magnification and distance based on the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system; determining the initial conversion matrix between the laser radar and the gimbal zoom camera based on the conversion relationship; and optimizing the initial conversion matrix to obtain the final conversion matrix.

[0083] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the calibration method between the gimbal zoom camera and the laser radar provided by the above-mentioned methods, the method comprising: obtaining the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system; fitting the conversion relationship between the magnification and distance of the gimbal zoom camera based on the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system; determining the initial conversion matrix between the laser radar and the gimbal zoom camera based on the conversion relationship; and optimizing the initial conversion matrix to obtain the final conversion matrix.

[0084] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0085] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method between a pan-tilt zoom camera and a laser radar, characterized in that: include: Get the position of the origin of the gimbal-zoom camera coordinate system in the lidar coordinate system; Fitting the conversion relationship between the magnification and distance of the gimbal-zoom camera based on the position of the origin of the gimbal-zoom camera coordinate system in the laser radar coordinate system; Determine an initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship; Optimizing the initial conversion matrix to obtain a final conversion matrix; The obtaining of the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system specifically includes: setting the vertical angle of the gimbal zoom camera to 0°, and then setting the horizontal angles of the gimbal zoom camera to 0°, 90°, and 180° respectively; at each horizontal angle, pointing the laser radar to the center of the gimbal zoom camera screen at the corresponding angle, and obtaining three laser radar coordinate points A, B, and C; Take any point P on the surface of the gimbal zoom camera, project P onto the line segment AC, and obtain the projection point P'; continuously adjust the position of the projection point P' so that the angle of ∠BP'A meets the preset conditions; the position of the projection point P' that meets the preset conditions is used as the position of the origin of the gimbal zoom camera coordinate system in the lidar coordinate system.

2. The calibration method between a pan-tilt zoom camera and a laser radar according to claim 1, characterized in that: The step of fitting the conversion relationship between the magnification and distance of the gimbal zoom camera based on the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system specifically includes: Place the target object at different distances from the pan / tilt zoom camera; Use laser radar to obtain the coordinates of target objects at multiple different distances; Obtaining the distance of the target object relative to the pan-tilt zoom camera according to the coordinates of the target object at the multiple different distances and the coordinates of the origin of the pan-tilt zoom camera; In the image of the pan-tilt zoom camera, magnify multiple targets at different distances to the same size and record the corresponding magnification values; The relationship between magnification and distance is fitted to a quadratic equation to determine the relationship between magnification and distance.

3. The calibration method between a pan-tilt zoom camera and a laser radar according to claim 1, characterized in that: The determining of the initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship specifically includes: Place a spherical object at at least n locations around the device to obtain the three-dimensional coordinates of the spherical object in a laser radar coordinate system; the device includes a laser radar and a ball camera; Obtaining a pan / tilt zoom value, converting the pan / tilt zoom value into Cartesian coordinates, and obtaining the three-dimensional coordinates of the spherical object in the pan / tilt zoom camera coordinate system; the pan / tilt zoom value includes a horizontal angle, a vertical angle, and a magnification; The three-dimensional coordinates of the spherical object in the laser radar coordinate system and the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system are optimally fitted to obtain the initial transformation matrix between the laser radar and the gimbal zoom camera.

4. The calibration method between a pan-tilt zoom camera and a laser radar according to claim 1, characterized in that: The optimizing the initial conversion matrix to obtain the final conversion matrix specifically includes: The error in determining the initial transformation matrix comes from the three-dimensional coordinate error of the pan-tilt zoom camera coordinate system; The objective function is obtained as the difference between the distance of each point in the laser radar coordinate system and the distance of each point in the gimbal zoom camera coordinate system; By minimizing the objective function, the scaling factor of each point in the three-dimensional coordinate system is obtained, and the coordinates of the pan-tilt zoom camera coordinate system are multiplied by the respective scaling factors; The three-dimensional coordinates of the spherical object in the laser radar coordinate system and the three-dimensional coordinates of the spherical object in the gimbal zoom camera coordinate system are best fitted again to obtain the final transformation matrix.

5. The calibration method between a pan-tilt zoom camera and a laser radar according to claim 3, characterized in that: The converting the pan / tilt zoom value into Cartesian coordinates specifically includes: The gimbal zoom value is converted into spherical coordinates, and then converted into Cartesian coordinates through trigonometric functions.

6. A calibration device between a pan-tilt zoom camera and a laser radar, characterized in that: include: The gimbal zoom camera coordinate origin acquisition module is used to obtain the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system, including: setting the vertical angle of the gimbal zoom camera to 0°, and then setting the horizontal angle of the gimbal zoom camera to 0°, 90° and 180° respectively; at each horizontal angle, pointing the laser radar to the center of the gimbal zoom camera screen at the corresponding angle, and obtaining three laser radar coordinate points A, B and C; taking any point P on the surface of the gimbal zoom camera, and projecting P to the line segment AC to obtain the projection point P'; continuously adjusting the position of the projection point P' so that the angle of ∠BP'A meets the preset conditions; taking the position of the projection point P' that meets the preset conditions as the position of the origin of the gimbal zoom camera coordinate system in the laser radar coordinate system; A fitting module, used for fitting the conversion relationship between the magnification and distance of the pan-tilt zoom camera based on the position of the origin of the pan-tilt zoom camera coordinate system in the laser radar coordinate system; An initial conversion matrix determination module, used to determine the initial conversion matrix between the laser radar and the pan-tilt zoom camera based on the conversion relationship; The final conversion matrix determination module is used to optimize the initial conversion matrix to obtain the final conversion matrix.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the calibration method between the pan-tilt zoom camera and the laser radar as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the calibration method between the pan-tilt zoom camera and the laser radar as described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the calibration method between the pan-tilt zoom camera and the laser radar as described in any one of claims 1 to 5 is implemented.

Images