Joint calibration method and calibration device based on fisheye camera and line laser
Through the joint calibration method of fisheye camera and line laser, the sliding target assembly and camera calibration model are used to extract the center pixel coordinates of the light bar and convert them into three-dimensional coordinates of the laser point, solving the problem of low calibration efficiency of fisheye camera and achieving high efficiency calibrates the linear laser parameters.
Patent Information
- Application Number
- CN202310660516.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
During the calibration process of fisheye cameras, it is difficult to ensure that the checkerboard image covers the entire field of view angle, resulting in low calibration efficiency. The traditional single checkerboard method is complicated, making it difficult to extract feature points in the edge area, affecting the calibration results. reliability.
Using a joint calibration method based on fisheye camera and line laser, the target assembly is controlled to slide relative to line laser module, and the internal and external parameters are output in combination with the camera calibration model, the center pixel coordinates of the light bar are extracted and converted into three-dimensional coordinates of laser points to realize the fish. The combined calibration of eye camera and line laser reduces artificial moving targets and improves calibration efficiency.
While ensuring the reliability of calibration results, the efficiency of the joint calibration of the fisheye camera and the line laser is improved, the calibration process is simplified, the calculation amount is reduced, and the stability and efficiency of calibration are improved.
Smart Images

Figure CN119090969A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of sensor parameter calibration, and in particular to a joint calibration method and a calibration device based on a fisheye camera and a line laser. Background Art
[0002] The structured light vision system is a measurement system that combines a line laser and a camera. Before use, the line laser surface needs to be calibrated, and the plane coefficients need to be calculated to express the spatial structure of the laser surface. Then the camera acquires the image and performs three-dimensional measurement through the imaging principle and the line laser surface. As a special corner point, the checkerboard corner point is widely used in camera calibration. The checkerboard pattern is widely used in camera calibration because of its obvious contrast, simple features, and easy detection and recognition. However, in the current structured light vision system, the camera cannot guarantee that the checkerboard image completely covers the entire camera's field of view, and it is not easy to use the checkerboard to extract feature points in the edge area. When calibrating the internal parameters of the fisheye camera, in order to ensure the reliability of the calibration results, it is often necessary to adjust the same checkerboard to be located at different angles of the camera for collection. Too many pictures are often used and the coordinate matrix conversion is too complicated, resulting in low calibration efficiency, which is not suitable for promoting the production and manufacturing process of the structured light vision system. Summary of the invention
[0003] This application discloses a method and a calibration device based on the joint calibration of a fisheye camera and a line laser. The specific technical solution is as follows:
[0004] The method for joint calibration based on a fisheye camera and a line laser comprises: step 1, controlling a target assembly to slide a plurality of distances relative to a line laser module; controlling a line laser to emit a line laser to the target assembly, projecting a light strip on the surface of the target at a corresponding angle; and controlling a fisheye camera to sequentially photograph a target or a light strip projected from its surface set at at least one angle in the target assembly at each sliding position, so as to obtain an image to be calibrated; wherein the line laser module comprises a line laser and a fisheye camera, and the relative position between the line laser and the fisheye camera is fixed; step 2, inputting pixel data of the image to be calibrated at different sliding positions into a camera calibration model, and outputting parameters of the fisheye camera; step 3, extracting the center pixel coordinates of the light strips from the images to be calibrated at different sliding positions respectively, and then converting all the extracted center pixel coordinates of the light strips into three-dimensional coordinates of laser points according to the parameters of the fisheye camera, and determining that the joint calibration of the line laser and the fisheye camera is completed.
[0005] In summary, in the case where the line laser emits a line laser to the target assembly and controls the fisheye camera to capture the image of the target set at various angles at various sliding positions or the image of the light strip projected from its surface, the target assembly is controlled to slide multiple distances relative to the line laser module, and the coordinate system of the center of the light strip is converted in combination with the internal and external parameters output by the camera calibration model to complete the joint calibration of the line laser and the fisheye camera. There is no need to manually move the target, and the images of multiple targets at different angles set in the target assembly can be captured at one time at the same position. Compared with the traditional single chessboard joint calibration method, the efficiency of the joint calibration of the line laser and the fisheye camera is improved while ensuring the reliability of the calibration results of the fisheye camera internal parameters. When the pixel coordinates of the center of the light strip are automatically extracted from the images to be calibrated at different sliding positions and converted into the three-dimensional coordinates of the laser point, the calibration operation of the laser point can be determined to be completed, forming a more efficient line laser parameter calibration method under the shooting angle of the fisheye camera, and improving the efficiency of the joint calibration of the line laser and the fisheye camera while ensuring the calibration accuracy.
[0006] Furthermore, the target assembly includes a plurality of calibration plates, any two calibration plates are set at an angle; the normal vectors of each calibration plate are different; a target is set on the surface of a calibration plate so that any two targets are also set at an angle; wherein, the closer the target assembly slides to the line laser module, the fewer targets are included in a frame of the image to be calibrated captured by the fisheye camera; wherein, the farther the target assembly slides to the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera. Considering the clarity of the fisheye camera when focusing, try to achieve a full-viewing angle balancing effect for calibration plates set at various angles.
[0007] Furthermore, in step 2, before the fisheye camera is calibrated, it also includes: whenever the target assembly slides to a sliding position, in a frame of the image to be calibrated taken in real time by the fisheye camera, searching for corner points in the complete target; in the same frame of the image to be calibrated, grouping the corner points searched out according to the complete target taken, a group of corner points corresponds to a complete target; and then numbering the complete targets where each group of corner points is located, so that the targets set at different angles in the same frame of the image to be calibrated have different numbers; the target assembly slides to different sliding positions, and the numbers of the complete targets in different frames of the image to be calibrated taken by the fisheye camera are different. In summary, before the internal parameters of the fisheye camera are calibrated, the corner point recognition of multiple targets and the numbering of the targets are realized. Because each calibration plate or each target in the target assembly has a specific distribution, after numbering, it is convenient to match the positional relationship between the light strip and the calibration plate where it is located in an orderly and fast manner during the coordinate system conversion process in step 3.
[0008] Furthermore, in the same frame of the image to be calibrated, if the number of corner points searched out from one of the targets is equal to a preset number threshold, one of the targets is set as a complete target; a complete target is a black and white square area; the black and white square area is composed of black cells and white cells on the surface of a calibration plate.
[0009] Furthermore, in the same complete target, a polygonal area is formed by expanding outward from the corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light bar, wherein the polygonal area includes all cells in the black and white square area and the white area outside the target; and then the polygonal area is set as the ROI area. A pixel area is framed around the target using the corner point, which improves the stability of calibration and reduces the amount of calculation.
[0010] Furthermore, the camera calibration model is a full-view camera model calibration correction model or a fisheye lens distortion correction model, which is used to calculate the parameters of the fisheye camera, including the intrinsic parameters of the fisheye camera and the extrinsic parameters of each target to the fisheye camera.
[0011] Further, in step 1, whenever the target assembly slides linearly to a sliding position relative to the line laser module, the relative position between the target assembly and the line laser module is kept unchanged, and the line laser and the fisheye camera are fixedly assembled, there is: firstly controlling the line laser to emit the line laser to the target assembly, and controlling the fisheye camera to capture at least one complete target, to obtain a frame of laser stripe image corresponding to a sliding position, a frame of laser stripe image is a frame of image to be calibrated including light stripes; then controlling the line laser to stop emitting the line laser, and turning on the infrared fill light to fill light for the target assembly, and controlling the fisheye camera to capture the at least one complete target. Target, obtain a frame of target image corresponding to the same sliding position, so that step 2 searches for corner points from the target image and expands the ROI area, and then frames the ROI area to the same pixel area of the laser stripe image, and the position relationship of the same pixel area in the laser stripe image is the position relationship of the ROI area in the target image; wherein, a frame of target image is a frame of image to be calibrated that does not include light strips; the center pixel coordinates of the light stripe are the projection position of the intersection line of the line laser emitted by the line laser and the target in the pixel plane of the fisheye camera, and are located in the laser stripe image; the intersection line of the line laser emitted by the line laser and the target is the light strip.
[0012] In summary, every time the target component moves a certain distance, two types of images must be collected, namely, a frame of laser stripe image and a frame of target image; when shooting a frame of target image, the infrared fill light is turned on, but the line laser is not turned on, so the target can only be identified from the target image, and the internal reference calibration of the fisheye camera can be performed through the camera calibration model. When shooting a frame of laser stripe image, the infrared fill light is not turned on, but the line laser is turned on, wherein only the light strips can be identified from the laser stripe image, but the target cannot be identified, and then combined with the target identified from the target image, it can be used for the joint calibration of the line laser and the fisheye camera, thereby utilizing the intermediate data in the internal reference calibration of the fisheye camera, which speeds up the calibration process.
[0013] Furthermore, in step 2, the external parameters from the target assembly at each sliding position to the fisheye camera are obtained by inputting the pixel data of the image to be calibrated into the camera calibration model, and the external parameters from the target assembly at each sliding position to the fisheye camera include the external parameters of the cells in the targets set at each angle at each sliding position; wherein the pixel data of the image to be calibrated includes but is not limited to the pixel coordinates of the corner points in the numbered targets. The camera calibration model realizes the conversion of the numbered corner points and the cell position information in the target into the internal and external parameters of the fisheye camera to obtain the true spatial scale on the target.
[0014] Furthermore, the step 3 specifically includes: extracting the center pixel coordinates of the light strip from the ROI area in the image to be calibrated at different sliding positions, and setting the center pixel coordinates of the light strip to be the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera; then based on the coordinate system conversion function set in the preset mathematical analysis software, using the parameters of the fisheye camera calibrated by the camera calibration model, mapping the two-dimensional pixel coordinates (u, v) to the light spot coordinates on the target surface with the corresponding number The light spot coordinates is located in the world coordinate system set on the corresponding calibration plate; then the light spot coordinates are converted to Convert the three-dimensional coordinates of the laser point into the camera coordinate system When the center pixel coordinates of all light stripes extracted in the ROI area of the image to be calibrated at different sliding positions are converted into the corresponding three-dimensional coordinates of the laser points in the camera coordinate system, the joint calibration of the line laser and the fisheye camera is completed. This is conducive to converting the depth information of a batch of laser points and completing the calibration of the laser point cloud data that constitutes the light stripes.
[0015] Furthermore, the coordinates of the corner points of the light bar center pixel coordinates extracted in the ROI region that have a projection relationship in the target are the light spot coordinates. The rotation matrix R and displacement vector T of the corner point generated by the camera calibration model, which has a projection relationship with the center pixel coordinates of the currently extracted light strip, are used to convert the corner point coordinates into the three-dimensional coordinates of the laser point in the camera coordinate system. The corresponding conversion formula is: Among them, the rotation matrix R and the displacement vector T are both external parameters of the fisheye camera, and the corner points that have a projection relationship with the center pixel coordinates of the currently extracted light strip are located on the target surface with the corresponding number. Based on the external parameters, the two-dimensional pixel coordinates corresponding to the targets set at different angles at different sliding positions (the targets pre-numbered in step 2) are transformed into world two-dimensional coordinates in sequence, and then transformed from world two-dimensional coordinates into camera coordinates, and the point cloud data in the camera coordinate system is calibrated.
[0016] Furthermore, in step 3, the pixel coordinates of the center of the light strip are extracted from the ROI area in the image to be calibrated at different positions, and the pixel coordinates of the center of the light strip are set as the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera. Then, according to the camera calibration model, the pixel coordinates of the center of the light strip are converted into distortion-free three-dimensional coordinates. Wherein, λ is the scaling factor; a0, a2, a3 and a4 are all the distortion coefficients of the fisheye camera, and are all the polynomial coefficients described by the camera calibration model; Then, the undistorted three-dimensional coordinates are obtained by using the corresponding numbered targets to obtain the external parameters required by the fisheye camera. Converted into the light spot coordinates of the target surface with the corresponding number (calibration plate with the corresponding number) This can eliminate the lens distortion in the image to be calibrated taken by the fisheye camera and obtain the distortion-free coordinate information of the corresponding target surface.
[0017] A calibration device, comprising a calibration drive component, a target component, a track, a fixed support platform, a line laser module, and a parameter calibration unit for executing the joint calibration method; wherein the target component comprises a plurality of targets arranged at an angle, and the line laser module comprises a line laser and a fisheye camera; the fixed support platform is fixedly assembled with the line laser and the fisheye camera; the fixed support platform is fixedly arranged at one end of the track, and the target component is slidably arranged at the other end of the track. The fisheye camera supports the expansion of the field of view angle, and the imaging range tries to catch up with the scanning range of the target of the line laser at various angles at multiple distances, thereby reducing the number of imaging sensors used, and the shooting process of the fisheye camera does not require manual movement of the target, and is easy to realize automation; thereby reducing the number of calibrations and the source of errors, and improving the efficiency of the joint calibration of the line laser and the fisheye camera.
[0018] Furthermore, there are five calibration plates in the target assembly; the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate; among the remaining four calibration plates, there is one calibration plate located above the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is one calibration plate located below the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is one calibration plate located to the left of the reference calibration plate and the reference calibration plate is set at a predicted observation angle, and there is one calibration plate located to the right of the reference calibration plate and the reference calibration plate is set at a predicted observation angle, so that when the target assembly slides along the track, the fisheye camera can capture at least one complete target in the target assembly and the white area set around it. Ensure that the field of view of the fisheye camera is equivalent to the size of the target set at each angle, so as to ensure that the target assembly is still within the field of view of the fisheye camera when it slides. On the surface of a calibration plate, the outside of the target is surrounded by a white area, leaving a certain amount of blank white background for a complete target on the surface of the calibration plate for line laser calibration, because pure white is conducive to the fisheye camera capturing clear light strips.
[0019] Furthermore, an infrared fill light is fixedly installed on the fixed support platform; whenever the target assembly slides to a sliding position along the track relative to the line laser module, the relative position between the target assembly and the line laser module is kept unchanged, and the line laser and the fisheye camera are fixedly installed, there is: the parameter calibration unit controls the line laser to emit the line laser to the target assembly, and controls the fisheye camera to capture at least one complete target, to obtain a frame of laser stripe image corresponding to a sliding position, and a frame of laser stripe image includes a frame of image to be calibrated including light strips; the parameter calibration unit controls the line laser to stop emitting the line laser, and turns on the infrared fill light to fill light for the target assembly, and controls the fisheye camera to capture the at least one complete target, to obtain a frame of target image corresponding to the same sliding position; wherein a frame of target image is a frame of image to be calibrated that does not include light strips; on the fixed support platform, the relative position between the infrared fill light and the line laser module remains unchanged.
[0020] In summary, the present technical solution uses the calibration plates in the target assembly as a medium to calibrate the fisheye camera and the line laser. On the one hand, the present embodiment can perform joint calibration of the camera and the line laser on the camera equipped with a fisheye lens (hereinafter referred to as the fisheye camera) without modifying the camera that needs to be calibrated. On the other hand, the calibration device used in the present technical solution is low-cost, and data acquisition is simple and direct. It integrates heterogeneous data through coordinate system transformation from the underlying hardware level, and calibrates the three-dimensional coordinate data of the laser points at various angles at various sliding positions in the camera coordinate system.
[0021] Furthermore, the track is arranged along the vertical direction, the fixed support platform is placed on the horizontal plane and fixedly arranged at the lower end of the track, the emission direction of the line laser and the shooting angle of the fisheye camera are both arranged upward relative to the fixed support platform; the reference calibration plate is arranged on the normal plane of the track to be parallel to the horizontal plane, and the five calibration plates are all fixed on the calibration drive assembly by the connecting bracket, and the calibration drive assembly drives the five calibration plates to slide up and down relative to the line laser module. By adjusting the relative positions of the five calibration plates on the track, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible near the maximum ranging boundary of the track, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the direction of the fisheye camera). BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 FIG. 1 is a schematic diagram of the arrangement of five calibration plates disclosed in one embodiment of the present invention.
[0023] Figure 2 It is a schematic diagram of the structure of a calibration device disclosed in one embodiment of the present invention.
[0024] Figure 3 It is a two-dimensional plane schematic diagram of a target set up with black and white square areas according to an embodiment of the present invention.
[0025] Figure 4 The present invention discloses a flow chart of a method for joint calibration based on a fisheye camera and a line laser according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. To further illustrate the embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.
[0027] In this application, the line laser module refers to any laser module including a line laser and a fisheye camera. The fisheye camera refers to a camera equipped with a fisheye lens, and the fisheye lens can generally refer to a wide-angle lens. In the line laser module, the line laser is used to emit a line laser outward. Among them, the line laser emitted by the line laser transmitter can be located in front of the object to be measured; the fisheye camera can collect environmental images, and can also receive the reflected light returned by the line laser hitting the object, wherein the line laser emitted by the line laser transmitter is located within the field of view of the fisheye camera; the line laser can help detect the contour, height and / or width of the object to be measured in its emission direction, collectively referred to as the position information of the laser spot or laser light strip. In this application, the object to be measured in the emission direction of the line laser transmitter is a target for calibration, and the image captured by the fisheye camera is the image to be calibrated. The fisheye camera is used to collect image data through a wide-angle lens. The fisheye camera can be an electronic device with image acquisition capabilities such as a camera or a video camera with a fisheye lens.
[0028] The present application discloses a method for joint calibration based on a fisheye camera and a line laser. The execution subject of the joint calibration method is a control unit for calibrating a line laser module, which can be set outside the line laser module. The line laser module includes a line laser and a fisheye camera, which are relatively fixedly assembled into one body, and the relative position between the line laser and the fisheye camera is fixed. Figure 4 As shown, the method includes:
[0029] Step 1, control the target assembly to slide multiple distances relative to the line laser module. In this embodiment, the distance between the target assembly and the line laser module is limited to change. Generally, the line laser module is fixed, and the target assembly moves relative to the line laser module. Whenever the target assembly slides a distance relative to the line laser module, the target assembly slides to a new sliding position. And control the fisheye camera to sequentially shoot the target set at at least one angle in the target assembly at each sliding position or the light strip projected from its surface to obtain the image to be calibrated. Then execute step 2.
[0030] In step 1, the line laser is controlled to emit a line laser to the target assembly, and a light strip is projected in the target at the corresponding angle, which can be a strip of bright light on the target surface; during the sliding process of the target assembly, the emission direction of the line laser emitted by the line laser to the target assembly remains unchanged. When the line laser is working, the emitted line laser forms a light strip on the measured surface of the target assembly (recorded as the target plane), which is also regarded as a collection of laser light spots; when the line laser is working, the emitted line laser remains to fall into the measured surface of the target assembly (recorded as the target plane) during the sliding process of the target assembly, that is, during the sliding process of the target assembly, the line laser emitted by the line laser always projects the range of the measured surface of the target assembly, including the target surface set at at least one angle in the target assembly. The line laser may not emit a line laser, so as to facilitate the acquisition of a complete target, and to determine the external parameters of each target to the fisheye camera in sequence during the calibration of the fisheye camera.
[0031] It should be noted that, in this embodiment, in order to implement the joint calibration of the fisheye camera and the line laser, targets at multiple angles are set in the target assembly. The fisheye camera can capture the target at at least one angle with its built-in fisheye lens, and can also capture the targets at all angles set in the target assembly at a distance from the target assembly at one time, thereby avoiding frequent manual adjustment of the target installation angle and the distance between the target and the fisheye camera.
[0032] In step 1, during the sliding process of the target assembly, the fisheye camera must not only take an image of the target when the line laser emits a line laser, as the image to be calibrated, which is specifically used for the calibration of laser parameters; it must also keep taking images of the target when the line laser does not emit a line laser, and when the line laser does not emit a line laser, when the target assembly slides to a new position, it is necessary to control the fisheye camera to sequentially take pictures of the complete target set at at least one angle in the target assembly, as the image to be calibrated, which is specifically used for fisheye camera calibration. It is worth noting that when the image to be calibrated includes light strips, a laser stripe image is formed, and there is at least a white area where the line laser is projected into the target, which is convenient for capturing light spots for laser point cloud calibration. When the image to be calibrated does not include light strips, a target image is formed.
[0033] The target can be designed with a checkerboard layout, and the corresponding external parameters can be obtained through the checkerboard, that is, the position and distance between the light spot and the corner point of the checkerboard can be found through the light spot of the line laser hitting the checkerboard, thus reducing the calibration error.
[0034] In some embodiments, the control unit for line laser module calibration contains an encoder, which controls the movement of the target assembly. Every time the encoder moves a preset unit distance, it triggers the fisheye camera to capture an image, and an image is captured once when the line laser emits a line laser and once when the line laser does not emit a line laser.
[0035] Step 2: Input the pixel data of the image to be calibrated at different sliding positions into the camera calibration model in sequence to calibrate the fisheye camera, and output the parameters of the fisheye camera. It can be determined that the calibration of the fisheye camera is completed, but the joint calibration of the line laser and the fisheye camera is not completed; then it is necessary to continue to perform step 3. Then, the two-dimensional pixel coordinates can be converted into three-dimensional coordinates in the camera coordinate system according to the parameters of the fisheye camera, and the depth information is given. The camera calibration model can be derived from the mathematical analysis calculation software disclosed in the prior art. The mathematical interface function set in the camera calibration model plays a role in calculating the pixel data of the image to be calibrated, which is used to calculate the internal parameters of the fisheye camera and the external parameters from the target to the fisheye camera, so as to complete the calibration of the fisheye camera before executing step 3; generally, after the internal parameters of the fisheye camera are calibrated first, the external parameters of the target set at different distances and angles to the fisheye camera can be calculated. After the internal parameters of the fisheye camera are calibrated, a rigid transformation pointing to the preset coordinate system (the so-called external parameter) can be specified to map the pixel coordinates to the preset coordinate system.
[0036] Preferably, the camera calibration model takes into account the distortion coefficient of the fisheye lens and uses a distortion coefficient elimination construction formula to overcome the image distortion error caused by the distortion in the direction of the corresponding coordinate axis.
[0037] Step 3, extract the pixel coordinates of the center of the light strip from the images to be calibrated at different sliding positions, and then convert all the extracted pixel coordinates of the center of the light strip into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera, that is, complete the calibration of the laser point cloud on the basis of completing the calibration of the fisheye camera, and regard it as and determine that the joint calibration of the line laser and the fisheye camera is completed. The three-dimensional coordinates of the laser point in step 3 are expressed as the coordinates of the light strip corresponding to each sliding position of the camera coordinate system, which is a three-dimensional coordinate with depth. This depth is provided by the external reference from the target to the fisheye camera and the internal reference of the fisheye camera, preferably the coordinate value in the Z-axis direction. In this way, the three-dimensional coordinates of the center pixel coordinates of the light strip captured at multiple different distances and angles in the camera coordinate system are used to calculate the joint calibration of the line laser and the fisheye camera.
[0038] Based on the above steps 1 and 2, the image used for fisheye camera calibration is the image to be calibrated that does not include the light strip. Specifically, the internal parameters of the fisheye camera are calibrated first, and then the calibrated internal parameters and the external parameters of the target to the fisheye camera are used to perform the coordinate system conversion operation of the laser point to complete the joint calibration of the line laser and the fisheye camera.
[0039] In summary, in this embodiment, when the line laser emits a line laser to the target assembly and controls the fisheye camera to capture the image of the target set at various angles at various sliding positions or the light strip projected from its surface, the target assembly is controlled to slide multiple distances relative to the line laser module, and the coordinate system of the center of the light strip is converted in combination with the internal and external parameters output by the camera calibration model, so as to complete the joint calibration of the line laser and the fisheye camera. There is no need to manually move the target, and the images of multiple targets at different angles set in the target assembly can be captured at one time at the same position. Compared with the traditional single chessboard joint calibration method, the efficiency of the joint calibration of the line laser and the fisheye camera is improved while ensuring the reliability of the calibration results of the fisheye camera internal parameters. When the pixel coordinates of the center of the light strip are automatically extracted from the images to be calibrated at different sliding positions and converted into the three-dimensional coordinates of the laser point, it can be determined that the calibration operation of the laser point is completed, forming a more efficient line laser parameter calibration method under the shooting angle of the fisheye camera, and improving the efficiency of the joint calibration of the line laser and the fisheye camera while ensuring the calibration accuracy.
[0040] As an embodiment, the target assembly includes a plurality of calibration plates, any two calibration plates are arranged at an angle, and the normal vectors of each calibration plate are different, so that each calibration plate included in the target assembly has a different orientation relative to the optical axis of the fisheye camera; a target is arranged on the surface of a calibration plate, and each target included in the target assembly has a different orientation relative to the optical axis of the fisheye camera. In this embodiment, the closer the target assembly slides to the line laser module, the fewer targets are included in a frame of the image to be calibrated taken by the fisheye camera; a frame of the image to be calibrated taken by the fisheye camera can be regarded as a frame of calibration plate image, which can include light strips. In combination Figure 1 and Figure 2It can be seen that not all targets can be captured in the image to be calibrated at a distance closer to the line laser module, but the target assembly can be photographed as a complete target by the fisheye camera at the minimum ranging boundary of the track 106. The minimum ranging boundary of the track 106 is the closest position that allows the targets in the target assembly to be imaged in the fisheye camera. In this embodiment, the farther the target assembly slides from the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera. The target assembly can be photographed as a complete target in all calibration plates by the fisheye camera at the maximum ranging boundary of the track 106. The maximum ranging boundary of the track 106 is the farthest position that allows all targets in the target assembly to be imaged in the fisheye camera. Therefore, when the target is imaged in the fisheye camera, it is necessary to consider the clarity of the fisheye camera when focusing, and try to achieve a full-viewing angle balancing effect for the calibration plates set at various angles.
[0041] exist Figure 1 In the target assembly, there are five calibration plates, and the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate 101 to reduce the error; the other four calibration plates are marked counterclockwise, namely calibration plate 102, calibration plate 103, calibration plate 104 and calibration plate 105. The sizes of calibration plates 102, 103, 104 and 105 are not necessarily equal. Calibration plate 102 is located above the reference calibration plate 101 and the reference calibration plate is set at a predicted observation angle. Calibration plate 104 is located below the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 103 is located to the left of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 105 is located to the right of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. The predicted observation angle is preferably 120 degrees. It is ensured that the field of view of the fisheye camera is comparable to the size of the target set at each angle, so as to ensure that the target assembly is still within the field of view of the fisheye camera when sliding.
[0042] like Figure 1 As shown, the reference calibration plate 101 and the calibration plate 102 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 103 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 104 are combined into a V-shaped calibration plate, and the reference calibration plate 101 and the calibration plate 105 are combined into a V-shaped calibration plate; the angle formed by the reference calibration plate 101 and the calibration plate 102 can be set according to actual conditions, that is, the predicted observation angle setting, which is not limited here. For example, the reference calibration plate 101 and the calibration plate 102 are preferably placed at an angle of 120 degrees; the reference calibration plate 101 and one side of the calibration plate 102 are arranged to fit each other, so that a part of the target assembly is formed by combining two calibration plates, and targets at two angles can be set accordingly.
[0043] To facilitate the placement of the V-shaped calibration plate, the two plates in the V-shaped calibration plate can be connected by hinges. The implementation form of the hinge connection can be formed by adding obtuse angle structures around the reference calibration plate to fix the connection with screws. In one embodiment, the connecting strip supports the two plates to be opened and closed to any angle to facilitate the external parameter calibration between the line laser and the fisheye camera. For example, it can be adjusted so that near the maximum ranging boundary of the track 106, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the direction of the fisheye camera).
[0044] To improve the calibration efficiency, the longer right-angled sides of the two calibration plates can be fixed to each other. In other embodiments, the shapes of the two plates in the V-shaped calibration plate are not limited to quadrilateral plates such as rectangular plates.
[0045] Preferably, the number of different positions that the target assembly slides along the track is greater than or equal to 4. Frequently capturing 4 to-be-calibrated images at different distances for camera calibration is more efficient than the traditional single chessboard joint calibration method.
[0046] Preferably, the normal vector of the reference calibration plate 101 is parallel to the optical axis of the fisheye camera. Considering the symmetry problem, the calibration plate 103 located on the left of the reference calibration plate 101 and the calibration plate 105 located on the right of the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate, and the targets on the left and right of the reference calibration plate 101 are symmetrically arranged; the calibration plate 102 located above the reference calibration plate 101 and the calibration plate 104 located below the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate, and the targets on the upper and lower sides of the reference calibration plate 101 are symmetrically arranged. The calibration error is further reduced.
[0047] In some embodiments, the key to the target assembly sliding along the track 106 is to allow the target to be photographed by the fisheye camera at several different distances, including far, medium and near. At least: when the target assembly slides to a sliding position at a first distance from the line laser module, the fisheye camera photographs 1 complete target; when the target assembly slides to a sliding position at a second distance from the line laser module, the fisheye camera photographs 1 to 3 complete targets; when the target assembly slides to a sliding position at a third distance from the line laser module, the fisheye camera photographs 5 complete targets; wherein, one target is set on a calibration plate, the first distance is smaller than the second distance, and the second distance is smaller than the third distance.
[0048] As an embodiment, in step 2, before the fisheye camera calibration is performed, the method further includes: whenever the target assembly slides to a sliding position, searching for corner points in a frame of the image to be calibrated taken in real time by the fisheye camera within the complete target, combining Figure 3It can be seen that when the target is a black and white square area, the corner point is the intersection of the edge lines of the black cell or the intersection of the edge lines of the white cell, which can be regarded as a two-dimensional coordinate located on the pixel plane. Then, in the same frame of the image to be calibrated, the searched corner points are grouped according to the complete target taken. A group of corner points corresponds to a complete target, and they can be numbered according to the calibration plate where the corner points are located to distinguish the angular orientation information of the target where the corner points are located. Therefore, a group of corner points is the corner points in a target set at an angle captured by the fisheye camera when the target assembly slides to a sliding position corresponding to a distance. Then the complete target where each group of corner points is located is numbered so that the calibration plates set at different angles in the same frame of the image to be calibrated have different numbers.
[0049] The target assembly slides to different sliding positions, and the numbers of each complete target in different frames of the image to be calibrated taken by the fisheye camera are different. Among them, incomplete targets are not numbered, and the corner points searched inside each numbered target are stored in a set of data. From the sensor hardware calibration level, the matching relationship between the three-dimensional space points obtained by the line laser and the two-dimensional pixel points obtained by the fisheye camera is improved, which speeds up the joint calibration of the line laser and the fisheye camera.
[0050] For example, the target assembly slides along the track 106 to a sliding position at a third distance from the line laser module, and the fisheye camera captures 5 complete targets and searches for corner points therefrom, then the corner points searched out in each complete target can be grouped together; then the complete targets in each group of corner points are numbered, that is, the 5 complete targets are numbered as target 1, target 2, target 3, target 4 and target 5 in sequence, and the specific order is not limited. Then the target assembly slides to a sliding position at a second distance from the line laser module, and the second distance is less than the third distance, and the fisheye camera captures 3 complete targets and searches for corner points therefrom, then the corner points searched out in each complete target can be grouped together; then the 5 complete targets are numbered as target 6, target 7 and target 8 in sequence, and the specific order is not limited, and it is only required to distinguish the various targets (or calibration plates) set at different angles at different distances for easy identification and extraction.
[0051] In summary, before calibrating the internal parameters of the fisheye camera, the corner points of multiple targets are identified and the targets are numbered and sorted. Because each calibration plate or each target in the target assembly has a specific distribution, numbering makes it easier to quickly and orderly match the positional relationship between the light bar and the calibration plate in step 3 during the coordinate system conversion process.
[0052] Preferably, the Harris corner detection method is used to search for corners in the complete target. It should be noted that the Harris corner detection method mainly searches for corners based on the gradient of the grayscale value change of the image pixels. Specifically, the area near the corner of the grayscale image is an area where the pixel grayscale value changes greatly, and its gradient is also very large. In other words, in the neighborhood of the non-corner position, the pixel values of each point do not change much, or are almost equal, and their gradients are relatively small. If the absolute value of the difference between the grayscale value of the point in the neighborhood and the grayscale value of the center point Image(i,j) is within a threshold t range, then this point is considered to be similar to the center point. At the same time, the similarity counter nlike(i,j) belonging to the Image(i,j) point is also increased by 1. After all the n neighborhoods of the Image(i,j) point are traversed once, the statistical value nlike(i,j) of the number of points similar to the center point in this neighborhood can be obtained. According to the size of nlike(i,j), it can be judged whether this center point is likely to be a corner point.
[0053] Based on the above embodiment, in the same frame of the image to be calibrated, if the number of corner points searched from one of the targets is equal to a preset number threshold, one of the targets is set as a complete target and assigned a number, wherein the preset number threshold is preferably 10.
[0054] A complete target is a black and white square area, such as Figure 3 As shown, similar to a chessboard, the calibration plate and the target are both rectangular, the size of the calibration plate is related to the size of the target, the black and white square area is composed of black cells and white cells, which are regarded as black cells and white cells arranged at intervals, and the aforementioned corner points are the corner points of the black cells and / or the corner points of the white cells; preferably, the number of cells constituting the square area is linearly related to the number of corner points searched out; the targets set in each calibration plate are consistent, which can be understood as providing the same calibration features for the line laser and the fisheye camera from each angle direction at the same distance.
[0055] On the surface of a calibration plate, the outside of the target is surrounded by a white area, leaving a certain amount of blank white background for a complete target on the surface of the calibration plate for line laser calibration, because pure white is conducive to the fisheye camera capturing clear light strips.
[0056] As an embodiment, in the same complete target, a polygonal area is formed by expanding outward with the corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light bar, wherein the polygonal area includes all cells in the black and white square area and the white area outside the target, so that the polygonal area frames the black and white square area, and the black and white square area is composed of all black and white cells in a single calibration plate to form a complete target or a complete checkerboard pattern; then the polygonal area is set as the ROI (Region of Interest) area, specifically, the occupied area in the pixel plane of the fisheye camera is set as the ROI area, i.e., the region of interest, so as to realize the use of the corner points to frame a pixel area around the target.
[0057] It should be noted that at multiple distances over which the target assembly slides, the line laser emitted by the line laser will fall inside the black and white square area at each sliding position, or on the blank area at the edge of the target, where the light bar is projected at the junction of black and white, and it is not easy to find the correct center of the light bar in the image to be calibrated captured by the fisheye camera. Therefore, the target is set in the middle of the calibration plate, and white space is left around the target. In this embodiment, in order to increase the ability of the fisheye camera to capture clear light bars, the ROI area covers the white area outside the target.
[0058] In this embodiment, on the surface of the calibration plate, after the reflected light of the polygon extending from the corner point changes the optical path through the fisheye camera, the occupied area in the pixel plane of the fisheye camera will be distorted and become different from the original extended polygon. For example, the square in front of the fisheye lens will be distorted after focusing and imaging, forming the ROI area. Subsequently, it becomes more accurate to directly search for the center pixel of the light bar in the ROI area, and accurately search for the light bar on the calibration plate set at the corresponding angle while adapting to the camera distortion condition, thereby improving the stability of the calibration and reducing the amount of calculation.
[0059] Preferably, the camera calibration model is a full-view camera model calibration correction model or a fisheye lens distortion correction model, which is used to calculate the parameters of the fisheye camera, including the internal parameters of the fisheye camera and the external parameters of each target to the fisheye camera, so as to calibrate the fisheye camera. Specifically, for the internal parameter calibration of the fisheye camera, the OCamCalib full-view camera model calibration correction algorithm proposed by David Scaramuzza can be used, or the calculation method based on the fisheye lens distortion correction model proposed by Juho Kannala and Sami Brandt in the open source algorithm library OpenCV can be used to calculate the camera internal parameters. Here, unless otherwise specified, this application will use the first method for explanation.
[0060] It should be noted that Scaramuzza D (David Scaramuzza), Martinelli A, Siegwart R., technicians in the relevant field, proposed a camera calibration model in the article "A toolbox for easily calibrating omnidirectional cameras" published in the "2006 IEEE / RSJ International Conference on Intelligent Robots and Systems". The model is a calibration and correction model for an omnidirectional camera model and is used to calculate the parameters of a fisheye camera, including the intrinsic parameters of the fisheye camera and the extrinsic parameters of the fisheye camera relative to each target.
[0061] As an embodiment, in step 1, whenever the target assembly slides linearly relative to the line laser module to a sliding position, the relative position between the target assembly and the line laser module is kept unchanged, and the line laser and the fisheye camera are fixedly assembled, there exists:
[0062] First, the line laser is controlled to emit a line laser to the target assembly, and the fisheye camera is controlled to capture at least one complete target to obtain a frame of laser stripe image corresponding to a sliding position. A frame of laser stripe image is a frame of image to be calibrated including light stripes. At this time, the target cannot be identified in the image to be calibrated, that is, the aforementioned black and white square area and the cells inside it, and only the central pixel coordinates of the light stripes in the calibration plate with the corresponding number can be extracted; the fisheye camera preferably adopts an infrared bandpass filter or an infrared high-pass filter to filter the reflected light of the line laser emitted by the line laser on the surface of the calibration plate, specifically, to transmit the infrared band carried by the line laser, and absorb or reflect the ambient light of other bands, so as to separate the infrared light carried by the reflected light from other interfering ambient light.
[0063] Then, neither the target assembly nor the line laser module moves, the line laser is controlled to stop emitting the line laser, and the infrared fill light is turned on to fill light for the target assembly, and the fill light increases the brightness. This is done to increase the depth of field during shooting. Because under the condition of relatively little ambient light in the factory, the fisheye camera is not able to clearly capture the information of the target and its corner points through external light, and a fill light is needed to illuminate the target. The relative position between the infrared fill light and the line laser module can remain unchanged or change, and there is no specific restriction here; and the fisheye camera is controlled to capture the at least one complete target to obtain a frame of target image corresponding to the same sliding position; specifically, in an environment where the ambient light interference is relatively small, the line laser is turned off and the target is filled with light using the infrared fill light, and then the image to be calibrated is captured. The advantage of doing this is that it facilitates image processing. Therefore, step 2 searches for corner points from the target image and expands the ROI area, and then frames the ROI area in the same pixel area of the laser stripe image. The positional relationship of the same pixel area in the laser stripe image is the positional relationship of the ROI area in the target image. Based on the pixel position information of the ROI area framed in the target image, the center pixel coordinates of the light stripe falling into the ROI area can be searched in the laser stripe image. On this basis, combined with the external parameters obtained by the fisheye camera calibration, the three-dimensional coordinates of the laser point on the intersection of the line laser and the target surface (in the camera coordinate system) can be converted, which corresponds to the spot position information of the light stripe searched in the area framed by the ROI area in the target, thereby narrowing the search range of the light stripe and improving the efficiency of the joint calibration.
[0064] In summary, every time the target component moves a certain distance, two types of images must be collected, namely, a frame of laser stripe image and a frame of target image; when shooting a frame of target image, the infrared fill light is turned on, but the line laser is not turned on, so the target can only be identified from the target image, and the internal reference calibration of the fisheye camera can be performed through the camera calibration model. When shooting a frame of laser stripe image, the infrared fill light is not turned on, but the line laser is turned on, wherein only the light strips can be identified from the laser stripe image, but the target cannot be identified, and then combined with the target identified from the target image, it can be used for the joint calibration of the line laser and the fisheye camera, thereby utilizing the intermediate data in the internal reference calibration of the fisheye camera, which speeds up the calibration process.
[0065] On the basis of the above embodiment, the number of targets covered by a frame of laser stripe image and a frame of target image taken by the target assembly at the same sliding position is equal, and the orientation of targets covered by a frame of laser stripe image and a frame of target image taken by the target assembly at the same sliding position is the same, so that when the fisheye camera takes a frame of laser stripe image and a frame of target image corresponding to the same sliding position, the external parameters from the target assembly to the fisheye camera are equal.
[0066] In step 2, the external parameters of the target assembly at each sliding position to the fisheye camera are the results obtained after the pixel data of the image to be calibrated is input into the camera calibration model. The corresponding external parameters can be composed of a rotation matrix R and an offset vector t, which can map the pixel coordinates (or the coordinate system of the imaging plane) to the coordinates of the target plane, and can also be mapped from the world coordinate system to the camera coordinate system to achieve the conversion of two-dimensional coordinates to three-dimensional vectors; wherein, the pixel data of the image to be calibrated includes but is not limited to the pixel coordinates of the corner points in the numbered targets, which are used to calibrate the fisheye camera and can also be used to calibrate the laser parameters. The result obtained by the camera calibration model also includes the internal parameters of the fisheye camera, which can be converted from the camera coordinate system to the pixel coordinate system based on the internal parameters, and the internal parameters output by the camera calibration model can also explain the stretching and deformation of the fisheye lens (the lens of the fisheye camera) to the graphics in the world coordinate system. The corresponding internal parameter parameters can use the stretching matrix to compensate for the misalignment of the sensor to the lens. The external parameters of the target assembly at each sliding position to the fisheye camera include the external parameters of the cells in the targets set at each angle at each sliding position. Therefore, the camera calibration model converts the numbered corner points and cell position information in the target into the intrinsic and extrinsic parameters of the fisheye camera to obtain the true spatial scale on the target.
[0067] As an embodiment, step 3 specifically includes: extracting the center pixel coordinates of the light strip from the ROI area in the image to be calibrated at different sliding positions, and setting the center pixel coordinates of the light strip as the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera as the ideal image projection of the light spot in the real world; searching for the light strip position in the ROI area to facilitate the calibration of the laser parameters after the camera is calibrated.
[0068] Then, based on the coordinate system conversion function set in the preset mathematical analysis software, the parameters of the fisheye camera calibrated by the camera calibration model are used to map the two-dimensional pixel coordinates (u, v) to the light spot coordinates of the corresponding numbered target surface (corresponding numbered calibration plate) The light spot coordinates It is located in the world coordinate system set on the corresponding calibration plate, Z W It can be equal to the value 0, so that the light spot coordinates become the world coordinates of the corresponding calibration plate plane or target plane, realizing the conversion of the two-dimensional pixel coordinates (u, v) to the plane of the calibration plate with the corresponding number, and restoring the two-dimensional coordinates in the real world.
[0069] Preferably, the preset mathematical analysis software includes but is not limited to Matlab (a combination of the two words matrix & laboratory, meaning matrix factory (matrix laboratory)), and a pointsToWorld function is provided in its internal toolkit for mapping two-dimensional pixel coordinates into coordinates in a world coordinate system. The pointsToWorld function may be derived from a paper “A toolbox for easily calibrating omnidirectional cameras” published by Scaramuzza D (David Scaramuzza), a technician in the relevant field; in this embodiment, the coordinates in the world coordinate system may be divided into world coordinate systems corresponding to different calibration plates (or different targets).
[0070] Then, the light spot coordinates are converted into Convert the three-dimensional coordinates of the laser point into the camera coordinate system The corresponding conversion formula is Among them, the external parameters include the rotation matrix R and offset vector T required to convert the world coordinate system set on the surface of the target with the corresponding number into the camera coordinate system of the fisheye camera, referred to as the external parameters R and T corresponding to the target to the camera, which are derived from a rigid body transformation specified by the camera calibration model and are conducive to converting the depth information.
[0071] When the center pixel coordinates of each light strip extracted in the ROI area of the image to be calibrated at different sliding positions are converted into the light spot coordinates of each target surface with different numbers (calibration plate with corresponding numbers) based on the coordinate system conversion function, the light spot coordinates of each target surface with different numbers (calibration plate with corresponding numbers) are converted into the rotation matrix R and offset vector T required for the camera coordinate system using the world coordinate system set by the target surface with corresponding numbers, and converted into the corresponding three-dimensional coordinates of the laser point in the camera coordinate system; therefore, when the center pixel coordinates of all light strips extracted in the ROI area of the image to be calibrated at different sliding positions are converted into the corresponding three-dimensional coordinates of the laser point in the camera coordinate system, it is determined that the joint calibration of the line laser and the fisheye camera is completed. This is conducive to converting the depth information of a batch of laser points and completing the calibration of the laser point cloud data that constitutes the light strips.
[0072] In some embodiments, it can be understood that the coordinates of the center pixel of the light strip extracted in the ROI area and the corner point coordinates of the projection relationship in the target are the coordinates of the corner point in the world coordinate system. The rotation matrix R and displacement vector T of the corner point generated by the camera calibration model, which has a projection relationship with the center pixel coordinates of the currently extracted light strip, are used to convert the corner point coordinates into the three-dimensional coordinates of the laser point in the camera coordinate system. The rotation matrix R and the displacement vector T are both external parameters of the fisheye camera. Among them, the corner point with a projection relationship with the center pixel coordinates of the currently extracted light strip is located on a calibration plate with a corresponding angle setting at a sliding position relative to the line laser module, specifically a target with a corresponding number in this embodiment. Based on the external parameters, the two-dimensional pixel coordinates corresponding to the targets (targets pre-numbered in step 2) set at various angles at different sliding positions are converted to world two-dimensional coordinates in sequence, and then converted from world two-dimensional coordinates to camera coordinates, and the point cloud data in the camera coordinate system is calibrated.
[0073] Considering the lens distortion suitable for fisheye cameras, combined with the camera calibration model, the following embodiments may exist:
[0074] Step 3 extracts the pixel coordinates of the center of the light strip from the ROI area in the image to be calibrated at different positions, and sets the pixel coordinates of the center of the light strip to the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera. Then, according to the camera calibration model, the pixel coordinates of the center of the light strip are converted into distortion-free three-dimensional coordinates. Wherein, λ is the scaling factor, which indicates the scale scaling degree of the pixel coordinates, that is, the numerical relationship between the physical length and the pixel length after the physical length in the actual space is scaled to the pixel length in the pixel plane; λ can also represent the scalar factor of the depth; a0, a2, a3 and a4 are all distortion coefficients of the fisheye camera, which are all polynomial coefficients described by the camera calibration model, where a1 is 0. Indicates the distance in pixels between the center of the light bar and the center of the image. The scaling factor may vary depending on the number of the target where the pixel coordinates of the center of the light bar are located (corresponding to the set angle).
[0075] Specifically, a0+a2ρ 2 +a3ρ 3 +a4ρ 4 It is used to correct a series of distortions of the fisheye camera to convert the ROI area in the distorted image (the image to be calibrated) into a normal undistorted image, and obtain the corresponding undistorted 3D coordinates. It can be used as an undistorted three-dimensional coordinate; then the undistorted three-dimensional coordinate is converted to the external parameters required by the fisheye camera through the corresponding numbered target (R and T can be used to represent a set of external parameters) Converted into the light spot coordinates of the target surface (calibration plate) with the corresponding number disclosed in the above embodiment This can eliminate the lens distortion in the image to be calibrated taken by the fisheye camera and obtain the distortion-free coordinate information of the corresponding target surface.
[0076] Specifically, It can be used as an intermediate quantity for transforming from the pixel coordinate system of a fisheye camera to the camera coordinate system.
[0077] The image to be calibrated is the image after the fisheye camera is distorted. Processing the two-dimensional pixel coordinates (u, v) through the aforementioned intermediate quantities is equivalent to flattening and straightening the target to form an anti-distorted image. Then, the depth information still needs to be calculated using external parameters, because the fisheye camera as a monocular camera in the online laser module cannot know the distance, at least it cannot independently know the depth information without the help of the line laser.
[0078] Preferably, starting from the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera, in the framework of the pointsToWorld function, the internal parameters of the fisheye camera are used to map to the normalized plane to obtain the normalized coordinates (Xnorm, Ynorm), Z = 1 (the depth on the normalized plane is 1, and the normalized plane is parallel to the imaging plane of the fisheye camera), the normalized coordinates (Xnorm, Ynorm) and Proportional relationship exists If we combine the laser surface function obtained in step 3 above, we can find Z C , you can get the three-dimensional coordinates of the laser point in the camera coordinate system The camera coordinates are inferred to obtain the depth information of the target.
[0079] The present application also discloses a calibration device, Figure 1 and Figure 2 It can be known that the calibration device includes a calibration drive component (not shown in the figure), a target component, a track 106, a fixed support platform 112, a line laser module, and a parameter calibration unit for the joint calibration method disclosed in the aforementioned embodiment; wherein the target component includes a plurality of targets arranged at an angle, namely a target arranged on the surface of the reference calibration plate 101, a target arranged on the surface of the first calibration plate 102, a target arranged on the surface of the second calibration plate 103, a target arranged on the surface of the third calibration plate 104, and a target arranged on the surface of the fourth calibration plate 105; the target component can be fixedly set on the calibration drive component by a connecting bracket 109 to drive the target to slide linearly along the track 106. The line laser module includes a line laser and a fisheye camera, and both the line laser and the fisheye camera are integrally assembled to the fixed support platform 112 through a sensor fixing position 111, and the fixed support platform is fixedly assembled with the line laser and the fisheye camera; the fixed support platform 112 is fixedly set at one end of the track 106, such as Figure 2 The lower end shown can be specifically Figure 1 The connector 108 shown is fixedly arranged; the target assembly can be slidably arranged at the other end of the track 106 (such as Figure 2For line laser modules, fisheye cameras support extended field of view, and the imaging range tries to catch up with the scanning range of the target at various angles at multiple distances of the line laser, reducing the number of imaging sensors used. The fisheye camera shooting process does not require manual movement of the target, which is easy to automate; thereby reducing the number of calibrations and error sources, and improving the efficiency of the joint calibration of the line laser and the fisheye camera.
[0080] In this embodiment, the parameter calibration unit is used to control the target assembly to slide multiple distances relative to the line laser module; control the line laser to emit a line laser to the target assembly, and project a light strip on the surface of the target at the corresponding angle; and control the fisheye camera to sequentially photograph the target or the light strip projected from its surface set at at least one angle in the target assembly at each sliding position to obtain the image to be calibrated. The parameter calibration unit is also used to input the pixel data of the image to be calibrated at different sliding positions into the camera calibration model to calibrate the fisheye camera, and output the parameters of the fisheye camera. The parameter calibration unit is also used to extract the center pixel coordinates of the light strip from the image to be calibrated at different sliding positions, respectively, and then convert all the extracted center pixel coordinates of the light strip into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera, and determine that the joint calibration of the line laser and the fisheye camera is completed.
[0081] As an embodiment, the target assembly includes a plurality of calibration plates, any two calibration plates are arranged at an angle, and the normal vectors of each calibration plate are different, and a target is arranged on a calibration plate, so that each calibration plate included in the target assembly has a different orientation relative to the optical axis of the fisheye camera; a target is arranged on the surface of a calibration plate, and then each target included in the target assembly has a different orientation relative to the optical axis of the fisheye camera. In some embodiments, on the surface of each calibration plate, the targets are arranged in a checkerboard pattern, and the edges of the checkerboard pattern are surrounded by a white area for line laser calibration; in the target assembly, the checkerboard pattern arranged on the surface of each calibration plate is the same; wherein, at multiple sliding positions of the same target assembly, the fisheye camera captures the complete target from the calibration plate arranged at the corresponding angle. In the same complete target, a polygonal area is formed by expanding outward from the corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light bar, wherein the polygonal area includes all cells of the black and white square area on the target and the white area outside the target, because the polygonal area needs to be configured to frame the entire checkerboard pattern on a single target, wherein all cells of the black and white square area form a complete checkerboard pattern; then the polygonal area is set as the ROI (Region of Interest) area, and the ROI frames the area around the checkerboard pattern, including the blank part on the calibration plate, and there will be a part of the light bar in the checkerboard and in the blank part of the area. Some light bars may be missed outside the ROI area, but it does not matter, as long as the light bars in the ROI area are sufficient to calculate the aforementioned laser surface function. Generally, the light bars are made to fall as much as possible in the ROI area according to the completeness of the target in the previous calibration image and the sliding distance of the target assembly in the track.
[0082] In this embodiment, the closer the target assembly slides along the track to the line laser module, the fewer complete targets are included in a frame of the image to be calibrated captured by the fisheye camera; conversely, the farther the target assembly slides along the track to the line laser module, the more complete targets are included in a frame of the image to be calibrated captured by the fisheye camera, so that the fisheye camera can capture complete targets in all calibration plates at the ranging boundary of the track. The number of different positions that the target assembly slides along the track is greater than or equal to 4. Figure 1 and Figure 2It can be seen that not all targets can be captured in the image to be calibrated at a distance closer to the line laser module, but the target assembly can be photographed as a complete target by the fisheye camera at the minimum ranging boundary of the track 106. The minimum ranging boundary of the track 106 is the closest position that allows the targets in the target assembly to be imaged in the fisheye camera. In this embodiment, the farther the target assembly slides from the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera. The target assembly can be photographed as a complete target in all calibration plates by the fisheye camera at the maximum ranging boundary of the track 106. The maximum ranging boundary of the track 106 is the farthest position that allows all targets in the target assembly to be imaged in the fisheye camera. Therefore, when the target is imaged in the fisheye camera, it is necessary to consider the clarity of the fisheye camera when focusing, and try to achieve a full-viewing angle balancing effect for the calibration plates set at various angles. Based on the specific target distribution structure and the sliding mode of the target assembly in the track, this embodiment can simplify the number of images to be calibrated to be collected to 3 to 4 frames. The corresponding technical effect is: while ensuring the reliability of the intrinsic calibration results of the fisheye camera, the joint calibration efficiency is improved compared with the traditional single chessboard calibration method.
[0083] As an example, combining Figure 1 and Figure 2 It can be seen that there are five calibration plates in the target assembly; the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate; among the other four calibration plates, there is a calibration plate located above the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is a calibration plate located below the reference calibration plate and is set at a predicted observation angle with the reference calibration plate, there is a calibration plate located to the left of the reference calibration plate and is set at a predicted observation angle with the reference calibration plate, and there is a calibration plate located to the right of the reference calibration plate and is set at a predicted observation angle with the reference calibration plate, so that when the target assembly slides along the track, the fisheye camera captures at least one complete target in the target assembly and the white area set around it. On the surface of a calibration plate, the outside of the target is surrounded by a white area, and a certain blank white background is left for a complete target on the surface of the calibration plate, which can be used for line laser calibration, because pure white is conducive to the fisheye camera capturing clear light strips.
[0084] Specifically, in Figure 1In the target assembly, there are five calibration plates, and the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate 101 to reduce the error; the other four calibration plates are marked counterclockwise, namely calibration plate 102, calibration plate 103, calibration plate 104 and calibration plate 105. The sizes of calibration plates 102, 103, 104 and 105 are not necessarily equal. Calibration plate 102 is located above the reference calibration plate 101 and the reference calibration plate is set at a predicted observation angle. Calibration plate 104 is located below the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 103 is located to the left of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 105 is located to the right of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. The predicted observation angle is preferably 120 degrees. It is ensured that the field of view of the fisheye camera is comparable to the size of the target set at each angle, so as to ensure that the target assembly is still within the field of view of the fisheye camera when sliding.
[0085] like Figure 1 As shown, the reference calibration plate 101 and the calibration plate 102 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 103 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 104 are combined into a V-shaped calibration plate, and the reference calibration plate 101 and the calibration plate 105 are combined into a V-shaped calibration plate; the angle formed by the reference calibration plate 101 and the calibration plate 102 can be set according to actual conditions, that is, the predicted observation angle setting, which is not limited here. For example, the reference calibration plate 101 and the calibration plate 102 are preferably placed at an angle of 120 degrees; the reference calibration plate 101 and one side of the calibration plate 102 are arranged to fit each other, so that a part of the target assembly is formed by combining the two calibration plates, and targets at two angles can be set accordingly.
[0086] In order to facilitate the placement of the V-shaped calibration plate, the two plates in the V-shaped calibration plate can be connected by hinges. The hinge connection can be in the form of Figure 2 The connecting strip 113 is fixed together at the predicted observation angle. Specifically, the calibration plate 103 and one side of the reference calibration plate are assembled by a connecting strip in an obtuse angle and a screw to form a connecting hinge. In one embodiment, the connecting strip supports the two plates to be opened and closed to any angle to facilitate the external parameter calibration between the line laser and the fisheye camera. For example, it can be adjusted so that near the maximum ranging boundary of the track 106, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the fisheye camera).
[0087] To improve the calibration efficiency, the longer right-angled sides of the two calibration plates can be fixed to each other. In other embodiments, the shapes of the two plates in the V-shaped calibration plate are not limited to quadrilateral plates such as rectangular plates.
[0088] Preferably, the normal vector of the reference calibration plate 101 is parallel to the optical axis of the fisheye camera. Considering the symmetry problem, the calibration plate 103 located on the left of the reference calibration plate 101 and the calibration plate 105 located on the right of the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate, and the targets on the left and right of the reference calibration plate 101 are symmetrically arranged; the calibration plate 102 located above the reference calibration plate 101 and the calibration plate 104 located below the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate, and the targets on the upper and lower sides of the reference calibration plate 101 are symmetrically arranged. The calibration error is further reduced.
[0089] In some embodiments, the key to the target assembly sliding along the track 106 is to allow the target to be photographed by the fisheye camera at several different distances, including far, medium and near. At least: when the target assembly slides to a sliding position at a first distance from the line laser module, the fisheye camera photographs 1 complete target; when the target assembly slides to a sliding position at a second distance from the line laser module, the fisheye camera photographs 1 to 3 complete targets; when the target assembly slides to a sliding position at a third distance from the line laser module, the fisheye camera photographs 5 complete targets; wherein, one target is set on a calibration plate, the first distance is smaller than the second distance, and the second distance is smaller than the third distance.
[0090] In the same frame of the image to be calibrated, if the number of corner points searched from one of the targets is equal to the preset number threshold, the one of the targets is set as a complete target and assigned a number, wherein the preset number threshold is preferably 10. A complete target is a black and white square area, such as Figure 3 As shown, similar to a chessboard, the calibration plate and the target are both rectangular, the size of the calibration plate is related to the size of the target, the black and white square area is composed of black cells and white cells, which are regarded as black cells and white cells arranged at intervals, and the aforementioned corner points are the corner points of black cells and / or the corner points of white cells; preferably, the number of cells constituting the square area is linearly related to the number of corner points searched out; the targets set in each calibration plate are consistent, which can be understood as providing the same calibration features for the line laser and the fisheye camera from each angle direction at the same distance, thereby ensuring the accuracy of the joint calibration of the line laser and the fisheye camera.
[0091] As an example, Figure 2As shown, the fixed support platform is also fixedly mounted with an infrared fill light 110. On the fixed support platform 112, the relative position between the infrared fill light 110 and the line laser module remains unchanged, that is, the sensor fixed position 111 and the infrared fill light 110 are both fixedly mounted in the fixed support platform 112, so that the emission direction of the line laser, the shooting angle of the fisheye camera, and the fill light direction of the infrared fill light 110 are all toward the target assembly (including the calibration plates set at various angles). Whenever the target assembly slides along the track to a sliding position relative to the line laser module, the relative position between the target assembly and the line laser module remains unchanged, and the line laser and the fisheye camera are both fixedly mounted, there are:
[0092] The parameter calibration unit controls the line laser to emit a line laser to the target assembly, and controls the fisheye camera to capture at least one complete target, to obtain a frame of laser stripe image corresponding to a sliding position, and a frame of laser stripe image includes a frame of image to be calibrated including light stripes. At this time, the target cannot be identified in the image to be calibrated, that is, the aforementioned black and white square area and the cells inside it, and only the central pixel coordinates of the light stripes in the calibration plate with the corresponding number can be extracted; the fisheye camera preferably adopts an infrared bandpass filter or an infrared high-pass filter to filter the reflected light of the line laser emitted by the line laser on the surface of the calibration plate, specifically to transmit the infrared band carried by the line laser, and absorb or reflect the ambient light of other bands, so as to separate the infrared light carried by the reflected light from other interfering ambient light.
[0093] The parameter calibration unit controls the line laser to stop emitting the line laser, turns on the infrared fill light to fill light for the target assembly, and controls the fisheye camera to capture the at least one complete target to obtain a frame of target image corresponding to the same sliding position; wherein a frame of target image is a frame of image to be calibrated that does not include light bars. The brightness of the target is increased by turning on the infrared fill light to increase the depth of field during shooting, because under the condition of low ambient light in the factory, the fisheye camera is not able to clearly capture the information of the target and its corner points through external light, so a fill light is needed to illuminate the target, wherein the relative position between the infrared fill light and the line laser module can remain unchanged or change, and there is no specific limitation here.
[0094] Specifically, keep in an environment with relatively small ambient light interference, turn off the line laser and use infrared fill light to fill light for the target, and then take the image to be calibrated. The advantage of doing this is that it is easy to perform image processing. Therefore, step 2 searches for corner points from the target image and expands the ROI area, and then frames the ROI area in the same pixel area of the laser stripe image. The position relationship of the same pixel area in the laser stripe image is the position relationship of the ROI area in the target image. Based on the pixel position information of the ROI area framed in the target image, the center pixel coordinates of the light stripe falling into the ROI area can be searched in the laser stripe image. On this basis, combined with the external parameters obtained by the fisheye camera calibration, the three-dimensional coordinates of the laser point on the intersection of the line laser and the target surface (in the camera coordinate system) can be converted, which corresponds to the spot position information of the light stripe searched in the area framed by the ROI area in the target, thereby narrowing the search range of the light stripe.
[0095] In summary, this embodiment uses the calibration plates in the target assembly as a medium to calibrate the fisheye camera and the line laser. On the one hand, this embodiment can perform joint calibration of the camera and the line laser on a camera equipped with a fisheye lens (hereinafter referred to as the fisheye camera) without modifying the camera that needs to be calibrated. On the other hand, the calibration device used in this embodiment is low-cost, and data acquisition is simple and direct. It integrates heterogeneous data through coordinate system transformation from the underlying hardware level, and calibrates the three-dimensional coordinate data of the laser points at various angles at various sliding positions in the camera coordinate system.
[0096] As an example, combining Figure 1 and Figure 2 It can be seen that the track 106 is arranged along the vertical direction, the fixed support platform 112 is placed on the horizontal plane and fixedly arranged at the lower end of the track 106, and the emission direction of the line laser and the shooting angle of the fisheye camera are both upward; the reference calibration plate 101 is arranged on the normal plane of the track 106 to be parallel to the horizontal plane, and the five calibration plates are fixed on the calibration drive assembly by the connecting bracket 109, and the calibration drive assembly drives the five calibration plates to slide up and down relative to the line laser module, which is actually sliding up and down relative to the sensor fixed position 111. Preferably, among the five calibration plates, if the top calibration plate is used as the reference calibration plate 101, the four calibration plates around it and the top reference calibration plate 101 form an obtuse angle of 120 degrees. Of course, the specific angle can be adjusted according to the actual situation without limitation. By adjusting the relative positions of the five calibration plates on the track 106, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible near the maximum ranging boundary of the track 106, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the direction of the fisheye camera).
[0097] Specifically, at the mechanical hardware operation level, the target assembly can slide up and down linearly along the motion axis 107 on the track 106 through the connecting bracket 109, and the calibration drive assembly can be provided with a wheel-axle mechanism that matches the motion axis 107, so that the target assembly can move on the track 106 along the axis of the motion axis 107 under the traction of the calibration drive assembly; in some embodiments, the aforementioned wheel-axle mechanism is arranged in the groove between the motion axis 107 and the raised rails on both sides of the track 106, and slides up and down in the groove under the traction of the calibration drive assembly. Preferably, the normal vector of the reference calibration plate 101 is set to be parallel to the motion axis 107, and the axial direction of the motion axis 107 is parallel to the laying direction of the track 106.
[0098] The following is a detailed introduction to several coordinate systems and coordinate transformation principles involved in this application.
[0099] World coordinate system: A reference coordinate system is also selected in the environment to describe the position of the camera and objects. This coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by the rotation matrix R and the translation vector t.
[0100] Camera coordinate system: The origin of the camera coordinate system is the optical center of the camera. The x-axis and y-axis are parallel to the X and Y axes of the image. The z-axis is the optical axis of the camera, which is perpendicular to the image plane. The spatial rectangular coordinate system formed by this is called the camera coordinate system, also known as the camera coordinate system. The camera coordinate system is a three-dimensional coordinate system. The intersection of the optical axis and the image plane is the origin of the image coordinate system. The rectangular coordinate system formed by the X and Y axes of the image is the image coordinate system. The image coordinate system is a two-dimensional coordinate system.
[0101] Pixel coordinate system (U, V): It is a two-dimensional rectangular coordinate system that reflects the arrangement of pixels in the imaging plane of the camera. The origin O is located in the upper left corner of the image, and the U axis and V axis are parallel to the two sides of the image plane.
[0102] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable technicians familiar with the technical field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.
Claims
1. A method based on joint calibration of fisheye camera and line laser, characterized in that: include: Step 1, controlling the target assembly to slide a plurality of distances relative to the line laser module; Control the line laser to emit a line laser to the target assembly, projecting a light strip on the surface of the target at the corresponding angle; and control the fisheye camera to sequentially photograph the target set at at least one angle in the target assembly at each sliding position or the light strip projected from its surface to obtain the image to be calibrated; wherein the line laser module includes a line laser and a fisheye camera, and the relative position between the line laser and the fisheye camera is fixed; Step 2: Input the pixel data of the image to be calibrated at different sliding positions into the camera calibration model, and output the parameters of the fisheye camera; Step 3: Extract the center pixel coordinates of the light stripes from the images to be calibrated at different sliding positions, and then convert all the extracted center pixel coordinates of the light stripes into three-dimensional coordinates of the laser points according to the parameters of the fisheye camera, and determine that the joint calibration of the line laser and the fisheye camera is completed.
2. The method according to claim 1, characterized in that: The target assembly includes a plurality of calibration plates, any two of which are arranged at an angle; the normal vectors of each calibration plate are different; a target is arranged on the surface of a calibration plate, so that any two targets are also arranged at an angle; The closer the target assembly slides to the line laser module, the fewer targets are included in a frame of the image to be calibrated captured by the fisheye camera. The farther the target assembly slides from the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera.
3. The method according to claim 2, characterized in that: In step 2, before the fisheye camera calibration is performed, the following steps are also included: Whenever the target assembly slides to a sliding position, a corner point is searched in the complete target in a frame of the image to be calibrated taken in real time by the fisheye camera; In the same frame of the image to be calibrated, the corner points searched are grouped according to the complete target photographed, and a group of corner points corresponds to a complete target; Then, the complete targets where each group of corner points is located are numbered so that the targets set at different angles in the same frame of the image to be calibrated have different numbers; when the target assembly slides to different sliding positions, the numbers of the complete targets in different frames of the image to be calibrated taken by the fisheye camera are different.
4. The method according to claim 3, characterized in that: In the same frame of the image to be calibrated, if the number of corner points searched from one of the targets is equal to a preset number threshold, then the one of the targets is set as a complete target; A complete target is a black and white square area; the black and white square area is composed of black cells and white cells on the surface of a calibration plate.
5. The method according to claim 4, characterized in that: In the same complete target, a polygonal area is formed by expanding outward from a corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light bar, wherein the polygonal area includes all cells in the black and white square area and the white area outside the target; Then set the polygonal area as the ROI area.
6. The method according to claim 5, characterized in that: The camera calibration model is a full-view camera model calibration correction model or a fisheye lens distortion correction model, which is used to calculate the parameters of the fisheye camera, including the internal parameters of the fisheye camera and the external parameters of each target to the fisheye camera.
7. The method according to claim 6, characterized in that: In step 1, whenever the target assembly slides linearly relative to the line laser module to a sliding position, the relative position between the target assembly and the line laser module remains unchanged, and the line laser and the fisheye camera are fixedly assembled, there exists: First, the line laser is controlled to emit a line laser to the target assembly, and the fisheye camera is controlled to capture at least one complete target, so as to obtain a frame of laser stripe image corresponding to a sliding position. A frame of laser stripe image is a frame of image to be calibrated including light stripes; Then, the line laser is controlled to stop emitting the line laser, and the infrared fill light is turned on to fill light for the target assembly, and the fisheye camera is controlled to capture the at least one complete target, and a frame of target image corresponding to the same sliding position is obtained, so that in step 2, the corner point is searched from the target image and the ROI area is expanded, and then the ROI area is framed to the same pixel area of the laser stripe image, and the position relationship of the same pixel area in the laser stripe image is the position relationship of the ROI area in the target image; wherein, a frame of target image is a frame of image to be calibrated that does not include light stripes; The pixel coordinates of the center of the light stripe are the projection positions of the intersection of the line laser emitted by the line laser and the target in the pixel plane of the fisheye camera, and are located in the laser stripe image; The intersection line of the line laser emitted by the line laser and the target is the light strip.
8. The method according to claim 7, characterized in that: In step 2, the external parameters from the target assembly at each sliding position to the fisheye camera are the results obtained by inputting the pixel data of the image to be calibrated into the camera calibration model, and the external parameters from the target assembly at each sliding position to the fisheye camera include the external parameters of the cells in the target set at each angle at each sliding position; The pixel data of the image to be calibrated includes but is not limited to the pixel coordinates of the corner points of the numbered targets.
9. The method according to claim 8, characterized in that: The step 3 specifically includes: The pixel coordinates of the center of the light strip are extracted from the ROI area of the image to be calibrated at different sliding positions, and the pixel coordinates of the center of the light strip are set as the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera; then, based on the coordinate system conversion function set in the preset mathematical analysis software, the parameters of the fisheye camera calibrated by the camera calibration model are used to map the two-dimensional pixel coordinates (u, v) to the light spot coordinates on the target surface with the corresponding number. The light spot coordinates It is located in the world coordinate system set on the corresponding calibration plate; Then, the light spot coordinates are converted into Convert the three-dimensional coordinates of the laser point into the camera coordinate system When the central pixel coordinates of all light stripes extracted in the ROI area of the to-be-calibrated image at different sliding positions are converted into the corresponding three-dimensional coordinates of the laser point in the camera coordinate system, it is determined that the joint calibration of the line laser and the fisheye camera is completed.
10. The method according to claim 9, characterized in that: The coordinates of the corner points of the light strip center pixel coordinates extracted in the ROI area that have a projection relationship in the target are the light spot coordinates The rotation matrix R and displacement vector T of the corner point generated by the camera calibration model, which has a projection relationship with the center pixel coordinates of the currently extracted light strip, are used to convert the corner point coordinates into the three-dimensional coordinates of the laser point in the camera coordinate system. The corresponding conversion formula is: Among them, the rotation matrix R and the displacement vector T are both external parameters of the fisheye camera, and the corner points that have a projection relationship with the center pixel coordinates of the currently extracted light strip are located on the target surface with the corresponding number.
11. The method according to claim 10, characterized in that: In step 3, the pixel coordinates of the center of the light strip are extracted from the ROI area in the image to be calibrated at different positions, and the pixel coordinates of the center of the light strip are set as the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera. Then, according to the camera calibration model, the pixel coordinates of the center of the light strip are converted into distortion-free three-dimensional coordinates. Wherein, λ is the scaling factor; a0, a2, a3 and a4 are all the distortion coefficients of the fisheye camera, and are all the polynomial coefficients described by the camera calibration model; Then, the undistorted three-dimensional coordinates are obtained by using the corresponding numbered targets to obtain the external parameters required by the fisheye camera. Convert to the light spot coordinates on the target surface corresponding to the number 12. A calibration device, characterized in that: The calibration device includes a calibration drive component, a target component, a track, a fixed supporting platform, a line laser module, and a parameter calibration unit for executing the method described in any one of claims 1 to 11; wherein, the target component includes a plurality of targets arranged at an angle, the line laser module includes a line laser and a fisheye camera; the fixed supporting platform is fixedly assembled with the line laser and the fisheye camera; the fixed supporting platform is fixedly arranged at one end of the track, and the target assembly is slidably arranged at the other end of the track.
13. The calibration device according to claim 12, characterized in that: There are five calibration plates in the target assembly; the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate; among the remaining four calibration plates, there is one calibration plate located above the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is one calibration plate located below the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is one calibration plate located to the left of the reference calibration plate and the reference calibration plate is set at a predicted observation angle, and there is one calibration plate located to the right of the reference calibration plate and the reference calibration plate is set at a predicted observation angle, so that when the target assembly slides along the track, the fisheye camera can capture at least one complete target in the target assembly and a white area set around it.
14. The calibration device according to claim 13, characterized in that: The fixed bearing platform is also equipped with an infrared fill light; Whenever the target assembly slides along the track relative to the line laser module to a sliding position, the relative position between the target assembly and the line laser module remains unchanged, and the line laser and the fisheye camera are fixedly assembled, there are: The parameter calibration unit controls the line laser to emit the line laser to the target assembly, and controls the fisheye camera to capture at least one complete target, to obtain a frame of laser stripe image corresponding to a sliding position, wherein a frame of laser stripe image includes a frame of image to be calibrated including light stripes; The parameter calibration unit controls the line laser to stop emitting the line laser, turns on the infrared fill light to fill light for the target assembly, and controls the fisheye camera to capture the at least one complete target, to obtain a frame of target image corresponding to the same sliding position; wherein a frame of target image is a frame of image to be calibrated that does not include light bars; On the fixed supporting platform, the relative position between the infrared fill light and the line laser module remains unchanged.
15. The calibration device according to claim 14, characterized in that: The track is arranged along the vertical direction, the fixed bearing platform is placed on a horizontal plane and fixedly arranged at the lower end of the track, and the emission direction of the line laser and the shooting angle of the fisheye camera are both arranged upward relative to the fixed bearing platform; The reference calibration plate is arranged on the normal plane of the track to be parallel to the horizontal plane. The five calibration plates are fixed on the calibration drive assembly by a connecting bracket, and the calibration drive assembly drives the five calibration plates to slide up and down relative to the line laser module.
