Parameter calibration method and calibration device based on line laser module
By using a parameter calibration method based on line laser module in a structured light vision system, the target assembly is sliding and light bar images of different angles are captured, and the three-dimensional coordinates of laser points are extracted and fitted, the efficiency of laser parameter calibration under wide-angle lenses is solved and Accuracy issues enable more accurate laser parameter modeling and calibration.
Patent Information
- Application Number
- CN202310660482.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
When using wide-angle lenses in existing structured light vision systems, there are large errors in the target point cloud, and the straightness of the laser cannot be guaranteed, resulting in inaccurate calibration of the calibration plane coefficient, inaccurate calibration of camera internal parameters and laser parameters, and low efficiency.
Using the parameter calibration method based on the linear laser module, by controlling the sliding of the target component relative to the linear laser module, the fisheye camera takes light bar images at different angles, extracts the pixel coordinates of the center of the light bar, and converts them into the three-dimensional coordinates of the laser point, based on The projection direction is fitted in the surface, the laser surface function is determined, and the laser parameters of the line laser module are calibrated.
It improves the efficiency and accuracy of linear laser module calibration, reduces the number of manual adjustments, and realizes more accurate laser parameter modeling, which is suitable for the production and manufacturing of structured light vision systems.
Smart Images

Figure CN119090968A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of sensor parameter calibration, and in particular to a parameter calibration method and a calibration device based on a line laser module, and specifically to an internal parameter calibration method of a fisheye camera and a laser parameter calibration method of a line laser and a calibration device of a line laser module. Background Art
[0002] The structured light vision system is a measurement system that combines a line laser and a camera. Before use, the line laser surface needs to be calibrated and the plane coefficients need to be calculated to express the spatial structure of the laser surface. The camera then acquires the image and performs three-dimensional measurement through the imaging principle and the line laser surface. In order to increase the field of view, the measurement system often uses a wide-angle lens as the receiving component. However, the edge distortion of the wide-angle lens is large, resulting in large errors in the target point cloud. When the quality of the laser cannot achieve good flatness, the edge points are curved, and the obtained plane coefficients cannot truly express the spatial structure of the laser surface, resulting in inaccurate camera internal parameters and laser parameters, and the final point cloud accuracy is not accurate enough.
[0003] As a special type of corner point, checkerboard corner points are widely used in camera calibration. Checkerboard patterns are widely used in camera calibration because of their obvious contrast, simple features, and easy detection and recognition. However, in the current structured light vision system, the camera cannot guarantee that the checkerboard image completely covers the entire camera's field of view, and it is not easy to use the checkerboard to extract feature points in the edge area. When calibrating the internal parameters of the fisheye camera, in order to ensure the reliability of the calibration results, it is often necessary to adjust the same checkerboard to be located at different angles of the camera for collection. Too many pictures are often used, and the edge points of the image are curved. The obtained plane coefficient cannot truly express the spatial structure of the laser surface, resulting in low calibration efficiency and low accuracy of the calibration results, which is not suitable for promoting the production and manufacturing process of structured light vision systems. Summary of the invention
[0004] This application discloses a parameter calibration method and a calibration device based on a line laser module. The specific technical solution is as follows:
[0005] The parameter calibration method based on the line laser module includes: step 1, controlling the target assembly to slide multiple distances relative to the line laser module; controlling the line laser to emit the line laser to the target assembly, and projecting a light strip on the surface of the target at the corresponding angle; and controlling the fisheye camera to sequentially shoot the target set at at least one angle in the target assembly at each sliding position or the light strip projected from its surface to obtain the image to be calibrated; step 2, inputting the pixel data of the image to be calibrated at different sliding positions into the camera calibration model to calibrate the fisheye camera, and outputting the parameters of the fisheye camera; step 3 , respectively extract the pixel coordinates of the center of the light strip from the to-be-calibrated images at different sliding positions, and then convert the pixel coordinates of the center of the light strip into three-dimensional coordinates of the laser point according to the parameters of the fisheye camera; then, according to the projection direction of the light strip in the imaging plane, perform surface fitting on the converted three-dimensional coordinates of the laser point so that the number of fittings in the direction of the coordinate axis closest to the projection direction of the light strip in the imaging plane is maximized, and the laser surface function is obtained, and the calibration of the laser parameters of the line laser module is completed to determine the functional relationship between the laser surface and the fisheye camera; wherein, the line laser module includes a line laser and a fisheye camera.
[0006] In summary, in the present technical solution, when the line laser emits a line laser to the target assembly and controls the fisheye camera to capture images of the targets set at various angles at various sliding positions or the light strips projected from their surfaces, the target assembly is controlled to slide multiple distances relative to the line laser module. In combination with the camera calibration model, the internal parameters of the fisheye camera are calculated and the laser surface function in the camera coordinate system is fitted as the calibrated line laser parameters. There is no need to manually move the target, and images of targets at multiple different angles set in the target assembly can be captured at the same position at one time. Compared with the traditional single chessboard calibration method, the calibration efficiency of the line laser module is improved while ensuring the reliability of the calibration results of the fisheye camera internal parameters. After the pixel coordinates of the center of the light strip are automatically extracted from the image to be calibrated at different sliding positions and converted into the three-dimensional coordinates of the laser point, a reasonable polynomial function is selected for surface fitting according to the coordinate axis closest to the projection direction of the light strip, so that the laser surface formed by the line laser emission is fitted as a laser surface function, and the functional relationship between the laser surface and the fisheye camera is determined. The calibration of the positional relationship between the line laser and the fisheye camera is completed, forming a more accurate and effective line laser parameter modeling and calibration method, improving the calibration accuracy, and solving the efficiency and accuracy problems of calibrating camera intrinsic parameters and line laser parameters when using a large-angle lens in the prior art.
[0007] A calibration device, comprising a calibration drive component, a target component, a track, a fixed support platform, a line laser module, and a parameter calibration unit for executing the parameter calibration method; wherein the target component comprises a plurality of targets arranged at an angle, and the line laser module comprises a line laser and a fisheye camera; the fixed support platform is fixedly assembled with the line laser and the fisheye camera; the fixed support platform is fixedly arranged at one end of the track, and the target component is slidably arranged at the other end of the track. For the line laser module, the fisheye camera supports an extended field of view, and the imaging range tries to catch up with the scanning range of the target at various angles at multiple distances of the line laser, thereby reducing the number of imaging sensors used, and the fisheye camera shooting process does not require manual movement of the target, and is easy to automate; thereby reducing the number of calibrations and the source of errors, and improving the calibration accuracy of the line laser module. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 FIG. 1 is a schematic diagram of the arrangement of five calibration plates disclosed in one embodiment of the present invention.
[0009] Figure 2 It is a schematic diagram of the structure of a calibration device disclosed in one embodiment of the present invention.
[0010] Figure 3 It is a two-dimensional plane schematic diagram of a target set up with black and white square areas according to an embodiment of the present invention.
[0011] Figure 4 It is a flow chart of a parameter calibration method based on a line laser module disclosed in one embodiment of the present invention. DETAILED DESCRIPTION
[0012] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. To further illustrate the embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.
[0013] In this application, the line laser module refers to any laser module including a line laser and a fisheye camera. The fisheye camera refers to a camera equipped with a fisheye lens, and the fisheye lens can generally refer to a wide-angle lens. In the line laser module, the line laser is used to emit a line laser outward. Among them, the line laser emitted by the line laser transmitter can be located in front of the object to be measured; the fisheye camera can collect environmental images, and can also receive the reflected light returned by the line laser hitting the object, wherein the line laser emitted by the line laser transmitter is located within the field of view of the fisheye camera; the line laser can help detect the contour, height and / or width of the object to be measured in its emission direction, collectively referred to as the position information of the laser spot or laser light strip. In this application, the object to be measured in the emission direction of the line laser transmitter is a target for calibration, and the image captured by the fisheye camera is the image to be calibrated. The fisheye camera is used to collect image data through a wide-angle lens. The fisheye camera can be an electronic device with image acquisition capabilities such as a camera or a video camera with a fisheye lens.
[0014] The internal calibration of the fisheye camera is for the reliability of the calibration results. It is often necessary to manually adjust a target plane to different angles of the fisheye camera for collection. Dozens of calibration pictures are generated, and there will be camera distortion. The quality of the line laser emitted by the line laser cannot scan the expected straightness. The result of calibration under such circumstances is often low calibration efficiency, which is not suitable for the actual production of line laser modules, and thus affects the accuracy of modeling and fitting of the laser point cloud collected by the calibrated line laser module.
[0015] In order to solve the efficiency and accuracy problems of calibrating camera internal parameters and line laser parameters when using a large-angle lens, especially a fisheye lens, in the prior art, the present application discloses a parameter calibration method based on a line laser module. The execution subject of the parameter calibration method is a control unit for calibrating the line laser module, which can be set outside the line laser module. The line laser module includes a line laser and a fisheye camera, which are relatively fixedly assembled into one. Figure 4 As shown, the parameter calibration method includes:
[0016] Step 1, control the target assembly to slide multiple distances relative to the line laser module. In this embodiment, the distance between the target assembly and the line laser module is limited to change. Generally, the line laser module remains fixed, while the target assembly moves relative to the line laser module. Whenever the target assembly slides a distance relative to the line laser module in a straight line, the target assembly slides to a new sliding position.
[0017] In step 1, the line laser is controlled to emit a line laser to the target assembly, and a light strip is projected in the target at the corresponding angle, which can be a strip of bright light on the target surface; during the sliding process of the target assembly, the emission direction of the line laser emitted by the line laser to the target assembly remains unchanged. When the line laser is working, the emitted line laser forms a light strip on the measured surface of the target assembly (recorded as the target plane), which is also regarded as a collection of laser light spots; when the line laser is working, the emitted line laser remains to fall into the measured surface of the target assembly (recorded as the target plane) during the sliding process of the target assembly, that is, during the sliding process of the target assembly, the projection range of the line laser line emitted by the line laser is always on the measured surface of the target assembly, including the target surface set at at least one angle in the target assembly. The line laser can also not emit a line laser, so as to facilitate the acquisition of a complete target, and to determine the external parameters of each target to the fisheye camera during the calibration of the fisheye camera.
[0018] It should be noted that, in this embodiment, in order to calibrate the fisheye camera and the laser parameters, targets at multiple angles are set in the target assembly. The fisheye camera can capture the target at at least one angle with its built-in fisheye lens, and can also capture the targets at all angles set in the target assembly at a distance from the target assembly at one time, thereby avoiding frequent manual adjustment of the target installation angle and the distance between the target and the fisheye camera.
[0019] Step 1 also controls the fisheye camera to sequentially photograph the target or the light strip projected from its surface set at at least one angle in the target assembly at each sliding position to obtain the image to be calibrated, and then executes step 2; specifically, during the sliding process of the target assembly, the fisheye camera must not only photograph the image of the target when the online laser emits the line laser, as the image to be calibrated, which is specifically used for the calibration of laser parameters; but also keep photographing the image of the target when the online laser does not emit the line laser, and when the online laser does not emit the line laser, when the target assembly slides to a new position, it is necessary to control the fisheye camera to sequentially photograph the complete target set at at least one angle in the target assembly, as the image to be calibrated, which is specifically used for the fisheye camera calibration.
[0020] It is worth noting that when the image to be calibrated includes light stripes, a laser stripe image is formed, and there is at least a white area where the line laser is projected into the target, which is convenient for capturing light spots for laser point cloud modeling and laser parameter calibration. When the image to be calibrated does not include light stripes, a target image is formed. The target can be designed with a checkerboard layout, and the corresponding external parameters can be obtained through the checkerboard, that is, the position and distance of the light spot and the corner point of the checkerboard can be found through the light spot projected by the line laser on the checkerboard. Reduce calibration error.
[0021] In some embodiments, the control unit for line laser module calibration contains an encoder, which controls the movement of the target assembly. Every time the encoder moves a preset unit distance, it triggers the fisheye camera to capture an image, and an image is captured once when the line laser emits a line laser and once when the line laser does not emit a line laser.
[0022] Step 2, the pixel data of the image to be calibrated at different sliding positions are sequentially input into the camera calibration model to calibrate the fisheye camera, and the parameters of the fisheye camera are output, which can be determined to complete the calibration of the fisheye camera; then step 3 is executed. Then the two-dimensional pixel coordinates can be converted into three-dimensional coordinates in the camera coordinate system according to the parameters of the fisheye camera, and the depth information is given. The camera calibration model can be derived from the mathematical analysis calculation software disclosed in the prior art, and the mathematical interface function set in the camera calibration model is used to calculate the internal parameters of the fisheye camera and the external parameters from the target to the fisheye camera, so as to complete the calibration of the fisheye camera before executing step 3; generally, after the internal parameters of the fisheye camera are calibrated first, the external parameters of the target set at different distances and angles to the fisheye camera can be calculated. After the internal parameters of the fisheye camera are calibrated, a rigid transformation pointing to the preset coordinate system (the so-called external parameter) can be specified to map the pixel coordinates to the preset coordinate system.
[0023] Therefore, the pixel coordinates formed by the line laser hitting the target surface are searched, and then the three-dimensional point cloud coordinates of the light strip in the camera coordinate system of the fisheye camera are calculated based on the calibrated internal and external parameters. From the original image to the three-dimensional coordinates, this process involves both internal and external parameters.
[0024] Preferably, the camera calibration model takes into account the distortion coefficient of the fisheye lens and uses a distortion coefficient elimination construction formula to overcome the image distortion error caused by the distortion in the direction of the corresponding coordinate axis.
[0025] Step 3, extract the pixel coordinates of the center of the light strip from the images to be calibrated at different sliding positions, and then convert the pixel coordinates of the center of the light strip into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera. The three-dimensional coordinates of the laser point here are represented by the coordinates of the camera coordinate system corresponding to the light strip at each sliding position, which is a three-dimensional coordinate with depth. The depth is provided by the external parameter from the target to the fisheye camera and the internal parameter of the fisheye camera, preferably the coordinate value in the Z-axis direction; then, according to the projection direction of the light strip in the imaging plane, the converted three-dimensional coordinates of the laser point are surface fitted so that the projection direction of the light strip in the imaging plane is consistent with the light strip. The number of fittings in the direction of the coordinate axis closest to the projection direction is the largest, and the laser surface function is obtained, which can also be regarded as obtaining the laser surface parameters, and determining the completion of the calibration of the laser parameters of the line laser module, so as to realize the calculation of the laser parameters through the three-dimensional coordinates of the light strips captured at multiple distances and angles in the camera coordinate system to complete the calibration of the laser parameters of the line laser module, specifically including determining the functional relationship between the laser surface and the fisheye camera (which can also be understood as the relative position relationship between the line laser and the fisheye camera), and completing the calibration of the position relationship between the line laser and the fisheye camera.
[0026] In this embodiment, the imaging plane is the XOY coordinate plane of the camera coordinate system. The coordinate axis where the highest degree independent variable in the laser surface function is located is determined by the coordinate axis closest to the projection direction of the light strip in the imaging plane. The coordinate axis closest to the projection direction of the light strip in the imaging plane is configured as the key coordinate axis for surface fitting, which directly affects the selection of the highest degree independent variable in the laser surface function, thereby improving the calibration accuracy of the line laser module.
[0027] If the calibration of the line laser module is divided into two parts: the internal parameter calibration of the fisheye camera and the laser parameter calibration, based on the above steps 1 and 2, the image used for the fisheye camera calibration is the image to be calibrated that does not include the light strip. Specifically, the internal parameters of the fisheye camera are calibrated first, and then the calibrated internal parameters and the external parameters of the fisheye camera are used to complete the calibration of the laser parameters by surface fitting.
[0028] In this embodiment, regarding the calibration of laser parameters and the fitting of the laser surface, the required mathematical model is a binary polynomial model, which is used to fit the laser surface, that is, by constructing a laser surface function, the calibration of the line laser module is completed. The binary polynomial model here is the laser parameter, which can be understood as a binary polynomial function z=f(x,y), which is an effective line laser parameter modeling and calibration method in this embodiment; in the binary polynomial function, in order to conform to the coordinate scalar form of the camera coordinate system, the independent variables can be selected as x and y, representing the coordinates of the X-axis and the Y-axis of the camera coordinate system respectively; the dependent variable is z, representing the coordinate of the Z-axis of the camera coordinate system; the highest degree of x and the highest degree of y will affect the shape of the surface that can be fitted by the binary polynomial function. In order to ensure the accuracy of parameter calibration, the coordinate variable on the coordinate axis closest to the projection direction of the light strip in the imaging plane is selected as the independent variable with the highest degree in the binary polynomial function. Then, the number of fitting times required on the coordinate axis closest to the projection direction of the light strip in the imaging plane is the largest, so as to select the surface fitting equation required for a relatively consistent laser surface, that is, to fit the laser surface function. Therefore, the fitting result of step 3 can represent the posture relationship between the fisheye camera and the laser surface emitted and formed by the line laser. The laser point can be positioned in the corresponding planning plane based on the laser surface function to obtain depth information.
[0029] It should be noted that the surface fitting equation, laser surface function, and binary polynomial function here all include binary polynomials with x and y as unknown quantities, which are equivalent to the polynomial f(x, y) outside the monomial where the variable z is located.
[0030] In summary, in the present application, when the line laser emits a line laser to the target assembly and controls the fisheye camera to capture images of the targets set at various angles at various sliding positions or the light strips projected from their surfaces, the target assembly is controlled to slide multiple distances relative to the line laser module, and in combination with the camera calibration model, the internal parameters of the fisheye camera are calculated and the laser surface function in the camera coordinate system is fitted as the calibrated line laser parameters. There is no need to manually move the target, and images of targets at multiple different angles set in the target assembly can be captured at the same position at one time. Compared with the traditional single chessboard calibration method, the calibration efficiency of the line laser module is improved while ensuring the reliability of the calibration results of the fisheye camera internal parameters. After the pixel coordinates of the center of the light strip are automatically extracted from the images to be calibrated at different sliding positions and converted into the three-dimensional coordinates of the laser point, a reasonable polynomial function (mathematical relationship) is selected for surface fitting based on the coordinate axis (key coordinate axis) closest to the projection direction of the light strip, so that the laser surface formed by the line laser emission is fitted as a laser surface function, and the functional relationship between the laser surface and the fisheye camera is determined. The calibration of the positional relationship between the line laser and the fisheye camera is completed, forming a more accurate and effective line laser parameter modeling and calibration method, improving the calibration accuracy, and solving the efficiency and accuracy problems of calibrating camera intrinsic parameters and line laser parameters when using a large-angle lens in the prior art.
[0031] As an embodiment, the target assembly includes a plurality of calibration plates, any two calibration plates are arranged at an angle, and the normal vectors of each calibration plate are different, so that each calibration plate included in the target assembly has a different orientation relative to the optical axis of the fisheye camera; a target is arranged on the surface of a calibration plate, and each target included in the target assembly has a different orientation relative to the optical axis of the fisheye camera. In this embodiment, the closer the target assembly slides to the line laser module, the fewer targets are included in a frame of the image to be calibrated taken by the fisheye camera; a frame of the image to be calibrated taken by the fisheye camera can be regarded as a frame of calibration plate image, which can include light strips. In combination Figure 1 and Figure 2It can be seen that not all targets can be captured in the image to be calibrated at a distance closer to the line laser module, but the target assembly can be photographed as a complete target by the fisheye camera at the minimum ranging boundary of the track 106. The minimum ranging boundary of the track 106 is the closest position that allows the targets in the target assembly to be imaged in the fisheye camera. In this embodiment, the farther the target assembly slides from the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera. The target assembly can be photographed as a complete target in all calibration plates by the fisheye camera at the maximum ranging boundary of the track 106. The maximum ranging boundary of the track 106 is the farthest position that allows all targets in the target assembly to be imaged in the fisheye camera. Therefore, when the target is imaged in the fisheye camera, it is necessary to consider the clarity of the fisheye camera when focusing, and try to achieve a full-viewing angle balancing effect for the calibration plates set at various angles.
[0032] exist Figure 1 In the target assembly, there are five calibration plates, and the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate 101 to reduce the error; the other four calibration plates are marked counterclockwise, namely calibration plate 102, calibration plate 103, calibration plate 104 and calibration plate 105. The sizes of calibration plates 102, 103, 104 and 105 are not necessarily equal. Calibration plate 102 is located above the reference calibration plate 101 and the reference calibration plate is set at a predicted observation angle. Calibration plate 104 is located below the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 103 is located to the left of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 105 is located to the right of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. The predicted observation angle is preferably 120 degrees. It is ensured that the field of view of the fisheye camera is comparable to the size of the target set at each angle, so as to ensure that the target assembly is still within the field of view of the fisheye camera when sliding.
[0033] like Figure 1 As shown, the reference calibration plate 101 and the calibration plate 102 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 103 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 104 are combined into a V-shaped calibration plate, and the reference calibration plate 101 and the calibration plate 105 are combined into a V-shaped calibration plate; the angle formed by the reference calibration plate 101 and the calibration plate 102 can be set according to actual conditions, that is, the predicted observation angle setting, which is not limited here. For example, the reference calibration plate 101 and the calibration plate 102 are preferably placed at an angle of 120 degrees; the reference calibration plate 101 and one side of the calibration plate 102 are arranged to fit each other, so that a part of the target assembly is formed by combining two calibration plates, and targets at two angles can be set accordingly.
[0034] To facilitate the placement of the V-shaped calibration plate, the two plates in the V-shaped calibration plate can be connected by a hinge. In one embodiment, the connecting strip supports the two plates to be opened and closed to any angle to facilitate the external parameter calibration between the line laser and the fisheye camera. For example, it can be adjusted so that near the maximum ranging boundary of the track 106, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the direction of the fisheye camera).
[0035] To improve the calibration efficiency, the longer right-angled sides of the two calibration plates can be fixed to each other. In other embodiments, the shapes of the two plates in the V-shaped calibration plate are not limited to quadrilateral plates such as rectangular plates.
[0036] Preferably, the number of different positions that the target assembly slides along the track is greater than or equal to 4. Frequently capturing 4 to-be-calibrated images at different distances for camera calibration is more efficient than the traditional single chessboard calibration method.
[0037] Preferably, the normal vector of the reference calibration plate 101 is parallel to the optical axis of the fisheye camera. Considering the symmetry problem, the calibration plate 103 located on the left of the reference calibration plate 101 and the calibration plate 105 located on the right of the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate, and the targets on the left and right of the reference calibration plate 101 are symmetrically arranged; the calibration plate 102 located above the reference calibration plate 101 and the calibration plate 104 located below the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate, and the targets on the upper and lower sides of the reference calibration plate 101 are symmetrically arranged. The calibration error is further reduced.
[0038] In some embodiments, the key to the target assembly sliding along the track 106 is to allow the target to be photographed by the fisheye camera at several different distances, including far, medium and near. At least: when the target assembly slides to a sliding position at a first distance from the line laser module, the fisheye camera photographs 1 complete target; when the target assembly slides to a sliding position at a second distance from the line laser module, the fisheye camera photographs 1 to 3 complete targets; when the target assembly slides to a sliding position at a third distance from the line laser module, the fisheye camera photographs 5 complete targets; wherein, one target is set on a calibration plate, the first distance is smaller than the second distance, and the second distance is smaller than the third distance.
[0039] As an embodiment, in step 2, before the fisheye camera calibration is performed, the method further includes: whenever the target assembly slides to a sliding position, searching for corner points in a frame of the image to be calibrated taken in real time by the fisheye camera within the complete target, combining Figure 3It can be seen that when the target is a black and white square area, the corner point is the intersection of the edge lines of the black cell or the intersection of the edge lines of the white cell, which can be regarded as a two-dimensional coordinate located on the pixel plane. Then, in the same frame of the image to be calibrated, the searched corner points are grouped according to the complete target taken. A group of corner points corresponds to a complete target, and they can be numbered according to the calibration plate where the corner points are located to distinguish the angular orientation information of the target where the corner points are located. Therefore, a group of corner points is the corner points in a target set at an angle captured by the fisheye camera when the target assembly slides to a sliding position corresponding to a distance. Then the complete target where each group of corner points is located is numbered so that the calibration plates set at different angles in the same frame of the image to be calibrated have different numbers.
[0040] When the target assembly slides to different sliding positions, the numbers of the complete targets in different frames of the images to be calibrated taken by the fisheye camera are different. Among them, incomplete targets are not numbered, and the corner points searched inside each numbered target are stored in a set of data. From the sensor hardware calibration level, the matching relationship between the three-dimensional space points obtained by the line laser and the two-dimensional pixel points obtained by the fisheye camera is improved.
[0041] For example, the target assembly slides along the track 106 to a sliding position at a third distance from the line laser module, and the fisheye camera captures 5 complete targets and searches for corner points therefrom, then the corner points searched out in each complete target can be grouped together; then the complete targets in each group of corner points are numbered, that is, the 5 complete targets are numbered as target 1, target 2, target 3, target 4 and target 5 in sequence, and the specific order is not limited. Then the target assembly slides to a sliding position at a second distance from the line laser module, and the second distance is less than the third distance, and the fisheye camera captures 3 complete targets and searches for corner points therefrom, then the corner points searched out in each complete target can be grouped together; then the 5 complete targets are numbered as target 6, target 7 and target 8 in sequence, and the specific order is not limited, and it is only required to distinguish the various targets (or calibration plates) set at different angles at different distances for easy identification and extraction.
[0042] In summary, before calibrating the internal parameters of the fisheye camera, the corner points of multiple targets are identified and the targets are numbered and sorted. Because each calibration plate or each target in the target assembly has a specific distribution, numbering makes it easier to match the position relationship between the light bar and the calibration plate it is on during the laser parameter calibration process in step 3.
[0043] Preferably, the Harris corner detection method is used to search for corners in the complete target. It should be noted that the Harris corner detection method mainly searches for corners based on the gradient of the grayscale value change of the image pixels. Specifically, the area near the corner of the grayscale image is an area where the pixel grayscale value changes greatly, and its gradient is also very large. In other words, in the neighborhood of the non-corner position, the pixel values of each point do not change much, or are almost equal, and their gradients are relatively small. If the absolute value of the difference between the grayscale value of the point in the neighborhood and the grayscale value of the center point Image(i,j) is within a threshold t range, then this point is considered to be similar to the center point. At the same time, the similarity counter nlike(i,j) belonging to the Image(i,j) point is also increased by 1. After all the n neighborhoods of the Image(i,j) point are traversed once, the statistical value nlike(i,j) of the number of points similar to the center point in this neighborhood can be obtained. According to the size of nlike(i,j), it can be judged whether this center point is likely to be a corner point.
[0044] Based on the above embodiment, in the same frame of the image to be calibrated, if the number of corner points searched from one of the targets is within a preset number threshold range, one of the targets is set as a complete target and assigned a number, wherein the preset number threshold range is preferably greater than or equal to 10 and less than or equal to 40.
[0045] A complete target is a black and white square area, such as Figure 3 As shown, similar to a chessboard, the calibration plate and the target are both rectangular, the size of the calibration plate is related to the size of the target, the black and white square area is composed of black cells and white cells, which are regarded as black cells and white cells arranged at intervals, and the aforementioned corner points are the corner points of the black cells and / or the corner points of the white cells; preferably, the number of cells constituting the square area is linearly related to the number of corner points searched out; the targets set in each calibration plate are consistent, which can be understood as providing the same calibration features for the line laser and the fisheye camera from each angle direction at the same distance.
[0046] On the surface of a calibration plate, the outside of the target is surrounded by a white area, leaving a certain amount of blank white background for a complete target on the surface of the calibration plate for line laser calibration, because pure white is conducive to the fisheye camera capturing clear light strips.
[0047] As an embodiment, in the same complete target, a polygonal area is formed by expanding outward with the corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light bar, wherein the polygonal area includes all cells in the black and white square area and the white area outside the target, so that the polygonal area frames the black and white square area, and the black and white square area is composed of all black and white cells in a single calibration plate to form a complete target or a complete checkerboard pattern; then the polygonal area is set as the ROI (Region of Interest) area, specifically, the occupied area in the pixel plane of the fisheye camera is set as the ROI area, i.e., the region of interest, so as to realize the use of the corner points to frame a pixel area around the target.
[0048] It should be noted that at multiple distances over which the target assembly slides, the line laser emitted by the line laser will fall inside the black and white square area at each sliding position, or on the blank area at the edge of the target, where the light bar is projected at the junction of black and white, and it is not easy to find the correct center of the light bar in the image to be calibrated captured by the fisheye camera. Therefore, the target is set in the middle of the calibration plate, and white space is left around the target. In this embodiment, in order to increase the ability of the fisheye camera to capture clear light bars, the ROI area covers the white area outside the target.
[0049] In this embodiment, on the surface of the calibration plate, after the reflected light of the polygon extending from the corner point changes the optical path through the fisheye camera, the occupied area in the pixel plane of the fisheye camera will be distorted and become different from the original extended polygon. For example, the square in front of the fisheye lens will be distorted after focusing and imaging, forming the ROI area. Subsequently, it becomes more accurate to directly search for the center pixel of the light bar in the ROI area, and accurately search for the light bar on the calibration plate set at the corresponding angle while adapting to the camera distortion condition, thereby improving the stability of the calibration and reducing the amount of calculation.
[0050] Preferably, the camera calibration model is a full-view camera model calibration correction model or a fisheye lens distortion correction model, which is used to calculate the parameters of the fisheye camera, including the internal parameters of the fisheye camera and the external parameters of each target to the fisheye camera, so as to calibrate the fisheye camera. Specifically, for the internal parameter calibration of the fisheye camera, the OCamCalib full-view camera model calibration correction algorithm proposed by David Scaramuzza can be used, or the calculation method based on the fisheye lens distortion correction model proposed by Juho Kannala and Sami Brandt in the open source algorithm library OpenCV can be used to calculate the camera internal parameters. Here, unless otherwise specified, this application will use the first method for explanation.
[0051] It should be noted that Scaramuzza D (David Scaramuzza), Martinelli A, Siegwart R., technicians in the relevant field, proposed a camera calibration model in the article "A toolbox for easily calibrating omnidirectional cameras" published in the "2006 IEEE / RSJ International Conference on Intelligent Robots and Systems". The model is a calibration and correction model for omnidirectional cameras and is used to calculate the parameters of fisheye cameras, including the intrinsic parameters of the fisheye camera and the extrinsic parameters of the fisheye camera relative to each target.
[0052] As an embodiment, in step 1, whenever the target assembly slides linearly relative to the line laser module to a sliding position, the relative position between the target assembly and the line laser module is kept unchanged, and the line laser and the fisheye camera are fixedly assembled, there exists:
[0053] First, the line laser is controlled to emit a line laser to the target assembly, and the fisheye camera is controlled to capture at least one complete target to obtain a frame of laser stripe image corresponding to a sliding position. A frame of laser stripe image is a frame of image to be calibrated including light stripes. At this time, the target cannot be identified in the image to be calibrated, that is, the aforementioned black and white square area and the cells inside it, and only the central pixel coordinates of the light stripes in the calibration plate with the corresponding number can be extracted; the fisheye camera preferably adopts an infrared bandpass filter or an infrared high-pass filter to filter the reflected light of the line laser emitted by the line laser on the surface of the calibration plate, specifically, to transmit the infrared band carried by the line laser, and absorb or reflect the ambient light of other bands, so as to separate the infrared light carried by the reflected light from other interfering ambient light.
[0054] Then, neither the target assembly nor the line laser module moves, the line laser is controlled to stop emitting the line laser, and the infrared fill light is turned on to fill light for the target assembly, and the fill light increases the brightness. This is done to increase the depth of field during shooting. Because under the condition of relatively little ambient light in the factory, the fisheye camera is not able to clearly capture the information of the target and its corner points through external light, and a fill light is needed to illuminate the target. The relative position between the infrared fill light and the line laser module can remain unchanged or change, and there is no specific restriction here; and the fisheye camera is controlled to capture the at least one complete target to obtain a frame of target image corresponding to the same sliding position; specifically, in an environment where the ambient light interference is relatively small, the line laser is turned off and the target is filled with light using the infrared fill light, and then the image to be calibrated is captured. The advantage of doing this is that it facilitates image processing. Therefore, step 2 searches for corner points from the target image and expands the ROI area, and then frames the ROI area in the same pixel area of the laser stripe image. The positional relationship of the same pixel area in the laser stripe image is the positional relationship of the ROI area in the target image. Based on the pixel position information of the ROI area framed in the target image, the center pixel coordinates of the light stripe falling into the ROI area can be searched in the laser stripe image. On this basis, combined with the external parameters obtained by the fisheye camera calibration, the three-dimensional coordinates of the laser point on the intersection of the line laser and the target surface (in the camera coordinate system) can be converted, which corresponds to the spot position information of the light stripe searched in the area framed by the ROI area in the target, thereby narrowing the search range of the light stripe and improving the calibration efficiency of the laser parameters.
[0055] In summary, every time the target component moves a certain distance, two types of images must be collected, namely, a frame of laser stripe image and a frame of target image; when shooting a frame of target image, the infrared fill light is turned on, but the line laser is not turned on, so the target can only be identified from the target image, and the internal parameter calibration of the fisheye camera can be performed through the camera calibration model. When shooting a frame of laser stripe image, the infrared fill light is not turned on, but the line laser is turned on, wherein only the light strips can be identified from the laser stripe image, but the target cannot be identified, and then combined with the target identified from the target image, it can be used for laser parameter calibration, thereby utilizing the intermediate data in the internal parameter calibration of the fisheye camera, which speeds up the calibration process.
[0056] On the basis of the above embodiment, the number of targets covered by a frame of laser stripe image and a frame of target image taken by the target assembly at the same sliding position is equal, and the orientation of targets covered by a frame of laser stripe image and a frame of target image taken by the target assembly at the same sliding position is the same, so that when the fisheye camera takes a frame of laser stripe image and a frame of target image corresponding to the same sliding position, the external parameters from the target assembly to the fisheye camera are equal.
[0057] In step 2, the external parameters of the target assembly at each sliding position to the fisheye camera are the results obtained after the pixel data of the image to be calibrated is input into the camera calibration model. The corresponding external parameters can be composed of a rotation matrix R and an offset vector t, which can map the pixel coordinates (or the coordinate system of the imaging plane) to the coordinates of the target plane, and can also be mapped from the world coordinate system to the camera coordinate system to achieve the conversion of two-dimensional coordinates to three-dimensional vectors; wherein, the pixel data of the image to be calibrated includes but is not limited to the pixel coordinates of the corner points in the numbered targets, which are used to calibrate the fisheye camera and can also be used to calibrate the laser parameters. The result obtained by the camera calibration model also includes the internal parameters of the fisheye camera, which can be converted from the camera coordinate system to the pixel coordinate system based on the internal parameters, and the internal parameters output by the camera calibration model can also explain the stretching and deformation of the fisheye lens (the lens of the fisheye camera) to the graphics in the world coordinate system. The corresponding internal parameter parameters can use the stretching matrix to compensate for the misalignment of the sensor to the lens. The external parameters of the target assembly at each sliding position to the fisheye camera include the external parameters of the cells in the targets set at each angle at each sliding position. Therefore, the camera calibration model converts the numbered corner points and cell position information in the target into the intrinsic and extrinsic parameters of the fisheye camera to obtain the true spatial scale on the target.
[0058] As an embodiment, in step 3, the method of extracting the center pixel coordinates of the light stripe from the images to be calibrated at different sliding positions, and then converting the center pixel coordinates of the light stripe into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera includes: extracting the center pixel coordinates of the light stripe from the ROI area in the images to be calibrated at different sliding positions, and setting the center pixel coordinates of the light stripe as the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera as the ideal image projection of the light spot in the real world; searching for the light stripe position in the ROI area to facilitate the calibration of the laser parameters after the camera calibration.
[0059] Then, based on the coordinate system conversion function set in the preset mathematical analysis software, the parameters of the fisheye camera calibrated by the camera calibration model are used to map the two-dimensional pixel coordinates (u, v) to the light spot coordinates of the corresponding numbered target surface (corresponding numbered calibration plate) The light spot coordinates It is located in the world coordinate system set on the corresponding calibration plate, Z W It can be equal to the value 0, so that the light spot coordinates become the world coordinates of the corresponding calibration plate plane or target plane, realizing the conversion of the two-dimensional pixel coordinates (u, v) to the plane of the calibration plate with the corresponding number, and restoring the two-dimensional coordinates in the real world.
[0060] Preferably, the preset mathematical analysis software includes but is not limited to Matlab (a combination of the two words matrix & laboratory, meaning matrix factory (matrix laboratory)), and a pointsToWorld function is provided in its internal toolkit for mapping two-dimensional pixel coordinates into coordinates in a world coordinate system. The pointsToWorld function may be derived from a paper “A toolbox for easily calibrating omnidirectional cameras” published by Scaramuzza D (David Scaramuzza), a technician in the relevant field; in this embodiment, the coordinates in the world coordinate system may be divided into world coordinate systems corresponding to different calibration plates (or different targets).
[0061] Then, the light spot coordinates are converted into Convert the three-dimensional coordinates of the laser point into the camera coordinate system The corresponding conversion formula is Among them, the external parameters include the rotation matrix R and offset vector T required to convert the world coordinate system set on the surface of the target with the corresponding number into the camera coordinate system of the fisheye camera, referred to as the external parameters R and T corresponding to the target to the camera, which are derived from a rigid body transformation specified by the camera calibration model and are conducive to converting the depth information.
[0062] In some embodiments, it can be understood that the coordinates of the center pixel of the light strip extracted in the ROI area and the corner point coordinates of the projection relationship in the target are the coordinates of the corner point in the world coordinate system. The rotation matrix R and displacement vector T of the corner point generated by the camera calibration model, which has a projection relationship with the center pixel coordinates of the currently extracted light strip, are used to convert the corner point coordinates into the three-dimensional coordinates of the laser point in the camera coordinate system. The rotation matrix R and the displacement vector T are both external parameters of the fisheye camera. Among them, the corner point that has a projection relationship with the center pixel coordinates of the currently extracted light strip is located on a calibration plate set at a corresponding angle relative to the line laser module at a sliding position, specifically a target with a corresponding number in this embodiment. Therefore, based on the external parameters and the internal parameters, the two-dimensional pixel coordinates corresponding to the targets (targets pre-numbered in step 2) set at various angles at different sliding positions are converted to world two-dimensional coordinates, and then converted from world two-dimensional coordinates to camera coordinates, providing the surface fitting in step 3 with the point cloud data to be fitted in the camera coordinate system.
[0063] Considering the lens distortion suitable for fisheye cameras, combined with the camera calibration model, the following embodiments may exist:
[0064] Step 3 extracts the pixel coordinates of the center of the light strip from the ROI area in the image to be calibrated at different positions, and sets the pixel coordinates of the center of the light strip to the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera. Then, according to the camera calibration model, the pixel coordinates of the center of the light strip are converted into distortion-free three-dimensional coordinates. Wherein, λ is the scaling factor, which indicates the scale scaling degree of the pixel coordinates, that is, the numerical relationship between the physical length and the pixel length after the physical length in the actual space is scaled to the pixel length in the pixel plane; λ can also represent the scalar factor of the depth; a0, a2, a3 and a4 are all distortion coefficients of the fisheye camera, which are all polynomial coefficients described by the camera calibration model, where a1 is 0. Indicates the distance in pixels between the center of the light bar and the center of the image. The scaling factor may vary depending on the number of the target where the pixel coordinates of the center of the light bar are located (corresponding to the set angle).
[0065] Specifically, a0+a2ρ 2 +a3ρ 3 +a4ρ 4 It is used to correct a series of distortions of the fisheye camera to convert the ROI area in the distorted image (the image to be calibrated) into a normal undistorted image, and obtain the corresponding undistorted 3D coordinates. It can be used as a distortion-free three-dimensional coordinate, and further converted to the light spot coordinate of the corresponding numbered target surface (corresponding numbered calibration plate) disclosed in the above embodiment through the aforementioned corresponding target to camera external parameters (which can be represented by R and T) This achieves the effect of eliminating the lens distortion in the image to be calibrated taken by the fisheye camera.
[0066] Specifically, It can be used as an intermediate quantity for transforming from the pixel coordinate system of a fisheye camera to the camera coordinate system.
[0067] The image to be calibrated is the image after the fisheye camera is distorted. Processing the two-dimensional pixel coordinates (u, v) through the aforementioned intermediate quantities is equivalent to flattening and straightening the target to form an anti-distorted image. Then, the depth information still needs to be calculated using external parameters, because the fisheye camera as a monocular camera in the online laser module cannot know the distance, at least it cannot independently know the depth information without the help of the line laser.
[0068] Preferably, starting from the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera, in the framework of the pointsToWorld function, the internal parameters of the fisheye camera are used to map to the normalized plane to obtain the normalized coordinates (Xnorm, Ynorm), Z = 1 (the depth on the normalized plane is 1, and the normalized plane is parallel to the imaging plane of the fisheye camera), the normalized coordinates (Xnorm, Ynorm) and Proportional relationship exists If we combine the laser surface function obtained in step 3 above, we can find Z C , you can get the three-dimensional coordinates of the laser point in the camera coordinate system The camera coordinates are inferred to obtain the depth information of the target.
[0069] As an embodiment, in step 3, the method for performing surface fitting on the converted three-dimensional coordinates of the laser point according to the projection direction of the light strip in the imaging plane includes: during the process of the target assembly sliding linearly for multiple distances relative to the line laser module, the line laser emits a line laser to the target assembly, which is reflected into the imaging plane of the fisheye camera through the target at the corresponding angle, and is projected to form a light stripe line, and the extension direction of the light stripe line is set to the projection direction of the light strip in the imaging plane, reflecting the relative relationship between the laser plane formed by the line laser through the fisheye camera and the camera coordinate system; preferably, the imaging plane is the image coordinate plane of the camera coordinate system, that is, the XOY coordinate system plane.
[0070] In the horizontal axis and the vertical axis of the image coordinate system corresponding to the imaging plane of the fisheye camera, the angles formed by the projection direction of the light strip in the imaging plane and the various coordinate axes of the image coordinate system are detected, including the angle formed by the projection direction and the positive direction of the horizontal axis of the image coordinate system, and the angle formed by the projection direction and the positive direction of the vertical axis of the image coordinate system.
[0071] In this embodiment, the binary polynomial function or the laser surface function includes only two independent variables, namely a first independent variable and a second independent variable; the first independent variable is used to represent the horizontal coordinate of the image coordinate system, and the second independent variable is used to represent the vertical coordinate of the image coordinate system.
[0072] When it is detected that the acute angle formed by the projection direction of the light strip in the imaging plane and the horizontal axis of the image coordinate system (the minimum angle formed by the horizontal axis) is less than the acute angle formed by the projection direction of the light strip in the imaging plane and the vertical axis of the image coordinate system (the minimum angle formed by the vertical axis), the three-dimensional coordinates of the laser point converted at each angle of the target assembly at multiple sliding positions are used to fit a binary polynomial function whose highest degree of the first independent variable is greater than the highest degree of the second independent variable, and the laser surface function is determined to fit the laser surface and determine the posture relationship between the laser surface and the fisheye camera, which is considered to complete the calibration of the laser parameters. It can be understood that the laser surface is the simulated surface formed by the line laser after transmission through the fisheye camera; and the fitting number of the first independent variable is greater than the fitting number of the second independent variable; at this time, the coordinate axis represented by the first independent variable is determined to be the key coordinate axis in the surface fitting, that is, the coordinate axis with the most fitting times in the binary polynomial function.
[0073] When it is detected that the acute angle formed by the projection direction of the light strip in the imaging plane and the ordinate axis of the image coordinate system is smaller than the acute angle formed by the projection direction of the light strip in the imaging plane and the abscissa axis of the image coordinate system, the three-dimensional coordinates of the laser points converted at various angles at multiple sliding positions of the target assembly are used to fit a binary polynomial function whose highest degree of the second independent variable is greater than the highest degree of the first independent variable, and the laser surface function is determined to fit the laser surface, so as to fit the laser surface and determine the posture relationship between the laser surface and the fisheye camera, which is considered to complete the calibration of the laser parameters. It can be understood that the laser surface is the surface formed by the simulated line laser after transmission through the fisheye camera, and the fitting number of the first independent variable is less than the fitting number of the second independent variable. At this time, the coordinate axis represented by the second independent variable is determined to be the key coordinate axis in the surface fitting, that is, the coordinate axis with the most fitting times in the binary polynomial function.
[0074] Therefore, this embodiment fits the laser surface from the horizontal axis and the vertical axis of the image coordinate system respectively and models the function whose highest power term satisfies the highest power of the independent variable in the direction of the corresponding coordinate axis, and realizes surface fitting of the converted three-dimensional coordinates of the laser point according to the projection direction of the light strip in the imaging plane, so that the number of fittings in the direction of the coordinate axis closest to the projection direction of the light strip in the imaging plane is the largest, and the camera coordinates of the light strips captured at multiple distances and angles are used to perform surface fitting calculations on the line laser parameters. Therefore, the following problems are overcome: First, in cameras with relatively large distortion, due to large edge distortion, plane fitting will lead to inaccurate calculation of the edge points of the light plane; second, when the quality of the laser cannot achieve good straightness, the edge points are curved and plane fitting is not applicable. This makes the laser parameter calibration results obtained by combining a fisheye camera with a line laser more accurate and the measurement effect better.
[0075] On the basis of the above embodiment, in the binary polynomial function whose highest degree of the first variable is greater than the highest degree of the second variable, the coordinate axis (horizontal axis) represented by the first independent variable deviates closest to the aforementioned light streak line, and the coordinate axis (horizontal axis) represented by the first independent variable becomes the key coordinate axis, and the highest degree of the first independent variable is greater than or equal to 2, and the highest degree of the second independent variable is equal to 1, so that the fitting times of the binary polynomial function in the horizontal axis direction of the image coordinate system is greater than the fitting times in the vertical axis direction of the image coordinate system, so the fitting accuracy of the binary polynomial function in the horizontal axis direction of the image coordinate system is higher than the fitting accuracy in the vertical axis direction of the image coordinate system. Preferably, the highest degree of the first independent variable is 3, and the highest degree of the second independent variable is 1, so as to save the amount of calculation; the specific surface fitting form can be selected according to the actual situation, such as using the least squares method, and this example does not make special restrictions.
[0076] In the binary polynomial function whose highest degree of the second independent variable is greater than the highest degree of the first independent variable, the coordinate axis (ordinate axis) represented by the second independent variable deviates closest to the aforementioned light streak line, and the coordinate axis (ordinate axis) represented by the second independent variable becomes the key coordinate axis. The highest degree of the second independent variable is greater than or equal to 2, and the highest degree of the first independent variable is equal to 1, so that the fitting times of the binary polynomial function in the direction of the ordinate axis of the image coordinate system is greater than the fitting times in the direction of the abscissa axis of the image coordinate system, so the fitting accuracy of the binary polynomial function in the direction of the abscissa axis of the image coordinate system is higher than the fitting accuracy in the direction of the ordinate axis of the image coordinate system. Preferably, the highest degree of the second independent variable is 3, and the highest degree of the first independent variable is 1, so as to save the amount of calculation; the specific surface fitting form can be selected according to the actual situation, such as using the least squares method, and this example does not make special restrictions.
[0077] Preferably, the binary polynomial function is expressed as z=f(x, y), forming a functional relationship between the laser surface and the fisheye camera, corresponding to the relative position relationship between the line laser and the fisheye camera; the image coordinate system is the plane coordinate system XOY formed by the camera coordinate system in the imaging plane, the imaging plane is the normal plane of the Z axis of the camera coordinate system, the ordinate axis of the image coordinate system is the Y axis, the abscissa axis of the image coordinate system is the X axis, the first independent variable of the binary polynomial function is represented by x, the second independent variable of the binary polynomial function is represented by y, and the dependent variable of the binary polynomial function is represented by z; in the binary polynomial function, the polynomial other than the term where the dependent variable z is located is f(x, y), for example, f(x, y)=x 3When y+xy+x, the highest power of the first independent variable x in f(x,y) is 3, and the highest power of the second independent variable y is 1. For example, f(x,y)=y 3 When x+x, the highest power of the second independent variable y in f(x,y) is 3, and the highest power of the first independent variable x is 1. Here, multiple sets of relative pose relationships are not calculated, but the three-dimensional coordinates of the laser point in the camera coordinate system obtained in step 3 are used to perform three-dimensional surface fitting to obtain the laser surface function in the camera coordinate system, which can be expressed as an equation in the form of the binary polynomial function z=f(x,y) or F(x,y,z)=0, thereby improving the calibration efficiency.
[0078] In summary, the surface fitting disclosed in this embodiment adopts a binary polynomial function to model, and only the number of fittings is increased for the key coordinate axes, thereby improving the modeling and fitting efficiency without increasing too much calculation. Among them, step 3 will specifically select the coordinate variables of a certain key coordinate axis for high-order fitting, so that the highest order of the coordinate variables affects the surface shape represented by the binary polynomial function, which is more consistent with the effect of the form fitting of the laser surface, improves the calibration accuracy of the laser parameters, and obtains a more accurate relative position relationship between the line laser (or the laser surface formed by it) and the fisheye camera.
[0079] The present application also discloses a calibration device, Figure 1 and Figure 2 It can be known that the calibration device includes a calibration drive component (not shown in the figure), a target component, a track 106, a fixed support platform 112, a line laser module, and a parameter calibration unit for the parameter calibration method disclosed in the aforementioned embodiment; wherein the target component includes a plurality of targets arranged at an angle, namely a target arranged on the surface of the reference calibration plate 101, a target arranged on the surface of the first calibration plate 102, a target arranged on the surface of the second calibration plate 103, a target arranged on the surface of the third calibration plate 104, and a target arranged on the surface of the fourth calibration plate 105; the target component can be fixedly set on the calibration drive component by a connecting bracket 109 to drive the target to slide linearly along the track 106. The line laser module includes a line laser and a fisheye camera, and both the line laser and the fisheye camera are integrally assembled to the fixed support platform 112 through a sensor fixing position 111, and the fixed support platform is fixedly assembled with the line laser and the fisheye camera; the fixed support platform 112 is fixedly set at one end of the track 106, such as Figure 2 The lower end shown can be specifically Figure 1 The connector 108 shown is fixedly arranged; the target assembly can be slidably arranged at the other end of the track 106 (such as Figure 2For line laser modules, fisheye cameras support extended field of view, and the imaging range tries to catch up with the scanning range of the target at various angles at multiple distances of the line laser, reducing the number of imaging sensors used. The fisheye camera shooting process does not require manual movement of the target, and is easy to automate; thereby reducing the number of calibrations and error sources, and improving the calibration accuracy of the line laser module.
[0080] In this embodiment, the parameter calibration unit is used to control the target assembly to slide multiple distances relative to the line laser module; control the line laser to emit a line laser to the target assembly, and project a light strip on the surface of the target at the corresponding angle; and control the fisheye camera to sequentially shoot the target or the light strip projected from its surface set at at least one angle in the target assembly at each sliding position to obtain the image to be calibrated. The parameter calibration unit is also used to input the pixel data of the image to be calibrated at different sliding positions into the camera calibration model to calibrate the fisheye camera and output the parameters of the fisheye camera. The parameter calibration unit is also used to extract the center pixel coordinates of the light strip from the image to be calibrated at different sliding positions, and then convert the center pixel coordinates of the light strip into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera; then, according to the projection direction of the light strip in the imaging plane, the converted three-dimensional coordinates of the laser point are surface fitted so that the number of fitting times in the coordinate axis direction closest to the projection direction of the light strip in the imaging plane is maximized, and the laser surface function is obtained, and the calibration of the laser parameters of the line laser module is determined to be completed.
[0081] As an embodiment, the target assembly includes a plurality of calibration plates, any two calibration plates are arranged at an angle, and the normal vectors of each calibration plate are different, and a target is arranged on a calibration plate, so that each calibration plate included in the target assembly has a different orientation relative to the optical axis of the fisheye camera; a target is arranged on the surface of a calibration plate, and then each target included in the target assembly has a different orientation relative to the optical axis of the fisheye camera. In some embodiments, on the surface of each calibration plate, the targets are arranged in a checkerboard pattern, and the edges of the checkerboard pattern are surrounded by a white area for line laser calibration; in the target assembly, the checkerboard pattern arranged on the surface of each calibration plate is the same; wherein, at multiple sliding positions of the same target assembly, the fisheye camera captures the complete target from the calibration plate arranged at the corresponding angle. In the same complete target, a polygonal area is formed by expanding outward from the corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light bar, wherein the polygonal area includes all cells of the black and white square area on a single target and the white area outside the target; and then the polygonal area is set as the ROI (Region of Interest) area, because the polygonal area needs to be configured to frame the entire checkerboard pattern on a single target, wherein all cells of the black and white square area form a complete checkerboard pattern. The ROI frames the area around the checkerboard pattern, including the blank part on the calibration plate, and there will be a part of the light bar in the checkerboard and in the blank part of the area. Some light bars may be missed outside the ROI area, but it does not matter, as long as the light bars in the ROI area are sufficient to calculate the aforementioned laser surface function. Generally, the light bars are made to fall as much as possible in the ROI area according to the completeness of the target in the previous calibration image and the sliding distance of the target assembly in the track.
[0082] In this embodiment, the closer the target assembly slides along the track to the line laser module, the fewer complete targets are included in a frame of the image to be calibrated captured by the fisheye camera; conversely, the farther the target assembly slides along the track to the line laser module, the more complete targets are included in a frame of the image to be calibrated captured by the fisheye camera, so that the fisheye camera can capture complete targets in all calibration plates at the ranging boundary of the track. The number of different positions that the target assembly slides along the track is greater than or equal to 4. Figure 1 and Figure 2It can be seen that not all targets can be captured in the image to be calibrated at a distance closer to the line laser module, but the target assembly can be photographed as a complete target by the fisheye camera at the minimum ranging boundary of the track 106. The minimum ranging boundary of the track 106 is the closest position that allows the targets in the target assembly to be imaged in the fisheye camera. In this embodiment, the farther the target assembly slides from the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera. The target assembly can be photographed as a complete target in all calibration plates by the fisheye camera at the maximum ranging boundary of the track 106. The maximum ranging boundary of the track 106 is the farthest position that allows all targets in the target assembly to be imaged in the fisheye camera. Therefore, when the target is imaged in the fisheye camera, it is necessary to consider the clarity of the fisheye camera when focusing, and try to achieve a full-viewing angle balancing effect for the calibration plates set at various angles. Based on the specific target distribution structure and the sliding mode of the target assembly in the track, this embodiment can simplify the number of images to be calibrated to be collected to 3 to 4 frames. The corresponding technical effect is: while ensuring the reliability of the internal calibration results of the fisheye camera, the calibration efficiency of the line laser module is improved compared with the traditional single chessboard calibration method.
[0083] As an example, combining Figure 1 and Figure 2 It can be seen that there are five calibration plates in the target assembly; the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate; among the other four calibration plates, there is a calibration plate located above the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is a calibration plate located below the reference calibration plate and is set at a predicted observation angle with the reference calibration plate, there is a calibration plate located to the left of the reference calibration plate and is set at a predicted observation angle with the reference calibration plate, and there is a calibration plate located to the right of the reference calibration plate and is set at a predicted observation angle with the reference calibration plate, so that when the target assembly slides along the track, the fisheye camera captures at least one complete target in the target assembly and the white area set around it. On the surface of a calibration plate, the outside of the target is surrounded by a white area, and a certain blank white background is left for a complete target on the surface of the calibration plate, which can be used for line laser calibration, because pure white is conducive to the fisheye camera capturing clear light strips.
[0084] Specifically, in Figure 1In the target assembly, there are five calibration plates, and the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate 101 to reduce the error; the other four calibration plates are marked counterclockwise, namely calibration plate 102, calibration plate 103, calibration plate 104 and calibration plate 105. The sizes of calibration plates 102, 103, 104 and 105 are not necessarily equal. Calibration plate 102 is located above the reference calibration plate 101 and the reference calibration plate is set at a predicted observation angle. Calibration plate 104 is located below the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 103 is located to the left of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. Calibration plate 105 is located to the right of the reference calibration plate 101 and is set at a predicted observation angle with the reference calibration plate 101. The predicted observation angle is preferably 120 degrees. It is ensured that the field of view of the fisheye camera is comparable to the size of the target set at each angle, so as to ensure that the target assembly is still within the field of view of the fisheye camera when sliding.
[0085] like Figure 1 As shown, the reference calibration plate 101 and the calibration plate 102 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 103 are combined into a V-shaped calibration plate, the reference calibration plate 101 and the calibration plate 104 are combined into a V-shaped calibration plate, and the reference calibration plate 101 and the calibration plate 105 are combined into a V-shaped calibration plate; the angle formed by the reference calibration plate 101 and the calibration plate 102 can be set according to actual conditions, that is, the predicted observation angle setting, which is not limited here. For example, the reference calibration plate 101 and the calibration plate 102 are preferably placed at an angle of 120 degrees; the reference calibration plate 101 and one side of the calibration plate 102 are arranged to fit each other, so that a part of the target assembly is formed by combining the two calibration plates, and targets at two angles can be set accordingly.
[0086] To facilitate placement of the V-shaped calibration plate, the two plates in the V-shaped calibration plate can be connected by hinges, and the hinge connection can be in the form of Figure 2 The connecting strip 113 is fixed together at the predicted observation angle. Specifically, the calibration plate 103 and one side of the reference calibration plate are assembled by a connecting strip in an obtuse angle and a screw to form a connecting hinge. In one embodiment, the connecting strip supports the two plates to be opened and closed to any angle to facilitate the external parameter calibration between the line laser and the fisheye camera. For example, it can be adjusted so that near the maximum ranging boundary of the track 106, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the fisheye camera).
[0087] To improve the calibration efficiency, the longer right-angled sides of the two calibration plates can be fixed to each other. In other embodiments, the shapes of the two plates in the V-shaped calibration plate are not limited to quadrilateral plates such as rectangular plates.
[0088] Preferably, the normal vector of the reference calibration plate 101 is parallel to the optical axis of the fisheye camera. Considering the symmetry problem, the calibration plate 103 located on the left of the reference calibration plate 101 and the calibration plate 105 located on the right of the reference calibration plate 101 can be symmetrically arranged about the center of the reference calibration plate 101, and the targets on the left and right of the reference calibration plate 101 are symmetrically arranged; the calibration plate 102 located above the reference calibration plate 101 and the calibration plate 104 located below the reference calibration plate 101 can be symmetrically arranged about the optical axis of the fisheye camera, and the targets on the upper and lower sides of the reference calibration plate 101 are symmetrically arranged. The calibration error is further reduced.
[0089] In some embodiments, the key to the target assembly sliding along the track 106 is to allow the target to be photographed by the fisheye camera at several different distances, including far, medium and near. At least: when the target assembly slides to a sliding position at a first distance from the line laser module, the fisheye camera photographs 1 complete target; when the target assembly slides to a sliding position at a second distance from the line laser module, the fisheye camera photographs 1 to 3 complete targets; when the target assembly slides to a sliding position at a third distance from the line laser module, the fisheye camera photographs 5 complete targets; wherein, one target is set on a calibration plate, the first distance is smaller than the second distance, and the second distance is smaller than the third distance.
[0090] In the same frame of the image to be calibrated, if the number of corner points searched from one of the targets is within the preset number threshold range, the one of the targets is set as a complete target and assigned a number, wherein the preset number threshold range is preferably greater than or equal to 10 and less than or equal to 40. A complete target is a black and white square area, such as Figure 3 As shown, similar to a chessboard, the calibration plate and the target are both rectangular, the size of the calibration plate is related to the size of the target, the black and white square area is composed of black cells and white cells, which are regarded as black cells and white cells arranged at intervals, and the aforementioned corner points are the corner points of the black cells and / or the corner points of the white cells; preferably, the number of cells constituting the square area is linearly related to the number of corner points searched out; the targets set in each calibration plate are consistent, which can be understood as providing the same calibration features for the line laser and the fisheye camera from each angle direction at the same distance.
[0091] As an example, Figure 2As shown, the fixed support platform is also fixedly mounted with an infrared fill light 110. On the fixed support platform 112, the relative position between the infrared fill light 110 and the line laser module remains unchanged, that is, the sensor fixed position 111 and the infrared fill light 110 are both fixedly mounted in the fixed support platform 112, so that the emission direction of the line laser, the shooting angle of the fisheye camera, and the fill light direction of the infrared fill light 110 are all toward the target assembly (including the calibration plates set at various angles). Whenever the target assembly slides along the track to a sliding position relative to the line laser module, the relative position between the target assembly and the line laser module remains unchanged, and the line laser and the fisheye camera are both fixedly mounted, there are:
[0092] The parameter calibration unit controls the line laser to emit a line laser to the target assembly, and controls the fisheye camera to capture at least one complete target, to obtain a frame of laser stripe image corresponding to a sliding position, and a frame of laser stripe image includes a frame of image to be calibrated including light stripes. At this time, the target cannot be identified in the image to be calibrated, that is, the aforementioned black and white square area and the cells inside it, and only the central pixel coordinates of the light stripes in the calibration plate with the corresponding number can be extracted; the fisheye camera preferably adopts an infrared bandpass filter or an infrared high-pass filter to filter the reflected light of the line laser emitted by the line laser on the surface of the calibration plate, specifically to transmit the infrared band carried by the line laser, and absorb or reflect the ambient light of other bands, so as to separate the infrared light carried by the reflected light from other interfering ambient light.
[0093] The parameter calibration unit controls the line laser to stop emitting the line laser, turns on the infrared fill light to fill light for the target assembly, and controls the fisheye camera to capture the at least one complete target to obtain a frame of target image corresponding to the same sliding position; wherein a frame of target image is a frame of image to be calibrated that does not include light bars. The brightness of the target is increased by turning on the infrared fill light to increase the depth of field during shooting, because under the condition of low ambient light in the factory, the fisheye camera is not able to clearly capture the information of the target and its corner points through external light, so a fill light is needed to illuminate the target, wherein the relative position between the infrared fill light and the line laser module can remain unchanged or change, and there is no specific limitation here.
[0094] Specifically, the laser is turned off and the target is illuminated with an infrared fill light, and then the image to be calibrated is taken. The advantage of this is that it is easy to process images. Therefore, step 2 searches for corner points from the target image and expands the ROI area, and then frames the ROI area in the same pixel area of the laser stripe image. The position relationship of the same pixel area in the laser stripe image is the position relationship of the ROI area in the target image. Based on the pixel position information of the ROI area framed in the target image, the center pixel coordinates of the light stripe falling into the ROI area can be searched in the laser stripe image. On this basis, combined with the external parameters obtained by the fisheye camera calibration, the three-dimensional coordinates of the laser point on the intersection of the line laser and the target surface (in the camera coordinate system) can be converted, which corresponds to the spot position information of the light stripe searched in the area framed by the ROI area in the target, narrowing the search range of the light stripe and improving the calibration efficiency of the laser parameters.
[0095] In summary, this embodiment uses the calibration plates in the target assembly as a medium to calibrate the fisheye camera and the line laser. On the one hand, this embodiment can calibrate the camera and the single-line laser of the camera equipped with a fisheye lens (hereinafter referred to as the fisheye camera) without modifying the camera that needs to be calibrated. On the other hand, the calibration device used in this embodiment is low-cost, and the data acquisition is simple and direct. It integrates heterogeneous data through coordinate system transformation from the underlying hardware level, and finally performs point cloud data fitting at each angle at each sliding position at one time, which has good calibration efficiency and accuracy.
[0096] As an example, combining Figure 1 and Figure 2 It can be seen that the track 106 is arranged along the vertical direction, the fixed support platform 112 is placed on the horizontal plane and fixedly arranged at the lower end of the track 106, and the emission direction of the line laser and the shooting angle of the fisheye camera are both upward; the reference calibration plate 101 is arranged on the normal plane of the track 106 to be parallel to the horizontal plane, and the five calibration plates are fixed on the calibration drive assembly by the connecting bracket 109, and the calibration drive assembly drives the five calibration plates to slide up and down relative to the line laser module, which is actually sliding up and down relative to the sensor fixed position 111. Preferably, among the five calibration plates, if the top calibration plate is used as the reference calibration plate 101, the four calibration plates around it and the top reference calibration plate 101 form an obtuse angle of 120 degrees. Of course, the specific angle can be adjusted according to the actual situation without limitation. By adjusting the relative positions of the five calibration plates on the track 106, the field of view of the fisheye camera is covered by the target pattern (i.e., the image to be calibrated) as much as possible near the maximum ranging boundary of the track 106, and the targets on each calibration plate can be completely photographed, and the line laser can hit the measured surface of the calibration plate (towards the direction of the fisheye camera).
[0097] Specifically, at the mechanical hardware operation level, the target assembly can slide up and down linearly along the motion axis 107 on the track 106 through the connecting bracket 109, and the calibration drive assembly can be provided with a wheel-axle mechanism that matches the motion axis 107, so that the target assembly can move on the track 106 along the axis of the motion axis 107 under the traction of the calibration drive assembly; in some embodiments, the aforementioned wheel-axle mechanism is arranged in the groove between the motion axis 107 and the raised rails on both sides of the track 106, and slides up and down in the groove under the traction of the calibration drive assembly. Preferably, the normal vector of the reference calibration plate 101 is set to be parallel to the motion axis 107, and the axial direction of the motion axis 107 is parallel to the laying direction of the track 106.
[0098] The following is a detailed introduction to several coordinate systems and coordinate transformation principles involved in this application.
[0099] World coordinate system: A reference coordinate system is also selected in the environment to describe the position of the camera and objects. This coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by the rotation matrix R and the translation vector t.
[0100] Camera coordinate system: The origin of the camera coordinate system is the optical center of the camera. The x-axis and y-axis are parallel to the X and Y axes of the image. The z-axis is the optical axis of the camera, which is perpendicular to the image plane. The spatial rectangular coordinate system formed by this is called the camera coordinate system, also known as the camera coordinate system. The camera coordinate system is a three-dimensional coordinate system. The intersection of the optical axis and the image plane is the origin of the image coordinate system. The rectangular coordinate system formed by the X and Y axes of the image is the image coordinate system. The image coordinate system is a two-dimensional coordinate system.
[0101] Pixel coordinate system (U, V): It is a two-dimensional rectangular coordinate system that reflects the arrangement of pixels in the imaging plane of the camera. The origin O is located in the upper left corner of the image, and the U axis and V axis are parallel to the two sides of the image plane.
[0102] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable technicians familiar with the technical field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.
Claims
1. A parameter calibration method based on a line laser module, characterized in that: Parameter calibration methods include: Step 1, control the target assembly to slide a plurality of distances relative to the line laser module; control the line laser to emit a line laser to the target assembly, projecting a light strip on the surface of the target at a corresponding angle; and control the fisheye camera to sequentially photograph the target set at at least one angle in the target assembly at each sliding position or the light strip projected from its surface, to obtain an image to be calibrated; Step 2: Input pixel data of the images to be calibrated at different sliding positions into the camera calibration model to calibrate the fisheye camera, and output the parameters of the fisheye camera; Step 3, respectively extract the pixel coordinates of the center of the light strip from the images to be calibrated at different sliding positions, and then convert the pixel coordinates of the center of the light strip into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera; then, according to the projection direction of the light strip in the imaging plane, perform surface fitting on the converted three-dimensional coordinates of the laser point so that the number of fittings in the direction of the coordinate axis closest to the projection direction of the light strip in the imaging plane is maximized, and the laser surface function is obtained, and the calibration of the laser parameters of the line laser module is determined to be completed; Among them, the line laser module includes a line laser and a fisheye camera.
2. The parameter calibration method according to claim 1, characterized in that: The target assembly includes a plurality of calibration plates, any two of which are arranged at an angle; the normal vectors of each calibration plate are different; a target is arranged on the surface of a calibration plate, so that any two targets are also arranged at an angle; The closer the target assembly slides to the line laser module, the fewer targets are included in a frame of the image to be calibrated captured by the fisheye camera. The farther the target assembly slides from the line laser module, the more targets are included in a frame of the image to be calibrated captured by the fisheye camera.
3. The parameter calibration method according to claim 2, characterized in that: In step 2, before the fisheye camera calibration is performed, the following steps are also included: Whenever the target assembly slides to a sliding position, a corner point is searched in the complete target in a frame of the image to be calibrated taken in real time by the fisheye camera; In the same frame of the image to be calibrated, the corner points searched are grouped according to the complete target photographed, and a group of corner points corresponds to a complete target; Then, the complete targets where each group of corner points is located are numbered so that the targets set at different angles in the same frame of the image to be calibrated have different numbers; when the target assembly slides to different sliding positions, the numbers of the complete targets in different frames of the image to be calibrated taken by the fisheye camera are different.
4. The parameter calibration method according to claim 3, characterized in that: In the same frame of the image to be calibrated, if the number of corner points searched from one of the targets is within a preset number threshold range, then the one of the targets is set as a complete target and assigned a number; A complete target is a black and white square area; the black and white square area is composed of black cells and white cells, and the outside of the target is surrounded by the white area.
5. The parameter calibration method according to claim 4, characterized in that: In the same complete target, a polygonal area is formed by expanding outward from a corner point as the search starting point, so that the polygonal area covers part of the spot area or the entire spot area of the light strip, wherein the polygonal area includes all cells in the black and white square area and the white area outside the target; Then set the polygonal area as the ROI area.
6. The parameter calibration method according to claim 5, characterized in that: The camera calibration model is a full-view camera model calibration correction model or a fisheye lens distortion correction model, which is used to calculate the parameters of the fisheye camera, including the internal parameters of the fisheye camera and the external parameters of each target to the fisheye camera.
7. The parameter calibration method according to claim 6, characterized in that: In step 1, whenever the target assembly slides linearly relative to the line laser module to a sliding position, the relative position between the target assembly and the line laser module remains unchanged, and the line laser and the fisheye camera are fixedly assembled, there exists: First, the line laser is controlled to emit a line laser to the target assembly, and the fisheye camera is controlled to capture at least one complete target, so as to obtain a frame of laser stripe image corresponding to a sliding position. A frame of laser stripe image is a frame of image to be calibrated including light stripes; Then, the line laser is controlled to stop emitting the line laser, and the infrared fill light is turned on to fill light for the target assembly, and the fisheye camera is controlled to capture the at least one complete target, and a frame of target image corresponding to the same sliding position is obtained, so that in step 2, the corner point is searched from the target image and the ROI area is expanded, and then the ROI area is framed to the same pixel area of the laser stripe image, and the position relationship of the same pixel area in the laser stripe image is the position relationship of the ROI area in the target image; wherein, a frame of target image is a frame of image to be calibrated that does not include light stripes; The pixel coordinates of the center of the light stripe are the projection positions of the intersection of the line laser emitted by the line laser and the target in the pixel plane of the fisheye camera, and are located in the laser stripe image; The intersection line of the line laser emitted by the line laser and the target is the light strip.
8. The parameter calibration method according to claim 7, characterized in that: In step 2, the external parameters from the target assembly at each sliding position to the fisheye camera are the results obtained by inputting the pixel data of the image to be calibrated into the camera calibration model, and the external parameters from the target assembly at each sliding position to the fisheye camera include the external parameters of the cells in the target set at each angle at each sliding position; The pixel data of the image to be calibrated includes but is not limited to the pixel coordinates of the corner points of the numbered targets.
9. The parameter calibration method according to claim 8, characterized in that: In step 3, the method of extracting the center pixel coordinates of the light strip from the images to be calibrated at different sliding positions, and then converting the center pixel coordinates of the light strip into the three-dimensional coordinates of the laser point according to the parameters of the fisheye camera includes: The pixel coordinates of the center of the light strip are extracted from the ROI area of the image to be calibrated at different sliding positions, and the pixel coordinates of the center of the light strip are set as the two-dimensional pixel coordinates (u, v) in the pixel coordinate system of the fisheye camera; then, based on the coordinate system conversion function set in the preset mathematical analysis software, the parameters of the fisheye camera calibrated by the camera calibration model are used to map the two-dimensional pixel coordinates (u, v) to the light spot coordinates on the target surface with the corresponding number. The light spot coordinates It is located in the world coordinate system set on the corresponding calibration plate; Then, the light spot coordinates are converted into Convert the three-dimensional coordinates of the laser point into the camera coordinate system The corresponding conversion formula is The external parameters generated by the camera calibration model calibration include the rotation matrix R and the offset vector T required for converting the world coordinate system set on the target surface with the corresponding number into the camera coordinate system of the fisheye camera.
10. The parameter calibration method according to claim 1, characterized in that: In step 3, according to the projection direction of the light strip in the imaging plane, the method for performing surface fitting on the converted three-dimensional coordinates of the laser point includes: During the process of the target assembly sliding linearly for multiple distances relative to the line laser module, after the line laser emits a line laser to the target assembly, it is reflected into the imaging plane of the fisheye camera through the target at the corresponding angle, and projected to form a light stripe line, and the extension direction of the light stripe line is set as the projection direction of the light stripe in the imaging plane; In the abscissa and ordinate axes of the image coordinate system corresponding to the imaging plane of the fisheye camera, detecting the angles formed by the projection direction of the light strip in the imaging plane and the respective coordinate axes of the image coordinate system; When it is detected that the acute angle formed by the projection direction of the light strip in the imaging plane and the abscissa axis of the image coordinate system is smaller than the acute angle formed by the projection direction of the light strip in the imaging plane and the ordinate axis of the image coordinate system, a two-variable polynomial function whose highest degree of the first independent variable is greater than the highest degree of the second independent variable is fitted using the three-dimensional coordinates of the laser point converted at each angle at multiple sliding positions of the target assembly, and a laser surface function is determined to fit the laser surface and determine the posture relationship between the laser surface and the fisheye camera; When it is detected that the acute angle formed by the projection direction of the light strip in the imaging plane and the ordinate axis of the image coordinate system is smaller than the acute angle formed by the projection direction of the light strip in the imaging plane and the abscissa axis of the image coordinate system, a two-variable polynomial function whose highest degree of the second independent variable is greater than the highest degree of the first independent variable is fitted using the three-dimensional coordinates of the laser point converted at each angle at multiple sliding positions of the target assembly, and a laser surface function is determined to fit the laser surface and determine the posture relationship between the laser surface and the fisheye camera; The first independent variable is used to represent the horizontal coordinate of the image coordinate system, and the second independent variable is used to represent the vertical coordinate of the image coordinate system.
11. The parameter calibration method according to claim 10, characterized in that: In a bivariate polynomial function in which the highest degree of the first variable is greater than the highest degree of the second variable, the highest degree of the first independent variable is greater than or equal to 2, and the highest degree of the second independent variable is equal to 1, so that the fitting number of the bivariate polynomial function in the direction of the abscissa axis of the image coordinate system is greater than the fitting number of the bivariate polynomial function in the direction of the ordinate axis of the image coordinate system; In the bivariate polynomial function in which the highest degree of the second independent variable is greater than the highest degree of the first independent variable, the highest degree of the second independent variable is greater than or equal to 2, and the highest degree of the first independent variable is equal to 1, so that the fitting number of the bivariate polynomial function in the vertical axis direction of the image coordinate system is greater than the fitting number in the horizontal axis direction of the image coordinate system.
12. The parameter calibration method according to claim 11, characterized in that: The bivariate polynomial function is expressed as z=f(x,y), forming a functional relationship between the laser surface and the fisheye camera; The image coordinate system is the plane coordinate system XOY formed by the camera coordinate system in the imaging plane, the imaging plane is the normal plane of the Z axis of the camera coordinate system, the ordinate axis of the image coordinate system is the Y axis, the abscissa axis of the image coordinate system is the X axis, the first independent variable of the bivariate polynomial function is represented by x, the second independent variable of the bivariate polynomial function is represented by y, and the dependent variable of the bivariate polynomial function is represented by z; In the bivariate polynomial function, the polynomial excluding the term where the dependent variable z is located is f(x,y).
13. A calibration device, characterized in that: The calibration device includes a calibration drive component, a target component, a track, a fixed supporting platform, a line laser module, and a parameter calibration unit for executing the parameter calibration method described in any one of claims 1 to 12; wherein, the target component includes a plurality of targets arranged at an angle, the line laser module includes a line laser and a fisheye camera; the fixed supporting platform is fixedly assembled with the line laser and the fisheye camera; the fixed supporting platform is fixedly arranged at one end of the track, and the target assembly is slidably arranged at the other end of the track.
14. The calibration device according to claim 13, characterized in that: The target assembly includes a plurality of calibration plates, any two calibration plates are arranged at an angle, and the normal vectors of each calibration plate are different, and one target is arranged on one calibration plate; The closer the target assembly slides along the track to the line laser module, the fewer complete targets are included in a frame of the image to be calibrated captured by the fisheye camera. The farther the target assembly slides along the track from the line laser module, the more complete targets are included in a frame of the image to be calibrated captured by the fisheye camera, so that the fisheye camera can capture complete targets in all calibration plates at the ranging boundary of the track; The number of different positions through which the target assembly slides along the track is greater than or equal to 4.
15. The calibration device according to claim 14, characterized in that: There are five calibration plates in the target assembly; the calibration plate whose normal vector is parallel to the track is set as the reference calibration plate; among the remaining four calibration plates, there is one calibration plate located above the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is one calibration plate located below the reference calibration plate and the reference calibration plate is set at a predicted observation angle, there is one calibration plate located to the left of the reference calibration plate and the reference calibration plate is set at a predicted observation angle, and there is one calibration plate located to the right of the reference calibration plate and the reference calibration plate is set at a predicted observation angle, so that when the target assembly slides along the track, the fisheye camera can capture at least one complete target in the target assembly and a white area set around it.
16. The calibration device according to claim 14, characterized in that: The fixed bearing platform is also equipped with an infrared fill light; Whenever the target assembly slides along the track relative to the line laser module to a sliding position, the relative position between the target assembly and the line laser module remains unchanged, and the line laser and the fisheye camera are fixedly assembled, there are: The parameter calibration unit controls the line laser to emit the line laser to the target assembly, and controls the fisheye camera to capture at least one complete target, to obtain a frame of laser stripe image corresponding to a sliding position, wherein a frame of laser stripe image includes a frame of image to be calibrated including light stripes; The parameter calibration unit controls the line laser to stop emitting the line laser, turns on the infrared fill light to fill light for the target assembly, and controls the fisheye camera to capture the at least one complete target, to obtain a frame of target image corresponding to the same sliding position; wherein a frame of target image is a frame of image to be calibrated that does not include light bars; On the fixed supporting platform, the relative position between the infrared fill light and the line laser module remains unchanged.
17. The calibration device according to claim 16, characterized in that: The track is arranged along the vertical direction, the fixed bearing platform is placed on a horizontal plane and fixedly arranged at the lower end of the track, and the emission direction of the line laser and the shooting angle of the fisheye camera are both arranged upward relative to the fixed bearing platform; The reference calibration plate is arranged on the normal plane of the track to be parallel to the horizontal plane. The five calibration plates are fixed on the calibration drive assembly by a connecting bracket, and the calibration drive assembly drives the five calibration plates to slide up and down relative to the line laser module.
Citation Information
Cited By
Internal reference calibration and triangulation equipment
CN121048574A
An internal reference calibration and triangulation device
CN121048574B
