Camera calibration method

Abstract

The invention discloses a camera calibration method, which is used for a camera of a mobile main body, and comprises camera internal reference calibration and camera external reference calibration, the camera internal reference calibration is performed before the camera is installed, and the camera external reference calibration improves the calibration efficiency and calibration precision of the camera through data acquisition, coarse calibration and fine calibration. According to the calibration method, the relative distance between the objects in the camera is determined through the distance between the objects in the satellite map, so that the scale information is determined in the aerial view of the camera, the external parameters of the camera are optimized through the plane hypothesis and the scale information in the aerial view, and the calibration result can be obtained without field measurement.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera calibration, and in particular relates to a camera calibration method. Background Art

[0002] Usually, during the navigation process of a ship, the installation and use of a camera is involved. During the use, the accuracy of the camera determines the navigation route of the ship. Figure 2 As shown in the figure, due to the large size of the ship and the large visual range required by the camera, the existing calibration accuracy is difficult to meet the calibration requirements of shipborne cameras. Before each calibration, existing shipborne cameras usually need to place a large number of special calibration objects on the water surface around the ship, which is time-consuming and inconvenient. In addition, during the calibration process, it is difficult for the camera to predict the change in position or direction between the ship and the calibration objects on the water surface, which may affect its accuracy. At the same time, the placement of calibration objects is not conducive to saving manpower and material resources. Summary of the invention

[0003] In view of the above problems, the main purpose of the present invention is to design a camera calibration method to solve the problem of difficult calibration and low accuracy when the camera is installed on a large mobile subject.

[0004] In order to achieve the above purpose, the present invention adopts the following technical solutions: A camera calibration method is used for a camera of a mobile subject, wherein the mobile subject includes a ship, a vehicle, a drone, and an aircraft carrier, and the calibration method includes calibration of camera intrinsic parameters and calibration of camera extrinsic parameters; Wherein, the calibration of the camera internal parameters is performed before the camera is installed; The calibration of the camera extrinsic parameters is performed after the camera is installed and relies on satellite maps and pre-existing calibration objects, which includes the following steps: Step 1: Collect camera data; Step 2: Measure the position relationship of the initial camera and construct constraints as prior information; Step 3: Use the pre-existing calibration objects and constraints to roughly calibrate the angle and translation between the camera and the moving subject, and obtain an uncalibrated bird's-eye view. Step 4: By comparing the bird's-eye view with the satellite map, the uncalibrated scale information is determined, and the camera calibration height information is obtained. The bird's-eye view with the correct scale and the rough calibration result are output to realize the external parameter calibration of the camera.

[0005] As a further description of the present invention, the camera internal parameters include focal length, optical center, and distortion coefficient, which are calibrated using a checkerboard calibration plate.

[0006] As a further description of the present invention, in step 1 of the camera extrinsic calibration, collecting camera data includes at least two times, the first time is when one side of the moving subject approaches the reference object to collect camera data, and the second time is when the other side of the moving subject approaches the pre-reference object to collect camera data; The area where the camera data is collected can be the same area or a different area, and the satellite map includes the area.

[0007] As a further description of the present invention, in step 2 of the camera extrinsic calibration, the positional relationship of the initial camera is measured, including the precise size of the mobile subject confirmed by the design drawing, and the installation position of the camera on the mobile subject, and the precise size of the mobile subject and the installation position of the mobile subject camera are used as constraints between the cameras of the mobile subject.

[0008] As a further description of the present invention, in step 3 of the camera extrinsic calibration, the rough calibration of the angle and translation between the camera and the moving subject is determined by the target point pair on the camera image and in the world coordinate system, and the target point pair is the point of the target object in the camera image and the point of the target object in the world coordinate system; wherein the point of the target object in the world coordinate system is converted to the point on the camera image, and the expression is: , Where K is the camera intrinsic parameter, T is the camera extrinsic parameter, the camera extrinsic parameter T includes the rotation matrix R and the translation matrix t, (u, v) is the pixel position of the point in the camera image, (Xw, Yw, Zw) is the coordinate of the point in the world coordinate system, and Zc is the scale factor, which is a constant; By substituting the above expressions into the target point pairs under multiple cameras, the angle and translation between the camera and the moving subject are obtained, and the uncalibrated bird's-eye view is obtained.

[0009] As a further description of the present invention, in step 4 of the camera extrinsic calibration, the bird's-eye view is preset with scale information, and the points on the bird's-eye view are matched one-to-one with the points on the camera image through a transformation matrix between the bird's-eye view image coordinates and the world coordinates.

[0010] As a further description of the present invention, in step 4 of the camera extrinsic calibration, the scale of the points in the bird's-eye view is compared with the points in the satellite map, and the translation Z in the camera extrinsic is adjusted so that the scale information of the bird's-eye view is consistent with that of the satellite map. Figure 1 To determine the camera height.

[0011] As a further description of the present invention, the calibration method also includes step 5: measuring the pose information between any markers around the moving subject and the pre-existing calibration objects in the field, and accurately constraining the translation amount (x, y, z) between the moving subject and the camera.

[0012] As a further description of the present invention, in step 5, accurate results are obtained by using a distance measuring instrument to accurately constrain the translation amount between the moving subject and the camera.

[0013] Compared with the prior art, the technical effects of the present invention are: The present invention provides a camera calibration method, which is used on a camera of a mobile subject, and includes camera intrinsic parameter calibration and camera extrinsic parameter calibration. The camera intrinsic parameter calibration is performed before the camera is installed. The camera extrinsic parameter calibration improves the calibration efficiency and calibration accuracy of the camera by collecting data, coarse calibration and fine calibration. The calibration method confirms the relative distance between objects in the camera through the distance between objects in a satellite map, thereby confirming the scale information in the camera's bird's-eye view. The camera's extrinsic parameters are optimized through the plane assumption and scale information in the bird's-eye view, and the calibration result can be obtained without on-site measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the external parameter calibration process of the present invention; Figure 2 Schematic diagram of the visual range of different cameras in the prior art. DETAILED DESCRIPTION

[0015] The present invention is described in detail below in conjunction with the accompanying drawings: In one embodiment of the present invention, a camera calibration method is disclosed, referring to Figure 1 As shown, the calibration method is used on a camera of a mobile subject, and the mobile subject includes but is not limited to ships, vehicles, drones, and aircraft carriers. In this embodiment, the calibration method is described by taking a shipborne camera as an example; specifically, the calibration method includes camera intrinsic parameter calibration and camera extrinsic parameter calibration; the specific calibration process includes: (1) Utilization of calibration objects extracted from satellite map content: This involves the use of high-resolution satellite maps containing detailed landmarks. The landmarks are used as reference points for calibrating the ship’s cameras and outputting preliminary coarse calibration results; (2) Utilization of pre-existing calibration objects, such as bridge cranes or lane markings on shore, rather than having to place them near the vessel before each calibration process, which saves time and makes the calibration process more convenient.

[0016] The satellite maps and pre-existing calibration objects described above define a world coordinate system.

[0017] It should be noted that, in this embodiment, the satellite map can also be substituted for the use of radar information: if the ship is equipped with a radar and the radar has been accurately calibrated, the radar distance measurement can be used instead of the satellite map; Satellite maps can also be used as a substitute for aerial maps taken by drones: if there is no satellite map information, the high-precision camera in the drone can be used to take aerial photos, and the position information of the drone camera and the hull can be used as a substitute for satellite maps.

[0018] Specifically, this embodiment provides a detailed description of the above-mentioned camera intrinsic calibration and camera extrinsic calibration, as follows: 1. Camera Intrinsic Calibration The camera's internal parameters can be calibrated before the camera is shipped, including focal length, optical center, distortion coefficient, etc., and are usually calibrated using a checkerboard calibration plate. In this embodiment, it is assumed that the internal parameters have been calibrated and the error is within a reasonable range.

[0019] 2. Camera extrinsic calibration The camera's external parameters are calibrated after the camera is loaded on board. The camera's external parameters include: angle and translation (x, y, z). The external parameter calibration of a camera on a large ship is usually carried out twice.

[0020] Specifically, in this embodiment, the camera extrinsic calibration steps are: 1. Collect camera data: The overall calibration process requires at least two data acquisitions from the ship's onboard camera. The first time the ship docks on one side (the quay crane is the reference object) to collect camera data, and the second time the ship docks on the other side to collect camera data. The docking area requires sufficient lane lines or markers. The two docking quay cranes can be in the same area or different areas. Try to choose the same area, and include the area in the satellite map.

[0021] 2. Rough calibration: (1) Measuring the initial camera position relationship: The precise dimensions of the ship and the installation positions of the hull cameras are confirmed through the ship design drawings, which serve as a priori information on the mutual constraints between the cameras as a whole.

[0022] Constraints: 1. The precise size of the hull; 2. The installation position of the hull camera, which is the installation position of all cameras on the hull. The transformation matrix T from the camera to the hull is usually used, or the translation (x, y, z) and Euler angles (yaw, pitch, roll) are used to confirm the relative position of the camera's six degrees of freedom.

[0023] (2) Using the existing calibration object information on the shore and the above constraints, a rough calibration of the angle and translation between the camera and the hull is performed to obtain an uncalibrated bird's-eye view.

[0024] Specifically, in this embodiment, the angle between the camera and the moving subject and the rough calibration of the translation amount are determined by the target point pair on the camera image and in the world coordinate system, where the target point pair is the point of the target object in the camera image and the point of the target object in the world coordinate system; wherein the point of the target object in the world coordinate system is converted to the point on the camera image, and the expression is: , Where K is the camera intrinsic parameter, T is the camera extrinsic parameter, the camera extrinsic parameter T includes the rotation matrix R and the translation matrix t, (u, v) is the pixel position of the point in the camera image, (Xw, Yw, Zw) is the coordinate of the point in the world coordinate system, and Zc is the scale factor, which is a constant; By substituting the above expressions into the target point pairs under multiple cameras, the angle and translation between the camera and the moving subject are obtained, and the uncalibrated bird's-eye view is obtained.

[0025] (3) By comparing the bird's-eye view with the satellite map, the above-mentioned uncalibrated scale information is determined, thereby calibrating the camera height information and outputting the initial coarse calibration result. By outputting the initial coarse calibration result, more accurate angle information and more accurate translation information between cameras are obtained, that is, the bird's-eye view and coarse calibration result with correct scale.

[0026] In this embodiment, the scale information (i.e., the distance represented by each pixel in the image) in the bird's-eye view can be combined with the camera installation height, and the true value of a certain distance in the bird's-eye view can be determined by using a satellite map, thereby determining the scale information.

[0027] Specifically, in this embodiment, the bird's-eye view is first preset with scale information (), and through the transformation matrix between the bird's-eye view image coordinate system (the origin is located in the upper left corner) and the world coordinate system (usually the origin of the camera installation target, that is, the origin of the coordinate system of the object to be solved for the camera extrinsic parameters, in this embodiment, it refers to the origin of the hull), the points on the bird's-eye view can be matched one-to-one with the pixel points of the camera image. This correspondence is related to the camera extrinsics, that is, the transformation relationship is correlated with the camera installation height. By comparing the scales of the target points in the generated bird's-eye view with the target points in the satellite map, the translation amount Z in the camera extrinsics is adjusted, so that the preset scale information () of the bird's-eye view is consistent with the satellite map. Figure 1 The camera height information is determined by , where is the scale information of the bird's-eye view, and each pixel represents the distance in the real world.

[0028] 3. Fine calibration: By measuring the position information between the hull markers and the shore markers on site, the translation (x, y, z) between the hull and the camera is precisely constrained.

[0029] Specifically, accurate results can be obtained through instruments such as rangefinders to accurately constrain the translation amount between the hull and the camera.

[0030] This step calibrates the precise translation information between the hull and the camera based on the rough calibration, and further improves the camera angle information and the translation information between cameras in the preliminary rough calibration.

[0031] In the subsequent use of the camera based on the above calibration method of this embodiment, since the generated surround view 2D and 3D stitched images are generated by transforming the camera original images through calibration parameters, the quality of the camera original images affects the final stitching effect. If the wide-angle camera original image is generated by stitching multiple cameras, the resulting stitching problems will also be displayed in the final surround view.

[0032] Through the above embodiments, the camera calibration method of the present invention is disclosed. Compared with the prior art, the present invention has the following advantages: 1. The camera calibration method of the present invention does not need to place multiple markers relatively stationary around the moving subject, thereby improving the calibration efficiency and reducing the demand for manpower and material resources; 2. The camera calibration method of the present invention does not require accurate measurement of the position information of several markers relative to the camera, and the calibration reference objects are flexible and changeable, thereby improving the calibration accuracy; 3. The camera calibration method of the present invention can select a longer distance that is currently difficult to measure on site through satellite maps, thereby improving the distance measurement accuracy.

[0033] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A camera calibration method for a camera of a moving subject, characterized in that: The mobile subject includes a ship, a vehicle, a drone, and an aircraft carrier, and the calibration method includes calibration of camera intrinsic parameters and calibration of camera extrinsic parameters; Wherein, the calibration of the camera internal parameters is performed before the camera is installed; The calibration of the camera extrinsic parameters is performed after the camera is installed and relies on satellite maps and pre-existing calibration objects, which includes the following steps: Step 1: Collect camera data; Step 2: Measure the position relationship of the initial camera and construct constraints as prior information; Step 3: Use the pre-existing calibration objects and constraints to roughly calibrate the angle and translation between the camera and the moving subject, and obtain an uncalibrated bird's-eye view. Step 4: By comparing the bird's-eye view with the satellite map, the uncalibrated scale information is determined, and the camera calibration height information is obtained. The bird's-eye view with the correct scale and the rough calibration result are output to realize the external parameter calibration of the camera.

2. A camera calibration method according to claim 1, characterized in that: The camera internal parameters include focal length, optical center, and distortion coefficient, which are calibrated using a checkerboard calibration plate.

3. A camera calibration method according to claim 1, characterized in that: In step 1 of the camera extrinsic calibration, collecting camera data includes at least two times, the first time is when one side of the moving subject approaches the reference object to collect camera data, and the second time is when the other side of the moving subject approaches the pre-reference object to collect camera data; The area where the camera data is collected can be the same area or a different area, and the satellite map includes the area.

4. A camera calibration method according to claim 1, characterized in that: In step 2 of the camera extrinsic calibration, the positional relationship of the initial camera is measured, including the precise size of the mobile body confirmed by the design drawing and the installation position of the camera on the mobile body, and the precise size of the mobile body and the installation position of the mobile body camera are used as constraints between the cameras of the mobile body.

5. A camera calibration method according to claim 1, characterized in that: In step 3 of the camera extrinsic calibration, the angle between the camera and the moving subject and the rough calibration of the translation amount are determined by the target point pair on the camera image and in the world coordinate system. The target point pair is the point of the target object in the camera image and the point of the target object in the world coordinate system; wherein the point of the target object in the world coordinate system is converted to the point on the camera image, and the expression is: ， Where K is the camera intrinsic parameter, T is the camera extrinsic parameter, the camera extrinsic parameter T includes the rotation matrix R and the translation matrix t, (u, v) is the pixel position of the point in the camera image, (Xw, Yw, Zw) is the coordinate of the point in the world coordinate system, and Zc is the scale factor, which is a constant; By substituting the above expressions into the target point pairs under multiple cameras, the angle and translation between the camera and the moving subject are obtained, and the uncalibrated bird's-eye view is obtained.

6. A camera calibration method according to claim 1, characterized in that: In step 4 of the camera extrinsic calibration, the bird's-eye view image is preset with scale information, and the points on the bird's-eye view image are matched one-to-one with the points on the camera image through a transformation matrix between the bird's-eye view image coordinates and the world coordinates.

7. A camera calibration method according to claim 6, characterized in that: In step 4 of the camera extrinsic calibration, the scale of the points in the bird's-eye view is compared with the points in the satellite map, and the Z of the translation amount in the camera extrinsic is adjusted so that the scale information of the bird's-eye view is consistent with the satellite map, thereby realizing the determination of the camera height.

8. A camera calibration method according to claim 1, characterized in that: The calibration method further comprises step 5: measuring the pose information between any markers around the moving subject and the pre-existing calibration objects in situ, and performing precise constraints on the translation amount (x, y, z) between the moving subject and the camera.

9. A camera calibration method according to claim 8, characterized in that: In step 5, accurate results are obtained by using a distance measuring instrument to accurately constrain the translation amount between the moving subject and the camera.

Images