A system and method for jointly measuring ship altitude using laser imaging radar and camera

Abstract

This invention discloses a system and method for jointly measuring ship altitude using laser imaging radar and a camera. Data collected by a camera sensor, a lidar sensor, and a tilt sensor are input into an industrial computer. Camera calibration, image processing, and point cloud processing are performed using a camera calibration module, an image processing module, and a point cloud data processing module, respectively. Then, a joint calibration module performs joint calibration on the image-processed data and the point cloud-processed data. Next, a coordinate transformation module performs coordinate transformation. The jointly calibrated and coordinate-transformed data are then input into a data fusion module for data fusion. An angle compensation module performs tilt compensation on the data detected by the tilt sensor. Finally, a height calculation module performs high-speed calculations based on the fused data and the tilt-compensated data to obtain the ship's altitude. This invention, through the joint measurement of a camera and lidar, enables rapid and accurate measurement of ship altitude.

Description

Technical Field

[0001] This invention relates to the field of ship altitude measurement technology, specifically to a system and method for jointly measuring ship altitude using laser imaging radar and a camera. Background Technology

[0002] With the continuous increase in inland waterway shipping volume, the phenomenon of inland waterway vessels exceeding height limits and violating regulations is also increasing, seriously affecting the navigation safety, efficiency, and economic benefits of inland waterways, and hindering the healthy development of the shipping economy.

[0003] For measuring the height of vessels in inland waterways, the current technology mainly involves setting up multiple sensors at different heights or using a single sensor at different heights. This includes methods such as lidar altimetry and binocular camera parallax measurement.

[0004] However, lidar altimetry can only obtain height information from the ship's surface, and the point cloud of lidar is relatively sparse in the vertical direction, making it impossible to guarantee that the highest point of the ship can be detected by lidar, thus compromising accuracy. While binocular camera parallax methods can obtain more accurate height information, they cannot accurately detect the boundaries of objects in areas where color or reflectivity changes are not significant, and they also suffer from blind spots and accuracy issues affected by lighting conditions.

[0005] Therefore, there is currently no effective technical means for rapid measurement of the height of ships during unloading operations, and a new and efficient method for measuring ship height is urgently needed to achieve rapid and accurate measurement of ships. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a system and method for jointly measuring ship height using laser imaging radar and a camera. By combining the measurement of the camera and laser radar, the height of a ship can be measured quickly and accurately. Compared with traditional ship height measurement methods, this method can effectively improve the accuracy of ship height measurement and has broad application prospects.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] This invention provides a system for jointly measuring the altitude of a ship using a laser imaging radar and a camera, the system comprising:

[0009] Camera sensor for acquiring ship image data;

[0010] A lidar sensor that uses laser imaging to obtain point cloud data of ships;

[0011] An inclination sensor that collects the ship's tilt angle;

[0012] A camera calibration module that connects to the camera sensor to calibrate the camera and obtain its intrinsic parameter data;

[0013] An image processing module connected to the camera calibration module, used to process ship image data calibrated by the camera calibration module;

[0014] A point cloud data processing module that connects to a lidar sensor and processes the point cloud data of the ship collected by the lidar sensor.

[0015] Connected to the point cloud data processing module and the image processing module, it is used to perform joint calibration on the processed point cloud data and processed image data to obtain the joint calibration module of the external parameter data of the camera and the lidar;

[0016] A coordinate transformation module connected to the joint calibration module, used to perform coordinate transformation on the jointly calibrated data;

[0017] A data fusion module connected to the joint calibration module and the coordinate transformation module, used to perform data fusion on the jointly calibrated data and the coordinate transformed data;

[0018] An angle compensation module, connected to the tilt sensor, is used to compensate for the ship's tilt angle acquired by the tilt sensor.

[0019] A height calculation module that connects to the data fusion module and the angle compensation module to calculate the ship's height based on the data after data fusion and the data after angle compensation.

[0020] Furthermore, the lidar and camera are horizontally mounted on the same bracket, with the lidar installed directly above the camera. A two-dimensional tilt sensor is installed on one side of the lidar to keep the camera and lidar horizontal.

[0021] Furthermore, the height calculation module obtains the ship's height relative to the water surface, i.e., the ship's clearance height, based on the distance and point cloud coordinates obtained by the data fusion module, as well as the installation height of the lidar and water level information.

[0022] Furthermore, the height calculation module, based on the highest point P0 of the ship found through image processing and the point cloud data projected onto the image pixels, searches downwards from the highest point of the ship for the nearest point projected onto the point cloud. When the first point projected onto the point cloud, P1, is found, the distance between the two is calculated, i.e., the number of pixels N between them. The coordinates of the point corresponding to P1 in the camera coordinate system are (ac, bc, cc), and the coordinates in the lidar coordinate system are (al, bl, cl). (al, bl, cl) are directly obtained from the lidar point cloud data, and (ac, bc, cc) are obtained from the lidar point cloud data through external parameter matrix transformation. Both are high-precision known data. In actual measurement, the coordinates of points between P0 and P1 on the Z-axis of the lidar coordinate system are equal to or approximately equal to cl, i.e., the distances from points between P0 and P1 to the planes where the camera and radar are located are the same. The Y-axis coordinates of the camera coordinate system and the Z-axis coordinates of the radar coordinate system represent the ship's height.

[0023] Furthermore, the data fusion module utilizes the obtained intrinsic parameter data of the camera and the extrinsic parameter data of the LiDAR and the camera to jointly process the point cloud data collected by the LiDAR and the image data of the camera. It extracts the highest point of the ship using image processing technology, and uses the intrinsic parameters of the camera and the extrinsic parameters calibrated by the LiDAR and the camera to project the point cloud collected by the LiDAR onto the image collected by the camera through projection algorithms and coordinate system transformation. Then, it detects the distance between the highest point of the ship and the nearest point projected by the point cloud below it, and uses relevant algorithms to calculate the true distance from the point to the point cloud projection point in the world coordinate system based on this distance.

[0024] Furthermore, the camera calibration module adopts the Zhang Zhengyou calibration method, which involves taking pictures of the checkerboard pattern at different locations and using a calibration algorithm to solve for the camera parameters, thereby obtaining the camera's intrinsic parameter matrix and distortion coefficients.

[0025] Furthermore, the image processing module performs background modeling on the two-dimensional image captured by the camera, and uses a background modeling algorithm to extract the foreground image, that is, to extract the moving ship in the image. Then, the image is binarized. The gray value of the moving ship target is 1, and the image is displayed as white. The gray value of the stationary background area is 0, and the image is displayed as black. The binarized image is traversed from top to bottom for each pixel. The first pixel obtained by traversal has a value of 1, and there is a pixel with a value of 1 in its 8 neighboring pixels. This is regarded as the pixel value of the highest point of the ship image. At this time, the coordinates of the corresponding highest point of the ship in the image are found.

[0026] Furthermore, the point cloud data processing module filters out noise and removes outliers and ground points from the point cloud data.

[0027] Furthermore, the joint calibration module uses a checkerboard calibration method to allow the lidar and camera to collect data synchronously. By finding the corner data of the checkerboard, the transformation relationship between the lidar coordinate system and the camera coordinate system is solved.

[0028] The present invention also provides a method for jointly measuring ship altitude using laser imaging radar and camera. Based on the above-mentioned system for jointly measuring ship altitude using laser imaging radar and camera, the method includes:

[0029] The camera's internal parameters are calibrated, and the camera's internal parameter data are calculated separately.

[0030] The camera and lidar are jointly calibrated to obtain the lidar's extrinsic parameters relative to the camera.

[0031] Simultaneous acquisition of image data and point cloud data of the ship was achieved using cameras and lidar.

[0032] The point cloud data is processed to remove outliers and ground points, and the remaining points are projected into a new two-dimensional matrix based on the camera's intrinsic parameter matrix and the camera and LiDAR extrinsic parameter matrix, while retaining the one-to-one correspondence between the points in the two-dimensional matrix and the three-dimensional point cloud data points before projection.

[0033] The projected two-dimensional matrix is ​​based on the size of the image captured by the camera. It projects the points of the two-dimensional matrix contained within the image area onto the image, with each point corresponding to a pixel.

[0034] Background modeling is performed on the 2D images captured by the camera to extract the ship targets in the images and obtain the 2D coordinates of the ship's highest point in the pixel coordinate system;

[0035] After finding the two-dimensional coordinates of the highest point on the image, find the nearest point projected by the point cloud in the positive direction of the v-axis of the pixel coordinate system, and record the distance between the highest point of the ship and the projected point. Based on this distance, calculate the distance between the highest point of the ship and the projected point in the world coordinate system, i.e., the true distance.

[0036] Based on the one-to-one correspondence between the points in the two-dimensional matrix after point cloud projection and the points in the three-dimensional point cloud before projection, the three-dimensional coordinates of the projected point are found. Combined with the angle compensation information of the tilt sensor, the true height value of the point relative to the installation height of the lidar and camera is obtained.

[0037] The height of the ship relative to the water surface is obtained based on the installation height of the camera and lidar, the water surface height, the height of the lidar projection point, and the height difference between the highest point of the ship and the lidar projection point.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention employs a combined camera and lidar system to measure ship altitude, integrating the information and advantages of both sensors. Based on image data from the camera and 3D point cloud data from the lidar, and combining the measurement data from both sensors, the invention acquires point cloud data of the ship's surface. Using algorithms such as highest point extraction, distance calculation, and coordinate transformation, it obtains the ship's altitude information. This invention fully leverages the advantages of lidar's high accuracy, stability, and insensitivity to lighting and color variations in 3D point cloud data, as well as the denser and more textured data acquired by the camera, facilitating target detection, thus improving measurement accuracy and reliability. This invention offers advantages such as real-time performance, high precision, and ease of operation, and can be effectively applied in the field of ship altitude measurement.

[0040] This invention preserves the one-to-one correspondence between the points in the two-dimensional matrix after point cloud projection and the points in the three-dimensional point cloud before projection, avoiding the back projection of the image (converting from pixel coordinate system to radar coordinate system) when calculating height, thus avoiding the errors caused by this and improving efficiency.

[0041] This invention uses a two-dimensional tilt sensor to compensate for the tilt angle of the camera and lidar, thereby improving measurement accuracy. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a system block diagram of a laser imaging radar and camera jointly measuring the altitude of a ship according to an embodiment of the present invention;

[0044] Figure 2 This is an overall flowchart of a method for jointly measuring ship altitude using a laser imaging radar and a camera, as described in an embodiment of the present invention.

[0045] Figure 3 This is an installation framework diagram in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of the joint calibration and transformation of the three-dimensional coordinates of the two sensors in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of the pixels projected from a point cloud onto an image in an embodiment of the present invention;

[0048] In the diagram, 1. Camera sensor; 2. LiDAR sensor; 3. Tilt sensor; 4. Point cloud data processing module; 5. Image processing module; 6. Angle compensation module; 7. Camera calibration module; 8. Joint calibration module; 9. Coordinate transformation module; 10. Data fusion module; 11. Height calculation module. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0050] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0051] like Figure 1 As shown, this embodiment of the invention provides a system for jointly measuring the altitude of a ship using a laser imaging radar and a camera. The system generally includes: a camera sensor 1, a laser radar sensor 2, a tilt sensor 3, a point cloud data processing module 4, an image processing module 5, an angle compensation module 6, a camera calibration module 7, a joint calibration module 8, a coordinate transformation module 9, a data fusion module 10, and an altitude calculation module 11; wherein:

[0052] Camera sensor 1 is used to acquire image data of the ship;

[0053] LiDAR sensor 2 is used for laser imaging to obtain point cloud data of the ship;

[0054] Tilt sensor 3 is used to collect the ship's tilt angle;

[0055] In specific implementation, such as Figure 3As shown, the lidar and camera are horizontally mounted on the same bracket, with the lidar installed directly above the camera. A two-dimensional tilt sensor 3 is installed on one side of the lidar to keep the camera and lidar horizontal.

[0056] The camera calibration module 7 is connected to the camera sensor 1 and is used to calibrate the camera to obtain information such as the camera's intrinsic parameter matrix and distortion coefficients.

[0057] Image processing module 5 is connected to camera calibration module 7 and is used to process the ship image data after the camera calibration module 7 has calibrated.

[0058] The point cloud data processing module 4 is connected to the lidar sensor 2 and is used to process the ship point cloud data collected by the lidar sensor 2.

[0059] The joint calibration module 8 is connected to the point cloud data processing module 4 and the image processing module 5. It is used to perform joint calibration on the processed point cloud data and the processed image data to obtain the extrinsic parameter data of the camera and the lidar, namely the rotation matrix and the translation matrix.

[0060] The coordinate transformation module 9 is connected to the joint calibration module 8 and is used to perform coordinate transformation on the jointly calibrated data.

[0061] The data fusion module 10 is connected to the joint calibration module 8 and the coordinate transformation module 9. It is used to fuse the data after joint calibration and the data after coordinate transformation. Specifically, it includes: using the obtained intrinsic parameter data of the camera and the extrinsic parameter data of the LiDAR and the camera, jointly processing the point cloud data collected by the LiDAR and the image data of the camera, extracting the highest point of the ship through image processing technology, and using the intrinsic parameters of the camera and the extrinsic parameters of the LiDAR and the camera calibration, projecting the point cloud collected by the LiDAR onto the image collected by the camera through projection algorithm and coordinate system transformation; then detecting the distance (the difference in pixels) between the highest point of the ship and the nearest point projected by the point cloud below it (in the positive direction of the v-axis of the pixel coordinate system), and using relevant algorithms to calculate the true distance from the point to the point cloud projection point in the world coordinate system based on this distance.

[0062] The angle compensation module 6 is connected to the tilt sensor 3 and is used to compensate for the tilt angle of the ship collected by the tilt sensor 3.

[0063] The height calculation module 11 is connected to the data fusion module 10 and the angle compensation module 6. It calculates the ship's height based on the data after data fusion and the data after angle compensation. Specifically, it calculates the ship's height relative to the water surface, i.e., the ship's clearance height, based on the distance and point cloud coordinates obtained by the data fusion module 10, as well as the installation height of the lidar and the water level.

[0064] The working process of the above system is as follows: The data collected by camera sensor 1, lidar sensor 2 and tilt sensor 3 are transmitted to the industrial computer. Camera calibration module 7, image processing module 5 and point cloud data processing module 4 are used to perform camera calibration, image processing and point cloud processing respectively. Then, the joint calibration module 8 is used to perform joint calibration on the image-processed data and the point cloud-processed data. Then, the coordinate transformation module 9 performs coordinate transformation. After that, the data after joint calibration and coordinate transformation is input into the data fusion module 10 for data fusion. The tilt compensation module 6 is used to perform tilt compensation on the data detected by tilt sensor 3. Finally, the height calculation module 11 performs high-speed calculation based on the data after data fusion and the tilt compensation data to obtain the ship's height.

[0065] like Figure 2 As shown, based on the above system, a method for jointly measuring ship altitude using a laser imaging radar and a camera in an embodiment of the present invention includes the following steps:

[0066] S1. Calibrate the camera's internal parameters and calculate the camera's internal parameter data respectively;

[0067] Specifically, camera calibration involves solving for the camera's intrinsic parameter matrix and distortion coefficients. Zhang Zhengyou's calibration method can be used, which involves photographing checkerboard patterns at different locations and using a calibration algorithm to solve for the camera parameters, thereby obtaining the camera's intrinsic parameter matrix and distortion coefficients.

[0068] S2. Perform joint calibration of the camera and lidar to obtain the external parameter data of the lidar relative to the camera;

[0069] Specifically, the lidar is positioned directly above the camera, ensuring both are horizontal. A checkerboard calibration method is used to synchronize data acquisition between the lidar and camera. By identifying the corner points of the checkerboard, the transformation relationship between the lidar coordinate system and the camera coordinate system is determined.

[0070] In order to project the points measured in the lidar coordinate system onto the camera coordinate system, an additional transformation needs to be added to the mapping operation, which consists of two parts: translation and rotation.

[0071] Translation describes a linear movement from point P to P′ in three-dimensional coordinates, which can be achieved by adding a translation vector t to P:

[0072]

[0073] The matrix representation is as follows:

[0074]

[0075] In a homogeneous coordinate system, this can be represented by concatenating an M-dimensional identity matrix I, where M is the number of elements in P and t, and T is the translation matrix.

[0076] In the rotation operation, the relationship after rotating about the x-axis is:

[0077] Y′ L =Y L cosα-Z L sinα (3)

[0078] Z′ L =Y L cosα+Z L sinα (4)

[0079]

[0080] Similarly, the relationship around the y-axis and around the z-axis is as follows:

[0081]

[0082]

[0083] Multiplying the rotation components in the three directions yields the rotation matrix between the two three-dimensional coordinates:

[0084]

[0085] From the formula above, the coordinate transformation required to completely map the LiDAR 3D point cloud onto the camera coordinate system is shown below, along with the joint calibration formula for the camera and LiDAR:

[0086]

[0087] Among them, P c (x c y z , z c ) and P l (x l y l , z l R and T are the point cloud data coordinates in the camera coordinate system and the lidar coordinate system, respectively, and the rotation and translation matrices from the lidar coordinate system to the camera coordinate system, respectively.

[0088] Equation (9) performs coordinate system transformation and correction on the two sensors based on the camera's intrinsic parameter matrix and the camera and lidar's extrinsic parameter matrices. Based on rigid body transformation, the transformation matrix is ​​used. The point cloud data acquired by the LiDAR is transformed into the camera coordinate system. Then, based on the camera's intrinsic parameter matrix and distortion coefficients, the camera coordinate system is transformed into the image coordinate system, and then into the pixel coordinate system. Finally, image fusion is performed with the image acquired by the camera to project the LiDAR point cloud data onto the image.

[0089] S3. Simultaneously collect image data and point cloud data of the ship using cameras and lidar;

[0090] S4. Process the point cloud data, remove outliers and ground points, and project the remaining points into a new two-dimensional matrix based on the camera's intrinsic parameter matrix and the camera and LiDAR extrinsic parameter matrix, while retaining the one-to-one correspondence between the points in the two-dimensional matrix and the three-dimensional point cloud data points before projection.

[0091] Point cloud data processing includes: filtering out noise, removing outliers and ground points from the point cloud data, as these points can affect the accuracy of subsequent calculations and therefore need to be removed.

[0092] S5. The projected two-dimensional matrix is ​​based on the size of the image captured by the camera. The points of the two-dimensional matrix contained within the image range are projected onto the image, with each point corresponding to a pixel.

[0093] To achieve data fusion, which involves projecting 3D point cloud data onto an image, the 3D point cloud is first transformed from the LiDAR coordinate system to the camera coordinate system using the calibrated R and T values ​​in formula (9). Then, based on the camera's intrinsic parameters, the camera coordinate system is converted to the pixel coordinate system. The point cloud data is then projected onto the pixels of the camera's image, preserving the one-to-one correspondence between the projected point cloud data and the original 3D point cloud in the LiDAR coordinate system. The projected image is shown below. Figure 5 As shown.

[0094] Specifically, the point cloud data P acquired by the lidar L =(X L Y L Z L Projecting this onto the camera coordinate system yields the corresponding coordinates P. C =(X C Y C Z C ):

[0095]

[0096] Projecting points from the camera coordinate system onto the pixel coordinate system, the projection formula is as follows:

[0097] u=(f x *X C / Z C )+c x(11)

[0098] v = (f y *Y C / Z C )+c y (12)

[0099] Where (u, v) are the point coordinates in the pixel coordinate system, (c x c y f is the coordinate offset of the camera's optical center in the pixel coordinate system. x and f y These are the camera's focal lengths, representing the horizontal and vertical focal lengths respectively. This formula allows the point cloud data from the LiDAR to be projected onto the pixels of the image.

[0100] Filter out points that are outside the image area based on the image size, and project the remaining points onto the image.

[0101] When projecting point cloud data onto the pixels of the camera image, it is necessary to preserve the one-to-one correspondence between the point cloud data projected onto the pixel coordinate system and the 3D point cloud in the lidar coordinate system before projection.

[0102] S6. Perform background modeling on the two-dimensional image captured by the camera, extract the ship target in the image, and obtain the two-dimensional coordinates of the ship's highest point in the pixel coordinate system.

[0103] Image processing includes: performing background modeling on the two-dimensional image captured by the camera, extracting the foreground image using a background modeling algorithm, that is, extracting the moving ship from the image, and then performing binarization on the image. The grayscale value of the moving ship target is 1, and the image is displayed as white. The grayscale value of the stationary background area is 0, and the image is displayed as black. The binarized image is traversed sequentially from top to bottom (positive direction of the v-axis of the pixel coordinate system) for each pixel point in the image. The first pixel point with a value of 1 and an 8-neighboring pixel point with a value of 1 is regarded as the pixel point of the highest point of the ship image. At this time, the coordinates of the corresponding highest point of the ship in the image are found.

[0104] S7. After finding the two-dimensional coordinates of the highest point on the image, find the nearest point projected by the point cloud in the positive direction of the v-axis of the pixel coordinate system, and record the distance between the highest point of the ship and the projected point. Calculate the distance between the highest point of the ship and the projected point in the world coordinate system based on this distance, which is the true distance.

[0105] S8. Based on the one-to-one correspondence between the points in the two-dimensional matrix after point cloud projection and the points in the three-dimensional point cloud before projection, find the three-dimensional coordinates of the projected point. Combined with the angle compensation information of the tilt sensor, obtain the true height value of the point relative to the installation height of the lidar and camera.

[0106] S9. Based on the installation height of the camera and lidar, the water surface height, the height of the lidar projection point, and the height difference between the highest point of the ship and the lidar projection point, the height of the ship relative to the water surface is obtained.

[0107] The altitude calculation specifically involves using the highest point P0 of the ship identified through the aforementioned image processing, along with the point cloud data projected onto the image pixels, such as... Figure 5 As shown, starting from the highest point of the ship, the system searches downwards (towards the positive v-axis of the pixel coordinate system) for the nearest point projected from the point cloud. Upon finding the first point P1 projected from the point cloud, the distance between the two points is calculated, which is the number of pixels N between them. The coordinates of point P1 in the camera coordinate system are (ac, bc, cc), and the coordinates in the lidar coordinate system are (al, bl, cl). (al, bl, cl) can be directly obtained from the lidar point cloud data, while (ac, bc, cc) can be obtained from the lidar point cloud data through an extrinsic parameter matrix transformation; both are high-precision known data. In actual measurements, the coordinates of points between P0 and P1 on the Z-axis of the lidar coordinate system are equal to or approximately equal to cl, meaning that the distances from points between P0 and P1 to the planes where the camera and lidar are located are the same. The orientation relationship between the coordinate axes of the camera and lidar coordinate systems is as follows: Figure 4 As shown, the Y-axis coordinate of the camera coordinate system and the Z-axis coordinate of the radar coordinate system can represent the ship's height.

[0108] Assuming the camera's field of view in the vertical direction is α, and the photograph taken by the camera has a total of n pixels in the vertical direction, then the actual distance d caused by moving each pixel at this distance is:

[0109] d=(b l *tan(α / n)) / cos(α / 2) (13)

[0110] b l The y-axis coordinate of the point in the lidar coordinate system corresponding to the point projected onto the image from the point cloud represents the distance from that point to the plane where the lidar and camera are located.

[0111] Therefore, the actual distance D between the highest point of the ship and the point cloud projection point is:

[0112] D=N*d (14)

[0113] At this point, the distance H1 from the highest point of the ship to the installation height H1 of the lidar and camera is:

[0114] H1 = D + c l (15)

[0115] c l The z-axis coordinate of the point in the lidar coordinate system corresponding to the point projected onto the image from the point cloud is given.

[0116] If the compensation angle transmitted by the tilt sensor at this time is θ (the lidar coordinate system rotates counterclockwise around the x-axis by θ degrees), then the installation height H2 of the highest point of the ship from the lidar and camera after compensation is:

[0117] H2=H*cosθ+b l *sinθ (16)

[0118] Therefore, the ship's clearance height H at this time is:

[0119] H = H2 + H0 + S (17)

[0120] Where H0 is the height of the camera mount and S is the distance from the bottom of the camera mount to the surface of the river.

[0121] In the above embodiments, the combined measurement of camera and lidar can quickly and accurately measure the height of a ship. Compared with traditional ship height measurement methods, this method can effectively improve the accuracy of ship height measurement and has broad application prospects.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for jointly measuring ship altitude using laser imaging radar and a camera, characterized in that, A system for measuring ship altitude based on a combination of laser imaging radar and camera, the system comprising: Camera sensor for acquiring ship image data (1); Laser imaging, obtaining point cloud data of ships by lidar sensor (2); An inclination sensor for collecting ship inclination angle (3); A camera calibration module (7) is connected to the camera sensor (1) and is used to calibrate the camera and obtain the camera's intrinsic parameter data. An image processing module (5) connected to the camera calibration module (7) is used to process the ship image data after the camera calibration module (7) has been calibrated. A point cloud data processing module (4) is connected to the lidar sensor (2) and is used to process the ship point cloud data collected by the lidar sensor (2). Connected to the point cloud data processing module (4) and the image processing module (5), it is used to perform joint calibration of the processed point cloud data and the processed image data to obtain the joint calibration module (8) of the external parameter data of the camera and the lidar. A coordinate transformation module (9) connected to the joint calibration module (8) is used to perform coordinate transformation on the data after joint calibration. A data fusion module (10) is connected to the joint calibration module (8) and the coordinate transformation module (9) to perform data fusion on the jointly calibrated data and the coordinate transformed data. An angle compensation module (6) connected to the tilt sensor (3) for angle compensation of the ship tilt angle collected by the tilt sensor (3), and A height calculation module (11) is connected to the data fusion module (10) and the angle compensation module (6) to calculate the ship's height based on the data after data fusion and the data after angle compensation. The method includes: The camera's internal parameters are calibrated, and the camera's internal parameter data are calculated separately. The camera and lidar are jointly calibrated to obtain the lidar's extrinsic parameters relative to the camera. Simultaneous acquisition of image data and point cloud data of the ship was achieved using cameras and lidar. The point cloud data is processed to remove outliers and ground points, and the remaining points are projected into a new two-dimensional matrix based on the camera's intrinsic parameter matrix and the camera and LiDAR extrinsic parameter matrix, while retaining the one-to-one correspondence between the points in the two-dimensional matrix and the three-dimensional point cloud data points before projection. The projected two-dimensional matrix is ​​based on the size of the image captured by the camera. It projects the points of the two-dimensional matrix contained within the image area onto the image, with each point corresponding to a pixel. Background modeling is performed on the 2D images captured by the camera to extract the ship targets in the images and obtain the 2D coordinates of the highest point of the ship in the pixel coordinate system; After finding the two-dimensional coordinates of the highest point on the image, the nearest point projected by the point cloud is found along the positive v-axis of the pixel coordinate system, and the distance between the highest point of the ship and the projected point is recorded. Based on this distance, the distance between the highest point of the ship and the projected point in the world coordinate system, i.e., the true distance, is calculated; the positive v-axis of the pixel coordinate system is downward from the highest point of the ship. Based on the one-to-one correspondence between the points in the two-dimensional matrix after point cloud projection and the points in the three-dimensional point cloud before projection, the three-dimensional coordinates of the projected point are found. Combined with the angle compensation information of the tilt sensor, the true height value of the point relative to the installation height of the lidar and camera is obtained. The height of the ship relative to the water surface is obtained based on the installation height of the camera and lidar, the water surface height, the height of the lidar projection point, and the height difference between the highest point of the ship and the lidar projection point.

2. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The lidar and camera are horizontally mounted on the same bracket. The lidar is mounted directly above the camera. A two-dimensional tilt sensor (3) is installed on one side of the lidar to keep the camera and lidar horizontal.

3. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The height calculation module (11) obtains the height of the ship relative to the water surface, i.e. the ship's clearance height, based on the distance and point cloud coordinates obtained by the data fusion module (10), the installation height of the lidar, and the water level information.

4. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 3, characterized in that, The height calculation module (11) finds the highest point P0 of the ship based on image processing and the point cloud data projected onto the image pixels. Starting from the highest point of the ship, it searches downwards for the nearest point projected by the point cloud. When the first point projected by the point cloud, P1, is found, the distance between the two is calculated, i.e., the number of pixels N between them. The coordinates of the point corresponding to P1 in the camera coordinate system are (ac, bc, cc), and the coordinates in the lidar coordinate system are (al, bl, cl). (al, bl, cl) are obtained directly from the lidar point cloud data, and (ac, bc, cc) are obtained from the lidar point cloud data through external parameter matrix transformation. Both are high-precision known data. In actual measurement, the coordinates of the points between P0 and P1 on the Z-axis of the lidar coordinate system are equal to or approximately equal to cl, i.e., the distances from the points between P0 and P1 to the plane where the camera and the radar are located are the same. The Y-axis coordinates of the camera coordinate system and the Z-axis coordinates of the radar coordinate system represent the ship height.

5. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The data fusion module (10) uses the obtained intrinsic parameter data of the camera and the extrinsic parameter data of the lidar and the camera to jointly process the point cloud data collected by the lidar and the image data of the camera. It extracts the highest point of the ship using image processing technology, and uses the intrinsic parameters of the camera and the extrinsic parameters calibrated by the lidar and the camera to project the point cloud collected by the lidar onto the image collected by the camera through projection algorithm and coordinate system transformation. Then, it detects the distance between the highest point of the ship and the nearest point projected by the point cloud below it, and calculates the real distance from the point to the point cloud projection point in the world coordinate system based on this distance using relevant algorithms.

6. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The camera calibration module (7) adopts the Zhang Zhengyou calibration method. By taking pictures of the checkerboard pattern at different positions and using the calibration algorithm to solve the camera parameters, the intrinsic parameter matrix and distortion coefficient of the camera are obtained.

7. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The image processing module (5) performs background modeling on the two-dimensional image captured by the camera, and uses the background modeling algorithm to extract the foreground image, that is, to extract the moving ship in the image, and then performs binarization processing on the image. The gray value of the moving ship target is 1, and the image is displayed as white. The gray value of the stationary background area is 0, and the image is displayed as black. The binarized image is traversed from top to bottom for each pixel point in the image. The first pixel obtained by traversal is 1, and there is a pixel with a value of 1 in its 8 neighboring pixels. This is regarded as the pixel value point of the highest point of the ship image. At this time, the coordinates of the highest point of the ship in the image are found.

8. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The point cloud data processing module (4) filters out noise and removes outliers and ground points from the point cloud data.

9. The method for jointly measuring ship altitude using laser imaging radar and camera according to claim 1, characterized in that, The joint calibration module (8) uses the checkerboard calibration method to allow the lidar and camera to collect data synchronously. By finding the corner data of the checkerboard, the transformation relationship between the lidar coordinate system and the camera coordinate system is solved.

Images