Ship posture coupling method and system based on binocular vision and laser radar

Abstract

The application provides a ship posture coupling method and system based on binocular vision and laser radar, and relates to the technical field of ships.Aiming at the problems that the ship monitoring operation in the prior art is greatly affected by the environment and weather, the specific position and movement trend of the ship cannot be analyzed, and the ship trend cannot be predicted in advance, the application designs to monitor through a non-contact binocular vision sensor and a laser radar, to realize three-dimensional monitoring of the ship, specifically, ship point cloud data collected in real time by the laser radar is used to extract feature information of the ship point cloud data, then the graphic information transmitted by the binocular vision sensor is coupled, to construct a three-dimensional coordinate system to fuse the precise monitoring data from the plane space to the three-dimensional space, so that the advantages of data and pictures are complementary, the ship motion posture data can be captured more accurately, and the monitoring performance and reliability of the system in all-weather and complex scenes are improved.

Description

Technical Field

[0001] This invention relates to the field of ship technology, and in particular to a ship attitude coupling method and system based on binocular vision and lidar. Background Technology

[0002] Current ship berthing operations typically rely on monitoring equipment to observe ship movement, which limits the depth of distance information and ship status perception for staff. This is because ordinary monitoring videos are two-dimensional images, unable to accurately determine the precise distance between the ship and the dock, or between the ship and adjacent vessels. This makes it difficult for staff to determine how many meters are remaining in the approach based on a single camera view, easily leading to misjudgments. Furthermore, such equipment is highly dependent on environmental conditions and limited space, making it impossible to confirm and assess the ship in poor weather or low visibility conditions such as nighttime. Unrelated objects such as waves, smoke, and birds are also easily captured in the video footage, exacerbating the risk of misjudgments of ship movement. Moreover, this reliance on video monitoring equipment for ship movement results in numerous blind spots, making it extremely dependent on the operational experience and attention of the staff.

[0003] The main reason for these problems is that current ship monitoring technology mainly focuses on two-dimensional planar monitoring of ships. In three-dimensional space, much important information is ignored, or unimportant information in the planar space is collected and applied. This leads to the inability to accurately analyze the ship's movement, making it difficult to predict the ship's trajectory in advance and to fully control the movement trends of important parts of the ship. Furthermore, based on the current information collection equipment, the above problems cannot be completely avoided.

[0004] Therefore, there is an urgent need for a method that can calculate and comprehensively control the real-time movement and changes of the ship in all directions during berthing, in order to reduce berthing difficulties and unpredictable impacts on mooring operations. This application proposes a real-time ship calculation method and system based on lidar point clouds. It analyzes, denoises, and identifies feature points from the data collected by the data acquisition equipment, performing motion measurements on the feature points and simultaneously calculating the overall motion of the ship. The system analyzes and predicts the movement of the feature points to form a new ship feature point recognition algorithm. Summary of the Invention

[0005] To address the difficulties and practical challenges in existing ship mooring operations, which make it difficult to grasp the overall ship's course and the impact of sea conditions, this invention first proposes a real-time calculation method for the six degrees of freedom of a ship based on lidar point clouds, specifically including the following steps: Step S1: Obtain point cloud feature information and image information. Extract feature information from the ship point cloud data collected in real time by the lidar, and simultaneously receive image information collected by the binocular vision sensor. Step S2: Process image information and track feature points. Based on the intrinsic parameters of the binocular vision sensor and the semi-global matching algorithm, process the image information to obtain feature points of the target area, and track the feature points. Step S3: Construct a spatial model and coordinate system, randomly sample the point cloud feature information, fit the ground plane, calculate the ship's forward vector based on the point cloud feature information, construct a three-dimensional coordinate system and corresponding transformation matrix based on the ground plane and the forward vector, and calculate the translation matrix in combination with the feature points; Step S4: Determine the target's three-dimensional coordinates, establish an auxiliary coordinate system and a target object coordinate system respectively, determine the Euclidean transformation relationship between the two, and calculate the target ship's three-dimensional coordinates based on the transformation matrix and the feature points; Step S5: Fuse data and output attitude results. The ship point cloud is fused with the target three-dimensional coordinates. The real-time attitude change results of the target ship are obtained through point cloud matching, and the calibration results are output based on the calibration extrinsic parameters.

[0006] Furthermore, in step S1, the feature information extraction includes distinguishing point cloud sets in the point cloud data according to the number of inner points, determining the point cloud set with the most inner points as the target ship point cloud, and obtaining the position and attitude of the target ship accordingly. The image information is a binocular stereo disparity map acquired by a binocular vision sensor.

[0007] Furthermore, in step S2, the generation of the binocular stereo disparity map includes imaging the target area left and right to obtain left and right imaging coordinate maps. Calculate the planar pixel difference between the left and right image coordinate maps, perform epipolar correction using a mapping table to make the baseline parallel to the horizontal direction of the image, and ensure that the pixels in the left and right images correspond in the row direction. Calculate the planar pixel difference to obtain the ranging model.

[0008] Furthermore, in step S2, the ranging model is: ; Where z is the distance between the spatial point and the vertical plane of the binocular vision sensor, f is the focal length, b is the baseline length, and u1 and u2 are the horizontal coordinates of the spatial point on the left and right pixel planes, respectively.

[0009] Furthermore, in step S2, the tracking of feature points includes, based on optical flow, assuming the coordinates of a feature point in the image at time t are (x, y) and the pixel brightness is I(x, y, t), and after time Δt, the feature point moves to (x+u, y+v), where u and v are the distances the feature point moves. The movement speeds u' and v' are obtained through Taylor expansion, thereby achieving the tracking of the feature point.

[0010] Furthermore, in step S3, the random sampling and fitting of the ground plane includes randomly selecting three points from the ship point cloud to fit the plane, and iteratively optimizing to obtain the optimal ground plane. Calculate the distances from the target ship point cloud and feature points to the plane to obtain the forward vector, construct a spatial model, and calculate the transformation matrix.

[0011] Furthermore, in step S4, the auxiliary coordinate system is established with the binocular vision sensor as the origin, X C Along the transverse direction of the sensor, Z C The axis is the shooting direction, Y C The axis is vertically upward; The target object coordinate system is referenced to the target ship, and its origin is the vertical projection point O of the origin of the auxiliary coordinate system. W Y W The axis is the forward direction of the ship, X. W The axis is perpendicular to the right.

[0012] Furthermore, in step S4, the Euclidean transformation relationship is calculated using the height H above the ground by the binocular vision sensor, and the auxiliary coordinate system and the target object coordinate system are fused to obtain the three-dimensional coordinates of the target ship.

[0013] Furthermore, in step S5, based on the calibrated intrinsic parameters of the binocular vision sensor, the formula is used: ; The depth information of the target vessel is calculated to obtain the real-time attitude change results.

[0014] According to another aspect of the present invention, a ship attitude coupling method system based on binocular vision and lidar is also proposed. The system is applicable to any of the above-mentioned ship attitude coupling methods based on binocular vision and lidar, and specifically includes the following modules: The lidar data processing module is used to process the collected point cloud data of the ship's hull and extract feature information; A binocular vision sensor processing module, connected to the lidar data processing module, is used for stereo correction, feature point extraction and tracking of image information; The control center module is connected to the binocular vision sensor processing module and is used to fuse point cloud features and image features to perform point cloud matching and pose calculation. The output module, connected to the control center module, is used to output the ship attitude calculation results in real time.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: Firstly, this invention is based on a ship attitude coupling method using binocular vision and lidar. It uses lidar to collect real-time ship point cloud data and extracts feature information from the ship point cloud data. Then, it couples the graphic information from the binocular vision sensor to construct a three-dimensional coordinate system that fuses planar space into three-dimensional space for accurate monitoring data. This achieves complementary advantages, improves monitoring efficiency and accuracy, and solves the problem that existing ship monitoring relies on a single monitoring device, resulting in inaccurate monitoring results.

[0016] Secondly, image data of the ship's hull is acquired through multiple industrial sensors. Nonlinear equations are used to solve for the sensor's intrinsic and extrinsic parameters, and the 3D coordinates (X, Y, Z) of feature corner points in the field of view are reconstructed. This allows for the derivation of the ship's 3D information, integrating the 2D observation point with the 3D space, taking into account all feature points in both the plane and space, achieving complementary advantages. Based on the pinhole camera principle, the desired result should be a straight line passing through the spatial point and the sensor's optical center. A stereo vision system can obtain multiple projection points of the same spatial point from different locations. These projection points can then be used to obtain multiple straight lines passing through that spatial point. The intersection of these lines represents the spatial point, thus enabling the stereo vision system to obtain depth information. This allows for the determination of the transformation relationships between different coordinate systems and the displacement in each direction, resulting in high safety, high reliability, and high adaptability in complex application scenarios.

[0017] Thirdly, while performing deep fusion calculations, this invention also eliminates temporary noise points such as birds and dust, ensuring data reliability. The fused feature points retain the original geometric information in 3D space, possessing powerful 3D representation capabilities. Furthermore, based on lidar point cloud-based target detection, through the extraction of ship features, obstacle detection, and perception of surrounding targets, it can provide accurate positioning and navigation information for ship monitoring, which can be directly used for ship berthing monitoring tasks, improving the level of intelligence in dock operations. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 The flowchart shows a method for coupling ship attitude based on binocular vision and lidar. Figure 2 This is a schematic diagram of the ranging model; Figure 3 This is a schematic diagram illustrating the Euclidean transformation relationship between the auxiliary coordinates and the target object coordinates. Figure 4 This is a structural diagram of a ship attitude coupling system based on binocular vision and lidar. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] The specific embodiments of the present invention will be described below.

[0022] To address the shortcomings of existing technologies that rely solely on video surveillance of ships, such as the lack of precise analysis of ship movement, difficulty in predicting ship trajectory, and inability to comprehensively monitor the movement trends of critical ship components, this paper proposes a real-time six-degree-of-freedom calculation method and system for ships based on lidar point clouds. This method analyzes, denoises, and identifies feature points from the data acquisition equipment, simultaneously measuring the motion of these feature points and calculating the overall ship motion. The system then analyzes and predicts the movement of the ship on-site, resulting in a novel ship feature point recognition algorithm.

[0023] Example 1 like Figure 1 As shown, this invention proposes a ship attitude coupling method based on binocular vision and lidar, which specifically includes the following steps: Step S1 involves acquiring point cloud feature information and image information. Feature information is extracted from the ship point cloud data collected in real-time by the lidar, while image information is received from a binocular vision sensor. Feature extraction involves distinguishing point cloud sets in the ship point cloud data according to the number of interior points, and selecting the point cloud set with the most interior points as the target ship point cloud. The ship's position and attitude are then determined based on the target ship point cloud. Additionally, the image information is a binocular stereo disparity map collected by the binocular vision sensor.

[0024] Feature information extraction includes distinguishing point cloud sets in point cloud data according to the number of interior points, identifying the point cloud set with the most interior points as the target ship point cloud, and obtaining the position and attitude of the target ship accordingly. The image information is a binocular stereo disparity map collected by a binocular vision sensor.

[0025] In practical applications, the process of acquiring point clouds using lidar is as follows: First, a laser beam is emitted from a laser transmitter. Upon encountering an object, the beam is reflected back to the laser receiver. Then, by analyzing the time it takes for the laser to return to the target object, the relative distance between the target object and the transmitter is calculated. This process of continuously scanning the target object with pulsed lasers yields accurate three-dimensional data after processing. When the lidar sensor rotates one full circle, the coordinates of all reflected points form a point cloud. Point cloud data is characterized by sparsity, disorder, and unstructured nature. Furthermore, planar target detection methods cannot be directly applied to three-dimensional target detection tasks. Therefore, in this embodiment, a method coupling lidar with a binocular vision sensor is used to achieve non-contact monitoring.

[0026] The binocular camera features high precision, wide dynamic range imaging, multi-sensor synchronization and fusion, and built-in IMU attitude assistance, making it particularly adaptable to harsh environments. Utilizing binocular stereo vision, it outputs dense 3D point cloud data of the ship's berthing area in real time, providing raw spatial information for six-degrees-of-freedom calculations. It can simultaneously and clearly capture details of brightly lit areas such as the ship's deck and hull, as well as underwater hull and shadowed areas, avoiding the loss of feature points due to uneven lighting. Through an external trigger mode, it synchronizes exposure with high-precision clock sources such as LiDAR and GPS within the system, achieving strict temporal alignment of multi-sensor data and improving the accuracy of fusion calculations. The collected graphic information is also of high precision, facilitating subsequent analysis.

[0027] Step S2: Processing image information and tracking feature points: Based on the intrinsic parameters of the binocular vision sensor and the semi-global matching algorithm, the image information is processed to obtain the feature points of the target area, and these feature points are then tracked. Graphic information is obtained through the intrinsic parameters and the semi-global matching algorithm. The binocular stereo disparity map is the left and right imaging coordinate map formed after the graphic information is imaged left and right in the target area. The planar pixel difference between the left and right imaging coordinate maps is calculated, and the left and right imaging coordinate maps are substituted into a mapping table for alignment correction to obtain the baselines of the left and right imaging coordinate maps. This ensures that the baselines are completely parallel to the horizontal direction of the left and right imaging coordinate maps, and that the pixels in the left and right imaging coordinate maps correspond left and right in the coordinate system. The planar pixel difference is then calculated, and a ranging model is obtained based on the planar pixel difference.

[0028] Furthermore, the generation of the binocular stereo disparity map includes: imaging the target area left and right to obtain left and right image coordinate maps; calculating the planar pixel difference between the left and right image coordinate maps; performing epipolar correction through a mapping table to ensure the baseline is parallel to the horizontal direction of the image and that the pixels in the left and right images correspond in the row direction; and then calculating the planar pixel difference to obtain the ranging model. The binocular stereo disparity map is the left and right image maps formed after the binocular vision sensor performs left and right imaging on the target area, and the planar pixel difference between the left and right image maps is calculated. The left and right image maps are substituted into a mapping table for alignment correction, resulting in the baseline of the left and right image maps being completely parallel to the horizontal direction of the image, ensuring the left and right coordinates of the feature points in the left and right image maps correspond, and then the planar pixel difference is calculated again. The binocular vision sensor ranging model is then obtained based on this pixel difference. The disparity map is a binocular vision sensor ranging model diagram obtained by calculating the planar pixel difference after the binocular vision sensor performs left and right imaging on the target area and then using this pixel difference.

[0029] like Figure 2 As shown, the ranging model is set as follows: c1 is the left optical center point, c2 is the right optical center point, and c1' is the projection of c1 onto the plane oxz, and c2' is the projection of c2 onto the plane oxz. Let f be the focal length, b be the baseline of the binocular sensor, P' be the projection of spatial point P onto the oxz plane, P1 and P2 be the projections of point P onto the left and right pixel planes, respectively, and u1 and u2 be the distances between P1, P2 and the line perpendicular to the two optical centers, respectively. The distance z between point P and the vertical plane of the binocular vision sensor is obtained.

[0030] The ranging model is then: ; Where z is the distance between the spatial point and the vertical plane of the binocular vision sensor, f is the focal length, b is the baseline length, and u1 and u2 are the horizontal coordinates of the spatial point on the left and right pixel planes, respectively.

[0031] The formula is used to convert pixel coordinates to range measurement model coordinates: ; ; Among them, (uc x ) and (vc y This is to eliminate the influence of point P offset.

[0032] Feature point tracking includes, based on optical flow, assuming the coordinates of a feature point in the image at time t are (x, y) and the pixel brightness is I(x, y, t), and after time Δt, it moves to (x+u, y+v), where u and v are the distances the feature point moves. The movement velocities u' and v' are obtained through Taylor expansion, thus achieving feature point tracking. Alternatively, feature points in the target region can be obtained through a ranging model to achieve feature point tracking. Setting the feature point vector as [u, v], and assuming the pixel brightness at (x, y) in frame t is I(x, y, t), and after moving to (x+u, y+v) in frame t+Δt, the following is obtained: ; Performing a first-order Taylor expansion on the left side of the above formula and rearranging, we obtain: ; Among them, I x I y For spatial gradient, I t Let u' and v' be the time gradient, and u' and v' be the velocities.

[0033] By identifying identical feature points in the left and right images and calculating their pixel position differences, combined with the camera's factory-calibrated parameters, including focal length and baseline distance, the precise coordinates (X, Y, Z) of these feature points in three-dimensional space can be calculated using triangulation. The system tracks the changes in the three-dimensional coordinates of these feature points in consecutive frames. By analyzing the position and angular changes of the rigid body formed by multiple feature points in three-dimensional space, the algorithm can calculate the precise motion of the ship in six degrees of freedom.

[0034] In practical applications, 180 vertically distributed laser emitting or receiving channels are scanned. A 120° field of view is achieved horizontally via a galvanometer or rotating mirror, while the vertical field of view covers -12.5° to +12.5°, approximately 25°. This typically yields an effective point cloud density of 1.65 million points per second. Combined with a 10Hz scanning frame rate, the point cloud resolution for the target area can reach centimeter-level within a 500m ranging range, with point spacing less than 1cm within 15m. The point cloud data includes three-dimensional coordinates (X, Y, Z), reflection intensity, and other information, supporting real-time transmission to the backend processing system. Precise and reliable calculations provide spatial data support for applications such as ship positioning and trajectory tracking, enabling real-time tracking of feature points. This avoids various problems caused by existing single-video monitoring and transmission methods. Furthermore, the stereo vision system in this embodiment can obtain multiple projection points of the same spatial point from different locations. Multiple projection points can be used to obtain multiple straight lines passing through that spatial point; the intersection of these lines represents the spatial point. Therefore, the stereo vision system can obtain the depth information of the spatial point. This allows for the determination of the transformation relationships between different coordinate systems and the displacement in each direction, ensuring high safety, high reliability, and high adaptability in complex application scenarios.

[0035] Step S3: Construct a spatial model and coordinate system. Randomly sample the point cloud feature information to fit the ground plane. Calculate the ship's forward vector based on the point cloud feature information. Construct a three-dimensional coordinate system and corresponding transformation matrix based on the ground plane and forward vector. Calculate the translation matrix using feature points. (The process involves randomly sampling the point cloud feature information, fitting the optimal ground plane value, calculating the ship's forward vector from the point cloud feature information, constructing a coordinate system and transformation matrix based on the ground plane value and forward vector, and calculating the translation matrix based on feature points and the transformation matrix.)

[0036] The random sampling and fitting of the ground plane involves randomly selecting three points from the ship point cloud to fit a plane, iteratively optimizing to obtain the optimal ground plane, calculating the distances from the target ship point cloud and feature points to this plane to obtain the forward vector, constructing a spatial model, and calculating the transformation matrix. The random sampling of point cloud feature information involves randomly selecting three points from the ship point cloud data, fitting these three randomly selected points to obtain the optimal ground plane, calculating the distances from the target ship point cloud and feature points to the optimal ground plane to obtain the forward vector, and constructing a spatial model. The spatial model is then transformed, the transformation matrix is ​​calculated, and the translation matrix is ​​calculated in the ranging model based on the feature points.

[0037] Taking a specific embodiment as an example, the transformation matrix refers to the rigid body transformation matrix that transforms a point cloud from one coordinate system to another. It includes a rotation part R and a translation part t, and is represented as a homogeneous matrix as follows: ; Where R is a 3×3 rotation matrix and t is a 3×1 translation vector. The translation matrix refers to the translation vector t, which directly describes the displacement of the origin. Therefore, t = T[0, 3, 3].

[0038] Taking a point F as an example, after initializing parameters such as a confidence level of 0.99 and an inlier threshold of 1.5 pixels, eight matching pairs are iteratively sampled to solve for point F, and its epipolar distance error is calculated. Points with an error ≤ 1.5 pixels are considered inliers. Iteration stops when the maximum number of iterations (100) or the proportion of inliers is ≥ 90%. After stopping iteration, the inliers are output as the correctly matched points. If the number of outliers is ≥ 2, feature points are re-detected, ultimately ensuring a 99% probability of finding the optimal model. Therefore, by using LiDAR to collect real-time ship point cloud data and extracting feature information from it, and then coupling it with graphic information from a binocular vision sensor, a three-dimensional coordinate system is constructed to fuse planar space into three-dimensional space for precise monitoring data. This achieves complementary advantages and improves monitoring efficiency and accuracy.

[0039] Step S4, as follows Figure 3As shown, the target's three-dimensional coordinates are determined, and an auxiliary coordinate system and a target object coordinate system are established respectively. The Euclidean transformation relationship between the two is determined, and the three-dimensional coordinates of the target ship are calculated based on the transformation matrix and the feature points. The auxiliary coordinate system is established with the binocular vision sensor as the origin, X... C Along the transverse direction of the sensor, Z C The axis is the shooting direction, Y C The axis points vertically upwards, and the target object coordinate system is referenced to the target ship, with the origin being the vertical projection point O of the auxiliary coordinate system origin. W Y W The axis is the forward direction of the ship, X. W The axis points vertically to the right. Specifically, an auxiliary coordinate system and a target object coordinate system are established separately. The auxiliary coordinate system is a left-handed rectangular coordinate system, and a three-dimensional rectangular coordinate system O is established with the binocular vision sensor as the origin. C , let X C The axis is along the lateral direction of the binocular vision sensor, Z C For the binocular vision sensor's shooting direction, Y C The binocular vision sensor is positioned vertically upwards. The target coordinate system is established with the target ship as a reference, and the origin of the target coordinate system is set as the projection point O perpendicular to the ground, which is the origin of the auxiliary coordinate system. W Target object coordinate system Y W The axis is the forward direction of the target ship, and the target object coordinate system is X. W The axis is perpendicular to the target and points to the right. The Euclidean transformation relationship is calculated using the height H of the binocular vision sensor above the ground. By fusing the auxiliary coordinate system and the target coordinate system, the three-dimensional coordinates of the target ship are obtained. To further fuse the auxiliary and target coordinate systems, let the height H of the binocular vision sensor above the ground be used, and calculate the Euclidean transformation relationship between the auxiliary and target coordinate systems to obtain the target's three-dimensional coordinates.

[0040] Because 3D spatial monitoring has more precise geometric properties than 2D spatial monitoring, the deep fusion of lidar monitoring and binocular vision stereo monitoring projects the generated 3D features onto feature points to determine changes in feature points within the target area, thereby identifying the target vessel. This complementary approach allows for more accurate capture of the vessel's six degrees of freedom attitude data, improving the system's monitoring performance and reliability in all-weather, complex scenarios.

[0041] Step S5: Fuse data and output attitude results. The ship point cloud is fused with the target's 3D coordinates. Point cloud matching is used to obtain the real-time attitude change of the target ship, and calibration results are output based on calibration extrinsic parameters. The ship point cloud is fused with the target coordinates, and point cloud matching is performed again in the target's 3D coordinates to obtain the real-time attitude change of the target ship. Calibration results are output based on calibration extrinsic parameters, thus obtaining the real-time attitude change of the target ship.

[0042] For parameter calibration of the binocular vision sensor, given the baseline as d and the focal length as f, substitute into the formula: ; according to: ; ; Substituting the values, we obtain the calibration result Z. We use nonlinear equations to solve for the sensor's intrinsic and extrinsic parameters, and reconstruct the three-dimensional coordinates (X, Y, Z) of the feature corner points in the field of view. This allows us to derive the ship's three-dimensional information. By obtaining multiple projection points of the same spatial point from different locations, we can obtain multiple straight lines passing through that spatial point. This allows us to obtain the spatial point's depth information, thereby determining the transformation relationships between different coordinate systems and the displacement in each direction.

[0043] This invention presents a ship attitude coupling method based on binocular vision and lidar. It utilizes non-contact 3D monitoring, specifically by extracting feature information from real-time collected ship point cloud data using lidar, and then coupling this data with graphic information from binocular vision sensors. This constructs a 3D coordinate system, fusing planar space into precise monitoring data, achieving complementary advantages and improving monitoring efficiency and accuracy. It addresses the inaccuracy caused by relying on single monitoring devices in existing ship monitoring systems. Multiple industrial sensors acquire image data of the ship's hull, and nonlinear equations are used to solve for the sensor's intrinsic and extrinsic parameters. This fuses the 2D observation point with 3D space, taking into account all feature points in both planar and spatial dimensions, achieving complementary advantages. Based on the pinhole imaging principle, the desired result should be a straight line passing through the spatial point and the sensor's optical center. The stereo vision system can obtain multiple projection points of the same spatial point from different locations. These projection points can then be used to obtain multiple straight lines passing through the spatial point; the intersection of these lines represents the spatial point. Therefore, the stereo vision system can obtain the depth information of the spatial point. This allows us to determine the transformation relationships between different coordinate systems and the displacement in each direction, ensuring safety and reliability in complex application scenarios.

[0044] Example 2 like Figure 4 As shown, the present invention also proposes a ship attitude coupling method system based on binocular vision and lidar, using the ship attitude coupling method based on binocular vision and lidar as described in any of Embodiment 1, including the following modules: The lidar data processing module is used to organize the collected hull point cloud data and extract the feature information of the hull point cloud data.

[0045] The binocular vision sensor processing module, connected to the lidar data processing module, is used for stereo correction, feature point extraction, and tracking of image information. Stereo correction is performed based on feature information, converting the feature information into stereo information and matching it with target coordinates to track the target vessel.

[0046] The control center module connects the lidar data processing module and the binocular vision sensor processing module, and is used to fuse point cloud features and image features to perform point cloud matching and attitude calculation. The output module connects to the control center module and outputs the ship attitude calculation results in real time.

[0047] This invention performs deep fusion calculations of lidar point cloud features and binocular visual image features while also eliminating temporary noise points such as birds and dust, ensuring data reliability. The fused feature points retain the original geometric information in 3D space, possessing powerful 3D representation capabilities. Furthermore, based on lidar point cloud-based target detection, through the extraction of ship features, obstacle detection, and perception of surrounding targets, it can provide accurate positioning and navigation information for ship monitoring. This information can be directly used for ship berthing monitoring tasks, improving the intelligence level of dock operations and providing stable and accurate data support for ship berthing operations, minimizing the impact of external factors.

[0048] Example 3 An electronic device, comprising: Processor and memory; The processor executes the steps of the binocular vision and lidar ship attitude coupling method as described in any of Embodiment 1 by calling programs or instructions stored in memory.

[0049] Example 4 A computer-readable storage medium includes computer program instructions that cause a computer to perform the steps of a binocular vision and lidar ship attitude coupling method as described in any of Embodiment 1.

[0050] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A ship attitude coupling method based on binocular vision and lidar, characterized in that, Includes the following steps: Step S1: Obtain point cloud feature information and image information. Extract feature information from the ship point cloud data collected in real time by the lidar, and simultaneously receive image information collected by the binocular vision sensor. Step S2: Process image information and track feature points. Based on the intrinsic parameters of the binocular vision sensor and the semi-global matching algorithm, process the image information to obtain feature points of the target area, and track the feature points. Step S3: Construct a spatial model and coordinate system, randomly sample the point cloud feature information, fit the ground plane, calculate the ship's forward vector based on the point cloud feature information, construct a three-dimensional coordinate system and corresponding transformation matrix based on the ground plane and the forward vector, and calculate the translation matrix in combination with the feature points; Step S4: Determine the target's three-dimensional coordinates, establish an auxiliary coordinate system and a target object coordinate system respectively, determine the Euclidean transformation relationship between the two, and calculate the target ship's three-dimensional coordinates based on the transformation matrix and the feature points; Step S5: Fuse data and output attitude results. The ship point cloud is fused with the target three-dimensional coordinates. The real-time attitude change results of the target ship are obtained through point cloud matching, and the calibration results are output based on the calibration extrinsic parameters.

2. The ship attitude coupling method of binocular vision and lidar according to claim 1, characterized in that, In step S1, the feature information extraction includes distinguishing point cloud sets in the point cloud data according to the number of inner points, determining the point cloud set with the most inner points as the target ship point cloud, and obtaining the position and attitude of the target ship accordingly. The image information is a binocular stereo disparity map acquired by a binocular vision sensor.

3. The ship attitude coupling method of binocular vision and lidar according to claim 2, characterized in that, In step S2, the generation of the binocular stereo disparity map includes imaging the target area left and right to obtain left and right imaging coordinate maps. Calculate the planar pixel difference between the left and right image coordinate maps, perform epipolar correction using a mapping table to make the baseline parallel to the horizontal direction of the image, and make the pixels in the left and right images correspond in the row direction. Calculate the planar pixel difference to obtain the ranging model.

4. The ship attitude coupling method of binocular vision and lidar according to claim 3, characterized in that, In step S2, the ranging model is: ； Where z is the distance between the spatial point and the vertical plane of the binocular vision sensor, f is the focal length, b is the baseline length, and u1 and u2 are the horizontal coordinates of the spatial point on the left and right pixel planes, respectively.

5. The ship attitude coupling method of binocular vision and lidar according to claim 4, characterized in that, In step S2, tracking the feature points includes, based on optical flow, assuming the coordinates of the feature points in the image at time t are (x, y) and the pixel brightness is I(x, y, t), and after time Δt, the feature points move to (x+u, y+v), where u and v are the distances the feature points move. The movement speeds u' and v' are obtained through Taylor expansion, thereby achieving the tracking of the feature points.

6. The ship attitude coupling method of binocular vision and lidar according to claim 5, characterized in that, In step S3, the random sampling and fitting of the ground plane includes randomly selecting three points from the ship point cloud to fit the plane, and iteratively optimizing to obtain the optimal ground plane; Calculate the distances from the target ship point cloud and feature points to the plane to obtain the forward vector, construct a spatial model, and calculate the transformation matrix.

7. The ship attitude coupling method of binocular vision and lidar according to claim 6, characterized in that, In step S4, the auxiliary coordinate system is established with the binocular vision sensor as the origin, X C Along the transverse direction of the sensor, Z C The axis is the shooting direction, Y C The axis is vertically upward; The target object coordinate system is referenced to the target ship, and its origin is the vertical projection point O of the origin of the auxiliary coordinate system. W Y W The axis is the forward direction of the ship, X. W The axis is perpendicular to the right.

8. The method for coupling binocular vision and lidar for ship attitude according to claim 7, characterized in that, In step S4, the Euclidean transformation relationship is calculated using the height H above the ground by the binocular vision sensor, and the auxiliary coordinate system and the target object coordinate system are fused to obtain the three-dimensional coordinates of the target ship.

9. The method for coupling binocular vision and lidar for ship attitude according to claim 8, characterized in that, In step S5, based on the calibrated intrinsic parameters of the binocular vision sensor, the formula is used: ； The depth information of the target vessel is calculated to obtain the real-time attitude change results.

10. A ship attitude coupling method system based on binocular vision and lidar, the system being applicable to the ship attitude coupling method based on binocular vision and lidar as described in any one of claims 1 to 9, characterized in that, Specifically, it includes the following modules: The lidar data processing module is used to process the collected point cloud data of the ship's hull and extract feature information; A binocular vision sensor processing module, connected to the lidar data processing module, is used for stereo correction, feature point extraction and tracking of image information; The control center module is connected to the binocular vision sensor processing module and is used to fuse point cloud features and image features to perform point cloud matching and pose calculation. The output module, connected to the control center module, is used to output the ship attitude calculation results in real time.

Images