A visual compensation-based berthing and unberthing auxiliary guiding method and system for a ship

Abstract

The application provides a ship berthing and unberthing auxiliary guidance method and system based on visual compensation, a world coordinate system is established first, and a homography matrix for mapping the world coordinate system to an image coordinate system is calculated; then pixel coordinates of a shore line reference line segment and a shore indication flag point input by a user are obtained as initial visual reference coordinates; a feature point set is initialized near the shore line, and an optical flow tracking method is used for inter-frame tracking, and a global average offset of the feature points is calculated; then the initial visual reference coordinates are inversely compensated based on the global average offset, and stable state reference coordinates are obtained; finally, the stable state reference coordinates are mapped to the world coordinate system by using an inverse matrix of the homography matrix, a ship bridge and shore distance, a ship bow and shore distance, a ship stern and shore distance, a ship bow and shore included angle, and a flag point and ship bridge vertical distance are calculated, and the above parameters are visually output, a berthing and unberthing auxiliary guidance interface is generated, and low-cost, high-precision and anti-interference berthing and unberthing state sensing and guidance are realized.

Description

Technical Field

[0001] This invention relates to the field of ship berthing and unberthing assistance technology, specifically to a ship berthing and unberthing assistance guidance method and system based on visual compensation. Background Technology

[0002] Berthing and unberthing operations are among the most complex and risky aspects of shipping, with their success heavily reliant on the experience and visual assessment of the crew. Traditional methods suffer from high subjectivity, a lack of precise quantitative data, and susceptibility to adverse weather and lighting conditions, posing significant risks to ship and port safety. Currently, the technological community urgently needs a solution that can provide real-time, accurate, stable, and intuitive berthing and unberthing status perception and guidance.

[0003] Existing ship berthing and unberthing status sensing solutions mainly fall into two categories: those based on high-precision sensors and those based on traditional image processing. Each solution has significant technical drawbacks. 1) Solutions based on high-precision sensors typically rely on high-precision sensors installed on the hull or dock, such as laser rangefinders, differential GPS positioning systems, millimeter-wave radar, or ultrasonic sensors. These sensors measure the absolute distance and relative speed between the ship and the dock to provide data reference for the operator. This solution has the following inherent disadvantages: high hardware costs, as the required specialized sensors are expensive, significantly increasing system deployment costs; complex deployment and poor applicability, requiring engineering installation and calibration on the hull or dock, potentially affecting the ship's original structure and making large-scale deployment difficult in the existing fleet; weak environmental adaptability, with lasers and radar performance drastically decreasing in adverse weather conditions such as rain, fog, and snow, resulting in shortened detection range, loss of accuracy, or even malfunction, failing to meet all-weather operational requirements.

[0004] 2) Traditional image processing-based solutions capture images of the scene using cameras and employ simple image processing algorithms (such as edge detection and template matching) to identify ships or shorelines and estimate their relative positions. This approach has the following inherent drawbacks: It lacks physical scale information; most solutions directly process pixel information, failing to establish a mapping between the image coordinate system and the world coordinate system. Its output (such as pixel distance) cannot be converted into data with real physical meaning (meters, centimeters), significantly reducing its practicality. It also exhibits extremely poor robustness and is susceptible to interference. Lacking an effective motion compensation mechanism, when ships experience continuous six-degree-of-freedom motion due to wind, waves, currents, and their own maneuvering, the image shakes violently, making traditional image processing algorithms (such as edge detection) prone to failure, resulting in false positives or false negatives. This leads to inaccurate or even interrupted output data, making it completely unable to provide reliable decision support in dynamic environments.

[0005] Therefore, there is an urgent need for a method and system that can effectively solve the above problems and achieve low cost, high precision, and strong anti-interference capabilities for sensing the berthing and unberthing status of ships. This system should be able to establish a precise mapping between images and the physical world, resist ship swaying interference, output stable and reliable berthing and unberthing parameters, and provide crew members with intuitive and quantitative operational guidance. Summary of the Invention

[0006] To address the technical problems of high hardware costs, lack of physical scale information, and poor robustness in existing ship berthing and unberthing status sensing methods, this invention provides a visual compensation-based ship berthing and unberthing auxiliary guidance method. This method can establish a precise mapping between image pixels and the physical world, resist ship sway interference through optical flow tracking and inverse coordinate translation compensation, and output berthing and unberthing parameters with real physical meaning. It features low cost, high accuracy, strong anti-interference, and intuitive results. This invention also relates to a visual compensation-based ship berthing and unberthing auxiliary guidance system.

[0007] The technical solution of the present invention is as follows:

[0008] A visual compensation-based auxiliary guidance method for ship berthing and unberthing, characterized by comprising the following steps:

[0009] Calibration and spatial mapping steps: A world coordinate system is established with the vertical projection of a fixed reference point on the ship's bridge as the origin, the direction parallel to the shoreline as the X-axis, the direction perpendicular to the shoreline pointing towards the ship as the Y-axis, and the ship's bridge perpendicular to the shoreline and upwards as the Z-axis. Multiple calibration points are selected on the shore, and the physical coordinates of each calibration point in the world coordinate system are measured. Images of each calibration point are then acquired using a camera pre-installed on the ship, and the pixel coordinates of each calibration point in the image coordinate system are obtained from the images. Based on the physical coordinates and pixel coordinates, the homography matrix mapping the world coordinate system to the image coordinate system is calculated using the least squares method. Simultaneously, the physical coordinates of the pre-set fixed reference point on the ship's bridge, the bow point, and the stern point in the world coordinate system are measured.

[0010] Visual reference coordinate input steps: Obtain the pixel coordinates of the two endpoints of the shoreline reference line segment marked by the user on the current frame video interface captured by the camera in real time, as well as the single pixel coordinates of the shoreline indicator flag point, as the initial visual reference coordinates.

[0011] Optical flow tracing steps: Using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points, construct a rectangular search region based on shoreline constraints in the current frame video interface. Initialize a set of feature points only within the rectangular search region, and retain only feature points whose distance from the shoreline is less than a preset first distance threshold as the initial feature point set. Then, use the optical flow tracing method to perform inter-frame tracking on the initial feature point set to obtain the positions of each feature point retained in the adjacent subsequent frame video interface of the current frame.

[0012] Reverse compensation steps: Calculate the vertical distance from each tracked feature point in the adjacent subsequent video interface of the current frame to the shoreline baseline segment of the current frame. If the vertical distance is greater than a preset second distance threshold, it is determined to be a mismatched point and is removed. Calculate the global average offset of the remaining feature points between adjacent frames after removal, and perform reverse coordinate translation compensation on the initial visual reference coordinates based on the global average offset to obtain the pixel coordinates of the two endpoints of the compensated shoreline baseline segment and the single pixel coordinates of the compensated shoreline indicator flag point, which are used as the steady-state reference coordinates.

[0013] The steps for calculating ship attitude parameters are as follows: Using the inverse of the homography matrix, the steady-state reference coordinates are transformed to the world coordinate system to obtain the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point; a shoreline straight line equation is established based on the world coordinates of the two endpoints of the shoreline reference segment; the vertical distances from the preset physical coordinates of the ship's bridge fixed reference point, bow point, and stern point to the shoreline straight line equation are calculated to obtain the ship's bridge distance, bow distance, and stern distance; the ship's hull line is then determined based on the physical coordinates of the bow and stern points, and the angle between the hull line and the shoreline reference segment is calculated to obtain the bow-to-shore angle; a perpendicular line is drawn from the world coordinates of the shoreline indicator flag point to the shoreline reference segment, and the distance between this perpendicular line and the straight line passing through the ship's bridge fixed reference point and perpendicular to the shoreline is calculated to obtain the vertical distance between the flag point and the ship's bridge.

[0014] Visualization output steps: The distance between the ship's bridge and the shore, the distance between the bow and the shore, the distance between the stern and the shore, the angle between the bow and the shore, and the vertical distance between the flag point and the ship's bridge are used as berthing and unberthing state parameters and visualized output to generate a ship berthing and unberthing auxiliary guidance interface to represent the real-time position and attitude of the ship during the berthing and unberthing process.

[0015] Preferably, in the reverse compensation step, the number of remaining feature points after removing mismatched points is also determined. When the number of remaining feature points after removal is determined to be lower than a preset safety point threshold, a re-initialization mechanism is triggered. The re-initialization mechanism is as follows: return to the optical flow tracking step, and the optical flow tracking step reconstructs a rectangular search area based on the shoreline constraint in the current frame video interface using the two endpoint pixel coordinates of the shoreline baseline segment as diagonal points. A new set of feature points is extracted in the reconstructed rectangular search area to replace the original initialized feature point set. Then, the inter-frame tracking operation is continued to maintain the continuity of optical flow tracking.

[0016] Preferably, in the optical flow tracking step, the Lucas-Kanade optical flow tracking method is used to perform inter-frame tracking on the initialized feature point set.

[0017] Preferably, in the ship attitude parameter calculation step, the ship's lateral velocity is calculated based on any one of the distances between the ship's bridge and the shore, the bow and the shore, and the stern and the shore corresponding to two consecutive frames, combined with the time difference between the two frames, and the ship's lateral velocity is output in the visualization output step.

[0018] Preferably, in the calibration and spatial mapping step, the plurality of calibration points are at least four shore calibration points; the camera is installed near the front centerline of the ship's bridge canopy, with the camera's field of view facing the berth ahead, and the camera's horizontal field of view is 100 degrees and its vertical field of view is 60 degrees.

[0019] In the visual reference coordinate input step, the shoreline indicator flag point is located on the shoreline.

[0020] A vision-compensated ship berthing and unberthing auxiliary guidance system is characterized by comprising, in sequence, a calibration and spatial mapping module, a visual reference coordinate input module, an optical flow tracking module, an inverse compensation module, a ship pose parameter calculation module, and a visualization output module.

[0021] The calibration and spatial mapping module establishes a world coordinate system with the vertical projection of a fixed reference point on the ship's bridge as the origin, the direction parallel to the shoreline as the X-axis, the direction perpendicular to the shoreline pointing towards the ship as the Y-axis, and the ship's bridge perpendicular to the shoreline and upward as the Z-axis. Multiple calibration points are selected on the shore, and the physical coordinates of each calibration point in the world coordinate system are measured. Images of each calibration point are then acquired using cameras pre-installed on the ship, and the pixel coordinates of each calibration point in the image coordinate system are obtained from the images. Based on the physical coordinates and pixel coordinates, the homography matrix mapping the world coordinate system to the image coordinate system is calculated using the least squares method. Simultaneously, the physical coordinates of the preset fixed reference point on the ship's bridge, the bow point, and the stern point in the world coordinate system are measured.

[0022] The visual reference coordinate input module obtains the pixel coordinates of the two endpoints of the shoreline reference line segment marked by the user on the current frame video interface captured by the camera in real time, as well as the single pixel coordinate of the shoreline indicator flag point, as the initial visual reference coordinates.

[0023] The optical flow tracking module constructs a rectangular search region based on shoreline constraints in the current frame video interface, using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. It initializes a set of feature points only within the rectangular search region and retains only feature points whose distance from the shoreline is less than a preset first distance threshold as the initial feature point set. Then, it uses the optical flow tracking method to perform inter-frame tracking on the initial feature point set to obtain the positions of each feature point retained in the adjacent subsequent frame video interface of the current frame.

[0024] The reverse compensation module calculates the vertical distance from each tracked feature point in the adjacent subsequent video interface of the current frame to the shoreline baseline segment of the current frame. If the vertical distance is greater than a preset second distance threshold, it is determined to be a mismatched point and is removed. The module calculates the global average offset of the remaining feature points between adjacent frames after removal, and performs reverse coordinate translation compensation on the initial visual reference coordinates based on the global average offset to obtain the pixel coordinates of the two endpoints of the compensated shoreline baseline segment and the single pixel coordinates of the compensated shoreline indicator flag point, which are used as the steady-state reference coordinates.

[0025] The ship attitude parameter calculation module uses the inverse matrix of the homography matrix to transform the steady-state reference coordinates to the world coordinate system, obtaining the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point; it establishes the shoreline straight line equation based on the world coordinates of the two endpoints of the shoreline reference segment; and calculates the perpendicular distances from the preset physical coordinates of the ship's bridge fixed reference point, bow point, and stern point to the shoreline straight line equation, obtaining the ship's bridge distance, bow distance, and stern distance; then, it determines the ship's hull line based on the physical coordinates of the bow and stern points, calculates the angle between the ship's hull line and the shoreline reference segment, obtaining the bow angle; and draws a perpendicular line to the shoreline reference segment with the world coordinates of the shoreline indicator flag point as the center point, calculating the distance between this perpendicular line and the straight line passing through the ship's bridge fixed reference point and perpendicular to the shoreline, obtaining the perpendicular distance between the flag point and the ship's bridge.

[0026] The visualization output module uses the distance between the ship's bridge and the shore, the distance between the bow and the shore, the distance between the stern and the shore, the angle between the bow and the shore, and the vertical distance between the flag point and the ship's bridge as berthing and unberthing status parameters and outputs them in a visualization manner to generate a ship berthing and unberthing auxiliary guidance interface to characterize the real-time position and attitude of the ship during the berthing and unberthing process.

[0027] Preferably, the reverse compensation module further determines the number of remaining feature points after removing mismatched points. When the number of remaining feature points after removal is lower than a preset safety point threshold, a re-initialization mechanism is triggered. The re-initialization mechanism is as follows: the optical flow tracking module returns to the optical flow tracking module, which uses the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points to reconstruct a rectangular search region based on shoreline constraints in the current frame video interface. A new set of feature points is extracted within the reconstructed rectangular search region to replace the original initialized feature point set. Then, the inter-frame tracking operation continues to be performed, thereby maintaining the continuity of optical flow tracking.

[0028] Preferably, in the optical flow tracking module, the Lucas-Kanade optical flow tracking method is used to perform inter-frame tracking on the initialized feature point set.

[0029] Preferably, in the ship attitude parameter calculation module, the ship's lateral velocity is calculated based on any one of the following distances corresponding to two consecutive frames: the distance between the ship's bridge and the shore, the distance between the ship's bow and the shore, and the distance between the ship's stern and the shore, combined with the time difference between the two frames, and the ship's lateral velocity is output in the visualization output step.

[0030] Preferably, in the calibration and spatial mapping module, the plurality of calibration points are at least four shore calibration points; the camera is installed near the front centerline of the ship's bridge canopy, with the camera's field of view facing the berth ahead, and the camera's horizontal field of view is 100 degrees and its vertical field of view is 60 degrees.

[0031] In the visual reference coordinate input module, the shoreline indicator flag is located on the shoreline.

[0032] The beneficial effects of this invention are as follows:

[0033] This invention provides a visual compensation-based auxiliary guidance method for ship berthing and unberthing. A fixed reference point is pre-selected on the ship's bridge, and a world coordinate system is established with the vertical projection of this reference point onto the shore as its origin. Images of the reference point are acquired using a camera mounted on the ship. The homography matrix mapping the world coordinate system to the image coordinate system is calculated using the least squares method. This homography matrix is ​​then used to convert the image pixel coordinates to physical coordinates in the world coordinate system using its inverse matrix. By establishing a precise mapping relationship between image pixel coordinates and physical coordinates in the world coordinate system through the homography matrix, the conversion of image pixel positions to physical coordinates within the berth shore is achieved. This completely solves the problem of measurement reference ambiguity caused by the lack of a unified coordinate system in traditional visual measurement methods, laying a solid geometric foundation for the subsequent conversion of visual observation data into navigation parameters with practical physical meaning.

[0034] Then, the pixel coordinates of the shoreline baseline segment and shoreline indicator flag points marked by the user on the video interface are obtained as the initial visual reference coordinates. By introducing prior knowledge from human-computer interaction, the "region of interest" and "absolute reference" that the algorithm focuses on are clearly defined through manual calibration. This not only defines precise geometric boundaries for subsequent feature extraction but also provides additional directional constraints using the indicator flag points, effectively avoiding matching ambiguities caused by repetitive shoreline textures or unclear features, ensuring the accuracy and relevance of the system during the initialization phase in complex shoreline environments. At the same time, this interactive calibration mechanism enables the system to adapt to the berthing and departure requirements of different berths and different voyages without requiring complex hardware calibration and parameter configuration for each berth in advance, significantly reducing the system's deployment difficulty and maintenance costs, and improving the system's versatility and practicality.

[0035] Building upon this foundation, this invention innovatively proposes an optical flow tracking and inverse compensation mechanism based on shoreline constraints. Through the optical flow tracking mechanism, a rectangular search region based on shoreline constraints is constructed using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. Only a set of feature points is initialized within this rectangular search region, and only feature points whose distance from the shoreline is less than a preset first distance threshold are retained as the initial feature point set. Inter-frame tracking is then performed using the optical flow tracking method to obtain the pixel offset. This shoreline-constrained feature point selection strategy limits the tracking range to the vicinity of the shoreline, effectively eliminating interference from moving parts of the ship itself (such as cranes and cables), water surface reflections, and other moving objects. This significantly improves the accuracy and robustness of feature point tracking, providing a high-quality, high-signal-to-noise ratio feature input foundation for subsequent motion compensation and pose calculation.

[0036] Then, through a reverse compensation mechanism, the vertical distance from each tracked feature point to the current frame's shoreline baseline segment is calculated. Mismatched points with a vertical distance greater than a preset second distance threshold are eliminated. The global average offset of the remaining feature points between adjacent frames is calculated, and the initial visual reference coordinates are compensated for by reverse coordinate translation based on this offset to obtain steady-state reference coordinates. This reverse compensation mechanism can offset the non-rigid displacement of the image caused by ship rolling, wind and waves, or camera shake in real time. This allows the system to output stable and reliable data even in complex actual berthing and unberthing operation environments. It overcomes the technical defects of traditional visual measurement methods, which suffer from drastic jumps and distortions in measurement data due to reference drift in harsh sea conditions. This significantly improves the anti-interference capability and measurement accuracy of the visual guidance system.

[0037] Furthermore, based on steady-state reference coordinates, the distances between the ship's bridge and the shore, bow and stern, the bow-to-shore angle, and the vertical distance between the flag point and the bridge are precisely calculated. Due to the introduction of the aforementioned inverse compensation and adaptive re-initialization mechanisms, the calculation of the ship's attitude parameters is no longer affected by image offset caused by the ship's dynamic rolling, accurately reflecting the ship's real-time relative position to the dock. This step achieves a precise mapping from two-dimensional image pixel coordinates to the ship's two-dimensional attitude parameters on the physical plane of the shore, that is, transforming the two-dimensional image pixel plane to the shore's XOY plane in the world coordinate system. This provides pilots with quantitative and objective berthing and unberthing status data, solving the problems of traditional visual observation methods relying on experience and lacking unified quantitative standards.

[0038] Finally, the distances between the ship's bridge and the shore, bow and stern, the angle between the bow and the shore, and the perpendicular distance between the flag point and the bridge are visualized as berthing and unberthing status parameters, generating a ship berthing and unberthing auxiliary guidance interface to represent the ship's real-time position and attitude during the berthing and unberthing process. This visualization output method transforms abstract algorithmic data into intuitive visual guidance information, enabling pilots to clearly grasp the relative position and movement trend of the ship relative to the dock. Especially in complex environments with limited visibility, such as at night or in foggy weather, this guidance system can act as an "electronic eye" to assist manual lookout, significantly reducing the risk of collisions and scrapes caused by blind spots or misjudgments.

[0039] This invention constructs a complete technical system for berthing and unberthing guidance, comprising "homophonic matrix calibration + shoreline-constrained optical flow tracking + inverse coordinate compensation + physical coordinate mapping + visualization output." This system completely overcomes the limitations of traditional ship berthing and unberthing guidance methods, such as unstable measurement references, easy loss of tracking features, and low measurement accuracy in dynamic ship rolling scenarios. The core points of this invention are: first, by using monocular vision calibration and calculating the homography matrix, a precise mapping relationship is established between image pixel coordinates and the physical coordinates of the wharf's XOY plane in the world coordinate system, achieving physical quantification output of visual measurement results; second, an innovative shoreline-constrained optical flow tracking strategy is proposed. This strategy delineates the feature point search area based on shoreline constraints and extracts the initial feature point set. During inter-frame tracking, mismatched feature points are filtered out. The global average offset of effective feature points is used to perform real-time, dynamic inverse compensation of the user-calibrated visual reference coordinates, effectively resisting image interference caused by ship rolling and stably outputting highly reliable ship berthing and unberthing parameters. Specifically, it includes the following aspects:

[0040] 1) Precise Accuracy and Quantitative Data: This invention establishes a precise mapping relationship between the image pixel coordinate system and the world physical coordinate system through a homography matrix, fundamentally solving the deficiency of traditional visual methods that can only provide pixel distances but cannot determine the true physical distances. The calculated distances between the ship's bridge and the shore, the bow and shore, the stern and shore (unit: meters), the angle between the bow and the shore (unit: degrees), and the perpendicular distance between the flag point and the bridge (unit: meters) are all physically meaningful true values ​​with high measurement accuracy. This provides the crew with precise and quantitative operational basis, upgrading berthing and unberthing operations from experience-based judgment to data-driven decision-making.

[0041] 2) Strong anti-interference and high robustness: This invention innovatively proposes an optical flow tracking and inverse compensation model based on shoreline constraints. Due to the continuous six-degree-of-freedom motion (roll, pitch, bow, sway, heave, and yaw) generated by ships under the influence of wind, waves, currents, and their own maneuvering, the camera image experiences severe shaking. Traditional edge detection and template matching algorithms are prone to failure in this dynamic environment—edges become blurred, templates deform, and features disappear. Therefore, this invention innovatively adopts an optical flow tracking method based on shoreline constraints: feature points are initialized only near the shoreline, focusing only on points with high consistency with the shoreline's motion, eliminating interference from the ship's own moving parts; the inter-frame motion of feature points is accurately calculated using an optical flow algorithm; and the global motion of the image is estimated through statistical averaging, avoiding the influence of errors from individual feature points. By tracking the motion of feature points near the shoreline to estimate the global motion offset of the image, accurate input is provided for subsequent inverse compensation. Building upon this foundation, by calculating the global motion offset and inversely compensating for the user's initial calibration reference, the system can effectively estimate and counteract the global image jitter caused by the ship's six degrees of freedom motion, outputting stable and accurate coordinate values. Simultaneously, a mismatch point elimination mechanism (eliminating feature points with a vertical distance greater than a preset threshold) further enhances tracking accuracy. This enables the system to output smooth, stable, and reliable pose data even in complex actual berthing and unberthing operation environments (such as windy and wavey weather, and swaying caused by the ship's own maneuvering), overcoming the key pain point of traditional image algorithms where measurement results drastically change or even fail due to ship swaying, achieving stable operation in all weather conditions and under all operating conditions.

[0042] 3) Intuitive Results and Efficient Decision-Making: This invention overlays and displays the calculated berthing and unberthing status parameters (distance, angle, etc.) in real time on a top-down view, generating a ship berthing and unberthing auxiliary guidance interface. The top-down view aligns with the thinking habits of ship operators, and the distance and angle values ​​are intuitive and clear. Crew members do not need to perform complex distance estimations and angle judgments; they can accurately grasp the relative position of the ship to the shoreline simply by observing the values ​​on the screen. This intuitive guidance method significantly reduces the cognitive burden on operators, making berthing and unberthing operations more relaxed and precise, effectively reducing the probability of berthing accidents and improving the safety and efficiency of berthing and unberthing operations.

[0043] 4) Low cost and easy deployment: This invention only requires the deployment of ordinary industrial cameras as signal acquisition units, eliminating the need for expensive external dedicated sensors such as LiDAR, ultrasonic sensors, and differential GPS, greatly reducing hardware costs and system deployment complexity. Construction and installation are limited to fixing and wiring the camera, requiring minimal modification to the original ship structure, which facilitates rapid implementation and widespread application on various types of vessels. Compared to solutions based on high-precision sensors, which often involve hardware investments of hundreds of thousands of yuan, this invention has a significant cost advantage.

[0044] In summary, this invention organically integrates the aforementioned mechanisms, including homography matrix calibration, shoreline-constrained optical flow tracking, inverse compensation, physical coordinate mapping, and visualization output, forming a complete berthing and unberthing auxiliary guidance technology system. This multi-mechanism collaborative architecture not only significantly improves the accuracy and stability of berthing and unberthing perception but also possesses advantages such as low cost, simple deployment, strong anti-interference capability, and intuitive results. It can adapt to the berthing and unberthing operation needs of different berths, different ship types, and different weather conditions. This invention can output high-confidence, real-time, and stable berthing and unberthing status parameters, providing ship operators with a reliable decision support tool. It has significant engineering application value and economic benefits for reducing berthing and unberthing accident rates, improving port operation efficiency, and ensuring shipping safety.

[0045] Furthermore, this invention introduces an adaptive re-initialization mechanism to monitor the number of remaining feature points after removal in real time. When the number is determined to be below a preset safety point threshold, a re-initialization process is triggered. This mechanism involves returning to the optical flow tracking step, whereby the optical flow tracking step uses the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonals to reconstruct a rectangular search area based on shoreline constraints in the current frame video interface, and extracts a new set of feature points to replace the original set. Through this geometric backtracking strategy based on endpoint coordinates, the system can quickly re-lock the target within the preset shoreline area and maintain the continuity of optical flow tracking even in extreme cases where a large number of feature points are lost due to occlusion or sudden changes in illumination. This not only avoids the tracking failure caused by feature point exhaustion in traditional algorithms but also ensures the robustness and uninterruptedness of the assisted guidance service throughout the berthing and unberthing process.

[0046] Furthermore, this invention calculates the ship's lateral velocity based on any one of the following distances from the ship's bridge to the shore, bow to the shore, or stern to the shore, combined with the time difference between the two frames, and outputs this velocity in the visualization output step. This feature enables dynamic quantitative monitoring of the ship's motion trend, overcoming the shortcomings of traditional visual guidance which only provides static position information. By intuitively displaying the lateral velocity, pilots can accurately grasp the instantaneous speed at which the ship approaches or moves away from the dock, thus more scientifically judging whether the current maneuvering actions are appropriate. Especially in the final stage of berthing, this data can effectively assist pilots in controlling the "soft landing" speed, preventing damage to the hull or facilities caused by excessive lateral velocity impacting the dock, or loss of control due to wind and current caused by insufficient speed, greatly improving the precision control level and safety of berthing and unberthing operations.

[0047] This invention also relates to a visual compensation-based auxiliary guidance system for ship berthing and unberthing. This system corresponds to the aforementioned visual compensation-based auxiliary guidance method for ship berthing and unberthing, and can be understood as a system that implements the aforementioned visual compensation-based auxiliary guidance method for ship berthing and unberthing. It includes a calibration and spatial mapping module, a visual reference coordinate input module, an optical flow tracking module, an inverse compensation module, a ship pose parameter calculation module, and a visualization output module, connected sequentially. These modules work collaboratively to construct a complete perception system from image acquisition to auxiliary guidance. First, the calibration and spatial mapping module establishes a world coordinate system and calculates the homography matrix, solving the mapping problem between image pixels and physical space. Then, the optical flow tracking module and the inverse compensation module achieve stable tracking against interference, solving the image jitter problem caused by ship swaying. Next, the ship pose parameter calculation module converts stable coordinates into physical parameters, solving the practical problem of output quantization. Finally, the visualization output module generates an auxiliary guidance interface, providing crew members with an intuitive and operable decision support tool. Attached Figure Description

[0048] Figure 1 This is a flowchart of the visual compensation-based auxiliary guidance method for berthing and unberthing of ships according to the present invention.

[0049] Figure 2 This is a schematic diagram of the distribution of calibration points in a real-world scenario according to the present invention.

[0050] Figure 3 This is a schematic diagram of the actual scenario of the world coordinate system establishment method and the preset points of the ship in this invention.

[0051] Figure 4 This is a three-view drawing of the camera installation position and the preset points on the ship according to the present invention.

[0052] Figure 5 This is a flowchart of the optical flow tracing method for tracking the shoreline according to the present invention.

[0053] Figure 6 This is a schematic diagram of the visual output interface of the present invention.

[0054] Figure 7 This is a schematic diagram of the hardware deployment of the ship berthing and unberthing auxiliary guidance system based on visual compensation according to the present invention. Detailed Implementation

[0055] The present invention will now be described with reference to the accompanying drawings.

[0056] This invention relates to a visual compensation-based auxiliary guidance method for ship berthing and unberthing. By employing monocular vision and optical flow tracking compensation technology, it addresses the problems of high hardware costs, lack of physical scale information, and poor anti-interference capabilities in traditional berthing and unberthing schemes. In terms of calibration and spatial mapping, by establishing a world coordinate system and calculating the homography matrix, a precise mapping relationship between image pixels and the physical plane of the dock surface is constructed, enabling the output of physical quantities such as distance and angle parameters. This fundamentally overcomes the deficiency of traditional visual methods, which can only provide pixel-level data and cannot determine the true physical distance. Regarding visual reference input and tracking, the shoreline reference segment and flag point are calibrated as initial references through user interaction. A feature point set is initialized near the shoreline, and inter-frame tracking is performed using optical flow tracking to calculate the global average offset of feature points, achieving accurate perception of image shake. In terms of anti-interference compensation, mismatched points are eliminated by setting a distance threshold, and the initial visual reference coordinates are inversely compensated based on the global average offset to obtain steady-state reference coordinates, effectively offsetting image shake interference caused by the six degrees of freedom motion of the ship. In terms of pose parameter calculation, the steady-state reference coordinates are mapped to the world coordinate system using the inverse homography matrix. Key physical parameters such as the distance between the bridge and the shore, the bow and shore, the stern and shore, the bow-to-shore angle, and the perpendicular distance between the flag point and the bridge are calculated and visualized, generating a berthing / unberthing auxiliary guidance interface. This invention, through the organic combination of monocular vision and optical flow tracking compensation technology, and a scientifically sound coordinate mapping and inverse compensation mechanism, can achieve high-precision, highly interference-resistant berthing / unberthing status perception and guidance at extremely low hardware costs, providing strong decision support for improving the safety and efficiency of ship berthing / unberthing operations. The flowchart of this method is as follows: Figure 1 As shown, the steps are as follows:

[0057] I. Calibration and Spatial Mapping Steps: A world coordinate system is established with the vertical projection of a fixed reference point on the ship's bridge as the origin, the direction parallel to the shoreline as the X-axis, the direction perpendicular to the shoreline pointing towards the ship as the Y-axis, and the ship's bridge perpendicular to the shoreline and upwards as the Z-axis. Multiple calibration points are selected on the shore, and the physical coordinates of each calibration point in the world coordinate system are measured. Images of each calibration point are then acquired using cameras pre-installed on the ship, and the pixel coordinates of each calibration point in the image coordinate system are obtained from the images. Based on the physical coordinates and pixel coordinates, the homography matrix mapping the world coordinate system to the image coordinate system is calculated using the least squares method. Simultaneously, the physical coordinates of the pre-set fixed reference point on the ship's bridge (referred to as the bridge point), the bow point, and the stern point in the world coordinate system are measured.

[0058] This step aims to establish a precise forward mapping between the world coordinate system and the image coordinate system. Based on the obtained homography matrix, it lays the foundation for subsequent inverse transformation of pixel locations in the image into physical coordinates in the world coordinate system. Specifically, before the ship begins berthing operations, a stable physical point near the bridge side window is selected as a reference point (fixed reference point), and its vertical projection point is the origin O of the world coordinate system. The positive X-axis is defined as parallel to the dock shoreline to the right, the positive Y-axis as perpendicular to the shoreline pointing towards the sea (i.e., the ship's berthing direction), and the Z-axis is vertically upward according to the right-hand rule, thus constructing a three-dimensional world coordinate system.

[0059] Subsequently, at least four distinct, non-collinear feature points (such as mooring bollards or quay crane reference points) are selected on the same shoreline as calibration points. These calibration points should be evenly distributed and cover the camera's field of view. The reason for requiring at least four calibration points is that the homography matrix has eight degrees of freedom (a 3×3 matrix, with the last element normalized to 1). Each calibration point provides two equations, requiring at least four points to solve the homography matrix. Then, a total station or laser rangefinder is used to accurately measure its physical coordinates in the XOY plane of the world coordinate system, such as... Figure 2 As shown, the physical coordinates of each calibration point in the XOY plane of the world coordinate system are measured and denoted as follows: Next, berth scene images containing the aforementioned calibration points are acquired using cameras installed on the ship. Sub-pixel-level edge detection algorithms are then used to extract the pixel coordinates of each calibration point in the image coordinate system from the berth scene images, denoted as... .

[0060] like Figure 3 A schematic diagram illustrating the actual scenario of establishing a world coordinate system and pre-set ship positions. Figure 4 The three-view diagram shows the camera installation location and the preset points on the ship. Figure 4 a is a top view, used to show the positional distribution of the ship's bridge reference point p1, bow reference point p2, and stern reference point p3 along the ship's length, as well as the installation orientation of the cameras; Figure 4 b is the front view, used to show the installation height of the camera relative to the ship's deck and the layout of the three preset points on the water surface; Figure 4 c is a side view, used to show the camera's field of view coverage and the camera's monitoring angle of the berth area when the ship is berthed. For example... Figure 3 and Figure 4 As shown, the camera is installed near the front centerline of the ship's bridge roof, with its field of view facing the berth area. Its purpose is to monitor the overall relative position of the ship to the berth area, providing a global perspective to capture images of the berth scene, ensuring the ship approaches the berth at the correct angle, and ensuring the image is parallel to the ship's side. The camera's field of view is 100 degrees horizontally and 60 degrees vertically.

[0061] Finally, based on the physical coordinates of each calibration point and pixel coordinates The homography matrix mapping the world coordinate system to the image coordinate system was calculated using the least squares method. Among them, the homography matrix It is a 3×3 matrix used to describe the mapping relationship between the world coordinate system XOY plane and the image coordinate system plane. Its homogeneous coordinate formula is:

[0062]

[0063] in, It is a non-zero arbitrary scale factor. Homography matrix This can be specifically expressed as:

[0064]

[0065] Therefore, the specific solution formula expands to:

[0066] =

[0067] In practical calculations, in order to eliminate the scale factor Influence and solve the matrix The eight independent parameters (usually set to 8) Based on the physical coordinates of at least 4 calibration points and pixel coordinates The correspondences between these equations are used to construct an overdetermined system of equations. This invention employs the least squares method to solve this overdetermined system of equations, thereby obtaining the optimal homography matrix. This matrix establishes the forward geometric mapping from physical coordinate points in the XOY plane of the world coordinate system to image pixels. Subsequently, the inverse matrix can be used to convert the pixel positions in the image into physical coordinates within the wharf shoreline, and then, combined with multi-frame time-series data, the actual physical distance changes of ships relative to the shoreline can be calculated.

[0068] In completing the homography matrix After calibration, the preset bridge point p1 was measured. ), bow point p2 ( ), stern point p3 ( The physical coordinates in the XOY plane of the world coordinate system (e.g.) Figure 4 (As shown in the top view).

[0069] II. Visual reference coordinate input steps: Obtain the pixel coordinates of the two endpoints of the shoreline reference line segment marked by the user on the current frame video interface captured by the camera in real time, as well as the single pixel coordinates of the shoreline indicator flag point, as the initial visual reference coordinates.

[0070] This step aims to provide an input benchmark for visual computing, allowing the system to adapt to different berth scenarios through user interaction. Specifically: the system provides a graphical user interface (such as a touchscreen) that prompts the pilot to mark key reference lines on the current video screen at the start of berthing or unberthing. The pilot stops a frame in the live video feed captured by the camera (i.e., the current frame video screen), and then uses the mouse to click on the current frame video screen to determine the two endpoints of the shoreline benchmark segment. The system records the pixel coordinates of these two endpoints. )and( The baseline segment of the shoreline (i.e., the shoreline position in the current image) should be parallel to the wharf shoreline and located on a prominent fixed structure. Then, use the mouse to click and select a prominent static feature point on the shoreline as the shoreline indicator flag point. The system records the pixel coordinates of this indicator flag point. Flag points are generally located on the shoreline and are used to help correct minor rotational deviations in the image.

[0071] Finally, the pixel coordinates of the two endpoints of the shoreline baseline segment and the single pixel coordinates of the shoreline indicator flag point are used as the initial visual reference coordinates. These reference coordinates remain unchanged throughout the berthing and unberthing process, and subsequent steps are based on these coordinates for compensation. Since the shoreline and flag point positions differ for different berths, the system adapts to different scenarios through user interaction, eliminating the need for pre-calibration for each berth, thus improving the system's versatility and ease of deployment.

[0072] III. Optical Flow Tracking Steps: Using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points, construct a rectangular search region based on shoreline constraints in the current frame video interface. Initialize a set of feature points only within the rectangular search region, and retain only feature points whose distance from the shoreline is less than a preset first distance threshold as the initial feature point set. Then, use the optical flow tracking method to perform inter-frame tracking on the initial feature point set to obtain the positions of each feature point retained in the adjacent subsequent frame video interface of the current frame.

[0073] This step aims to estimate the global motion offset of the image using optical flow tracing, providing a basis for subsequent inverse compensation. Specifically, such as... Figure 5As shown, firstly, a rectangular area is enclosed by the pixel coordinates of the two endpoints of the shoreline baseline segment (shoreline) as the lower left and upper right points. That is, using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points, the minimum x-coordinate of the two endpoints is taken as the left boundary of the rectangle, the maximum x-coordinate as the right boundary, the minimum y-coordinate as the lower boundary, and the maximum y-coordinate as the upper boundary, forming a rectangular search area (Box). This rectangular area serves as the initial shoreline proximity region.

[0074] Then, a corner detection algorithm (such as Shi-Tomasi corner detection) is used to initialize a set of feature points within the rectangular search area. Feature points are pixels in the image with obvious texture, corners, edges, etc., such as the corners of fixed facilities like mooring bollards and crash pads on the shoreline. This set of feature points is further filtered, retaining only those with a distance from the shoreline less than a preset first distance threshold (e.g., 10 pixels) as the initial feature point set. This reduces the number of feature points and improves computation speed; simultaneously, it focuses only on points moving in sync with the shoreline, avoiding interference from moving parts of the ship itself (such as cranes and cables) or reflections from water waves.

[0075] Next, in each frame, optical flow tracing is used to perform inter-frame tracking (i.e., tracing the movement trajectory of these feature points) on the initialized feature point set. Preferably, the Lucas-Kanade optical flow tracing method is used. Its basic principle is to assume that the pixel brightness of the same feature point remains unchanged between two adjacent frames, and the position change is continuous. The motion vector of the feature point (the pixel displacement vector of each feature point) is obtained by solving the optical flow equation. For each feature point in the initialized feature point set, its corresponding position in the next frame (adjacent subsequent frame) of the video interface is found, and the positions of each feature point retained in the next frame (adjacent subsequent frame) of the video interface are obtained.

[0076] IV. Reverse Compensation Steps: Calculate the vertical distance from each tracked feature point in the adjacent subsequent video interface of the current frame to the shoreline baseline segment of the current frame. If the vertical distance is greater than or equal to a preset second distance threshold, it is determined to be a mismatched point and is removed. Calculate the global average offset of the remaining feature points between adjacent frames after removal, and perform reverse coordinate translation compensation on the initial visual reference coordinates based on the global average offset to obtain the pixel coordinates of the two endpoints of the compensated shoreline baseline segment and the single pixel coordinates of the compensated shoreline indicator flag point, which are used as the steady-state reference coordinates.

[0077] This step aims to eliminate image jitter interference caused by ship swaying through reverse compensation, and to ensure the continuity and stability of tracking through mismatch point elimination and re-initialization mechanisms. Specifically, firstly, the vertical distance from each tracked feature point in the video interface of the next frame (adjacent subsequent frame) to the shoreline baseline segment of the current frame is calculated. If this vertical distance is greater than or equal to a preset second distance threshold (e.g., 10 pixels), the feature point is determined to be a mismatch point and is eliminated. Only feature points with high consistency with shoreline movement (i.e., feature points with a vertical distance < 10 pixels) are retained, and interference points caused by the following reasons are eliminated: movement of the ship's own moving parts, reflection of water waves, occlusion by other moving objects, and points with large optical flow tracking errors.

[0078] Then, for the remaining feature points after removal, the displacement vector of each feature point between adjacent frames is calculated. The displacement vector is calculated as: Current frame position - Previous frame position. The average value of all displacement vectors is then calculated as follows:

[0079]

[0080]

[0081] in, This represents the number of feature points after filtering. , The value represents the global average offset of the remaining feature points between adjacent frames after removal. Since the swaying of the ship causes an overall displacement of the entire image, and the movement trend of each feature point is consistent, using the global average offset can effectively eliminate local noise.

[0082] Finally, due to the ship's swaying, the camera moves with the ship, causing the position of the shoreline in the frame to constantly change, but the shoreline's position in the physical world is fixed. Therefore, it is necessary to eliminate the influence of camera movement and restore the true position of the shoreline in a stable coordinate system. Let the initial position of the shoreline in the frame be P. Due to the ship's swaying, the camera in the current frame produces ( , The displacement of ); in the image, the shoreline appears to have moved to P + ( , To obtain the true position of the shoreline in a stable coordinate system, the motion component needs to be subtracted from the observed position. The compensation formula is as follows:

[0083] P_comp = P - ( , )

[0084] And perform reverse coordinate translation compensation on the initial visual reference coordinates, that is, perform reverse coordinate translation compensation on the two endpoint pixel coordinates of the initially calibrated shoreline reference line segment (shoreline). )and( ), and the pixel coordinates of the shoreline indicator flags ( Perform the above reverse compensation respectively. The pixel coordinates of the first endpoint of the compensated shoreline baseline segment are ( _comp, _comp) is:

[0085] _comp = -

[0086] _comp = -

[0087] The second endpoint pixel coordinates of the compensated shoreline baseline segment ( _comp, _comp) is:

[0088] _comp = -

[0089] _comp = -

[0090] Compensated pixel coordinates of the shoreline indicator flag ( _comp, _comp) is:

[0091] _comp = -

[0092] _comp= -

[0093] Finally, the pixel coordinates of the two endpoints of the compensated shoreline baseline segment and the single pixel coordinates of the compensated shoreline indicator flag point are used as the steady-state reference coordinates.

[0094] Preferably, the number of remaining feature points after removal is monitored in real time. When the number of remaining feature points after removal is determined to be lower than a preset safety point threshold T (e.g., T=20), it is determined that the feature point loss is severe, possibly due to large waves or sudden changes in illumination, triggering a re-initialization mechanism. The re-initialization mechanism is as follows: return to the optical flow tracking step, where the optical flow tracking step uses the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points to reconstruct a rectangular search area based on shoreline constraints in the current frame video interface, and extracts a new set of feature points within the reconstructed rectangular search area to replace the original initialized feature point set. Then, the inter-frame tracking operation continues to be performed, thereby maintaining the continuity of optical flow tracking. This mechanism effectively solves the problem of tracking failure caused by feature point loss, ensures the stability of the system during long-term operation, and achieves seamless recovery of the tracking state.

[0095] V. Calculation Steps for Ship Attitude Parameters: Using the inverse matrix of the homography matrix, the steady-state reference coordinates are transformed to the world coordinate system to obtain the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point; the shoreline straight line equation is established based on the world coordinates of the two endpoints of the shoreline reference segment; the vertical distances from the physical coordinates of the preset fixed reference point of the ship's bridge (referred to as the bridge point), the bow point, and the stern point to the shoreline straight line equation are calculated respectively to obtain the distances between the ship's bridge and the shore, the bow point and the shore, and the stern point and the shore; the ship's hull line is determined based on the physical coordinates of the bow point and the stern point, and the angle between the ship's hull line and the shoreline reference segment is calculated to obtain the bow-to-shore angle; a perpendicular line to the shoreline reference segment is drawn with the world coordinates of the shoreline indicator flag point as the center point, and the distance between this perpendicular line and the straight line passing through the bridge point and perpendicular to the shoreline is calculated to obtain the vertical distance between the flag point and the ship's bridge.

[0096] This step aims to convert the compensated image coordinates into world coordinates with real physical meaning and to calculate key physical values. Specifically, it first utilizes the inverse matrix of the homography matrix calculated in the calibration and spatial mapping steps. By mapping the steady-state reference coordinates to the XOY plane of the world coordinate system, the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point are obtained. This is achieved using the inverse matrix of the homography matrix. The pixel coordinates of the first endpoint of the compensated shoreline baseline segment ( _comp, Converting _comp to world coordinates uses the following formula:

[0097]

[0098] The specific calculation steps are as follows: First, calculate the intermediate value: Then normalize: X = X' / W', Y = Y' / W'. This gives the world coordinates (X_line1, Y_line1) and (X_line2, Y_line2) of the two endpoints of the shoreline baseline segment, as well as the world coordinates (X_flag, Y_flag) of the shoreline indicator flag point.

[0099] Then, the equation of the shoreline straight line is established based on the world coordinates of the two endpoints of the shoreline baseline segment. First, the slope k of the straight line is calculated, as shown in the following formula:

[0100] k = (Y_line2 - Y_line1) / (X_line2 - X_line1)

[0101] Next, calculate the line intercept, as shown in the following formula:

[0102] b = Y_line1 - k · X_line1

[0103] Thus, we obtain the equation of the shoreline straight line: Y = k·X + b, or in general form: Ax + By + C = 0, where A = k, B = -1, and C = b.

[0104] Next, calculate the pre-set bridge point ( ), bow point ( ), stern point ( The perpendicular distance from the point to the shoreline is calculated using the point-to-line distance formula shown below:

[0105]

[0106] This yields the distances between the bridge and the shore, D_bridge, the bow and the shore, D_bow, and the stern and the shore, respectively (unit: meters).

[0107] Then based on the bow point ( ) and stern point ( Determine the ship's sideline and calculate the angle between it and the shoreline baseline segment, using the formula for the angle between two straight lines as shown below:

[0108] θ = arctan(|k_ship - k_line| / |1 + k_ship·k_line|)

[0109] Where k_ship is the slope of the ship's side line. k_line is the slope of the shoreline, k_line = k;

[0110] Then, convert the radians θ to degrees: θ_deg = θ×(180 / π), to obtain the angle θ_deg between the bow and the shore (unit: degrees). A positive value of this angle θ_deg indicates that the bow is facing the shoreline, and a negative value indicates that the bow is moving away from the shoreline.

[0111] Finally, using the world coordinates (X_flag, Y_flag) of the shoreline indicator flag point as the center point, draw a perpendicular line to the shoreline baseline segment, and calculate the distance between this perpendicular line and the straight line passing through the bridge point and perpendicular to the shoreline. The specific calculation method is as follows: first calculate the foot coordinates (X_foot_bridge, Y_foot_bridge) from the bridge point to the shoreline and the foot coordinates (X_foot_flag, Y_foot_flag) from the flag point to the shoreline. The formula for calculating the foot coordinates is:

[0112]

[0113]

[0114] Then calculate the distance between the two points:

[0115] D_flag =

[0116] Obtain the vertical distance D_flag (in meters) between the flag point and the bridge. This distance reflects the lateral offset of the flag point relative to the bridge and is used to assist in ship alignment.

[0117] Preferably, the ship's lateral velocity is calculated based on any one of the following distances corresponding to two consecutive frames: the distance between the ship's bridge and the shore, the distance between the ship's bow and the shore, and the distance between the ship's stern and the shore. This velocity is then output in the subsequent visualization output step. For example, the distance from a point to the shore calculated in the previous frame (such as the distance between the ship's bow and the shore, denoted as D_bow_prev), the distance between the ship's bow and the shore in the current frame, denoted as D_bow_current, and the time difference between the two frames, denoted as Δt, is usually the reciprocal of the camera's frame rate, such as 1 / 30 of a second.

[0118] The ship's lateral speed V is calculated using the following formula:

[0119] V = (D_bow_current - D_bow_prev) / Δt

[0120] The speed is then output in centimeters per second in the subsequent visualization output step. The ship's lateral speed reflects the rate at which the ship approaches or moves away from the shoreline, providing a quantitative basis for the crew to control the ship's speed.

[0121] VI. Visualization Output Steps: (e.g.) Figure 6As shown, the distance between the ship's bridge and the shore, the distance between the bow and the shore (i.e., the distance between the bow and the shore is 6.84m, and the lateral velocity is 85.14cm / s), the distance between the stern and the shore (i.e., the distance between the bow and the shore is 13.35m, and the lateral velocity is -66.33cm / s), the angle between the bow and the shore (5.17°), and the vertical distance between the flag point and the ship's bridge (10.56m) are used as berthing and unberthing state parameters and visualized to generate a ship berthing and unberthing auxiliary guidance interface to characterize the real-time position and attitude of the ship during the berthing and unberthing process.

[0122] This top-down visualization method eliminates the need for crew members to perform complex distance estimations and angle judgments. They can accurately determine the ship's relative position to the shoreline simply by observing the numerical values ​​and graphics on the screen. This includes crucial information such as the distance between the bow and stern, whether the ship is parallel to the shoreline, the lateral offset of the flag point relative to the bridge, and the ship's speed as it approaches the shore. This significantly improves the safety and decision-making efficiency of berthing and unberthing operations, upgrading these processes from experience-based judgment to data-driven decision-making.

[0123] This invention also relates to a visual compensation-based auxiliary guidance system for berthing and unberthing of ships. This system corresponds to the aforementioned visual compensation-based auxiliary guidance method for berthing and unberthing of ships, and can be understood as a system that implements the aforementioned method, such as... Figure 7 As shown, the entire set of hardware equipment is installed and deployed on the ship.

[0124] The system hardware mainly consists of a camera, a network switch, a server, and a client, connected via a ship's local area network. The camera, serving as a ship-side view camera, is installed near the front centerline of the ship's bridge canopy, with its field of view facing the berth ahead. The camera's horizontal field of view is preferably 100 degrees, and its vertical field of view is preferably 60 degrees. The camera captures real-time video streams of the berth scene and transmits these streams to the server via the network switch. This server is an algorithm server, internally carrying the various software functional modules of this invention, specifically including: a calibration and spatial mapping module, a visual reference coordinate input module, an optical flow tracking module, and an inverse compensation module. The system includes a module for calculating ship pose parameters and a visualization output module. After receiving the video stream, the server uses the aforementioned software modules to perform coordinate calibration, feature tracking, visual compensation, pose parameter calculation, and visualization rendering, generating a video processing result that includes berthing and departure guidance parameters. The server then sends the processed visualization result to the client via a network switch. The client, also known as a client workstation, is where operators perform user input operations such as marking shoreline endpoints and shoreline indicator flags. The corresponding user input commands are transmitted back to the server via the network switch, enabling bidirectional data interaction between the server and the client. Specifically,

[0125] The calibration and spatial mapping module establishes a world coordinate system with the vertical projection of a fixed reference point on the ship's bridge as the origin, the direction parallel to the shoreline as the X-axis, the direction perpendicular to the shoreline pointing towards the ship as the Y-axis, and the ship's bridge perpendicular to the shoreline and upward as the Z-axis. Multiple calibration points are selected on the shore, and the physical coordinates of each calibration point in the world coordinate system are measured. Images of each calibration point are then acquired using cameras pre-installed on the ship, and the pixel coordinates of each calibration point in the image coordinate system are obtained from the images. Based on the physical coordinates and pixel coordinates, the homography matrix mapping the world coordinate system to the image coordinate system is calculated using the least squares method. Simultaneously, the physical coordinates of the preset fixed reference point on the ship's bridge, the bow point, and the stern point in the world coordinate system are measured.

[0126] The visual reference coordinate input module obtains the pixel coordinates of the two endpoints of the shoreline reference line segment marked by the user on the current frame video interface captured by the camera in real time, as well as the single pixel coordinate of the shoreline indicator flag point, as the initial visual reference coordinates.

[0127] The optical flow tracking module constructs a rectangular search region based on shoreline constraints in the current frame video interface, using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. It initializes a set of feature points only within the rectangular search region and retains only feature points whose distance from the shoreline is less than a preset first distance threshold as the initial feature point set. Then, it uses the optical flow tracking method to perform inter-frame tracking on the initial feature point set to obtain the positions of each feature point retained in the adjacent subsequent frame video interface of the current frame.

[0128] The reverse compensation module calculates the vertical distance from each tracked feature point in the adjacent subsequent video interface of the current frame to the shoreline baseline segment of the current frame. If the vertical distance is greater than a preset second distance threshold, it is determined to be a mismatched point and is removed. The module calculates the global average offset of the remaining feature points between adjacent frames after removal, and performs reverse coordinate translation compensation on the initial visual reference coordinates based on the global average offset to obtain the pixel coordinates of the two endpoints of the compensated shoreline baseline segment and the single pixel coordinates of the compensated shoreline indicator flag point, which are used as the steady-state reference coordinates.

[0129] The ship attitude parameter calculation module uses the inverse matrix of the homography matrix to transform the steady-state reference coordinates to the world coordinate system, obtaining the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point; it establishes the shoreline straight line equation based on the world coordinates of the two endpoints of the shoreline reference segment; and calculates the perpendicular distances from the preset physical coordinates of the ship's bridge fixed reference point, bow point, and stern point to the shoreline straight line equation, obtaining the ship's bridge distance, bow distance, and stern distance; then, it determines the ship's hull line based on the physical coordinates of the bow and stern points, calculates the angle between the ship's hull line and the shoreline reference segment, obtaining the bow angle; and draws a perpendicular line to the shoreline reference segment with the world coordinates of the shoreline indicator flag point as the center point, calculating the distance between this perpendicular line and the straight line passing through the ship's bridge fixed reference point and perpendicular to the shoreline, obtaining the perpendicular distance between the flag point and the ship's bridge.

[0130] The visualization output module uses the distance between the ship's bridge and the shore, the distance between the bow and the shore, the distance between the stern and the shore, the angle between the bow and the shore, and the vertical distance between the flag point and the ship's bridge as berthing and unberthing status parameters and outputs them in a visualization manner to generate a ship berthing and unberthing auxiliary guidance interface to characterize the real-time position and attitude of the ship during the berthing and unberthing process.

[0131] Preferably, in the reverse compensation module, the number of remaining feature points after removing mismatched points is also determined. When the number of remaining feature points after removal is determined to be lower than a preset safety point threshold, a re-initialization mechanism is triggered. The re-initialization mechanism is as follows: return to the optical flow tracking module, and the optical flow tracking module reconstructs a rectangular search area based on the shoreline constraint in the current frame video interface using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. A new set of feature points is extracted in the reconstructed rectangular search area to replace the original initialized feature point set. Then, the inter-frame tracking operation is continued to maintain the continuity of optical flow tracking.

[0132] Preferably, in the optical flow tracking module, the Lucas-Kanade optical flow tracking method is used to perform inter-frame tracking on the initialized feature point set.

[0133] Preferably, in the ship attitude parameter calculation module, the ship's lateral velocity is calculated based on any one of the following distances corresponding to two consecutive frames: the distance between the ship's bridge and the shore, the distance between the ship's bow and the shore, and the distance between the ship's stern and the shore, combined with the time difference between the two frames, and the ship's lateral velocity is output in the visualization output step.

[0134] Preferably, in the calibration and spatial mapping module, the plurality of calibration points are at least four shore calibration points; the camera is installed near the front centerline of the ship's bridge canopy, with the camera's field of view facing the berth ahead, and the camera's horizontal field of view is 100 degrees and its vertical field of view is 60 degrees.

[0135] In the visual reference coordinate input module, the shoreline indicator flag is located on the shoreline.

[0136] This invention provides an objective and scientific visual compensation-based auxiliary guidance method and system for ship berthing and unberthing. Through the organic coordination of six steps or functional modules—calibration and spatial mapping, visual reference coordinate input, optical flow tracking, inverse compensation, ship pose parameter calculation, and visualization output—a complete visual perception system from image acquisition to auxiliary guidance is constructed. First, the calibration and spatial mapping step / module establishes a world coordinate system and calculates the homography matrix, achieving precise mapping between image pixel coordinates and the physical coordinates of the dock surface. Then, the optical flow tracking and inverse compensation steps / modules achieve stable tracking against interference, solving the image jitter problem caused by ship swaying. Next, the ship pose parameter calculation step / module converts the compensated steady-state reference coordinates into standardized physical quantification parameters, solving the practical problem of not being able to quantify the output of berthing and unberthing states. Finally, the visualization output step / module generates an auxiliary guidance interface, providing crew members with an intuitive and operable decision support tool.

[0137] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail with reference to the accompanying drawings and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention. In short, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention patent.

Claims

1. A method for assisting ship berthing and unberthing guidance based on visual compensation, characterized in that, Includes the following steps: Calibration and spatial mapping steps: Establish a world coordinate system with the vertical projection of a fixed reference point on the ship's bridge as the origin, the direction parallel to the shoreline as the X-axis, the direction perpendicular to the shoreline pointing towards the ship as the Y-axis, and the direction perpendicular to the shoreline pointing upwards as the Z-axis. Multiple calibration points were selected on shore, and the physical coordinates of each calibration point in the world coordinate system were measured. Images of each calibration point were then acquired using cameras pre-installed on the ship, and the pixel coordinates of each calibration point in the image coordinate system were obtained from the images. Based on the physical coordinates and pixel coordinates, the homography matrix mapping the world coordinate system to the image coordinate system was calculated using the least squares method. At the same time, the physical coordinates of the preset fixed reference point of the ship's bridge, the bow point, and the stern point in the world coordinate system were measured. Visual reference coordinate input steps: Obtain the pixel coordinates of the two endpoints of the shoreline reference line segment marked by the user on the current frame video interface captured by the camera in real time, as well as the single pixel coordinates of the shoreline indicator flag point, as the initial visual reference coordinates. Optical flow tracking steps: Using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points, construct a rectangular search region based on shoreline constraints in the current frame video interface, initialize a set of feature points only within the rectangular search region, and retain only feature points whose distance from the shoreline is less than a preset first distance threshold as the initial feature point set; Then, optical flow tracing is used to perform inter-frame tracking on the initialized feature point set to obtain the positions of each feature point retained in the video interface of the adjacent subsequent frames of the current frame; Reverse compensation step: Calculate the vertical distance from each tracked feature point in the adjacent subsequent frame video interface of the current frame to the shoreline baseline segment of the current frame. If the vertical distance is greater than the preset second distance threshold, it is determined to be a mismatched point and is removed. Calculate the global average offset of the remaining feature points between adjacent frames after removal, and perform reverse coordinate translation compensation on the initial visual reference coordinates based on the global average offset to obtain the pixel coordinates of the two endpoints of the compensated shoreline reference line segment and the single pixel coordinates of the compensated shoreline indicator flag point, which are used as the steady-state reference coordinates. The steps for calculating ship attitude parameters are as follows: Using the inverse of the homography matrix, the steady-state reference coordinates are transformed to the world coordinate system to obtain the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point; a shoreline straight line equation is established based on the world coordinates of the two endpoints of the shoreline reference segment; the vertical distances from the preset physical coordinates of the ship's bridge fixed reference point, bow point, and stern point to the shoreline straight line equation are calculated to obtain the ship's bridge distance, bow distance, and stern distance; the ship's hull line is then determined based on the physical coordinates of the bow and stern points, and the angle between the hull line and the shoreline reference segment is calculated to obtain the bow-to-shore angle; a perpendicular line is drawn from the world coordinates of the shoreline indicator flag point to the shoreline reference segment, and the distance between this perpendicular line and the straight line passing through the ship's bridge fixed reference point and perpendicular to the shoreline is calculated to obtain the vertical distance between the flag point and the ship's bridge. Visualization output steps: The distance between the ship's bridge and the shore, the distance between the bow and the shore, the distance between the stern and the shore, the angle between the bow and the shore, and the vertical distance between the flag point and the ship's bridge are used as berthing and unberthing state parameters and visualized output to generate a ship berthing and unberthing auxiliary guidance interface to represent the real-time position and attitude of the ship during the berthing and unberthing process.

2. The visual compensation-based auxiliary guidance method for ship berthing and unberthing according to claim 1, characterized in that, In the reverse compensation step, the number of remaining feature points after removing mismatched points is also determined. When the number of remaining feature points after removal is lower than the preset safety point threshold, a re-initialization mechanism is triggered. The re-initialization mechanism is as follows: return to the optical flow tracking step, and the optical flow tracking step reconstructs a rectangular search area based on the shoreline constraint in the current frame video interface using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. A new set of feature points is extracted in the reconstructed rectangular search area to replace the original initialized feature point set. Then, the inter-frame tracking operation is continued to maintain the continuity of optical flow tracking.

3. The visual compensation-based auxiliary guidance method for ship berthing and unberthing according to claim 1, characterized in that, In the optical flow tracking step, the Lucas-Kanade optical flow tracking method is used to perform inter-frame tracking on the initialized feature point set.

4. The visual compensation-based auxiliary guidance method for ship berthing and unberthing according to claim 1 or 2, characterized in that, In the ship attitude parameter calculation step, the ship's lateral velocity is calculated based on any one of the following distances in two consecutive frames: the distance between the ship's bridge and the shore, the distance between the ship's bow and the shore, and the distance between the ship's stern and the shore, combined with the time difference between the two frames. The ship's lateral velocity is then output in the visualization output step.

5. The visual compensation-based auxiliary guidance method for ship berthing and unberthing according to claim 1 or 2, characterized in that, In the calibration and spatial mapping steps, the plurality of calibration points are at least four shore calibration points; the camera is installed near the front centerline of the ship's bridge canopy, with the camera's field of view facing the berth ahead, and the camera's horizontal field of view is 100 degrees and its vertical field of view is 60 degrees. In the visual reference coordinate input step, the shoreline indicator flag point is located on the shoreline.

6. A ship berthing and unberthing auxiliary guidance system based on vision compensation, characterized in that, It includes, in sequence, a calibration and spatial mapping module, a visual reference coordinate input module, an optical flow tracking module, an inverse compensation module, a ship pose parameter calculation module, and a visualization output module. The calibration and spatial mapping module establishes a world coordinate system with the vertical projection of a fixed reference point on the ship's bridge as the origin, the direction parallel to the shoreline as the X-axis, the direction perpendicular to the shoreline and pointing towards the ship as the Y-axis, and the ship's bridge perpendicular to the shoreline and pointing upwards as the Z-axis. Multiple calibration points were selected on shore, and the physical coordinates of each calibration point in the world coordinate system were measured. Images of each calibration point were then acquired using cameras pre-installed on the ship, and the pixel coordinates of each calibration point in the image coordinate system were obtained from the images. Based on the physical coordinates and pixel coordinates, the homography matrix mapping the world coordinate system to the image coordinate system was calculated using the least squares method. At the same time, the physical coordinates of the preset fixed reference point of the ship's bridge, the bow point, and the stern point in the world coordinate system were measured. The visual reference coordinate input module obtains the pixel coordinates of the two endpoints of the shoreline reference line segment marked by the user on the current frame video interface captured by the camera in real time, as well as the single pixel coordinate of the shoreline indicator flag point, as the initial visual reference coordinates. The optical flow tracking module constructs a rectangular search region based on shoreline constraints in the current frame video interface, using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. It initializes a set of feature points only within the rectangular search region and retains only feature points whose distance from the shoreline is less than a preset first distance threshold as the initial feature point set. Then, optical flow tracing is used to perform inter-frame tracking on the initialized feature point set to obtain the positions of each feature point retained in the video interface of the adjacent subsequent frames of the current frame; The reverse compensation module calculates the vertical distance from each tracked feature point in the adjacent subsequent video interface of the current frame to the shoreline baseline segment of the current frame. If the vertical distance is greater than a preset second distance threshold, it is determined to be a mismatched point and is removed. Calculate the global average offset of the remaining feature points between adjacent frames after removal, and perform reverse coordinate translation compensation on the initial visual reference coordinates based on the global average offset to obtain the pixel coordinates of the two endpoints of the compensated shoreline reference line segment and the single pixel coordinates of the compensated shoreline indicator flag point, which are used as the steady-state reference coordinates. The ship attitude parameter calculation module uses the inverse matrix of the homography matrix to transform the steady-state reference coordinates to the world coordinate system, obtaining the world coordinates of the two endpoints of the shoreline reference segment and the world coordinates of the shoreline indicator flag point; it establishes the shoreline straight line equation based on the world coordinates of the two endpoints of the shoreline reference segment; and calculates the perpendicular distances from the preset physical coordinates of the ship's bridge fixed reference point, bow point, and stern point to the shoreline straight line equation, obtaining the ship's bridge distance, bow distance, and stern distance; then, it determines the ship's hull line based on the physical coordinates of the bow and stern points, calculates the angle between the ship's hull line and the shoreline reference segment, obtaining the bow angle; and draws a perpendicular line to the shoreline reference segment with the world coordinates of the shoreline indicator flag point as the center point, calculating the distance between this perpendicular line and the straight line passing through the ship's bridge fixed reference point and perpendicular to the shoreline, obtaining the perpendicular distance between the flag point and the ship's bridge. The visualization output module uses the distance between the ship's bridge and the shore, the distance between the bow and the shore, the distance between the stern and the shore, the angle between the bow and the shore, and the vertical distance between the flag point and the ship's bridge as berthing and unberthing status parameters and outputs them in a visualization manner to generate a ship berthing and unberthing auxiliary guidance interface to characterize the real-time position and attitude of the ship during the berthing and unberthing process.

7. The visual compensation-based ship berthing and unberthing auxiliary guidance system according to claim 6, characterized in that, In the reverse compensation module, the number of remaining feature points after removing mismatched points is also determined. When the number of remaining feature points after removal is lower than the preset safety point threshold, a re-initialization mechanism is triggered. The re-initialization mechanism is as follows: return to the optical flow tracking module, and the optical flow tracking module reconstructs a rectangular search area based on the shoreline constraint in the current frame video interface using the pixel coordinates of the two endpoints of the shoreline baseline segment as diagonal points. A new set of feature points is extracted in the reconstructed rectangular search area to replace the original initialized feature point set. Then, the inter-frame tracking operation is continued to maintain the continuity of optical flow tracking.

8. The visual compensation-based ship berthing and unberthing auxiliary guidance system according to claim 6, characterized in that, In the optical flow tracking module, the Lucas-Kanade optical flow tracking method is used to perform inter-frame tracking on the initialized feature point set.

9. The visual compensation-based ship berthing and unberthing auxiliary guidance system according to claim 6 or 7, characterized in that, The ship attitude parameter calculation module also calculates the ship's lateral velocity based on any one of the following distances in two consecutive frames: the distance between the ship's bridge and the shore, the distance between the ship's bow and the shore, and the distance between the ship's stern and the shore, combined with the time difference between the two frames, and outputs the ship's lateral velocity in the visualization output step.

10. The visual compensation-based ship berthing and unberthing auxiliary guidance system according to claim 6 or 7, characterized in that, In the calibration and spatial mapping module, the plurality of calibration points are at least four shore calibration points; the camera is installed near the front centerline of the ship's bridge canopy, with the camera's field of view facing the berth ahead, and the camera's horizontal field of view is 100 degrees and its vertical field of view is 60 degrees. In the visual reference coordinate input module, the shoreline indicator flag is located on the shoreline.

Images