Unmanned ship autonomous berthing control method based on visual identification and related equipment

Abstract

The embodiment of the invention provides an unmanned ship autonomous berthing control method based on visual identification and related equipment, and belongs to the technical field of unmanned ship intelligent control. The method comprises the following steps: acquiring pose information of a parking marker through visual identification, wherein the pose information comprises a marker position and a marker pose; determining a target pose according to the pose information and the safe distance from the shoreline, and updating the target pose to a memory variable; and the target pose is read from the memory variables, and path planning and path tracking control are performed on the unmanned ship according to the target pose, so that the unmanned ship berths towards the berth marker. According to the scheme, the robustness of the unmanned ship in the autonomous berthing process is improved on the whole.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology for unmanned vessels, and in particular to an autonomous berthing control method and related equipment for unmanned vessels based on visual recognition. Background Technology

[0002] With the rapid development of computer and communication technologies, unmanned surface vessels (USVs) are increasingly involved in various maritime operations. However, achieving safe and reliable autonomous berthing remains a key bottleneck restricting their comprehensive intelligent upgrade. Currently, USV berthing mainly relies on manual remote control or single GNSS navigation to approach the target shoreline. These technologies have significant drawbacks: the former suffers from high labor costs, insufficient control precision, and susceptibility to collisions and damage due to human error; the latter suffers from large positioning errors, susceptibility to signal blockage, and weak dynamic obstacle avoidance capabilities. In recent years, some research has attempted to introduce visual or lidar-assisted navigation, but this typically relies on simple marker points to identify berth locations. In scenarios with multiple obstacles or narrow berths, it's difficult to accurately berth the USV at a designated location. Furthermore, during the USV's tracking of the target berth, target recognition is limited to the visual field, and system failures are prone to occur when the target is lost. Summary of the Invention

[0003] The main objective of this application is to propose a vision-based autonomous berthing control method and related equipment for unmanned vessels, aiming to improve the robustness of unmanned vessels during autonomous berthing.

[0004] To achieve the above objectives, one aspect of this application proposes a vision-based autonomous berthing control method for unmanned vessels, comprising the following steps: The position and orientation information of berth markers is obtained through visual recognition, and the position and orientation information includes the position and orientation of the markers. Based on the pose information and the safe distance from the shoreline, the target pose is determined and updated in the memory variables; The target pose is read from the memory variable, and the unmanned vessel is used for path planning and path tracking control based on the target pose so that the unmanned vessel can dock at the berth marker.

[0005] In some embodiments, the vision recognition-based unmanned vessel autonomous berthing control method further includes the following steps: Laser point cloud data of the environment is collected by lidar, and depth image point cloud data of the environment is collected by depth camera. Global loop closure detection is performed based on the current point cloud data and local map to correct perceived motion pose, and continuous pose state estimation is performed based on the continuous depth image point cloud data to obtain visual odometry data. Based on the visual odometry data, the laser point cloud data, and the depth image point cloud data, SLAM mapping and global loop closure detection are performed to obtain a global 3D map.

[0006] In some embodiments, obtaining the position and orientation information of berth markers through visual recognition includes the following steps: The target detection model is used to detect berth markers in the currently acquired image data to obtain the image detection results; Based on the image detection results, the pose of the berth marker in the camera coordinate system is calculated to obtain the pose calculation result. The pose calculation result is mapped onto the coordinate system of the global 3D map to obtain the pose information of the berth marker in the global 3D map.

[0007] In some embodiments, the memory variable is updated through the following steps: Determine whether a berth marker has been detected or whether the task status has changed; If a berth marker is detected, the target pose corresponding to the detected berth marker is updated in the memory variable; or, if the task state changes, the memory variable is changed according to the task endpoint indicated by the task state.

[0008] In some embodiments, the path planning and path tracking control of the unmanned vessel based on the target pose includes the following steps: Based on the current pose of the unmanned vessel and the pose of the target, a global path is obtained by performing global path planning on the global 3D map. The local 3D map in the global 3D map is updated based on the real-time collected point cloud data, and the global path is locally optimized based on the local 3D map to obtain the optimized path. The optimized path is subjected to path tracking control to obtain and apply ship control commands.

[0009] In some embodiments, the step of locally optimizing the global path based on the local 3D map to obtain an optimized path includes the following steps: Based on the obstacle information indicated by the local 3D map and the reference path indicated by the global path, multiple candidate motion trajectories are planned to obtain the motion trajectory; the candidate motion trajectory includes the motion speed. The candidate motion trajectory is scored using a multi-dimensional weighted scoring method to obtain a trajectory score; the dimensions of the trajectory score include target alignment, in-situ rotation capability, and motion oscillation, and the weight of the in-situ rotation capability dimension is dynamically adjusted according to the distance between the unmanned vessel and the berth marker; The global path is locally optimized based on the candidate motion trajectory with the highest score to obtain the optimized path.

[0010] In some embodiments, performing path tracking control on the optimized path to obtain and apply ship control commands includes the following steps: A linear velocity control loop for calculating the total thrust rotational speed is constructed based on the desired linear velocity indicated by the optimized path, and an angular velocity control loop for steering force is constructed based on the desired angular velocity indicated by the optimized path. The weighting coefficients for the rudder angle and the difference in thruster speed are dynamically allocated based on the current linear velocity of the unmanned vessel. The desired rudder angle and desired thruster speed difference are calculated based on the weight allocation result and the steering force calculated by the angular velocity control loop. The desired dual-side thruster speed is calculated based on the desired thruster speed difference and the total thrust speed calculated by the linear velocity control loop. Ship control commands are generated based on the desired rudder angle and the desired dual-side thruster speeds.

[0011] To achieve the above objectives, another aspect of this application proposes a vision-based autonomous berthing control system for unmanned vessels, comprising: The first module is used to obtain the position and pose information of berth markers through visual recognition, wherein the position and pose information includes the position and pose of the markers. The second module is used to determine the target pose based on the pose information and the safe distance from the shoreline, and update the target pose to the memory variables; The third module is used to read the target pose from the memory variable and perform path planning and path tracking control on the unmanned vessel based on the target pose, so that the unmanned vessel can dock at the berth marker.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0014] The embodiments of this application include at least the following beneficial effects: This application provides a vision-based autonomous berthing control method, system, electronic device, and program product for unmanned surface vessels (USVs). This scheme first acquires the pose information of berth markers through visual recognition, including the marker's position and orientation. Then, based on the pose information and the safe distance from the shoreline, the target pose is determined and updated in a memory variable. The target pose is then read from the memory variable, and path planning and path tracking control are performed on the USV based on the target pose to enable it to berth towards the berth marker. This scheme considers both the marker's position and orientation during berthing control, allowing the USV to accurately berth at the designated berth even in complex environments. Furthermore, this scheme uses a memory variable to store the identified target orientation, enabling continued berthing operations even if the target is lost. Overall, this scheme improves the robustness of the USV during autonomous berthing. Attached Figure Description

[0015] Figure 1 This is a flowchart of the vision recognition-based autonomous berthing control method for unmanned vessels provided in the embodiments of this application; Figure 2 This is a schematic diagram of a multi-source sensor sensing system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0018] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0019] SLAM (Simultaneous Localization and Mapping) can be described as follows: An agent is placed in an unknown location within an unknown environment. As the agent moves, it simultaneously perceives the environment and estimates its own position and orientation using sensors (such as LiDAR and cameras) (localization), while gradually building a map of the environment based on this observation data (mapping). These two processes are interdependent and occur concurrently; localization relies on map information, while building an accurate map requires the agent's precise location.

[0020] GNSS (Global Navigation Satellite System) is a general term for all satellite navigation and positioning systems that can provide global coverage. It is a space-based radio navigation and positioning system that can provide users with accurate three-dimensional coordinates, velocity, and time information at any location on the Earth's surface or in near-Earth space, 24 / 7. Its working principle is that user equipment receives signals from multiple satellites and uses these signals to calculate its own position.

[0021] An RGB-D camera, also known as a 3D depth camera, is an advanced sensor capable of simultaneously acquiring color images (RGB) and depth information of a scene.

[0022] An IMU (Inertial Measurement Unit) is a core sensor device used to measure the acceleration, angular velocity, and attitude (or angular rate) of an object in three-dimensional space.

[0023] The ICP (Iterative ClosestPoint) algorithm is a core algorithm for 3D point cloud registration. It achieves point cloud alignment and matching by iteratively finding the nearest point correspondence between two point clouds and calculating the optimal rigid body transformation (rotation and translation).

[0024] ORB feature extraction algorithm is an efficient feature point detection and descriptor generation algorithm with rotation and scale invariance. It achieves multi-scale detection by constructing an image pyramid, calculates the orientation of key points using the gray-scale centroid method, and finally generates binary rBRIEF descriptors for fast matching.

[0025] EKF (Extended Kalman Filter) is a state estimator for nonlinear systems. It performs point cloud data fusion by fusing data from different sensors and at different frequencies through two steps: prediction and update.

[0026] DWB (Dynamic Window Approach-Based) is a local path planning algorithm that, under the robot's current state and dynamic constraints, samples a set of candidate trajectories in the velocity space (composed of linear velocity v and angular velocity w) and selects the optimal trajectory through a multi-index evaluation function.

[0027] PID (Proportional-Integral-Derivative Controller) is an automatic control algorithm based on the principle of negative feedback. Its core idea is to correct system deviations by linearly combining the proportional (P), integral (I), and derivative (D) control actions of the controlled object.

[0028] This application provides a vision-based autonomous berthing control method and related equipment for unmanned vessels, which can improve the robustness of unmanned vessels during autonomous berthing.

[0029] The unmanned vessel autonomous berthing control method based on visual recognition provided in this application relates to the field of unmanned vessel intelligent control technology. This method can be applied to a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or shipborne terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] Figure 1 This is an optional flowchart of the vision recognition-based autonomous berthing control method for unmanned vessels provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S103.

[0032] S101, obtain the position and pose information of the berth markers through visual recognition, including the position and pose of the markers; S102, determine the target pose based on the pose information and the safe distance from the shoreline, and update the target pose to the memory variables; S103 reads the target pose from the memory variable and performs path planning and path tracking control on the unmanned vessel based on the target pose so that the unmanned vessel can dock at the berth marker.

[0033] In the steps S101 to S103 shown in this embodiment, in addition to considering the position of the marker, the attitude of the marker is also considered during the berthing process for berthing control. Even in complex environments, the unmanned vessel can be accurately berthed at the designated berth. This embodiment also sets a memory variable to store the identified target attitude, so that the berthing operation can continue even if the target is lost. This embodiment breaks through the adaptability bottleneck of traditional methods in complex environments and significantly improves the reliability, robustness and generalization ability of unmanned vessels' autonomous berthing.

[0034] In some embodiments, the autonomous berthing control method for unmanned surface vessels (USVs) according to this application can be applied to USVs equipped with a multi-source sensor perception system, a dynamic decision-making and planning system, and a motion control and execution system. The multi-source sensor perception system is used to acquire multi-source sensing data such as images and laser point clouds, and for 3D map construction, berth marker detection, and pose estimation. The dynamic decision-making and planning system is mainly used to continuously optimize the berthing path in real time based on the data analyzed by the multi-source sensing system. The motion control and execution system is used to convert the ship's pose sequence indicated by the path planned by the dynamic decision-making and planning system into actual ship motion commands, thereby enabling the ship to berth. Please refer to... Figure 2 The multi-source sensor perception system can be based on LiDAR, RGB-D camera and IMU. The system mainly includes three modules: multi-source data acquisition and synchronization module, high-precision fusion positioning and mapping module and visual berth detection and pose estimation module.

[0035] In some embodiments, the vision-based autonomous berthing control method for unmanned vessels according to this application may also include, but is not limited to, the following steps: S201 collects laser point cloud data in the environment using a lidar and depth image point cloud data in the environment using a depth camera. S202, global loop closure detection is performed based on the current point cloud data and local map to correct the perceived motion pose, and continuous pose state estimation is performed based on the continuous depth image point cloud data to obtain visual odometry data. S203 performs SLAM mapping and global loop closure detection based on visual odometry data, laser point cloud data, and depth image point cloud data to obtain a global 3D map.

[0036] Traditional sensor data lacks deep fusion, resulting in insufficient robustness under conditions such as water surface reflection and dynamic surges. This embodiment uses a tightly coupled LiDAR, RGB-D, and IMU to construct a scene map with dense point clouds via SLAM, and introduces visual odometry and ICP closed-loop detection. Even under interference such as GNSS rejection, water surface reflection, and surge impact, it can still stably build a scene map and maintain centimeter-level positioning accuracy.

[0037] In this embodiment, the multi-source data acquisition and synchronization module consists of a lidar, an RGB-D camera, and an IMU. It performs timestamp alignment and soft triggering before data output to achieve millisecond-level synchronization.

[0038] The high-precision fusion positioning and mapping module can perform SLAM mapping by fusing 2D LiDAR and RGB-D camera data, which are respectively fixed to the hull and bow of the ship. It fuses LiDAR point cloud data with RGB-D depth image point cloud data (including depth image data and RGB image data) to jointly construct a global 3D map. Furthermore, it reduces accumulated positioning errors by performing global loop closure detection using the ICP algorithm, which compares the current point cloud data with historically visited local maps. The current point cloud data can refer to LiDAR point cloud data, depth image point cloud data, or a fusion of both. The core registration formula of ICP is expressed as follows: ; in, It is a 3×3 rotation matrix, representing the rotational transformation relationship between the point cloud coordinate systems; A 3×1 translation vector represents the translation transformation between point cloud coordinate systems; and These are the corresponding 3D feature points in the target point cloud and the source point cloud, respectively. To determine the number of matching point pairs.

[0039] Furthermore, RGB-D images (i.e., depth image point cloud data) generate feature descriptors using the ORB feature extraction algorithm. Then, by matching ORB features between consecutive frames, a correspondence between points in three-dimensional space can be established. Using this inter-frame correspondence, the relative motion (rotation and translation) of the camera between two frames can be calculated, thus forming visual odometry data. Further, the visual odometry data and the built-in IMU, which maintains hardware synchronization with the camera data, can be fused using EKF filtering. The former provides global, low-drift motion information for the system, while the latter primarily provides high-frequency, instantaneous motion information and attitude stabilization, resulting in more robust and stable visual odometry data. The EKF state estimation is set to 12 dimensions, and the state estimation vector is represented as follows: ; in, and This provides global positioning information on a 2D plane. , , These are the heading angle, roll angle, and pitch angle, respectively. and These are the linear velocities along the x-axis and y-axis, respectively. The angular velocity is the heading angle. The angular velocity is the roll angle. ω is the angular velocity of the pitch angle.

[0040] The fused visual odometry data, along with the depth image point cloud data (including depth images and RGB images) acquired by RGB-D and the laser point cloud data acquired by LiDAR, are input into SLAM mapping based on the RTAB-Map framework. The image resolution is limited to 640×480 to accommodate the shipborne computer with limited computing power, thereby constructing a robust and stable global 3D map.

[0041] In step S101 of some embodiments, the pose information of the berth markers is obtained through visual recognition. The pose information includes the marker's position and orientation. Specifically, the unmanned surface vessel (USV) obtains real-time environmental images through a visual sensor, then inputs the environmental images into a trained convolutional neural network model for berth marker recognition. The recognized berth marker pixels are matched with a global 3D map to determine the position and orientation of the berth markers. Traditional solutions for berth visual positioning use simple marker points or monocular methods, lacking accurate estimation of the berth's three degrees of freedom pose. This makes it impossible to accurately berth the USV in scenarios with multiple obstacles or narrow berths. This embodiment uses visual recognition to detect berths and their orientations in real time, supporting the system in real-time planning of berthing paths.

[0042] In some embodiments, step S101 may include, but is not limited to, the following steps: S301, The target detection model is used to detect berth markers in the currently acquired image data to obtain the image detection results; S302, Based on the image detection results, perform pose calculation of the berth marker in the camera coordinate system to obtain the pose calculation result; S303 maps the pose calculation results to the coordinate system of the global 3D map to obtain the pose information of the berth marker in the global 3D map.

[0043] In this embodiment, the visual recognition module performs real-time detection of berth markers (such as AprilTag markers) on the RGB-D image using a target detection model to obtain image detection results. The image detection results are used to determine whether there are berth markers in the current image, and if there are berth markers, output berth marker detection boxes. The target detection model can be a YOLOv series model.

[0044] When a berth marker is detected, the pose of the berth marker in the camera coordinate system is calculated using the PnP algorithm to obtain the pose calculation result. The core calculation formula is as follows: ; in, Scale factor (dimensionless, the projection scaling ratio of a point in three-dimensional space onto the image plane). Image plane pixel coordinates (unit: pixels, homogeneous coordinate form) ); The camera intrinsic parameter matrix (a 3×3 matrix containing the focal length) and principal point coordinates ); It is a 3×3 rotation matrix (describing the rotation transformation from the marker coordinate system to the camera coordinate system); It is a 3×1 translation vector (unit: m, representing the position of the origin of the marker coordinate system in the camera coordinate system); The world coordinates (unit: m, homogeneous coordinate form) of the three-dimensional feature points on the marker. ).

[0045] The pose calculation results are converted into a global 3D map by combining the camera extrinsic parameter calibration results, thus obtaining the three-degree-of-freedom pose of the berth marker in the global 3D map coordinate system, i.e., pose information. ,in, This refers to the location information in the global 3D map coordinate system. The attitude angle of the berth marker is determined, and this attitude information is sent to the dynamic decision-making and planning system.

[0046] In step S102 of some embodiments, the target pose is determined based on the pose information and the safe distance from the shoreline, and the target pose is updated in the memory variables. Traditional berth target recognition is limited to the visual field of view, and system failure is prone to occur when the target is lost. In this embodiment, when the visual target is lost, the system immediately activates the target pose memory and retention mechanism, continuously publishes valid berth target pose information based on historical pose trajectories, and ensures the continuity of berthing tasks.

[0047] Specifically, after receiving pose information, the dynamic decision-making and planning system uses the identified landmark pose as a reference, moves forward a set distance along its normal vector (simulating the safe distance between the berth and the shoreline), generates information such as the center, size, and orientation of the ROI area, and continuously optimizes the berthing trajectory in real time based on the ship's size and channel constraints, generating a target pose sequence that meets kinematic feasibility requirements. The dynamic decision-making and planning system adopts a three-level logical architecture: berth coordinate transformation, local path optimization, and global path planning and cost map construction. The berth coordinate transformation is detailed below: After receiving the pose information of the berth marker in the global map coordinate system, the position and pose of the ROI area (i.e., the target pose) are calculated by reverse translation and rotation using a preset safe berthing distance (i.e., safe distance from the shoreline), as shown below: ; in, This refers to the safe forward movement distance in the direction of the normal vector of the berth marker, which is the simulated safe distance between the berth and the shoreline. This embodiment avoids collisions caused by the unmanned vessel directly approaching the shoreline by setting a safe forward movement distance. It also provides a reasonable starting reference for berthing trajectory planning and leaves room for the unmanned vessel to turn.

[0048] In some embodiments, the memory variable in step S102 can be updated through, but is not limited to, the following steps: S401, determine whether a berth marker has been detected or whether the task status has changed; S402, if a berth marker is detected, update the target pose corresponding to the detected berth marker to the memory variable; or, if the task state changes, change the memory variable according to the task endpoint indicated by the task state.

[0049] In this embodiment, a memory variable is created to store the desired target of the path. The value of this memory variable is then read during subsequent path planning. The dynamic decision-making and planning system determines whether a berth marker has been detected or whether the task status has changed. If a berth marker is detected, the target pose corresponding to the detected berth marker is updated in the memory variable. Even if the marker is out of sight, the planning layer can still perform real-time path updates because the target pose corresponding to the berth marker is already stored in the memory variable. Alternatively, if the task status changes, such as changing a berthing task to a cruising task, the memory variable can be modified according to the task endpoint indicated by the new task status, thereby realizing path planning for the new task.

[0050] Specifically, the aforementioned target pose is continuously published to the target receiving interface of the navigation system. Furthermore, considering the limited field of view of the camera, a target pose memory and retention mechanism is incorporated into the system: when the target's field of view is lost, the pose of the last received valid berth target is continuously published as a temporary navigation target to maintain the continuity of the navigation task. That is, the system internally sets up a memory variable to store the global pose information of the last validly detected berth. During visual detection interruptions, this memory pose is repeatedly recalled and sent as a navigation target to the path planning module, enabling the vessel to continue along the predetermined berthing path even without visual updates, until the visual target is detected again or the task status changes.

[0051] In some embodiments, the real-time pose of the unmanned vessel (USV) is continuously output after fusing data from the inertial measurement unit and visual odometry using an extended Kalman filter. This output includes the vessel's real-time position coordinates, heading angle, linear velocity, and angular velocity in the global three-dimensional map coordinate system, serving as a real-time positioning reference during navigation.

[0052] In step S103 of some embodiments, the system reads the target pose from the memory variable and performs path planning and path tracking control on the unmanned vessel based on the target pose, so that the unmanned vessel berths towards the berth marker.

[0053] In some embodiments, the step of performing path planning and path tracking control of the unmanned vessel based on the target pose in step S103 may include, but is not limited to, the following steps: S501, based on the current pose of the unmanned vessel and the pose of the target, performs global path planning on the global 3D map to obtain the global path; S502 updates the local 3D map in the global 3D map based on the real-time collected point cloud data, and optimizes the global path locally based on the local 3D map to obtain the optimized path; S503 performs path tracking control on the optimized path to obtain and apply ship control commands.

[0054] In this embodiment, global path planning is performed on the global 3D map based on the current pose and target pose of the unmanned vessel. While moving along the path, the local 3D map in the global 3D map is updated based on the real-time collected point cloud data (including laser point cloud and depth image point cloud). The global path is also locally optimized based on the local 3D map, so that the global path can take into account the dynamic water surface environment more accurately, realize the precise obstacle avoidance of the vessel, and improve berthing safety.

[0055] In step S501 of some embodiments, the global path planning module generates an initial global path from the current pose of the unmanned vessel to the target pose at the berth based on a prior map and an improved A* algorithm. The prior map can be a global 3D map established by a multi-source data perception system of the vessel itself or other vessels. This global path subsequently serves as a reference trajectory for the local planner, and is dynamically adjusted in the dynamic environment through a real-time updated local cost map to improve the feasibility and optimality of the global path.

[0056] In step S502 of some embodiments, the global path is locally optimized based on the local 3D map to obtain an optimized path, which may include, but is not limited to, the following steps: S601, based on obstacle information indicated by the local 3D map and reference path indicated by the global path, performs motion trajectory planning to obtain multiple candidate motion trajectories; the candidate motion trajectories include motion speed; S602, the candidate motion trajectory is weighted and scored in multiple dimensions to obtain the trajectory score; the dimensions of the trajectory score include target alignment, in-situ rotation capability and motion oscillation, and the weight of the in-situ rotation capability dimension is dynamically adjusted according to the distance between the unmanned vessel and the berth marker. S603: Based on the candidate motion trajectory with the highest score, perform local optimization on the global path to obtain the optimized path.

[0057] In this embodiment, the high-precision fusion positioning and mapping module dynamically updates the corresponding local areas in the global 3D map based on real-time collected point cloud data. The local planner can use the DWB algorithm for local path planning. Specifically, based on the obstacle information indicated by the updated local 3D map, and with the global path as a reference, it plans the motion trajectory locally based on the ship's kinematics model, obtaining multiple candidate motion trajectories within the local area. In addition to indicating the trajectory points, the candidate motion trajectories also indicate the motion speed and attitude at those points. The candidate motion trajectories are scored using multi-dimensional evaluation metrics, thereby selecting the optimal candidate motion trajectory to locally optimize the global path, improving the safety and accuracy of the berthing process. The trajectory score of the candidate motion trajectory considers dimensions including target alignment, in-situ rotation capability, and motion oscillation, as detailed below: GoalAlign: The core evaluation measure is the deviation between the end of the trajectory and the target orientation (determined by the global path), and this indicator is given a high weight to ensure that the USV approaches the berth at the correct angle. RotateToGoal: When the unmanned vessel is close to the target and the linear velocity is extremely low, the forced planner generates pure angular velocity commands, which are specifically used for attitude correction under zero-speed conditions; that is, the closer the unmanned vessel is to the berth marker, the higher the weight of the rotation capability dimension when calculating the trajectory score. Oscillation assessment: Limits the rate of change of the local planner's output speed commands (e.g., maximum translational speed change ≤ 0.01), improving the ship's stability during low-speed surface movement and avoiding frequent switching of steering or propeller commands. Simultaneously, the planner strictly adheres to USV physical motion constraints, setting the maximum permissible linear and angular velocities to conservatively low values ​​(e.g., maximum linear velocity ≤ 0.3 m / s) in berthing mode, ensuring that the output speed commands meet low-speed safety requirements.

[0058] In some embodiments, this embodiment also includes an exception handling mechanism. When it is detected that the local planner cannot generate a valid trajectory or the global path is completely blocked, the replanning process can be automatically triggered to recalculate the feasible global path or adjust the berthing strategy (such as determining other berthing locations through visual recognition) so that the autonomous berthing task can be completed smoothly.

[0059] In step S503 of some embodiments, path tracking control of the optimized path may include, but is not limited to, the following steps: S701, constructs a linear velocity control loop for calculating the total thrust rotational speed based on the desired linear velocity indicated by the optimized path, and constructs an angular velocity control loop for steering force based on the desired angular velocity indicated by the optimized path; S702 dynamically allocates the weighting coefficients of the rudder angle and the difference in thruster speed based on the current linear velocity of the unmanned vessel; S703 calculates the desired rudder angle and desired thruster speed difference based on the weight allocation result and the steering force calculated by the angular velocity control loop, and calculates the desired dual-side thruster speed based on the desired thruster speed difference and the total thrust speed calculated by the linear velocity control loop. S703 generates ship control commands based on the desired rudder angle and the desired rotational speed of both propellers.

[0060] Traditional controller solutions are mostly based on simple differential speed or rudder angle allocation, failing to consider the coupled maneuvering characteristics of ships under multiple operating conditions such as low speed and zero speed. The motion control and execution system in this embodiment adopts a "speed-steering" dual closed-loop PID architecture and designs a control command generation strategy based on the ship's low-speed operating conditions when considering berthing, solving the coupled maneuvering problem under low-speed conditions and improving the feasibility of precise berthing.

[0061] Specifically, the dynamic decision-making and planning system generates an optimized path in real time. This optimized path indicates the trajectory point expected to be reached at a future time, as well as the expected linear velocity and angular velocity at that trajectory point. The motion control and execution system constructs two independent PID closed loops based on the optimized path: a linear velocity control loop and an angular velocity control loop, as follows: The linear velocity control loop calculates the total thrust and rotational speed. ,as follows: ; in, This is the linear velocity proportionality coefficient. The linear velocity integral coefficient, The linear velocity differential coefficient, This is the linear velocity error, which is the difference between the expected linear velocity and the actual linear velocity.

[0062] Angular velocity control loop calculates normalized steering force ,as follows: ; in, This is the angular velocity proportionality coefficient. The integral coefficient of angular velocity. These are the differential coefficients of angular velocity. This refers to the angular velocity error, which is the difference between the expected angular velocity and the actual angular velocity.

[0063] The controller dynamically adjusts the rudder angle of S according to the current linear velocity v of the USV. Difference in thruster speed The allocation of weighting coefficients, the core of which is to determine low-speed operating conditions through the zero-speed threshold judgment formula, the allocation of weighting coefficients for rudder angle and propeller speed difference, and the generation of ship control commands are as follows: When |v|≥v0 (a preset speed threshold, which can be zero or close to zero), the steering force S is mainly converted into the rudder angle. And part of it is converted into thruster speed difference. The rudder angle allows the servo to dominate at higher speeds to maintain course stability, while the difference in propeller speeds serves as an auxiliary torque for fine-tuning of the steering. , ,in, and These are the weighting coefficients corresponding to the difference in rudder angle and thruster speed, respectively. Greater than At this point, the final rotational speed of the left and right thrusters is obtained by adding or subtracting the difference between the total thrust rotational speed and the auxiliary rotational speed. The specific rotational speeds of the left and right thrusters are as follows: ; ; When |v| < v0, it is in the zero-speed / precise fine-tuning condition (berth alignment). The system automatically suppresses and zeros the output of the linear velocity control loop , and at the same time forces the rudder angle to be centered. At this time, the system completely relies on the pure rotational speed difference generated by the normalized steering force S to drive the left and right thrusters. That is, the weight coefficient of the rudder angle is 0, and the weight coefficient of the rotational speed difference of the thrusters is 1, to achieve in-place turning and attitude fine-tuning, enabling the USV to remain parallel to the shoreline. Finally, the calculated ship control instructions are sent to the thruster and rudder controller through the serial port.

[0064] Through the collaborative action of the multi-sensor fusion SLAM system, the dynamic decision-making and planning system, and the motion control execution system in the embodiments of this application, high-precision positioning and robust control of visual-guided unmanned ships for autonomous berthing in complex water environments are achieved: the SLAM system provides centimeter-level positioning accuracy and avoids mission interruption through the target pose memory and retention module when the visual target is lost. The dynamic decision-making and planning system ensures the continuity and safety of the path. The motion control system adapts to the requirements of multiple working conditions such as low-speed differential turning, significantly improving the automation level and safety redundancy of the berthing operation.

[0065] The SLAM system constructs a dense environmental point cloud map through the tight coupling mechanism of lidar-RGBD-IMU, and uses visual odometry and ICP closed-loop detection to achieve global pose optimization. This method significantly reduces the drift error compared with traditional single-sensor solutions and achieves centimeter-level positioning accuracy. In addition, the designated berth area is located through visual positioning, providing perceptual data support for the subsequent system to guide the unmanned ship to berth stably. At the same time, the system is built with a target pose memory and retention module, which continuously outputs ROI pose information through the prior map and historical pose data when the visual target is lost, effectively avoiding the interruption of the berthing task and improving the stability of the berthing task.

[0066] Based on the three-degree-of-freedom pose of the berth output by the SLAM, the dynamic decision-making and planning system uses the improved A* algorithm in the NAV2 system for global path search, and generates a spatio-temporal trajectory including speed and heading angle constraints in combination with the ship kinematic model. And when the USV approaches the berthing area, the local path is updated in real time through the dynamic window method (DWB) to improve the continuity and safety of the berthing process.

[0067] The motion control execution system adopts a dual-closed-loop PID architecture: the outer loop calculates the normalized steering force S based on the heading error and lateral deviation, while the inner loop outputs control commands based on the dynamic model of the thrusters and obtains the ship's current speed and attitude information in real time as feedback input through the fusion of data from the IMU and visual odometry. Under low-speed conditions, it automatically switches to pure differential mode, achieving on-the-spot steering by adjusting the speed difference Δn between the left and right thrusters, meeting the high-precision berthing requirements of narrow berths.

[0068] This application also provides a vision-based autonomous berthing control system for unmanned vessels, including: The first module is used to obtain the position and pose information of berth markers through visual recognition. The position and pose information includes the position and pose of the markers. The second module is used to determine the target pose based on the pose information and the safe distance from the shoreline, and update the target pose to the memory variables; The third module is used to read the target pose from the memory variables and perform path planning and path tracking control on the unmanned vessel based on the target pose, so that the unmanned vessel can dock at the berth marker.

[0069] It is understood that the methods described in the above method embodiments are applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0070] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including a tablet computer, a shipboard computer, or similar device.

[0071] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0072] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0073] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0074] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0075] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0076] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0077] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0079] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0080] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0082] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0084] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0085] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0087] If the integrated modules are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for autonomous berthing control of unmanned vessels based on visual recognition, characterized in that, Includes the following steps: The position and orientation information of berth markers is obtained through visual recognition, and the position and orientation information includes the position and orientation of the markers. Based on the pose information and the safe distance from the shoreline, the target pose is determined and updated in the memory variables; The target pose is read from the memory variable, and the unmanned vessel is used for path planning and path tracking control based on the target pose so that the unmanned vessel can dock at the berth marker.

2. The unmanned vessel autonomous berthing control method based on visual recognition according to claim 1, characterized in that, The vision-based autonomous docking control method for unmanned vessels also includes the following steps: Laser point cloud data of the environment is collected by lidar, and depth image point cloud data of the environment is collected by depth camera. Global loop closure detection is performed based on the current point cloud data and local map to correct perceived motion pose, and continuous pose state estimation is performed based on the continuous depth image point cloud data to obtain visual odometry data. Based on the visual odometry data, the laser point cloud data, and the depth image point cloud data, SLAM mapping and global loop closure detection are performed to obtain a global 3D map.

3. The unmanned vessel autonomous berthing control method based on visual recognition according to claim 2, characterized in that, The method of obtaining the position and pose information of berth markers through visual recognition includes the following steps: The target detection model is used to detect berth markers in the currently acquired image data to obtain the image detection results; Based on the image detection results, the pose of the berth marker in the camera coordinate system is calculated to obtain the pose calculation result. The pose calculation result is mapped onto the coordinate system of the global 3D map to obtain the pose information of the berth marker in the global 3D map.

4. The unmanned vessel autonomous berthing control method based on visual recognition according to claim 1, characterized in that, The memory variable is updated through the following steps: Determine whether a berth marker has been detected or whether the task status has changed; If a berth marker is detected, the target pose corresponding to the detected berth marker is updated in the memory variable; or, if the task state changes, the memory variable is changed according to the task endpoint indicated by the task state.

5. The unmanned vessel autonomous berthing control method based on visual recognition according to claim 1, characterized in that, The path planning and path tracking control of the unmanned vessel based on the target pose includes the following steps: Based on the current pose of the unmanned vessel and the pose of the target, a global path is obtained by performing global path planning on the global 3D map. The local 3D map in the global 3D map is updated based on the real-time collected point cloud data, and the global path is locally optimized based on the local 3D map to obtain the optimized path; The optimized path is subjected to path tracking control to obtain and apply ship control commands.

6. The unmanned vessel autonomous berthing control method based on visual recognition according to claim 5, characterized in that, The step of locally optimizing the global path based on the local 3D map to obtain an optimized path includes the following steps: Based on the obstacle information indicated by the local 3D map and the reference path indicated by the global path, multiple candidate motion trajectories are planned to obtain the motion trajectory; the candidate motion trajectory includes the motion speed. The candidate motion trajectory is weighted and scored in multiple dimensions to obtain a trajectory score; the dimensions of the trajectory score include target alignment, in-situ rotation capability, and motion oscillation, and the weight of the in-situ rotation capability dimension is dynamically adjusted according to the distance between the unmanned vessel and the berth marker. The global path is locally optimized based on the candidate motion trajectory with the highest score to obtain the optimized path.

7. The unmanned vessel autonomous berthing control method based on visual recognition according to claim 6, characterized in that, The process of performing path tracking control on the optimized path to obtain and apply ship control commands includes the following steps: A linear velocity control loop for calculating the total thrust rotational speed is constructed based on the desired linear velocity indicated by the optimized path, and an angular velocity control loop for steering force is constructed based on the desired angular velocity indicated by the optimized path. The weighting coefficients for the rudder angle and the difference in thruster speed are dynamically allocated based on the current linear velocity of the unmanned vessel. The desired rudder angle and desired thruster speed difference are calculated based on the weight allocation result and the steering force calculated by the angular velocity control loop. The desired dual-side thruster speed is calculated based on the desired thruster speed difference and the total thrust speed calculated by the linear velocity control loop. Ship control commands are generated based on the desired rudder angle and the desired dual-side thruster speeds.

8. A vision-based autonomous berthing control system for unmanned vessels, characterized in that, include: The first module is used to obtain the position and pose information of berth markers through visual recognition, wherein the position and pose information includes the position and pose of the markers. The second module is used to determine the target pose based on the pose information and the safe distance from the shoreline, and update the target pose to the memory variables; The third module is used to read the target pose from the memory variable and perform path planning and path tracking control on the unmanned vessel based on the target pose, so that the unmanned vessel can dock at the berth marker.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images