Unmanned ship autonomous docking wharf method and system based on self-supervised learning
Through self-supervised learning and deep learning algorithms, combined with PID controllers, the autonomous mooring of unmanned ships in complex marine environments is achieved, which solves the problems of low efficiency and relying on artificial feature engineering in traditional technology, and improves the accuracy and operation efficiency of autonomous docking .
Patent Information
- Application Number
- CN202411081787.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional unmanned ship autonomous docking technology has problems such as low efficiency and reliance on a large number of artificial feature engineering and data annotation in complex and dynamic marine environments, making it difficult to achieve efficient autonomous docking of docks.
Using a self-supervised learning method, by obtaining the image data of the unmanned ship relative to the dock, establishing a prediction model of a deep learning algorithm, performing data augmentation and training, predicting the relative attitude position error of the unmanned ship, and realizing motion through a PID controller. Closed-loop control to reduce human intervention.
It realizes accurate and reliable autonomous docking of unmanned ships in complex marine environments, improves operational efficiency, reduces human label intervention, improves autonomous docking accuracy, and is suitable for autonomous docking tasks in dynamic environments.
Smart Images

Figure CN119091294A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned ship applications, and more specifically to a method and system for an unmanned ship to autonomously dock with a dock based on self-supervised learning. Background Art
[0002] In recent years, unmanned vessels have been increasingly used in areas such as ocean monitoring, environmental monitoring, and cargo transportation. Traditional unmanned vessel operations usually require human control, especially during re-berthing after mission completion. This reliance on human control methods based on a first-person perspective has limitations in terms of operational efficiency and autonomy. Especially for fully unmanned deployments, the development of autonomous berthing is expected to be a key advancement in improving the functionality and application range of unmanned vessels.
[0003] The main challenge of autonomous mooring lies in coping with dynamic and unpredictable marine environments. These environments include complex meteorological conditions, water flow dynamics, and various obstacles, requiring complex perception and control strategies. Traditional methods such as position-based visual servoing and image-based visual servoing frameworks still require a lot of manual feature engineering, data annotation, and precise positioning technology in these situations, which still requires a lot of manpower investment.
[0004] Therefore, it is an urgent problem for technicians in this field to propose a method and system for autonomous docking of unmanned boats with docks based on self-supervised learning to solve the difficulties existing in the prior art. Summary of the invention
[0005] In view of this, the present invention provides a method and system for autonomous docking of an unmanned ship with a wharf based on self-supervised learning, which realizes visual mooring of the unmanned ship through the self-supervised learning method and minimizes human intervention.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A method for autonomous docking of an unmanned ship with a dock based on self-supervised learning, comprising the following steps:
[0008] S1. Data acquisition step: obtaining an image of the unmanned ship relative to the center of the dock to obtain relevant data of the unmanned ship;
[0009] S2, data processing step: perform data enhancement on the collected relevant data, and divide the data after data enhancement into training set and test set;
[0010] S3, model building and training steps: building a prediction model based on a deep learning algorithm, including an encoding network and a prediction network, and inputting the training set and test set in S2 into the prediction model to implement model training and testing;
[0011] S4, error prediction step: the posture of the unmanned ship is estimated through the trained prediction model, and the relative posture position error of the unmanned ship moving relative to the dock is predicted;
[0012] S5, unmanned ship control step: adjust the attitude of the unmanned ship according to the predicted relative attitude position error to achieve closed-loop motion control of the unmanned ship.
[0013] In the above method, optionally, the unmanned ship related data obtained by acquiring an image of the unmanned ship relative to the center of the dock in S1 includes a distance and a deflection angle of the unmanned ship relative to the center of the docking position.
[0014] In the above method, optionally, the prediction model in S3 includes an encoding network and a prediction network, the encoding network downsamples the image data to obtain an implicitly expressed feature vector; the prediction network estimates the position error based on the implicitly expressed feature vector.
[0015] In the above method, optionally, the error prediction in S4 includes:
[0016] Calculate the weights of the attitude estimation of the relevant data of the unmanned ship to be predicted, and estimate the attitude of the unmanned ship to be predicted based on the weights; obtain the latest image of the unmanned ship and predict the attitude of the target end effector, and based on the image of each time step, predict the relative attitude position error of the unmanned ship relative to the dock online.
[0017] The above method, optionally, S4 also includes the following conversion relationship between the unmanned ship coordinate systems:
[0018] ΔT:= {B} T {D} = {B} T {W} {W} T {D}
[0019] in, {B} T {W} is the positional relationship between an image I and the current posture of the unmanned ship, {W} T {D} is the desired initial docking posture defined by the user, and ΔT is the relative posture between the unmanned ship and the dock.
[0020] In the above method, optionally, in S5, closed-loop motion control is performed by a PID controller, and the PID controller includes an external controller and an internal controller; the external controller input is the relative posture position error of the unmanned ship relative to the dock, and the output is the expected speed; the internal controller input is the error between the current speed and the expected speed, and the control signal is gradually calculated and adjusted.
[0021] In the above method, optionally, the unmanned ship control in S5 further includes: for the differential thrust of the unmanned ship with two fixed rear thrusters, the simplified motion distribution model is:
[0022] v usv =α(T r +T l )
[0023] ω usv =β(T r -T l )
[0024] Among them, v usv ,ω usv Represent the linear velocity and angular velocity of the unmanned ship, T r ,T l represent the thrust ratio of the right engine and the left engine respectively, and α and β are scaling factors.
[0025] An unmanned ship autonomous docking dock system based on self-supervised learning, using any of the above unmanned ship autonomous docking dock method based on self-supervised learning, including a data acquisition module, a data processing module, a model building and training module, an error prediction module, and an unmanned ship control module connected in sequence; wherein,
[0026] A data acquisition module, used to acquire an image of the unmanned ship relative to the center of the dock to obtain relevant data of the unmanned ship;
[0027] The data processing module is used to perform data enhancement on the collected relevant data and divide the data after data enhancement into a training set and a test set;
[0028] Model building and training module, used to build a prediction model based on deep learning algorithm, including encoding network and prediction network, and input training set and test set into the prediction model to realize model training and testing;
[0029] The error prediction module is used to estimate the attitude of the unmanned ship through the trained prediction model and predict the relative attitude position error of the unmanned ship moving relative to the dock;
[0030] The unmanned ship control module is used to adjust the attitude of the unmanned ship according to the predicted relative attitude position error to achieve closed-loop motion control of the unmanned ship.
[0031] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and system for autonomous docking of an unmanned ship with a dock based on self-supervised learning, which has the following beneficial effects:
[0032] (1) This invention achieves visual mooring of unmanned vessels through a self-supervised learning method, minimizing human intervention; the study aims to determine the feasibility of self-supervised learning to achieve accurate and reliable autonomous mooring in complex and changing marine environments; through self-supervised data collection, minimal human label intervention, and a self-supervised dock posture estimator, these techniques can improve operational efficiency without losing accuracy. The contribution of this invention is to transfer robotic learning technology to real-world applications and operations. As technology continues to advance, we expect that future unmanned vessels will be more intelligent and autonomous, and can accurately and autonomously perform docking tasks in more complex and dynamic environments.
[0033] (2) The present invention aims to improve the autonomous docking accuracy of unmanned vessels through a method based on self-supervised learning; the method includes data collection, model training, and accurate docking pose prediction using a neural docking pose estimator. The proposed framework does not require manual annotation and camera calibration, making it efficient and reliable in actual water environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0035] Figure 1 A flow chart of a method for autonomous docking of an unmanned ship with a dock based on self-supervised learning provided by the present invention;
[0036] Figure 2 It is a data collection flow chart of the present invention;
[0037] Figure 3 This is a control block diagram of the unmanned ship of the present invention;
[0038] Figure 4 It is a model training framework diagram of the present invention;
[0039] Figure 5 It is the coordinate system conversion diagram of the present invention. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] Reference Figure 1 As shown, the present invention discloses a method for autonomous docking of an unmanned ship with a dock based on self-supervised learning, comprising the following steps:
[0042] S1. Data acquisition step: obtaining an image of the unmanned ship relative to the center of the dock to obtain relevant data of the unmanned ship;
[0043] S2. Data processing step: Perform data enhancement on the collected relevant data. Figure 4 As shown, the data after data enhancement is divided into training set and test set;
[0044] S3, model building and training steps: building a prediction model based on a deep learning algorithm, including an encoding network and a prediction network, and inputting the training set and test set in S2 into the prediction model to implement model training and testing;
[0045] S4, error prediction step: the posture of the unmanned ship is estimated through the trained prediction model, and the relative posture position error of the unmanned ship moving relative to the dock is predicted;
[0046] S5, unmanned ship control step: adjust the attitude of the unmanned ship according to the predicted relative attitude position error to achieve closed-loop motion control of the unmanned ship.
[0047] Furthermore, the unmanned ship related data obtained by acquiring the image of the unmanned ship relative to the center of the dock in S1 includes the distance and the deflection angle of the unmanned ship relative to the center of the docking position.
[0048] Specifically, Figure 2 As shown in the figure, the distance and deflection angle of the unmanned boat relative to the docking position center are collected autonomously, and the unmanned boat is remotely controlled by the keyboard to move randomly in the pre-docking area to collect data; the RGB lens in Intel D435 and the Intel T265 visual odometer are used to collect image data sets and relative position information.
[0049] (I,ΔT) t →D src
[0050] I is an RGB image, ΔT is a label for supervised learning, which collects the relative posture information between the unmanned ship and the dock, t is a time series, and D src It is the data set collected by all I,ΔT.
[0051] Specifically, data enhancement in S2 includes operations such as changing brightness, motion blur, and image distortion to ensure data diversity;
[0052] Furthermore, the prediction model in S3 includes an encoding network and a prediction network. The encoding network downsamples the image data to obtain an implicitly expressed feature vector; the prediction network estimates the position error based on the implicitly expressed feature vector; the encoding network is specifically a convolutional neural network CNN, and the prediction network is specifically a multi-layer perceptron MLP.
[0053]
[0054] Where f is the neural network model, and the superscript ^ represents the estimated value that is distinguished from the true value.
[0055] Furthermore, the error prediction in S4 includes:
[0056] Calculate the weights of the attitude estimation of the relevant data of the unmanned ship to be predicted, and estimate the attitude of the unmanned ship to be predicted based on the weights; obtain the latest image of the unmanned ship and predict the attitude of the target end effector, and based on the image of each time step, predict the relative attitude position error of the unmanned ship relative to the dock online.
[0057] Furthermore, S4 also includes the conversion between unmanned ship coordinate systems such as Figure 5 The relationships shown are as follows:
[0058] ΔT:= {B} T {D} = {B} T {W} {W} T {D}
[0059] in, {B} T {W} is the positional relationship between an image I and the current posture of the unmanned ship, {W} T {D} is the desired initial docking posture defined by the user, and ΔT is the relative posture between the unmanned ship and the dock.
[0060] Specifically, the visual odometry module estimates the current position of the unmanned ship, and combined with the error predicted in the previous step, estimates the expected position of the unmanned ship at the next moment by the above formula; the position-based visual servo is integrated to achieve tracking control of the unmanned ship. Assuming that the estimation accuracy is higher when the unmanned ship approaches the dock, the target posture is updated through the latest single-frame RGB image.
[0061] Furthermore, the PID controller is used in S5 to perform closed-loop motion control. Figure 3 As shown, the PID controller includes an external controller and an internal controller; the external controller input is the relative attitude position error of the unmanned ship relative to the dock, and the output is the desired speed; the internal controller input is the error between the current speed and the desired speed, and the control signal is gradually calculated and adjusted.
[0062] Furthermore, the unmanned ship control in S5 also includes: for the unmanned ship differential thrust with a fixed two rear thrusters configuration, the simplified motion distribution model is:
[0063] v usv =α(T r +T l )
[0064] ω usv =β(T r -T l )
[0065] Among them, v usv ,ω usv Represent the linear velocity and angular velocity of the unmanned ship, T r ,T l represent the thrust ratio of the right engine and the left engine respectively, and α and β are scaling factors.
[0066] Through algebraic operations, the motion of the unmanned ship is represented by the variables v and the hull speed ω, so that the motion control can be intuitively assigned. After calculating the heading angle of the unmanned ship, the formula for converting the heading angle into the thrust of the unmanned ship's steering gear is:
[0067]
[0068] The thrust command is proportional to the pulse width modulation (PWM) setpoint. Therefore, the PWM setpoint for the actuator is defined as
[0069]
[0070] Among them, u v is the linear speed of the unmanned ship, u ω is the angular velocity of the unmanned ship. In order to ensure that the PWM value is valid, the PWM value is controlled within a valid range:
[0071]
[0072] pwm out,i Output control pwm value for unmanned boat, pwm i To calculate the value, pwm min Output the minimum safety value for the unmanned ship, pwm max Output the maximum safety value for the unmanned ship.
[0073] An unmanned ship autonomous docking dock system based on self-supervised learning, using any of the above unmanned ship autonomous docking dock method based on self-supervised learning, including a data acquisition module, a data processing module, a model building and training module, an error prediction module, and an unmanned ship control module connected in sequence; wherein,
[0074] A data acquisition module, used to acquire an image of the unmanned ship relative to the center of the dock to obtain relevant data of the unmanned ship;
[0075] The data processing module is used to perform data enhancement on the collected relevant data and divide the data after data enhancement into a training set and a test set;
[0076] Model building and training module, used to build a prediction model based on deep learning algorithm, including encoding network and prediction network, and input training set and test set into the prediction model to realize model training and testing;
[0077] The error prediction module is used to estimate the attitude of the unmanned ship through the trained prediction model and predict the relative attitude position error of the unmanned ship moving relative to the dock;
[0078] The unmanned ship control module is used to adjust the attitude of the unmanned ship according to the predicted relative attitude position error to achieve closed-loop motion control of the unmanned ship.
[0079] In a specific embodiment, the following steps are included: Target point determination and path planning: The initial task of the unmanned ship is to autonomously dock at a designated dock. First, the location and posture of the dock are determined according to the docking requirements, and then a path from the current location to the dock is planned.
[0080] Environmental factor assessment and adjustment: During navigation, the unmanned ship is affected in real time by external natural factors such as wind, water flow and obstacles. If these factors cause the unmanned ship to deviate from the planned path, the autonomous mooring system will automatically make adjustments to ensure that the unmanned ship can arrive at the dock smoothly.
[0081] PID heading angle adjustment: Once the unmanned boat detects a deviation from the dock, the PID control system of the unmanned boat immediately adjusts the heading angle to reduce the degree of deviation and gradually guide the unmanned boat back to the original planned path. This adjustment takes into account the current direction of the unmanned boat, the location of the dock, and the degree of deviation.
[0082] Mission execution and autonomous docking: The unmanned vessel navigates along the planned path and begins autonomous docking when approaching the dock. The self-supervised dock posture estimator predicts the dock posture based on real-time camera images and performs precise docking through a position-based visual servo system.
[0083] Real-time path monitoring and adjustment: During the entire docking process, the unmanned vessel's navigation system continuously monitors the consistency of the actual route with the planned path. Any necessary fine-tuning is performed in real time to ensure the accuracy of the path and the safety of the docking.
[0084] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0085] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for autonomous docking of an unmanned ship with a dock based on self-supervised learning, characterized in that: The following steps are involved: S1. Data acquisition step: obtaining an image of the unmanned ship relative to the center of the dock to obtain relevant data of the unmanned ship; S2, data processing step: perform data enhancement on the collected relevant data, and divide the data after data enhancement into training set and test set; S3, model building and training steps: building a prediction model based on a deep learning algorithm, including an encoding network and a prediction network, and inputting the training set and test set in S2 into the prediction model to implement model training and testing; S4, error prediction step: the posture of the unmanned ship is estimated through the trained prediction model, and the relative posture position error of the unmanned ship moving relative to the dock is predicted; S5, unmanned ship control step: adjust the attitude of the unmanned ship according to the predicted relative attitude position error to achieve closed-loop motion control of the unmanned ship.
2. The method for autonomous docking of an unmanned ship with a dock based on self-supervised learning according to claim 1, characterized in that: The unmanned ship related data obtained by acquiring the image of the unmanned ship relative to the center of the dock in S1 includes the distance and the deflection angle of the unmanned ship relative to the center of the docking position.
3. The method for autonomous docking of an unmanned ship with a dock based on self-supervised learning according to claim 1, characterized in that: The prediction model in S3 includes an encoding network and a prediction network. The encoding network downsamples the image data to obtain an implicitly expressed feature vector; the prediction network estimates the position error based on the implicitly expressed feature vector.
4. The method for autonomous docking of an unmanned ship with a dock based on self-supervised learning according to claim 1, characterized in that: The error prediction in S4 includes: Calculate the weights of the attitude estimation of the relevant data of the unmanned ship to be predicted, and estimate the attitude of the unmanned ship to be predicted based on the weights; obtain the latest image of the unmanned ship and predict the attitude of the target end effector, and based on the image of each time step, predict the relative attitude position error of the unmanned ship relative to the dock online.
5. The method for autonomous docking of an unmanned ship with a dock based on self-supervised learning according to claim 3, characterized in that: S4 also includes the conversion relationship between the unmanned ship coordinate systems as follows: ΔT:= {B} T {D} = {B} T {W} {W} T {D} in, {B} T {W} is the positional relationship between an image I and the current posture of the unmanned ship, {W} T {D} is the desired initial docking posture defined by the user, and ΔT is the relative posture between the unmanned ship and the dock.
6. The method for autonomous docking of an unmanned ship with a dock based on self-supervised learning according to claim 1, characterized in that: In S5, motion closed-loop control is performed through a PID controller, which includes an external controller and an internal controller; the external controller input is the relative attitude position error of the unmanned ship relative to the dock, and the output is the desired speed; the internal controller input is the error between the current speed and the desired speed, and the control signal is gradually calculated and adjusted.
7. The method for autonomous docking of an unmanned ship with a dock based on self-supervised learning according to claim 6, characterized in that: The unmanned ship control in S5 also includes: For the differential thrust of the unmanned ship with a fixed two rear thrusters, the simplified motion distribution model is: v usv =α(T r +T l ) oh usv =β(T r -T l ) Among them, v usv ,ω usv Represent the linear velocity and angular velocity of the unmanned ship, T r ,T l represent the thrust ratio of the right engine and the left engine respectively, and α and β are scaling factors.
8. An unmanned ship autonomous docking system based on self-supervised learning, characterized in that: A method for autonomous docking of an unmanned ship with a dock based on self-supervised learning as described in any one of claims 1 to 7, comprising a data acquisition module, a data processing module, a model building and training module, an error prediction module, and an unmanned ship control module connected in sequence; wherein: A data acquisition module, used to acquire an image of the unmanned ship relative to the center of the dock to obtain relevant data of the unmanned ship; The data processing module is used to perform data enhancement on the collected relevant data and divide the data after data enhancement into a training set and a test set; Model building and training module, used to build a prediction model based on deep learning algorithm, including encoding network and prediction network, and input training set and test set into the prediction model to realize model training and testing; The error prediction module is used to estimate the attitude of the unmanned ship through the trained prediction model and predict the relative attitude position error of the unmanned ship moving relative to the dock; The unmanned ship control module is used to adjust the attitude of the unmanned ship according to the predicted relative attitude position error to achieve closed-loop motion control of the unmanned ship.
Citation Information
Cited By
Unmanned ship positioning method and system for automatic refueling robot
CN119472690A
