Unmanned ship docking method and system based on air-ground collaborative visual identification

Abstract

The invention discloses an unmanned ship docking method and system based on air-ground collaborative visual identification. An unmanned aerial vehicle-mounted visual system collects a first position image of an unmanned ship and a dock cabin; carrying out target detection model detection, connecting an auxiliary line according to key points of a target obtained through detection, and calculating a relative distance and a course angle between the dock cabin door and the unmanned ship; whether the relative distance and the course angle reach a preset range or not is judged, the unmanned ship is controlled to move through the first propelling speed and the first course adjusting angle or the second propelling speed and the second course adjusting angle, and docking of the unmanned ship is repeatedly achieved. Through visual systems of the unmanned aerial vehicle and the unmanned ship, air-ground collaborative visual identification is realized. The images of the unmanned ship and the cabin door are captured in real time, the inclination angle and the relative position of the unmanned ship and the cabin door are accurately recognized in combination with a rotating target detection algorithm, and the safety and the efficiency of automatic docking of the unmanned ship are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to intelligent water transportation, in particular to an unmanned ship docking method and system based on air-ground collaborative visual recognition. BACKGROUND

[0002] With the continuous development of unmanned ship technology, automatic docking has become an important part of intelligent water transportation and port automation. Traditional unmanned ship docking methods rely on shore-based fixed cameras for visual assistance, combined with GPS positioning and inertial navigation systems for path planning and control. However, the shore-based camera has a fixed view angle, which is easily affected by obstruction and limited vision, making it difficult to obtain the relative position and attitude information of the unmanned ship and the hatch in a comprehensive and real-time manner, resulting in difficulty in effectively correcting attitude deviation during docking.

[0003] In addition, GPS signals in complex environments such as ports are easily affected by obstruction, multipath interference, and other factors, resulting in decreased positioning accuracy, which restricts the safety and reliability of unmanned ship docking. Existing visual assistance mostly uses two-dimensional target detection algorithms, which cannot accurately obtain the inclination angle and three-dimensional spatial relationship of the target, and cannot meet the needs of complex and dynamic docking scenarios. The existing unmanned ship docking methods have the problems of large positioning error, limited visual range, and inability to accurately determine the hatch orientation and relative attitude.

[0004] Current visual-based unmanned ship short-range guidance techniques mainly include:

[0005] For example, Chinese patent application No. CN202411574448.4 discloses an unmanned ship visual guidance system, method, and computer readable storage medium, belonging to the field of unmanned ship autonomous recovery technology. The system mainly includes an unmanned ship equipped with passive and active infrared vision systems, and a mother ship equipped with a reflective near-infrared cooperative target at the docking position. The method uses the passive infrared vision system to detect and guide the mother ship at a long distance (30m away), and switches to the active infrared vision system at a short distance (within 30m), illuminates the cooperative target, and uses high-brightness images to solve the precise pose, finally completing the docking recovery. This scheme uses a dual-mode infrared vision system and a reflective cooperative target to improve the stability and accuracy of guidance in complex sea conditions and all-weather conditions. However, this scheme has low algorithm intelligence, insufficient environmental robustness and system integration, and is difficult to operate stably in long-term recovery tasks without human intervention.

[0006] For example, in Chinese patent application No. CN202311490179.9, a water surface unmanned boat visual guidance docking recovery method, system and device are disclosed, belonging to the technical field of unmanned boat recovery. The method includes preliminarily guiding the unmanned boat in the heading direction until the first STag reference mark of the opening end of the recovery device is detected by the camera of the unmanned boat. Then, visual guidance is performed with the first mark as the target, so that the bow of the unmanned boat is directly opposite the opening end. When the distance enters the preset range, the second STag reference mark of the opening end of the recovery device is switched to for final guidance until docking is completed. This scheme realizes the "coarse tuning and fine tuning" guidance strategy through the two markers with different codes, aiming to improve the accuracy and success rate of the unmanned boat terminal docking. However, the feature extraction method relying on manual markers and fixed geometric constraints limits the scalability, and lacks robust redundancy design in dynamic environments, low visibility and marker failure conditions. SUMMARY

[0007] The present application provides a safe and efficient unmanned boat docking method and system based on air-ground collaborative visual recognition. It involves the joint use of computer vision technology by unmanned aerial vehicles and unmanned boats to realize accurate recognition and position relationship judgment of the unmanned boat and the hatch through rotating target detection (YOLOV11-OBB) and monocular vision ranging, thereby intelligently controlling the unmanned boat to complete safe and efficient docking operations.

[0008] Technical solution: To solve the above problems, the present application adopts an unmanned boat docking method based on air-ground collaborative visual recognition, comprising the following steps:

[0009] (1) An unmanned aerial vehicle carries a visual system to collect a first position image of the unmanned boat and the dock hatch;

[0010] (2) The collected first position image is detected using a target detection model, and the detection result includes the key points of each target;

[0011] (3) Auxiliary lines are connected according to the key points of the target, and the relative distance and heading angle between the dock hatch and the unmanned boat are calculated through the auxiliary lines;

[0012] (4) It is judged whether the calculated relative distance and heading angle reach the preset range. If not, the first propulsion speed and first heading adjustment angle are calculated to control the movement of the unmanned boat, and then step (1) is returned. If the preset range is reached, step (5) is performed;

[0013] (5) The second position image of the dock hatch collected by the visual system carried by the unmanned boat is obtained, and the relative distance and heading angle between the dock hatch and the unmanned boat are determined according to the second position image, and the second propulsion speed and second heading adjustment angle are calculated to control the movement of the unmanned boat;

[0014] (6) Repeat step (5) to realize the unmanned ship into the dock.

[0015] Further, the preset range includes a transition range and a fine adjustment range, if the fine adjustment range is reached, step (5) is performed, if the transition range is reached, the following steps are performed:

[0016] According to the relative distance and the heading angle between the dock door and the unmanned ship determined by the first position image, the first propulsion speed and the first heading adjustment angle are calculated;

[0017] Obtain the second position image of the dock door collected by the vision system carried by the unmanned ship, and determine the relative distance and the heading angle between the dock door and the unmanned ship according to the second position image, and calculate the second propulsion speed and the second heading;

[0018] The first propulsion speed and the first heading adjustment angle and the second propulsion speed and the second heading adjustment angle are weighted and fused to obtain the joint propulsion speed and the joint heading adjustment angle to control the movement of the unmanned ship.

[0019] Further, after the movement of the unmanned ship is controlled by the second propulsion speed and the second heading adjustment angle in step (5), the second position image of the dock door collected by the vision system carried by the unmanned ship is obtained, the relative distance and the heading angle between the dock door and the unmanned ship are determined according to the second position image, and the heading angle between the dock door and the unmanned ship is judged, if the heading angle does not reach the fine adjustment range, the joint propulsion speed and the joint heading adjustment angle are calculated to control the movement of the unmanned ship, otherwise the second propulsion speed and the second heading adjustment angle are calculated to control the movement of the unmanned ship.

[0020] Further, the step (6) specifically includes:

[0021] (61) Determine whether the relative distance and the heading angle between the dock door and the unmanned ship reach the anti-collision range, if not, return to step (5); if the anti-collision range is reached, step (62) is performed;

[0022] (62) Get the distance between the corner point of the unmanned ship and the edge of the dock door through the vision system of the unmanned aerial vehicle, compare it with the preset safety distance, get the heading angle correction amount and the propulsion speed of the unmanned ship, generate the optimal obstacle avoidance path, and control the unmanned ship into the dock.

[0023] Further, the calculation formula of the heading angle correction amount of the unmanned ship in step (62) is:

[0024]

[0025] Wherein, is the control gain, is the heading angle correction direction indicator.

[0026] Furthermore, the target detection model adopts the YOLOV11-OBB model.

[0027] Furthermore, the target includes an unmanned surface vessel (USV) and a dock. The key points include the center point of the dock entrance, the center point of the front of the USV, and the center point of the rear of the USV. The auxiliary lines include an axial centerline, a bow line, and a stern line. The axial centerline is the line connecting the center points of the front and rear of the USV. The bow line is the line connecting the center points of the dock entrance and the front of the USV. The stern line is the line connecting the center points of the dock entrance and the rear of the USV.

[0028] Furthermore, the second position image of the dock door acquired by the vision system on the unmanned surface vessel includes a calibration plate set on the door. The position coordinates of the calibration plate on the second position image are obtained, and the depth distance and left and right offset distance between the unmanned surface vessel and the calibration plate are calculated according to the size of the calibration plate, thereby determining the relative distance and heading angle between the dock door and the unmanned surface vessel.

[0029] Furthermore, if the unmanned surface vessel's vision system fails to recognize the hatch calibration plate due to external factors, it will directly return to step (1).

[0030] This invention also employs an unmanned surface vessel docking system based on air-ground collaborative visual recognition, comprising a drone and an unmanned surface vessel equipped with a vision system, as well as a shore-based operations center; the shore-based operations center includes a detection module, a calculation module, a status discrimination module, and a control module;

[0031] The drone's vision system is used to acquire initial positional images of the unmanned surface vessel and its dock.

[0032] The detection module is used to detect targets in the first location image using a target detection model. The detection results include the rotated bounding box of each target.

[0033] The calculation module is used to determine the corner points and key points of the target based on the target's rotated bounding box, connect auxiliary lines based on the corner points and key points, calculate the relative distance and heading angle between the dock hatch and the unmanned surface vessel through the auxiliary lines, and calculate the first propulsion speed and the first heading adjustment angle.

[0034] The state determination module is used to determine whether the calculated relative distance and heading angle are within the preset range.

[0035] The control module is used to control the movement of the unmanned surface vessel (USV) by means of a first propulsion speed and a first heading adjustment angle, based on the judgment result of the state discrimination module not reaching the preset range; and to control the movement of the USV by means of a second propulsion speed and a second heading adjustment angle, based on the judgment result of the state discrimination module reaching the preset range.

[0036] The visual system of the unmanned ship is used to collect a second position image of the dock cabin door, the relative distance and the heading angle between the dock cabin door and the unmanned ship are determined according to the second position image, and the second propulsion speed and the second heading adjustment angle are calculated to control the movement of the unmanned ship.

[0037] Beneficial effects: The present application has the remarkable advantage over the prior art that by introducing a camera-carrying unmanned aerial vehicle as a mobile visual platform, air-ground collaborative visual identification is realized. The unmanned aerial vehicle camera captures images of the unmanned ship and the cabin door vertically downward in real time, combines a rotating target detection algorithm (such as YOLOV11-OBB), accurately identifies the inclination angle and relative position of the unmanned ship and the cabin door, combines the camera internal height information to convert the spatial relative position and attitude angle, constructs an angle deviation and distance deviation judgment model, and accordingly performs the docking state identification and control right switching. This method overcomes the problem of limited visual angle of traditional shore-based cameras, improves the timeliness and accuracy of visual information, realizes intelligent assistance and dynamic guidance of the unmanned ship docking process, and improves the safety and efficiency of the unmanned ship automatic docking. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 The flowchart of the docking method in the present application.

[0039] Figure 2 The structural schematic diagram of the unmanned ship docking from the perspective of the unmanned aerial vehicle in the present application.

[0040] Figure 3 The structural schematic diagram of deploying Aruco code for flight height estimation from the perspective of the unmanned aerial vehicle in the present application.

[0041] Figure 4 The schematic diagram of deploying Aruco code for positioning, distance measurement and heading angle calculation from the perspective of the unmanned ship in the present application.

[0042] Figure 5 The positioning and distance measurement schematic diagram of deploying YOLOV11-OBB oblique annotation box from the perspective of the unmanned aerial vehicle in the present application.

[0043] Figure 6 The heading angle calculation schematic diagram of deploying YOLOV11-OBBB oblique annotation box from the perspective of the unmanned aerial vehicle in the present application.

[0044] Figure 7 The close-range driving anti-collision distance calculation schematic diagram of deploying YOLOV11-OBB from the perspective of the unmanned aerial vehicle in the present application.

[0045] Figure 8 The cabin entry anti-collision distance calculation schematic diagram of deploying YOLOV11-OBB from the perspective of the unmanned aerial vehicle in the present application.

[0046] Figure 9This is a simulation diagram of the experimental scenario of unmanned surface vessel entering the dock in this invention. Detailed Implementation

[0047] Example 1

[0048] like Figure 1 As shown in this embodiment, an unmanned surface vessel (USV) docking method based on air-ground collaborative visual recognition is proposed. The aim is to achieve a closed-loop perception, judgment, and control of the entire USV docking process. It is applicable to autonomous docking tasks of USVs in complex environments such as ports and docks. Figure 2 and Figure 9 As shown, this method uses an aerial drone equipped with a camera to take overhead photos of the water surface scene, and combines this with an advanced rotating target detection algorithm (YOLOV11-OBB) to achieve accurate identification and attitude estimation of the unmanned surface vessel and the hatch, thereby improving the automation, safety and stability of the unmanned surface vessel docking process.

[0049] The method for docking an unmanned surface vessel includes the following steps:

[0050] (1) The UAV is equipped with a vision system to collect first position images of the unmanned surface vessel and the dock;

[0051] (2) The target detection model is used to detect the first location image. The detection results include the rotated bounding box of each target.

[0052] (3) Determine the corner points and key points of the target based on the rotating bounding box of the target, connect the auxiliary lines based on the corner points and key points, and calculate the relative distance and heading angle between the dock door and the unmanned surface vessel through the auxiliary lines;

[0053] (4) Determine whether the calculated relative distance and heading angle are within the preset range. If they are not within the preset range, calculate the first propulsion speed and the first heading adjustment angle to control the movement of the unmanned surface vessel, and then return to step (1); if they are within the preset range, proceed to step (5).

[0054] (5) Obtain the second position image of the dock door collected by the vision system on the unmanned surface vessel, determine the relative distance and heading angle between the dock door and the unmanned surface vessel based on the second position image, and calculate the second propulsion speed and the second heading adjustment angle to control the movement of the unmanned surface vessel;

[0055] (6) Repeat step (5) to achieve unmanned surface vessel docking.

[0056] Before drones can collect images, camera calibration is required: by calibrating the internal and external parameters of the cameras carried by drones and unmanned surface vessels, an accurate imaging model is obtained, which provides accuracy for the spatial positioning of targets in subsequent images.

[0057] And need to train and verify the target detection model in advance, the target detection model in this embodiment adopts YOLOV11-OBB model. Based on the rotating box target detection model (YOLOV11-OBB), the key targets including the dock cabin, the unmanned ship and the like are labeled and trained, the recognition model with the rotating target detection capability is constructed, and the model precision is verified through the data set and the measured images.

[0058] Then the distance and heading angle estimation and evaluation of the unmanned aerial vehicle and the unmanned ship are carried out. The relative distance and heading angle between the hatch and the unmanned ship are estimated by using the rotating box information of the target in the detection result and the camera calibration parameters, combining with the geometric projection method, and the estimation precision is evaluated by using the experiment, so as to provide a basis for control decision.

[0059] In the process of entering the dock, according to the distance and heading angle error between the unmanned ship and the hatch, the control right allocation between the unmanned aerial vehicle and the unmanned ship is dynamically adjusted. When the unmanned ship is far away from the hatch, the system preferentially uses the global visual information provided by the unmanned aerial vehicle for remote guidance; when the unmanned ship approaches the hatch and the heading angle is stable within a reasonable range, the control right is gradually switched to the local vision module of the unmanned ship, and fine berthing control is realized. In order to ensure the smooth transition of the control right, the system designs a weight fusion mechanism based on distance and heading angle, realizing the gradual switching from "unmanned aerial vehicle leading" to "unmanned ship leading".

[0060] In addition, when the system detects that the local perception of the unmanned ship is abnormal or the heading angle deviation exceeds the set threshold, the fallback mechanism will be triggered, and the control right will be returned to the unmanned aerial vehicle for auxiliary correction. After the state is restored, the control right is returned to the unmanned ship again, realizing the closed-loop control process of "guidance - autonomy - intervention - again autonomy", and improving the stability and robustness of the entering process.

[0061] In order to improve the operation safety of the unmanned ship in the approach stage of the dock cabin, the system introduces a short-distance anti-collision auxiliary control module. When the unmanned ship enters the fine adjustment area in front of the hatch (such as the distance is less than the set threshold 2m), the system further introduces anti-collision auxiliary logic on the basis of the entering control, so as to prevent the collision between the ship body and the hatch structure (especially the two side guardrails).

[0062] Specifically, by continuously monitoring the transverse distance between the two side angles of the front edge of the dock cabin and the edge of the unmanned ship. When it is detected that the ship body deviates from the center line or approaches any side boundary too close, the system will dynamically adjust the propulsion speed and the heading correction angle, preferentially avoiding the risk of collision, while taking into account the entering target. The anti-collision control module can be executed with the main control instruction, providing a layer of redundant safety protection for the unmanned ship without affecting the overall entering strategy, improving the adaptability and robustness in complex environment.

[0063] (1) as Figure 3Camera calibration: The cameras mounted on the USV and UAV are calibrated. The USV camera calibration adopts Zhang Zhengyou camera calibration method, which obtains the camera intrinsic parameters (including focal length, principal point position, distortion coefficient) by shooting a large number of standard chessboard patterns combined with the MATLAB calibration toolbox. The calibration data is used for subsequent image geometric calculation. The UAV camera calibration also adopts Zhang Zhengyou camera calibration method, supplemented by Aruco calibration board, so as to realize the visual estimation of spatial position and height. The system uses the estimatePoseSingleMarkers function in OpenCV to estimate the rotation matrix and translation vector. The Z-axis component of the translation vector can be used to estimate the distance from the camera to the calibration board, thereby assisting the UAV in height control and visual spatial perception.

[0064] (2) YOLOV11-OBB model training and verification

[0065] YOLOV11-OBB model training: In order to realize the rotation target detection of the hatch and the USV, the system is based on the deep learning detection model of YOLOV11-OBB. First, the dock and the USV are arranged in the experimental scene. The images of the dock and the USV are captured by the UAV at different heights through the Bluetooth remote controller, covering the multi-scale and multi-angle appearance changes of the target as much as possible. Then, the LabelImg tool is used to label the collected images in the format of Oriented Bounding Box (OBB), generating image and label data that meet the input requirements of YOLOV11-OBB. After completing the labeling, the Python script program is used to divide the training set, test set and validation set in the ratio of 8:1:1. After configuring the model training hyperparameters (such as training rounds, learning rate, batch size), the training is started. After completing the training task, the system will automatically save the detection model with the best performance (such as best.pt) for subsequent deployment and inference of the visual recognition module.

[0066] Model verification and deployment: The best.pt is loaded on the shore-based computer and connected to the real-time image of the UAV overhead camera for inference verification, which outputs the rotated bounding box, its category and confidence score in real time, assisting the operator in evaluating the target detection effect. If the model detection result fails to meet the accuracy requirements (such as false positives, missed detection or obvious deviation of the rotation angle of the bounding box), the image collection and labeling process will be re-executed to expand the training samples and improve the robustness of the model. The entire training and verification process fully considers the actual lighting conditions and target appearance changes in the experimental scene, ensuring that the trained model has good adaptability and universality for the dock and the USV target.

[0067] (3) UAV / USV distance, heading angle estimation and evaluation

[0068] UAV positioning and ranging model evaluation: UAV Aruco height estimation verification - first, before takeoff, the high-precision barometer is zero-point calibrated, then the UAV is started, and the UAV is slowly lifted to the specified height (e.g. 2m, 3m, 4m, etc. height layer) through the Bluetooth remote control, ensuring that the onboard camera view is vertically downward, and the calibration board is located in the center of the camera view. Then at each fixed height flight point, the system calls the estimatePoseSingleMarkers() function in OpenCV to monitor the height of the Aruco calibration board, and obtains the translation vector of the corresponding frame . The Z component represents the estimated spatial distance between the camera optical center and the center point of the calibration board. Finally, the height barometer readings and visual estimation values are recorded at each fixed height flight point respectively

[0069]

[0070] Evaluate the estimation accuracy. If the relative error is within the allowable range (e.g. <5%), the visual estimation result can be used for subsequent flight height control and spatial positioning tasks.

[0071] UAV positioning and ranging function verification: The calibration obtained camera parameters are imported into the image processing module to realize image positioning and ranging function. The camera parameters (including focal length , , principal point coordinates , ) are preloaded into the image processing program before the UAV platform takes off, for real-time use during flight. Then, using the pinhole imaging principle, the relative distance between the unmanned boat and the dock cabin under the UAV view is calculated, and the calculation formula is as follows:

[0072]

[0073] Where:

[0074] ( , ), ( , ) are the coordinates of two target pixels in the image;

[0075] 、 are the focal lengths of the camera in the x and y directions (units: pixels);

[0076] 、 are the principal point positions (units: pixels);

[0077] , is the horizontal distance between two targets, and

[0078] is the vertical distance between camera and target;

[0079] To verify the accuracy of the pinhole imaging ranging model, a pair of targets with a fixed length, such as two markers with a known distance of , are set up on an outdoor or indoor platform. The targets should be arranged roughly parallel to the flight plane of the UAV to ensure that their image projections are as distortion-free as possible.

[0080] After the UAV takes off, the targets are photographed at different heights, and two pixel coordinates , ) and , ) are selected. The distance between the two targets is estimated according to the pinhole imaging formula , and the specific formula is as follows:

[0081]

[0082] where is the estimated height value by Aruco. If the monitoring target has a certain height difference from the Aruco calibration plate plane, the value will be increased or decreased accordingly. Finally, compared with the known true value , the relative error is calculated:

[0083]

[0084] If the error is controlled within a reasonable range (e.g., less than 5%), the UAV visual ranging model is considered effective and reliable, and can be applied to subsequent unmanned boat positioning and guidance.

[0085] UAV heading angle estimation: To accurately identify the heading angle between the unmanned boat and the dock, the system uses the output characteristics of the YOLOV11-OBB model to the target's rotated bounding box in the image after the model is deployed. The axis of the dock and the unmanned boat target outline is calculated, and the angle is calculated. This method includes the following steps:

[0086] YOLO-OBB annotation format is (class_index, x1, y1, x2, y2, x3, y3, x4, y4). The coordinates of the four corners of the bounding box are normalized between 0 and 1. The numbering order of the corners is usually a convention when data labeling. It is always in the order of the upper right corner (corner 1), then in clockwise order, the lower right corner (corner 2), the lower left corner (corner 3), and the upper left corner (corner 4). And since the model will learn the local geometric features of the corners, the corners of the actually used model are dynamic and follow the rotation. The system only needs to load the trained YOLOV11-OBB target detection model (best.pt) and perform inference on the overhead image obtained by the unmanned aerial vehicle on-board camera. In the detection result, each target (such as a dock or an unmanned boat) is output in the form of a rotating bounding box. As shown in Figure 5 and Figure 6 , the midpoint of corner 1 and corner 4 (denoted as M1) and the midpoint of corner 2 and corner 3 (denoted as M2) are extracted, and the midpoint M1 and M2 are connected to obtain the axial center line of the target.

[0087] Then the system further extracts the key point coordinates according to the detection position: for the dock, the midpoint of corner 1 and corner 4 is taken as the center point of the rear end (the entrance of the hatch) of the dock; for the unmanned boat, the midpoint of corner 1 and corner 4 is taken as the center point of the front end (the bow end) of the unmanned boat, and the midpoint of corner 2 and corner 3 is taken as the center point of the rear end (the stern end) of the unmanned boat. The center points of the hatch entrance, the bow, and the stern are connected to obtain three key lines on the image plane: the axial center line (M1-M2), the hatch-to-stern connecting line, and the hatch-to-bow connecting line. The specific determination steps are as follows:

[0088] ① Target bounding box corner reading

[0089] The model inference outputs the rotating bounding box coordinates of each target (such as a dock or an unmanned boat) in the format (x1, y1, x2, y2, x3, y3, x4, y4). The system reads the coordinates according to the fixed corner order (right up → right down → left down → left up) and restores the normalized coordinates to pixel coordinates.

[0090] ② Axial center line extraction

[0091] Calculate the midpoint coordinates of corner 1 and corner 4 , and the midpoint coordinates of corner 2 and corner 3 . Connect the midpoints M1 and M2 to obtain the main axis direction line of the target, which is used to represent the attitude axis of the unmanned boat or the dock.

[0092] ③ Target center point and front and rear end differentiation

[0093] The system takes the main axis line The direction of the image is the basis for the determination. According to the orientation of the unmanned ship or the dock cabin in the image, the end point close to the top or the left side of the image is determined as the "front end", and the other end is the "rear end".

[0094] ④ Key point coordinate conversion

[0095] For the above extracted key points, the system converts the camera intrinsic parameters (focal length fx, f_y and principal point Cx, Cy) and Aruco estimated height Z C , the image pixel coordinates (u, v) are converted into physical coordinates (X C , Y C ) in the camera coordinate system:

[0096]

[0097] Thus, the physical positions of the dock cabin entrance center point, the ship bow center point and the ship stern center point in space are obtained.

[0098] ⑤ Construction of three key lines

[0099] L1: Axial center line (M1-M2) - describes the attitude direction of the unmanned ship itself;

[0100] L2: Cabin door to stern connecting line - the center point of the dock cabin entrance to the center point of the stern end of the unmanned ship;

[0101] L3: Cabin door to bow connecting line - the center point of the dock cabin entrance to the center point of the bow end of the unmanned ship.

[0102] In order to calculate the angle, the system converts the pixel distance of the three key lines in the image plane into physical distance according to the pinhole imaging model, which is denoted as side length a, b and c respectively. According to the law of cosines, the included angle of the triangle formed by the three sides is calculated, and the supplementary angle of the heading angle of the dock cabin relative to the bow end of the unmanned ship is derived, and the calculation formula is as follows:

[0103]

[0104] And the heading angle of the dock cabin relative to the bow end of the unmanned ship is derived, and the calculation formula is as follows:

[0105]

[0106] Unmanned ship positioning and distance measurement model evaluation: In order to realize the accurate positioning and navigation control of the unmanned ship when approaching the entrance of the dock cabin, the system identifies the Aruco calibration board set at the dock cabin based on the onboard camera, and combines the pose estimation algorithm of OpenCV to realize the accurate calculation of the relative distance and angle of the cabin door within 5 meters, and then realizes the accurate docking of the unmanned ship. The specific verification method is as follows:

[0107] As Figure 4 shown, unmanned boat Aruco distance value estimation verification: install the coded Aruco calibration board at the cabin door entrance position, the size of the calibration board needs to be pre-set for subsequent unmanned boat camera accurate ranging calculation. The standard camera is mounted in front of the unmanned boat, and the video stream image is collected to calculate the three-dimensional pose of the Aruco calibration board through the estimatePoseSingleMarkers() function in OpenCV, and the translation vector of the corresponding frame is obtained . The Z component represents the depth distance estimation value between the camera (i.e. the unmanned boat) and the calibration board (i.e. the cabin door); the X component represents the left-right offset distance between the camera (i.e. the unmanned boat) and the calibration board (i.e. the cabin door). Finally, the unmanned boat is fixed at different known depth distance (such as 1m, 2m, 3m, 4m) position points to shoot the calibration board, and the known depth distance value and the visual estimation value are calculated.

[0108]

[0109] If the ranging error is controlled within 5%, it is considered that the visual estimation model meets the accuracy requirements of short-distance navigation control.

[0110] Unmanned boat heading angle deviation estimation: during the process of the unmanned boat approaching the cabin door, the system identifies the Aruco calibration board on the cabin door and extracts its position horizontal coordinate u in the image plane. Considering that the camera is fixedly installed in front of the boat body, if the Aruco calibration board deviates from the principal point (i.e. the center of the camera optical axis) position horizontal coordinate cx in the image, it indicates that there is a deviation in the heading of the boat body. The angle deviation in radian is the heading deviation angle of the cabin door relative to the unmanned boat, and its calculation formula is as follows:

[0111] If the degree system is used, the angle deviation is:

[0112]

[0113] (4) Unmanned aerial vehicle and unmanned boat control right switching

[0114] To improve the control accuracy and task robustness of the unmanned boat in the automatic docking process, a control right dynamic switching and instruction weighting mechanism based on multi-source information fusion is proposed. The mechanism introduces two state variables, distance and heading angle , dynamically calculates the fusion weight coefficient, and realizes the joint scheduling of the control suggestions of the unmanned aerial vehicle and the unmanned boat.

[0115] The total control instruction of the system is represented as follows:

[0116]

[0117] wherein: is the total control vector output to the propulsion control module, containing the speed and the heading angle correction angle The unmanned boat calculates the control suggestion for the speed of the unmanned boat according to local perception and the heading angle correction angle ; The unmanned aerial vehicle calculates the control suggestion for the speed of the unmanned boat according to the overhead view and the heading angle correction angle ; is a fusion weight coefficient function, satisfying .

[0118] In order to balance the navigation stability at long distances and the fine maneuverability at short distances, the system dynamically adjusts the weight coefficient according to the state interval of D and . The specific method is as follows:

[0119] ① Long-distance guidance stage (D ≥ 5.5m)

[0120] When the unmanned boat is more than 5.5m away from the hatch, the system completely relies on the overhead view provided by the unmanned aerial vehicle for global guidance. The unmanned aerial vehicle obtains the relative position and attitude information of the unmanned boat and the dock hatch through a rotating box detection model (such as YOLOv11-OBB), and outputs the appropriate propulsion speed and heading adjustment angle of the unmanned boat, quickly guiding the unmanned boat to the hatch area.

[0121] The set weight function is:

[0122]

[0123] At this time, the control formula is:

[0124]

[0125] In this stage, the propulsion speed of the unmanned boat is set to 0.5-1.0m / s. Considering the rapid response and heading adjustment stability

[0126] ② Weight fusion excessive stage (D )

[0127] In order to avoid the smooth transition from long-distance unmanned aerial vehicle guidance to short-distance boat end precise adjustment, and prevent system instability caused by sudden change of control right. In this stage, the system no longer uses single control source output, but introduces multi-source weighted hybrid control strategy:

[0128] Set weight parameters:

[0129]

[0130] Control formula at this time:

[0131]

[0132] where

[0133]

[0134] Weight function According to the distance And the heading angle error Together determine. Its design is based on the two-dimensional linear weight function of distance and heading angle .

[0135] The linear weight function based on distance can be expressed as:

[0136]

[0137] where , The corresponding transition interval is [4.5m, 5.5m].

[0138] The linear weight function based on heading angle can be expressed as:

[0139]

[0140] where

[0141] To further improve the stability and adaptability of the transition stage control strategy, the distance factor and the heading angle factor are jointly modeled to output the final control weight:

[0142]

[0143] where the parameter is the balance coefficient, and

[0144] ③ Close-range fine adjustment stage (D ≤4.5m ∩ ≤10°)

[0145] When the unmanned ship is less than the set threshold of 4.5m and the heading angle is within the threshold of 10°, the system completely relies on the front camera of the unmanned ship for accurate adjustment. The unmanned ship obtains the relative position of the unmanned ship and the dock through the Aruco code through the front camera. The attitude information between the unmanned ship and the dock is obtained through the visual model. Thus, the appropriate output of the unmanned ship's propulsion speed is output and the heading adjustment angle , so as to accurately guide the unmanned ship to complete the docking operation.

[0146] The set weight function:

[0147]

[0148] The control formula at this time:

[0149]

[0150] In this stage, the propulsion speed of the unmanned ship is set to 0.1-0.2 m / s to improve stability and docking accuracy.

[0151] ④ Heading angle deviation back correction stage )

[0152] If the attitude angle error of the unmanned ship exceeds the heading angle deviation tolerance range in the close-range segment, the system designs a linear weight control back mechanism based on the heading angle deviation, gradually transfers the control right to the unmanned aerial vehicle end, detects the heading angle through the overhead view of the unmanned aerial vehicle, adjusts the attitude angle of the unmanned ship to , and then transfers the control right again.

[0153] The set weight parameter:

[0154]

[0155] The control formula at this time:

[0156]

[0157] The linear weight function based on the heading angle can be expressed as

[0158]

[0159] wherein , , and the corresponding transition interval is [8 , 12 ].

[0160] ⑤ Visual perception abnormal back mechanism

[0161] When the front camera of the unmanned boat fails to recognize the Aruco calibration board on the hatch door due to external factors such as surges, water mist or light interference (e.g. no valid hatch door calibration board features in the image, or the recognition result confidence is lower than the set threshold), the system will determine that the current perception module is not available. To ensure that the task is not interrupted, the system will immediately enable the unmanned aerial vehicle image source as the main perception channel, and switch control back to the unmanned aerial vehicle side to continue performing the heading angle and distance correction task until the vision conditions return to normal.

[0162] The set weight function:

[0163]

[0164] The control formula at this time:

[0165]

[0166] This fallback mechanism ensures that the system has self-diagnosis and control strategy switching capability during the critical close-in berthing phase, effectively avoiding the failure of entering the dock due to the failure of a single sensor or excessive attitude error, and improving the robustness and fault tolerance of the overall control system.

[0167] In this fallback correction phase, a cruise value of 0.3 m / s is used to ensure the stability and responsiveness of the heading angle and distance correction.

[0168] (5) Close-range collision avoidance assistance

[0169] During the unmanned boat docking control process, to ensure the berthing accuracy and the safety of the ship, the system introduces a close-range collision avoidance assistance control module during the close-range fine adjustment phase (e.g. distance D < 2m). Based on the straight-line distance difference between the left and right corner points of the ship's bow (i.e. the left front corner point 4 and the right front corner point 1) and the edge corner points of the hatch door entrance on both sides (i.e. the left front corner point 4 and the right front corner point 1 of the dock entrance), the module realizes collision avoidance control and heading correction.

[0170] ① Drive-in phase corner point distance calculation and dynamic control

[0171] When the unmanned boat enters the entrance area of the dock (usually within 1-2 meters outside the hatch), as shown in Figure 7 , the unmanned aerial vehicle vision system recognizes and tracks the following key points:

[0172] Left front corner point of the bow to the right front corner point of the entrance ;

[0173] Right front corner point of the bow to the right front corner point of the entrance .

[0174] The system calculates two key geometric distances in real time:

[0175] The minimum distance between the left front corner point of the bow and the right front corner point of the entrance to the dock . ;

[0176] The minimum distance between the right front corner point of the bow and the right front corner point of the entrance to the dock . .

[0177] In the anti-collision control, as the distances between the left and right corner points of the bow and the boundaries of the dock on both sides and become smaller, it means that the ship is gradually approaching the boundaries of the dock, and it is necessary to adjust the speed and heading angle to avoid collision.

[0178] If it is detected that: indicates that the left side is close to the guardrail, and a right deviation adjustment should be triggered; indicates that the left side is close to the guardrail, and a left deviation adjustment should be triggered;

[0179] If , speed reduction operation and heading angle adjustment are performed; if , immediate emergency stop or manual takeover mode is triggered.

[0180] where , (which can be determined according to the width of the dock and the ship width)

[0181] First, speed control

[0182]

[0183] where

[0184] is the current control speed (adjusted dynamically according to the safety distance).

[0185] is the basic navigation speed, which is 0.1-0.2 m / s in this close-range precision control phase

[0186] Then calculate the distance deviation to obtain the heading angle correction . The goal is to drive into the center of the dock, and if the left and right distances are not the same:

[0187]

[0188] If , it indicates that the ship is deviating to the right, and a left deviation is needed;

[0189] If This indicates that the ship is veering to the left and needs to veer to the right.

[0190] Finally d is mapped to the heading angle correction. The hyperbolic tangent function is used for nonlinear mapping:

[0191]

[0192] in,

[0193] To control the gain, adjustments can be made empirically based on the hull width and reaction sensitivity.

[0194] Positive and negative indicate which side is biased towards, and the unit is meters;

[0195] The unit is radians or degrees.

[0196] Finally, the total heading angle is obtained.

[0197]

[0198] in,

[0199] This represents the reference heading angle calculated by the unmanned surface vessel based on the center point of its approach to the target.

[0200] It is a heading correction angle introduced to prevent collisions and achieve close-range obstacle avoidance assistance.

[0201] This module does not completely replace the guidance system, but rather adjusts the guidance heading angle incrementally. Make corrections to adjust the overall control angle. While maintaining the accuracy of the docking target, improve the safety tolerance for nearby guardrails.

[0202] ② Edge distance calculation and dynamic control during docking phase

[0203] After the unmanned surface vessel enters the dock, as Figure 8 As shown, the system uses a UAV-based vision system to identify the four corners of the ship's hull and the two side lines of the hatches (i.e., the two long sides of YOLOv11-OBB). The system then estimates four key distances:

[0204] bow left front corner Minimum distance to the right side line of the dock entrance ;

[0205] bow right front corner Minimum distance to the left side line of the dock entrance .

[0206] Bow Right Rear Corner Point Minimum distance to the left side line of the docking entrance ;

[0207] Bow Left Rear Corner Point Minimum distance to the right side line of the docking entrance .

[0208] In the collision avoidance control, as the distance between the four corner points of the bow and the boundary of the dock decreases, it means that the ship is gradually approaching the boundary of the dock, and it is necessary to adjust the speed and heading angle to avoid collision.

[0209] If detected:

[0210] It means that the left side is close to the guardrail, and the right deviation adjustment should be triggered;

[0211] It means that the right side is close to the guardrail, and the left deviation adjustment should be triggered;

[0212] If , speed reduction operation and heading angle adjustment are performed.

[0213] If , immediate emergency stop or manual takeover mode is triggered.

[0214] Where , (the width of the dock and the width of the ship can be determined)

[0215] First, speed control

[0216]

[0217] Where

[0218] is the current control speed (adjusted dynamically according to the safety distance).

[0219] is the basic navigation speed, which is 0.1-0.2m / s in the close-range precise control phase

[0220] Then calculate the distance deviation to obtain the heading angle correction . The goal is to drive into the center of the dock, if the left and right distances are not consistent:

[0221]

[0222]

[0223] If , it means that the ship is deviated to the right, and the left deviation is required.

[0224] If , it means the ship is deviating to the left, and the right rudder is needed.

[0225] Finally, the is mapped to the heading angle correction , using the hyperbolic tangent function for nonlinear mapping:

[0226]

[0227] Where:

[0228] is the control gain, which can be adjusted empirically according to the ship width and response sensitivity;

[0229] Positive and negative represent which side to deviate, and the unit is meters;

[0230] The unit is radians or degrees.

[0231] Finally, the total heading angle

[0232]

[0233] Where:

[0234] represents the reference heading angle calculated by the unmanned ship according to the front center point of the target

[0235] is the heading correction angle introduced to prevent collision and achieve close-range auxiliary obstacle avoidance.

[0236] This module does not completely replace the guidance system, but rather corrects the guided heading angle in an incremental manner, so that the total control angle maintains the alignment of the target while improving the safety tolerance of the nearby guardrail. Embodiment 2

[0237] An unmanned ship docking system based on air-ground collaborative visual recognition in this embodiment includes a UAV equipped with a high-definition camera, an unmanned ship performing docking operations, and a shore-based control center. The shore-based control center includes a detection module, a calculation module, a state discrimination module, and a control module.

[0238] The vision system of the UAV is used to collect the first position image of the unmanned ship and the dock.

[0239] The detection module is used to detect the collected first position image using a target detection model, and the detection result includes the rotation bounding box of each target.

[0240] The computing module is configured to determine an angle point and a key point of the target according to a rotating bounding box of the target, connect an auxiliary line according to the angle point and the key point, calculate a relative distance and a heading angle between the dock cabin door and the unmanned ship through the auxiliary line, and calculate a first propulsion speed and a first heading adjustment angle;

[0241] The state discrimination module is configured to determine whether the calculated relative distance and heading angle reach a preset range,

[0242] The control module is configured to control the unmanned ship to move through the first propulsion speed and the first heading adjustment angle according to a determination result that the state discrimination module does not reach the preset range, and control the unmanned ship to move through a second propulsion speed and a second heading adjustment angle according to the state discrimination module reaching the preset range.

[0243] The vision system of the unmanned ship is configured to collect a second position image of the dock cabin door, determine the relative distance and the heading angle between the dock cabin door and the unmanned ship according to the second position image, and calculate the second propulsion speed and the second heading adjustment angle to control the unmanned ship to move.

[0244] The unmanned aerial vehicle is equipped with a high-definition camera, a Pixhawk flight controller, a Bluetooth remote controller, a high-precision barometer, a laser range finder, an IMU module, a GPU module, and an edge computing platform. The high-definition camera is used for image acquisition; the Pixhawk flight control module is used for controlling flight attitude and autonomous path execution; the Bluetooth remote controller is used for manual and remote control during the task; the high-precision barometer, the laser range finder, the IMU module, and the GPU module are used for real-time monitoring of the flight state of the unmanned aerial vehicle; the edge computing platform is used for data preprocessing; the unmanned ship is internally integrated with a self-navigation control module and a wireless communication module, and can accurately control the propeller to perform attitude adjustment and path navigation operation according to control instructions. The shore-based operation center serves as the core information processing and control center of the system, is responsible for establishing stable communication connection with the unmanned aerial vehicle and the unmanned ship, receiving image and state data returned by the unmanned aerial vehicle, performing key tasks such as image recognition, attitude judgment, and control instruction generation, and simultaneously serving as an interface for manual intervention and remote monitoring.

[0245] The system integrates an advanced rotating target detection algorithm YOLOV11-OBB, realizes target monitoring and attitude detection of the unmanned ship and the cabin door in the image, combines camera calibration and monocular imaging geometric model, and can calculate the relative distance and heading angle deviation between the unmanned ship and the dock cabin. The state discrimination module built in the shore-based center dynamically judges whether the unmanned aerial vehicle vision guidance or the unmanned ship autonomous execution is performed in the current stage according to the heading angle and distance data, and switches the corresponding control strategy. The system also has function modules such as task recording, abnormality detection, path planning, and instruction issuing, and constitutes a collaborative and efficient, stable and reliable intelligent docking control system.

Claims

1. A method for docking unmanned surface vessels based on air-ground collaborative visual recognition, characterized in that, Includes the following steps: (1) The UAV is equipped with a vision system to collect first position images of the unmanned surface vessel and the dock; (2) The target detection model is used to detect the first location image, and the detection results include the key points of each target; (3) Connect auxiliary lines according to the key points of the target, and calculate the relative distance and heading angle between the dock door and the unmanned surface vessel using the auxiliary lines; (4) Determine whether the calculated relative distance and heading angle are within the preset range. If they are not within the preset range, calculate the first propulsion speed and the first heading adjustment angle to control the movement of the unmanned surface vessel, and then return to step (1); if they are within the preset range, proceed to step (5). (5) Obtain the second position image of the dock door collected by the vision system on the unmanned surface vessel, determine the relative distance and heading angle between the dock door and the unmanned surface vessel based on the second position image, and calculate the second propulsion speed and the second heading adjustment angle to control the movement of the unmanned surface vessel; (6) Repeat step (5) to achieve unmanned surface vessel docking.

2. The unmanned surface vessel docking method according to claim 1, characterized in that, The preset range includes a transition range and a fine-tuning range. If the fine-tuning range is reached, then step (5) is performed; if the transition range is reached, then: Based on the relative distance and heading angle between the dock hatch and the unmanned surface vessel determined from the first position image, calculate the first propulsion speed and the first heading adjustment angle. The second position image of the dock door is acquired by the vision system on the unmanned surface vessel (USV). The relative distance and heading angle between the dock door and the USV are determined based on the second position image. The second propulsion speed and the second heading are then calculated. The first propulsion speed and the first heading adjustment angle, as well as the second propulsion speed and the second heading adjustment angle, are weighted and fused to obtain the joint propulsion speed and the joint heading adjustment angle to control the movement of the unmanned surface vessel.

3. The unmanned surface vessel docking method according to claim 2, characterized in that, In step (5), after controlling the movement of the unmanned surface vessel (USV) with the second propulsion speed and the second heading adjustment angle, the second position image of the dock door collected by the vision system on the USV is obtained. The relative distance and heading angle between the dock door and the USV are determined based on the second position image. The heading angle between the dock door and the USV is judged. If the heading angle does not reach the fine adjustment range, the combined propulsion speed and the combined heading adjustment angle are calculated to control the movement of the USV. Otherwise, the second propulsion speed and the second heading adjustment angle are calculated to control the movement of the USV.

4. The unmanned surface vessel docking method according to claim 1, characterized in that, Step (6) specifically includes: (61) Determine whether the relative distance and heading angle between the dock hatch and the unmanned surface vessel reach the collision avoidance range. If not, return to step (5); if the collision avoidance range is reached, proceed to step (62). (62) The distance between the corner point of the unmanned surface vessel and the edge of the dock door is obtained through the vision system of the UAV. By comparing it with the preset safe distance, the heading angle correction and propulsion speed of the unmanned surface vessel are obtained, the optimal obstacle avoidance path is generated, and the unmanned surface vessel is controlled to enter the dock.

5. The unmanned surface vessel docking method according to claim 4, characterized in that, The formula for calculating the heading angle correction of the unmanned surface vessel in step (62) is as follows: in, To control the gain, This is a quantity representing the direction of the heading angle correction.

6. The unmanned surface vessel docking method according to claim 1, characterized in that, The target detection model adopted is the YOLOV11-OBB model.

7. The unmanned surface vessel docking method according to claim 1, characterized in that, The target includes an unmanned surface vessel (USV) and a dock. The key points include the center point of the dock entrance, the center point of the front of the USV, and the center point of the rear of the USV. The auxiliary lines include an axial centerline, a bow line, and a stern line. The axial centerline is the line connecting the center points of the front and rear of the USV. The bow line is the line connecting the center points of the dock entrance and the front of the USV. The stern line is the line connecting the center points of the dock entrance and the rear of the USV.

8. The unmanned surface vessel docking method according to claim 1, characterized in that, The second position image of the dock door acquired by the vision system on the unmanned surface vessel includes a calibration plate set on the door. The position coordinates of the calibration plate on the second position image are obtained, and the depth distance and left and right offset distance between the unmanned surface vessel and the calibration plate are calculated according to the size of the calibration plate, thereby determining the relative distance and heading angle between the dock door and the unmanned surface vessel.

9. The unmanned surface vessel docking method according to claim 8, characterized in that, If the unmanned surface vessel's vision system fails to recognize the hatch calibration plate due to external factors, it will directly return to step (1).

10. A docking system for unmanned surface vessels based on air-ground collaborative visual recognition, characterized in that, The system includes unmanned aerial vehicles (UAVs) and unmanned surface vessels (USVs) equipped with vision systems, as well as a shore-based operations center. The shore-based operations center includes a detection module, a calculation module, a status determination module, and a control module. The drone's vision system is used to acquire initial positional images of the unmanned surface vessel and its dock. The detection module is used to detect targets in the first location image using a target detection model. The detection results include the rotated bounding box of each target. The calculation module is used to determine the corner points and key points of the target based on the target's rotated bounding box, connect auxiliary lines based on the corner points and key points, calculate the relative distance and heading angle between the dock hatch and the unmanned surface vessel through the auxiliary lines, and calculate the first propulsion speed and the first heading adjustment angle. The state determination module is used to determine whether the calculated relative distance and heading angle are within the preset range. The control module is used to control the movement of the unmanned surface vessel by means of a first propulsion speed and a first heading adjustment angle, based on the judgment result of the state discrimination module that the preset range has not been reached. If the state discrimination module reaches the preset range, the unmanned surface vessel is controlled to move by the second propulsion speed and the second heading adjustment angle. The unmanned surface vessel's vision system is used to acquire a second position image of the dock door, determine the relative distance and heading angle between the dock door and the unmanned surface vessel based on the second position image, and calculate a second propulsion speed and a second heading adjustment angle to control the movement of the unmanned surface vessel.

Images