An unmanned aerial vehicle cooperative ship berthing situation monitoring and early warning method and system based on a retinal segmentation mask task framework

Abstract

The application discloses a kind of unmanned plane cooperative ship berthing situation monitoring and early warning method and system based on retina segmentation mask.First, obtain ship berthing image dataset, and construct the berthing segmentation model that is fused retina segmentation mask task framework and SE attention mechanism to train.Then, real-time acquisition ship berthing video using unmanned plane, and accurately identify the polygon of ship and berth polygon in video by trained model.Next, the nearest distance solving method is used to calculate the nearest pixel distance between the two polygons, and it is converted into the actual distance between ship and berth, to realize accurate ranging.At the same time, the pixel displacement of the ship is calculated using Deep SORT tracking algorithm, to realize real-time measurement of berthing speed.Finally, referring to the standard of International Maritime Organization, combined with the actual performance of the ship, a three-layer electronic early warning fence containing risk alarm, early warning and emergency refuge is designed, forming a depth defense system.This method realizes the automatic monitoring and intelligent early warning of the distance and speed of the whole process of ship berthing, effectively improves the safety and efficiency of berthing operation.

Description

Technical Field

[0001] This invention relates to the field of ship berthing technology, and in particular to a method and system for monitoring and early warning of ship berthing status in collaboration with unmanned aerial vehicles (UAVs) based on a retinal segmentation mask mission framework. Background Technology

[0002] Berthing operations are a critical aspect of navigation. Especially with the increasing size of ships, the blind spots on the bridge have significantly expanded, severely weakening the helmsman's situational awareness and reducing the accuracy of ship-to-shore distance judgments, thus greatly increasing the safety risks of berthing operations. Therefore, developing environmental perception technologies suitable for large or high-blind-spot vessels, providing accurate, real-time, and intuitive environmental information, is a core prerequisite for achieving safe, precise, and efficient autonomous berthing.

[0003] The increasing size of ships leads to larger blind spots on the bridge, posing a significant challenge to the pilot's situational awareness and distance judgment capabilities using traditional berthing methods, thus significantly increasing operational risks. Therefore, it is necessary to develop high-precision, real-time, and visualized environmental perception technologies to support safe berthing.

[0004] Currently, berthing perception mainly relies on the following technical approaches: 1) Fixed ranging: using shore-based / shipborne sonar, laser rangefinders, etc. to obtain ship-shore distance data, but the deployment is rigid and the information dimension is limited; 2) Shipborne multi-sensor fusion: integrating cameras, radar, and other equipment to improve perception robustness, but the system cost is high, data processing is complex, and lidar is expensive; 3) Shore-based visual monitoring: deploying cameras at the dock to monitor ships, but it is easily obstructed and the reliability of identifying general ship targets is insufficient; 4) Artificial marker recognition: setting up special markers at the berth for shipborne cameras to recognize, although it improves the recognition rate, there are problems such as cumbersome deployment and maintenance and impact on the port environment; 5) Satellite navigation assistance: susceptible to interference from the multipath effect at the dock, and the accuracy is unstable.

[0005] However, existing technologies still have the following limitations: 1) The perspective is fixed and cannot dynamically cover the blind spots of large ships; 2) The information is mainly numerical and lacks intuitive visualization of the situation, resulting in a high cognitive load for the driver; 3) High-precision solutions are costly, while low-cost solutions are constrained by environmental adaptability (such as lighting and weather); 4) Visual algorithms are not robust enough in complex scenarios; 5) The linkage between the perception system and the early warning and control links is weak, and an intelligent closed loop has not been formed.

[0006] In summary, while current ship berthing sensing technologies have made progress in terms of single measurement accuracy or application in specific scenarios, they still have significant shortcomings in terms of system flexibility, situational awareness, environmental robustness, and closed-loop intelligence. Summary of the Invention

[0007] This invention provides a method and system for monitoring and early warning of ship berthing status in collaboration with unmanned aerial vehicles (UAVs) based on a retinal segmentation mask mission framework, in order to overcome the above-mentioned technical problems.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows: A method for UAV-assisted ship berthing situation monitoring and early warning based on a retinal segmentation mask task framework, comprising the following steps: S1: Obtain a dataset of images of ships berthing; S2: Establish a berthing segmentation model based on the retinal segmentation mask task framework, and train the berthing segmentation model based on the ship berthing image dataset to obtain the trained berthing segmentation model; S3: Use a drone to acquire actual ship berthing videos, and based on the trained berthing segmentation model, identify the instance segmentation results of ships and berths in the actual ship berthing videos. The instance segmentation results of ships and berths include bounding box coordinates, classification probabilities, and instance masks. The instance masks include ship polygons and berth polygons. S4: Use the nearest distance solution method to obtain the nearest pixel distance between the ship polygon and the berth polygon; S5: Convert the nearest pixel distance between the ship polygon and the berth polygon into the actual nearest distance between the ship and the berth; S6: Using the Deep SORT tracking algorithm and the instance segmentation results, the pixel displacement during the ship's berthing process is obtained, thereby obtaining the ship's real-time ground speed during the berthing process. S7: Based on the actual closest distance and real-time ground speed, combined with the preset multi-level electronic early warning fence rules, generate berthing safety early warning information. The multi-level electronic early warning fence rules are designed according to the actual ship performance.

[0009] Furthermore, the berthing segmentation model includes a preliminary instance segmentation model and a refined instance segmentation model; The preliminary instance segmentation model is used to perform preliminary instance detection and segmentation on the ship berthing image dataset to obtain initial instance segmentation features; The refined instance segmentation model is a model built based on the Retina Mask model architecture, used to refine the initial instance segmentation features to obtain instance segmentation results for ships and berths.

[0010] Furthermore, the preliminary instance segmentation model includes: a backbone network, a neck network, and a detection head network; The backbone network is used to extract multi-scale ship feature maps from ship berthing images and transmit them to the neck network; The neck network is used to extract enhanced features with fused contextual information from multi-scale ship feature maps and transmit them to the detection head network; The detection head network is used to decode the enhanced features that fuse contextual information, and to perform bounding box regression, class confidence prediction and pixel-level mask generation operations for ship instances in parallel, thereby outputting initial instance segmentation features.

[0011] Furthermore, the backbone network includes a first convolutional block, a second convolutional block, a first convolutional extraction layer, a third convolutional block, a second convolutional extraction layer, a fourth convolutional block, a third convolutional extraction layer, a fifth convolutional block, a fourth convolutional extraction layer, a spatial pyramid module, and a mask prediction branch; The neck network includes a first upsampling layer, a first SE attention mechanism, a fifth convolutional extraction layer, a second upsampling layer, a second SE attention mechanism, a first edge extraction layer, a sixth convolutional block, a connection layer, a second edge extraction layer, a seventh convolutional block, a third SE attention mechanism, and a third edge extraction layer; The detection head network includes a first instance segmentation intersection detection module, a second instance segmentation intersection detection module, and a third instance segmentation intersection detection module; The first convolutional block is used to extract primary spatial features from the ship berthing image and transmit them to the second convolutional block. The second convolutional block is used to extract low-level features from the output features of the first convolutional block and transmit them to the first convolutional extraction layer; The first convolutional extraction layer is used to extract the first refined features from the output features of the second convolutional block and transmit them to the third convolutional block; The third convolutional block is used to extract deep features from the output features of the first convolutional extraction layer and transmit them to the second convolutional extraction layer; The second convolutional extraction layer is used to extract the refined features from the output features of the third convolutional block and transmit them to the fourth convolutional block and the first SE attention mechanism, respectively. The fourth convolutional block is used to extract the first abstract feature from the output features of the second convolutional extraction layer and transmit it to the third convolutional extraction layer; The third convolutional extraction layer is used to extract the refined features from the output features of the fourth convolutional block and transmit them to the fifth convolutional block and the second SE attention mechanism. The fifth convolutional block is used to extract the second abstract feature from the output feature of the third convolutional extraction layer and transmit it to the fourth convolutional extraction layer; The fourth convolutional extraction layer is used to extract the final backbone output features from the output features of the fifth convolutional block and transmit them to the spatial pyramid module. The spatial pyramid module is used to extract pooled features that incorporate multi-scale contextual information from the output features of the fourth convolutional extraction layer and transmit them to the mask prediction branch. The mask prediction branch is used to extract mask prototype features for predicting the fine boundaries of the object from the multi-scale pooling features output by the spatial pyramid module, and transmits them to the first upsampling layer and the third SE attention mechanism respectively. The first upsampling layer is used to upsample the output features of the mask prediction branch and output a first upsampled feature whose size matches the output features of the third convolution extraction layer; The second SE attention mechanism is used to reweight the output features of the third convolutional extraction layer and the first upsampled features through channels to enhance the target-related features and suppress background interference before transmitting them to the fifth convolutional extraction layer. The fifth convolutional extraction layer is used to extract the first convolutional features from the output features of the second SE attention mechanism and then transmit them to the second upsampling layer; The second upsampling layer is used to upsample the output features of the fifth convolutional extraction layer and output a second upsampled feature with a size that matches the output features of the second convolutional extraction layer. The first SE attention mechanism is used to perform channel reweighting on the output features of the second convolutional extraction layer and the second upsampled features to enhance the target-related features and suppress background interference before transmitting them to the first edge extraction layer; The first edge extraction layer is used to enhance the edge features of the output features of the first SE attention mechanism to obtain high-level features with clear edge information, and use them as the input features of the first instance segmentation intersection detection module; The first instance segmentation intersection detection module outputs preliminary target category, bounding box, and initial mask prediction based on the input features; The sixth convolutional block is used to perform deep feature extraction on the output features of the first edge extraction layer and output a high-dimensional abstract feature. The connection layer is used to concatenate the output features of the sixth convolutional block and the output features of the fifth convolutional extraction layer according to the channel dimension; The second edge extraction layer is used to perform edge-aware feature fusion on the output features of the connection layer to obtain context features that integrate deep semantics and detailed edge information, and use them as input features of the second instance segmentation intersection detection module; The second instance segmentation intersection detection module outputs further refined target categories, optimized bounding boxes, and mask predictions based on the input features; The seventh convolutional block is used to compress and optimize the output features of the second edge extraction layer, and output features that further improve feature resolution and target semantics; The third SE attention mechanism is used to reweight the output features of the seventh convolutional block and the output features of the mask prediction branch to enhance the target-related features and suppress background interference before transmitting them to the third edge extraction layer. The third edge extraction layer is used to refine the output features of the third SE attention mechanism to obtain the final features with accurate contour and detail information, and uses them as the input features of the third instance segmentation intersection detection module. The third instance segmentation intersection detection module outputs the final, high-confidence target category, bounding box, and instance segmentation mask based on the input features.

[0012] Furthermore, the specific steps for obtaining the nearest pixel distance between the ship polygon and the berth polygon using the nearest distance solution method include: Calculate the Euclidean distance between all point pairs in the ship polygon and berth polygon to construct the point set distance matrix. The formula for calculating the Euclidean distance is:

[0013] In the formula, and For the first point in the polygonal point set of the ship i The point and the first j One point; and For the berth point in the polygonal point set of the berth i The point and the first j One point; Based on the point set distance matrix, obtain the index of the ship polygon point corresponding to the minimum Euclidean distance. and the index of the berth polygon points , is represented as:

[0014] Based on the indices of the ship polygon points and the berth polygon points, the coordinates of the nearest point on the berth polygon and the coordinates of the nearest point on the berth polygon are obtained respectively. Calculate the distance between the nearest point on the ship polygon and the nearest point on the berth polygon to obtain the nearest pixel distance between the ship polygon and the berth polygon.

[0015] Furthermore, the specific steps for converting the nearest pixel distance between the ship polygon and the berth polygon into the actual nearest distance between the ship and the berth include: Obtain the pixel lengths of the ship's length and width, as well as the actual lengths of the ship's length and width; The scaling factor is determined based on the pixel length of the ship's length and width, as well as the actual length and width of the ship. The nearest pixel distance between the ship polygon and the berth polygon is dynamically matched in real time based on the scaling factor to convert it into the actual nearest distance between the ship polygon and the berth polygon.

[0016] Furthermore, the specific steps for obtaining the pixel lengths of the ship's length and width include: Based on the instance segmentation results, obtain the mask point set, center point, width, height, and rotation angle; The length and width of the minimum area bounding rectangle of the ship are calculated using the minAreaRect function in the OpenCV library, along with the mask point set, center point, width and height, and rotation angle. The length and width of the minimum area bounding rectangle of the ship are then used as the pixel length of the ship's length and width.

[0017] Furthermore, the specific steps for obtaining the real-time ground velocity of the ship during berthing by utilizing the Deep SORT tracking algorithm and combining it with the instance segmentation results include: Initialize the Deep SORT tracker, which performs Kalman filter prediction and data association based on the detection boxes and confidence scores of ships in the instance segmentation results, and outputs the tracking results; Based on the tracking results, the ship positions in the current frame and the previous frame under the same tracking ID are extracted, and then the pixel displacement is obtained; Determine the ship's heading based on the major axis of the ship's smallest circumscribed rectangle. Establish a ship coordinate system with the center of the ship's smallest circumscribed rectangle as the origin, where the longitudinal axis is parallel to the ship's heading and the transverse axis is perpendicular to the ship's heading. The pixel displacement is projected onto the ship's coordinate system, and through coordinate transformation, the pixel displacement is decomposed into displacement along the longitudinal axis of the ship and displacement along the transverse axis of the ship. The ship's displacement along the longitudinal axis and the transverse axis is calculated using a pre-calibrated scaling factor. The actual distance traveled in the berthing direction during the time interval; Based on the actual travel distance and The real-time ground speed of the ship during the berthing process is calculated by time interval calculation.

[0018] Furthermore, in S7, the specific steps for designing multi-level electronic early warning fence rules based on actual ship performance include: Establish a core alert zone, a near-end alert zone, and a far-end alert zone, wherein: The core warning zone is defined by the actual boundary of the vessel. If the core warning zone is touched, a level one alarm will be triggered. The near-end warning zone is obtained by expanding the core warning zone based on a polygon vertex extrapolation algorithm. If the near-end warning zone is touched, a secondary alarm will be triggered. The remote warning zone is obtained by expanding the core warning zone based on a polygon vertex extrapolation algorithm. If the remote warning zone is touched, a level three alarm will be triggered. The expansion of the core warning zone based on the polygon vertex extrapolation algorithm includes: For each vertex on the ship polygon Calculate the vector formed by its adjacent points: ; Obtain the unit normal vector by orthogonalization: ; in, Representing vectors Counterclockwise orthogonal vectors; Extrapolate each vertex of the ship polygon to the target distance along the unit normal vector direction: ; in, D This is the pixel distance based on the outward expansion distance transformation.

[0019] A UAV-cooperative ship berthing situation monitoring and early warning system based on a retinal segmentation mask task framework includes: The monitoring and display module is used to acquire real-time or non-real-time video of ships berthing. The berthing status monitoring module is used to obtain the actual closest distance between the vessel and the berthing position and the berthing speed based on real-time or non-real-time vessel berthing video. The risk warning module is used to assess safety risks and issue alarms based on the actual closest distance and speed between the ship and the berth; The alarm log is used to record alarm events, including the time the alarm was triggered, the video frame index, the geofence category, and the confidence level. The recording, playback, and export module is used to play back key frames before and after the alarm and save the alarm footage as a video to a specified folder.

[0020] Beneficial Effects: This invention utilizes UAVs to acquire actual ship berthing videos, and identifies instance segmentation results from the established berthing segmentation model. It employs a nearest-nearest-distance solution method to obtain the nearest pixel distance between the ship's polygon and the berth's polygon. This nearest-nearest-pixel distance is converted into the actual nearest distance between the ship and the berth, achieving distance measurement during berthing. The Deep SORT tracking algorithm, combined with instance segmentation results, is used to obtain pixel displacements during berthing, thus determining the ship's real-time ground speed and achieving speed measurement. Electronic warning fence rules are designed based on actual ship performance to provide safety warnings during berthing. This invention can intuitively obtain the nearest distance between the ship and the berth line, as well as the closest point of the ship to the berth line, assisting users in real-time monitoring of the ship's surroundings. It solves the problem of limited visibility during berthing, improving berthing efficiency and safety. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a UAV-cooperative ship berthing situation monitoring and early warning method based on a retinal segmentation mask task framework in this invention; Figure 2 This is a structural diagram of the berthing segmentation model in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the closest distance solution in an embodiment of the present invention; Figure 4 This is a comparison diagram of the ship and berth before and after treatment in an embodiment of the present invention; Figure 5 This is a schematic diagram of the minimum circumscribed rectangle of the ship in an embodiment of the present invention; Figure 6 This is a schematic diagram of the ship berthing distance measurement results in an embodiment of the present invention; Figure 7 This is a schematic diagram of a three-level electronic fence for ships in an embodiment of the present invention; Figure 8 This is a schematic diagram of the login module page of the early warning system in an embodiment of the present invention; Figure 9 This is a schematic diagram of the parameter settings page in an embodiment of the present invention; Figure 10This is a schematic diagram illustrating risk identification during ship berthing in an embodiment of the present invention; Figure 11 This is a schematic diagram of the real-time alarm and recording interface in an embodiment of the present invention; Figure 12 This is a schematic diagram of the playback alarm and recording interface in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This embodiment provides a method for UAV-cooperative ship berthing situation monitoring and early warning based on a retinal segmentation mask task framework, such as... Figure 1 As shown, the specific steps include: S1: Obtain a dataset of images of ships berthing; S2: Establish a berthing segmentation model based on the retinal segmentation mask task framework, and train the berthing segmentation model based on the ship berthing image dataset to obtain the trained berthing segmentation model; Specifically, the trained berthing segmentation model can quickly detect and segment ships and berths in videos, while reducing the impact of complex factors on ship segmentation.

[0025] S3: As Figure 2 As shown, an actual ship berthing video is acquired using an unmanned aerial vehicle (UAV). Based on the trained berthing segmentation model, the instance segmentation results of the ship and berth in the actual ship berthing video are identified. The instance segmentation results of the ship and berth include bounding box coordinates, classification probabilities, and instance masks. The instance masks include ship polygons and berth polygons.

[0026] Specifically, drones can not only flexibly monitor the environment around a ship, but also calculate the ship's berthing parameters by processing aerial video and transmit them to both the ship and shore ends, helping the driver to adjust the ship's berthing plan in real time.

[0027] S4: Use the nearest distance solution method to obtain the nearest pixel distance between the ship polygon and the berth polygon; S5: Convert the nearest pixel distance between the ship polygon and the berth polygon into the actual nearest distance between the ship and the berth, so as to achieve the purpose of distance measurement during the ship berthing process; S6: Using the Deep SORT tracking algorithm and combining the instance segmentation results, the pixel displacement during the ship's berthing process is obtained, thereby obtaining the ship's real-time ground speed during the berthing process, so as to achieve the purpose of measuring the ship's speed during berthing. S7: Based on the actual closest distance and real-time ground speed, combined with the preset multi-level electronic early warning fence rules, generate berthing safety early warning information. The multi-level electronic early warning fence rules are designed according to the actual ship performance.

[0028] In a specific embodiment, the berthing segmentation model includes a preliminary instance segmentation model and a fine instance segmentation model; The preliminary instance segmentation model is used to perform preliminary instance detection and segmentation on the ship berthing image dataset to obtain initial instance segmentation features; The refined instance segmentation model is a model built based on the Retina Mask model architecture, used to refine the initial instance segmentation features to obtain instance segmentation results for ships and berths.

[0029] Specifically, Retina Mask is a framework for instance segmentation that extends the classic Retina Net architecture, achieving high-quality instance segmentation on top of a single-stage detector. The core principle of Retina Mask lies in the Mask R-CNN framework, enabling it to effectively capture targets such as ships at different scales. Furthermore, it utilizes a Feature Pyramid Network (FPN) and introduces additional branches for mask prediction. The Retina Mask model extracts feature maps at multiple levels and passes them to the corresponding decoding paths, allowing the instance segmentation model to maintain attention to object details at different spatial resolutions. In addition, to address the class imbalance problem, the Retina Mask model uses a focal loss function instead of the traditional cross-entropy loss function, enhancing its learning ability for difficult samples such as berths and berth lines.

[0030] Specifically, in videos captured by drones, due to common low visibility and weak light conditions, it is generally difficult to accurately identify the details of ships and berths using conventional image recognition algorithms. Mask extraction, in instance segmentation tasks, can not only identify objects in the image but also provide fine boundary information of those objects. Therefore, this embodiment constructs a berthing segmentation model based on the retinal segmentation mask task framework and the SE attention mechanism to solve the aforementioned problems.

[0031] In a specific embodiment, the preliminary instance segmentation model includes: a backbone network, a neck network, and a detection head network; The backbone network is used to extract multi-scale ship feature maps from ship berthing images and transmit them to the neck network; Specifically, the backbone network can extract visual features at different levels from the original image, forming a feature pyramid that includes information from edges and textures to semantics, providing a foundation for subsequent processing.

[0032] The neck network is used to extract enhanced features with fused contextual information from multi-scale ship feature maps and transmit them to the detection head network; Specifically, the neck network fuses and enhances the multi-scale features output by the backbone network, enabling the feature map to simultaneously possess high-resolution detail information and rich semantic context information, which is crucial for accurately segmenting ship outlines against complex backgrounds.

[0033] The detection head network is used to decode the enhanced features that fuse contextual information, and to perform bounding box regression, class confidence prediction and pixel-level mask generation operations for ship instances in parallel, thereby outputting initial instance segmentation features.

[0034] Specifically, in this embodiment, the backbone network is used to deal with water surface reflection and weather changes; the fusion capability of the neck network is particularly important for distinguishing densely moored ships and handling partial occlusion; the detection head network directly outputs basic target features that can be used for subsequent tracking and situational analysis. The entire process conforms to the actual technical link from image input to masked ship instance output.

[0035] In a specific embodiment, such as Figure 2 As shown, the backbone network includes a first convolutional block, a second convolutional block, a first convolutional extraction layer, a third convolutional block, a second convolutional extraction layer, a fourth convolutional block, a third convolutional extraction layer, a fifth convolutional block, a fourth convolutional extraction layer, a spatial pyramid module, and a mask prediction branch. Specifically, each convolutional block includes two branches of CSP, namely a 3×3 standard convolution and a 3×3 depthwise separable convolution, which combines speed and accuracy.

[0036] The neck network includes a first upsampling layer, a first SE attention mechanism, a fifth convolutional extraction layer, a second upsampling layer, a second SE attention mechanism, a first edge extraction layer, a sixth convolutional block, a connection layer, a second edge extraction layer, a seventh convolutional block, a third SE attention mechanism, and a third edge extraction layer; The detection head network includes a first instance segmentation intersection detection module, a second instance segmentation intersection detection module, and a third instance segmentation intersection detection module; The first convolutional block is used to extract primary spatial features, such as edge and texture features, from the ship berthing image and transmit them to the second convolutional block. The second convolutional block is used to extract richer low-level features from the output features of the first convolutional block and transmit them to the first convolutional extraction layer; The first convolutional extraction layer is used to extract the first refined features from the output features of the second convolutional block, focusing on texture, edge and small object details, and then transmits them to the third convolutional block; The third convolutional block is used to extract deep features from the output features of the first convolutional extraction layer and transmit them to the second convolutional extraction layer; The second convolutional extraction layer is used to extract the refined features from the output features of the third convolutional block, focusing on local semantics and component combination, and is transmitted to the fourth convolutional block and the first SE attention mechanism respectively. The fourth convolutional block is used to extract the first abstract feature from the output features of the second convolutional extraction layer and transmit it to the third convolutional extraction layer; The third convolutional extraction layer is used to extract the refined features from the output features of the fourth convolutional block, focusing on global semantics and large target contours, and then transmits them to the fifth convolutional block and the second SE attention mechanism. The fifth convolutional block is used to extract the second abstract feature from the output feature of the third convolutional extraction layer and transmit it to the fourth convolutional extraction layer; The fourth convolutional extraction layer is used to extract the final backbone output features from the output features of the fifth convolutional block. These features contain the richest semantic information and are then transmitted to the spatial pyramid module. The spatial pyramid module is used to extract pooled features that incorporate multi-scale contextual information from the output features of the fourth convolutional extraction layer and transmit them to the mask prediction branch. The mask prediction branch is used to extract mask prototype features for predicting the fine boundaries of the object from the multi-scale pooling features output by the spatial pyramid module, and transmits them to the first upsampling layer and the third SE attention mechanism respectively. The first upsampling layer is used to upsample the output features of the mask prediction branch and output a first upsampled feature whose size matches the output features of the third convolution extraction layer; The second SE attention mechanism is used to reweight the output features of the third convolutional extraction layer and the first upsampled features through channels to enhance the target-related features and suppress background interference before transmitting them to the fifth convolutional extraction layer. The fifth convolutional extraction layer is used to extract the first convolutional features from the output features of the second SE attention mechanism and then transmit them to the second upsampling layer; The second upsampling layer is used to upsample the output features of the fifth convolutional extraction layer and output a second upsampled feature with a size that matches the output features of the second convolutional extraction layer. The first SE attention mechanism is used to perform channel reweighting on the output features of the second convolutional extraction layer and the second upsampled features to enhance the target-related features and suppress background interference before transmitting them to the first edge extraction layer; The first edge extraction layer is used to enhance the edge features of the output features of the first SE attention mechanism to obtain high-level features with clear edge information, and use them as the input features of the first instance segmentation intersection detection module; The first instance segmentation intersection detection module outputs preliminary target category, bounding box, and initial mask prediction based on the input features; The sixth convolutional block is used to perform deep feature extraction on the output features of the first edge extraction layer and output a higher-dimensional abstract feature. The connection layer is used to concatenate the output features of the sixth convolutional block and the output features of the fifth convolutional extraction layer according to the channel dimension; The second edge extraction layer is used to perform edge-aware feature fusion on the output features of the connection layer to obtain context features that integrate deep semantics and detailed edge information, and use them as input features of the second instance segmentation intersection detection module; The second instance segmentation intersection detection module outputs more refined target categories, optimized bounding boxes, and mask predictions based on the input features; The seventh convolutional block is used to compress and optimize the output features of the second edge extraction layer, and output a feature with higher feature resolution and more focused target semantics; The third SE attention mechanism is used to reweight the output features of the seventh convolutional block and the output features of the mask prediction branch to enhance the target-related features and suppress background interference before transmitting them to the third edge extraction layer. The third edge extraction layer is used to refine the output features of the third SE attention mechanism to obtain the final features with accurate contour and detail information, and uses them as the input features of the third instance segmentation intersection detection module. The third instance segmentation intersection detection module outputs the final, high-confidence target category, accurate bounding box, and high-quality instance segmentation mask based on the input features.

[0037] Specifically, this embodiment adds a mask prediction function after the spatial pyramid module of the backbone network of the berthing segmentation model, fusing them into a single branch. To avoid the possibility of distance self-identification after a collision between the ship and the berth in extreme cases, a polygon intersection detection instruction is added to the head network to determine whether the ship and the berth intersect; if they intersect, the minimum distance between them is set to 0 meters. An SE attention mechanism is added to the neck network to improve the overall detection accuracy of the algorithm. Furthermore, to address the class imbalance problem, this embodiment employs a focal loss function to improve the berthing segmentation model's learning ability for difficult samples such as berths and berth lines, further enhancing its robustness.

[0038] Specifically, the instance segmentation intersection detection module outputs mask prediction results based on prototype masks and mask coefficients. The prototype mask is the underlying mask, while the mask coefficients correspond one-to-one with each detected object. By combining the mask coefficients of each detected object with the prototype mask, the corresponding object mask can be generated. Specifically, the prototype mask generator processes the feature map obtained after forward propagation to generate several prototype masks. At the same time, the mask coefficient convolutional layer processes the feature map obtained after forward propagation to generate mask coefficients. Finally, by combining the mask coefficients of each detected object with the prototype mask, the masks corresponding to ships and berths are generated, which are the initial instance segmentation features.

[0039] Specifically, the SE (squeeze and excitation) attention mechanism is a module designed to improve the performance of convolutional neural networks. It aims to enhance the network's expressive power by dynamically adjusting the weights of channel features. The main design idea of ​​the SE module is to reweight the features of each channel, highlighting important features and suppressing unimportant ones through learned weights, thereby improving the network's recognition ability. The SE module includes two main operations: compression and excitation. Global average pooling is used to obtain global information for each channel, and two fully connected layers are used to generate weights for each channel. Then, a weighting operation is used to reweight the original features. The excitation operation aims to generate weights for each channel, adjusting the importance of each channel based on these weights. Through this weighting operation, the network can dynamically adjust the response of each channel to improve the expressive power of key features. This mechanism effectively enhances the network's feature selection ability in the channel dimension, enabling the model to achieve better performance in tasks such as image classification and object detection.

[0040] Specifically, in the feature map of a convolutional neural network, each channel contains information about the spatial dimension, but there is a lack of global interaction between different channels. Assume the input feature map is... X R H W C ， in H and W These are the height and width of the feature map. C This represents the number of channels. The compression operation calculates the global feature vector for each channel using global average pooling, as shown in the following formula:

[0041] In the formula, z c It is C A dimensional vector, representing the global feature representation for each channel. z c R c .

[0042] In a specific embodiment, the steps for obtaining the nearest pixel distance between the ship polygon and the berth polygon using the nearest distance solution method include: Calculate the Euclidean distance between all pairs of points in the ship polygon and the berth polygon to construct a point-set distance matrix, which is a... n × m A matrix, where each element Represents a point set A Middle (point set) A have n (point) i Points and point sets B Middle (point set) B have m (point) j The Euclidean distance between points is calculated using the following formula: ; In the formula, and For the first point in the polygonal point set of the ship i The point and the first j One point; and For the berth point in the polygonal point set of the berth i The point and the first j One point.

[0043] Specifically, this embodiment calculates the Euclidean distance between all point pairs in the ship polygon and the berth polygon using scipy.spatial.distance.cdist, thereby evaluating the distance relationship between them.

[0044] Based on the point set distance matrix, obtain the index of the ship polygon point corresponding to the minimum Euclidean distance. and the index of the berth polygon points , is represented as:

[0045] Based on the indices of the ship polygon points and the berth polygon points, the coordinates of the nearest point on the berth polygon and the coordinates of the nearest point on the berth polygon are obtained respectively. Calculate the distance between the nearest point on the ship polygon and the nearest point on the berth polygon to obtain the nearest pixel distance between the ship polygon and the berth polygon.

[0046] Specifically, this embodiment proposes a distance measurement process for calculating the minimum distance from the edge of the ship to the berth line, including a nearest point extraction method and an actual distance conversion method. First, the global index of the minimum value is searched to obtain the specific location of the nearest point, thus obtaining the pixel distance between the two points. Then, through dynamic distance matching, the nearest actual distance is obtained, as follows: Figure 3 As shown.

[0047] In a specific embodiment, the specific steps for converting the nearest pixel distance between the ship polygon and the berth polygon into the actual nearest distance between the ship and the berth include: Obtain the pixel lengths of the ship's length and width, as well as the actual lengths of the ship's length and width; The scaling factor is determined based on the pixel length of the ship's length and width, as well as the actual length and width of the ship. The nearest pixel distance between the ship polygon and the berth polygon is dynamically matched in real time based on the scaling factor to convert it into the actual nearest distance between the ship polygon and the berth polygon.

[0048] Specifically, this embodiment provides real-time monitoring support for ship berthing operations by using high-precision conversion from pixel-level segmentation to actual physical distance.

[0049] In a specific embodiment, the specific steps for obtaining the pixel displacement of the ship during the berthing process using the Deep SORT tracking algorithm and combined with the instance segmentation results, and thus obtaining the real-time ground speed of the ship during the berthing process, include: Initialize the Deep SORT tracker, which performs Kalman filter prediction and data association based on the detection boxes and confidence scores of ships in the instance segmentation results, and outputs the tracking results; Specifically, the tracking results not only include the pixel positions corresponding to the detection boxes, but more importantly, they assign a unique and continuous tracking ID to the same ship and record its historical motion trajectory. At this point, a target position sequence with temporal continuity is obtained.

[0050] Based on the tracking results, the ship positions in the current frame and the previous frame under the same tracking ID are extracted, and then the pixel displacement is obtained; Determine the ship's heading based on the major axis of the ship's smallest circumscribed rectangle. Establish a ship coordinate system with the center of the ship's smallest circumscribed rectangle as the origin, where the longitudinal axis is parallel to the ship's heading and the transverse axis is perpendicular to the ship's heading. The pixel displacement is projected onto the ship's coordinate system, and through coordinate transformation, the pixel displacement is decomposed into displacement along the longitudinal axis of the ship and displacement along the transverse axis of the ship. The ship's displacement along the longitudinal axis and the transverse axis is calculated using a pre-calibrated scaling factor. The actual distance traveled in the berthing direction during the time interval; Based on the actual travel distance and The real-time ground speed of the ship during the berthing process is calculated by time interval calculation.

[0051] In a specific embodiment, the specific steps for obtaining the pixel lengths of the ship's length and width include: Based on the instance segmentation results, obtain the mask point set, center point, width, height, and rotation angle; The length and width of the minimum area bounding rectangle of the ship are calculated using the `minAreaRect` function in the OpenCV library, along with the mask point set, center point, width and height, and rotation angle. Figure 4 and 5 As shown, the length and width of the bounding rectangle of the ship's minimum area are used as the pixel length of the ship's length and width.

[0052] Specifically, in this embodiment, the nearest point of the ship and the nearest point of the berth line are visualized, such as... Figure 6 As shown, the actual distance value is displayed as text in the upper left corner of the image for easy reading by the user. The steps for visualization include: (1) Draw the nearest point: On the image, the nearest point of the vessel and the nearest point of the berth line are marked with solid red circles.

[0053] (2) Draw the connecting lines: On the image, a yellow line connects the nearest point of the ship to the nearest point of the berth line, visually demonstrating the minimum distance between the ship and the berth.

[0054] (3) Add text annotations: Add a text label to the top left corner of the image showing the minimum distance between the vessel and the berth, rounded to two decimal places.

[0055] In this embodiment, S7, the specific steps for designing multi-level electronic early warning fence rules based on actual ship performance include: Establish a core alert zone, a near-end alert zone, and a far-end alert zone, wherein: The core warning zone is defined by the actual boundary of the vessel. If the core warning zone is touched, a level one alarm will be triggered. The near-end warning zone is obtained by expanding the core warning zone based on a polygon vertex extrapolation algorithm. If the near-end warning zone is touched, a secondary alarm will be triggered. The remote warning zone is obtained by expanding the core warning zone based on a polygon vertex extrapolation algorithm. If the remote warning zone is touched, a level three alarm will be triggered. Specifically, the remote warning zone is mainly set up to alert large ships (length > 250m) of risk. The expansion of the core warning zone based on the polygon vertex extrapolation algorithm includes: For each vertex on the ship polygon Calculate the vector formed by its adjacent points: ; Obtain the unit normal vector by orthogonalization: ; in, Representing vectors Counterclockwise orthogonal vectors; Extrapolate each vertex of the ship polygon to the target distance along the unit normal vector direction: ; in, D The pixel distance is based on the outward expansion distance conversion. For the near-end warning zone, the outward expansion distance is 6 times the maximum ship length. For the far-end warning zone, the outward expansion distance is 6 times the maximum ship length plus the set buffer safety distance area.

[0056] Specifically, a maritime electronic fence utilizes electronic equipment and communication technology to monitor and protect the virtual boundary of a specific area. By comparing the position of the Automatic Identification System (AIS) signal with the spatial position of the electronic fence, it is possible to determine if a vessel has crossed the boundary and issue an alarm, thereby effectively preventing vessels from entering unsafe waters. Because vessels have significant inertia, steering and braking require a certain amount of time and distance. Therefore, when setting up an electronic fence, a safe distance must be allowed between the vessel and the berth line. This ensures that maritime regulators and ship owners have sufficient time to take action when they receive an alarm.

[0057] In this embodiment, the core basis for calculating the safe distance between the ship and obstacles or berth line boundaries is the International Maritime Organization (IMO) MSC.137(76) turning performance standard: still water turning diameter, ship turning... The vertical distance of the center of gravity trajectory at 180° must be ≤ 5.0 times the ship length (L). This standard is only a theoretical reference value and does not cover the dynamic disturbances in actual navigation or the specific circumstances of large ships or ships with limited maneuverability. Therefore, this embodiment is modified as follows: Crew response delay: IMO MSC / Circ.1053 indicates that it takes 30-60 seconds from the discovery of a risk to full rudder maneuvering, during which time the ship moves forward by 0.5L due to inertia.

[0058] Wind and wave interference: Under level 7 wind (wind speed 13.9~17.1m / s), the lateral wind-induced drift increases the circumference diameter by 0.5L; Emergency operation redundancy: When avoiding imminent danger, exceeding the standard full rudder (rudder angle > 35°) will result in an additional increase in the turning radius of 0.5L.

[0059] Large vessels: For bulk carriers of 300,000 tons or more (length > 327 m) and oil tankers of 100,000 tons or more (length > 260 m), the starboard buffer zone should be increased by 0.3 nautical miles (556 m) in accordance with the major operation requirements of COLREG, and the safety zone distance should be increased by 500 m in accordance with the mandatory requirement of IMO A.671(16).

[0060] After the above rule correction, this embodiment roughly divides the setting of the safety distance into two categories: For conventional ships (length <250 m), it is calculated as 6 times the maximum length, and needs to be extended by about 1500 m to ensure the ship's basic turning space; For large ships (length between 250 and 350 m), it is calculated as 6 times the maximum length plus a buffer safety distance, and needs to be extended by about 3000 m to cover the triple requirements of large ships for turning, buffer, and safety zone.

[0061] Specifically, such as Figure 7 As shown, this embodiment uses ship boundary data segmented from retinal mask instances, combines a polygon vertex extrapolation algorithm to expand the polygon, and draws the fence outline and performs semi-transparent filling.

[0062] This embodiment also provides a UAV-cooperative ship berthing situation monitoring and early warning system based on a retinal segmentation mask task framework, including: Monitoring display module, such as Figure 8 As shown, this is used to acquire real-time or non-real-time video of ship berthing; The berthing status monitoring module is used to obtain the actual closest distance between the vessel and the berthing position and the berthing speed based on real-time or non-real-time vessel berthing video. The risk warning module is used to assess safety risks and issue alarms based on the actual closest distance and speed between the ship and the berth; The alarm log is used to record alarm events in detail, including the time the alarm was triggered, the video frame index, the geofence category, and the confidence level. The recording, playback, and export module is used to play back key frames before and after the alarm and save the alarm screen as a video to a specified folder for subsequent data archiving or evidence preservation.

[0063] Specifically, this system implements front-end video processing and acquisition functions based on the OpenCV library, supporting operations such as capturing real-time camera footage, playback control, and image scaling. The system integrates a berthing segmentation model for real-time risk identification during the berthing process within the video stream. The user interface is developed using PyQt5, with the QtDesigner visualization tool used for layout. PyOt5, as a combination of Python and Ot libraries, supports multiple operating systems including Windows and Linux, and provides rich interface components such as buttons, tables, and charts, meeting the system's application requirements.

[0064] like Figure 9 As shown, the berthing status monitoring module also includes a parameter setting unit, which allows users to design data based on the actual dimensions of the berthing vessel. In extreme weather conditions, the distance between the three alarm fences can be adjusted appropriately according to wind speed and current velocity to improve the reliability of the system.

[0065] Specifically, this system utilizes OpenCV to implement real-time video processing and combines it with a berthing segmentation model to detect risk situations during berthing operations. The detection results are displayed on the system interface as an overlay of bounding boxes and category labels, facilitating quick identification of key behavioral events and helping users concisely and clearly identify risky operational behaviors.

[0066] like Figure 10 As shown, the main interface of the constructed UAV-assisted docking early warning system includes a video display area and a function button interface below. It also includes functions for importing video and opening the camera, allowing users to select between offline video and real-time monitoring docking operation recognition modes. The video display area can play real-time camera footage or local video files and features a zoom function to ensure clear visibility of the docking detection. The system uses a docking segmentation model to identify real-time docking operation footage and issues alarms for docking operation risk situations. The alarm information displayed in the pop-up window includes the alarm time, index, risk behavior category, risk behavior coordinates, and confidence level of the risk behavior. Through the alarm pop-up information, users can easily and quickly obtain information about risk behaviors and thus respond rapidly.

[0067] Specifically, the alarm mechanism of this system operates through two methods: pop-up alerts and sound prompts. Once the triggering conditions are met, a pop-up notification box appears on the interface, displaying the alarm type, occurrence time, and target location coordinates, allowing on-site personnel to understand the situation and take immediate action. To prevent alarm pop-ups from becoming too frequent, the system also sets a minimum trigger interval. The triggering time interval of alarm pop-ups does not affect alarm records; all alarms are fully recorded, supporting subsequent querying and analysis.

[0068] To reduce false alarm rates and improve the reliability of the system in identifying risky behaviors, this system employs an adjustable confidence level mechanism. The default threshold is set to 0.80. Only when the confidence level of a detection result exceeds this value will the system classify the behavior as risky and trigger an alarm. Detection results below this value will be recorded in the "log area" without triggering pop-up windows or audible alerts, thus reducing the occurrence of invalid warnings. This default value was determined through multiple sets of real-ship video tests and comparative experiments: after testing at different thresholds (0.70, 0.75, 0.80, and 0.85), 0.80 significantly reduced false alarms in non-warning situations while maintaining a high recall rate, effectively balancing the system's sensitivity and stability.

[0069] like Figure 11 As shown, real-time alarm information is recorded in a table format at the bottom of the system interface. The table includes key information such as time, category, and confidence level, as well as vessel information such as the closest distance of the vessel to the berth line at the time of the alarm, vessel heading, vessel speed, and the angle between the vessel and the berth line, facilitating user viewing and management. This system has record playback and export functions, allowing users to view previous alarm screens and review detection results; for example... Figure 12 As shown, the playback function will pop up in the system in the form of a pane, load a video clip of several seconds before and after the alarm, realize dynamic playback, and automatically load the corresponding alarm frame according to the previously recorded alarm, and draw a detection box in the playback screen to intuitively mark the location and category of the risk target.

[0070] In summary, this system has successfully implemented an intelligent early warning process based on deep learning, providing technical feasibility support for safety risk management in ship berthing operation environments, and has good prospects for deployment and application in actual operation environments.

[0071] This embodiment extracts images targeting ships and berths from a large number of drone aerial videos of ships berthing, and collects images of similar unmanned surface vessels from the internet, totaling 4100 images. All images are segmented and labeled using Labellmg software. The dataset is divided into training, validation, and test sets in an 8:1:1 ratio, including one category: ships. The computer used to train the instance segmentation model in this embodiment has a 16 vCPU Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz and a VGPU-32GB (32GB) GPU. 1. The system has 32GB of RAM, runs Windows 11, and uses Python 3.11, CUDA 12.1, and PyTorch 2.1. This embodiment uses pre-trained weights from the docking segmentation model for transfer learning. Detailed parameters during training are shown in Table 1.

[0072] Table 1 Experimental parameters during the training process

[0073] To effectively evaluate the accuracy of image segmentation, this embodiment selects mean average precision (mAP), the number of model parameters (params), and frames per second (FPS) as evaluation metrics. Params refers to the number of parameters in the model, and FPS refers to the number of images that can be processed per second. The calculation process is shown in the following formula:

[0074] In the formula: P For accuracy; R Recall rate; TP This represents the quantity for which both the actual value and the forecast are true. FP FN represents the number of values ​​that were actually false but predicted to be true; FN represents the number of values ​​that were actually true but predicted to be false. k This represents the number of categories.

[0075] To further verify the effectiveness of the method proposed in this embodiment, the current classic image segmentation models SAM-YOLO, YOLACT, YOLOv5n, YOLOv7 and YOLOv8n were selected for comparative experiments under the same experimental conditions. The experimental results are shown in Table 2.

[0076] Table 2 Performance Comparison Results of Different Models

[0077] As shown in Table 2, compared with the two-stage segmentation model Mask R-CNN, the model in this embodiment shows significant improvements in both mAP and mAP50:95. Compared with the one-stage segmentation model YOLACT, the mAP is improved by 5.5 percentage points, demonstrating a significant improvement in accuracy. Compared with the basic segmentation models YOLOv5n, YOLOv7, and YOLOv8n, both mAP and mAP50:95 are improved to some extent, and the model has fewer parameters, resulting in higher segmentation accuracy. Based on the experimental results, the segmentation model in this embodiment, after incorporating the Retina Mask task framework, exhibits high detection accuracy while maintaining a small number of model parameters, making it suitable for high-precision segmentation of ships and berths. The berthing distance detection experiment used the experimental vessel "Minghu" and a DJI mini 4k UAV. Detailed equipment parameters are shown in Table 3, and camera intrinsic information is shown in Table 4.

[0078] Table 3 Experimental Equipment Parameters

[0079] Table 4 Camera Intrinsic Parameter Information

[0080] In the unmanned surface vessel (USV) test, when the vertical distance between the USV and the vessel was less than 3m, a complete image could not be formed; above 22m, the entire vessel appeared small in the video image, making accurate identification difficult. To compare the ranging accuracy of the USV at different altitudes, four altitudes—4m, 10m, 15m, and 20m—were selected. To fully compare the accuracy of ranging and speed measurement, 10 sets of measurement data were taken at different altitudes and compared with the actual values. The results are shown in Tables 5 and 6.

[0081] Table 5. Maximum and average deviation data for ranging at different flight altitudes.

[0082] Table 6. Maximum and average deviation data of air speed measurement at different flight altitudes

[0083] Multiple tests revealed that the higher the drone's altitude, the greater the distance between the onboard camera and the ship, resulting in the inability to extract detailed information about the ship. Conversely, with a fixed camera position, a drone flying too low made it difficult to identify the entire berthing maneuver of the ship, and factors such as viewing angle increased related errors. After several rounds of measurement and comparison, it was found that a drone altitude of 10 meters was optimal.

[0084] Experiments show that the berthing segmentation model incorporating the Retina Mask task framework achieves an accuracy of 92.9% in ship segmentation tasks, improving the accuracy of ship segmentation by 15% compared to conventional masks, and its performance is significantly better than traditional methods.

[0085] Multiple rounds of experiments revealed that as the drone's flight altitude increased and the onboard camera moved further away from the ship, the ship's size in the image decreased, making it impossible to capture more ship details. Conversely, when the drone's flight altitude was too low, with a fixed camera position, the entire berthing process of the ship could not be identified, and errors were amplified due to factors such as viewing angle. Therefore, after multiple rounds of measurement and comparison, a drone altitude of 10m was found to be optimal. At this altitude and camera position, the ship's size was suitable, allowing the drone to capture the entire berthing process of the experimental unmanned surface vessel (USV) and identify the shortest distance from the USV's edge to the berth line throughout the process. The error in the closest distance between the ship and the berth was less than 0.04m, and the measurement error of the ship's speed was less than 0.03m / s, achieving minimum error. These results indicate that this method can not only utilize drones to assist in monitoring the environment around berthing ships but also achieve visualization of the distance between the ship and the berth with high measurement accuracy, ensuring berthing safety. Furthermore, it possesses real-time and intelligent characteristics in practical port operations, showing great promise for berthing operations.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring and early warning of ship berthing status using unmanned aerial vehicles (UAVs) in collaboration with a retinal segmentation mask task framework, characterized in that, The specific steps include: S1: Obtain a dataset of images of ships berthing; S2: Establish a berthing segmentation model based on the retinal segmentation mask task framework, and train the berthing segmentation model based on the ship berthing image dataset to obtain the trained berthing segmentation model; S3: Use a drone to acquire actual ship berthing videos, and based on the trained berthing segmentation model, identify the instance segmentation results of ships and berths in the actual ship berthing videos. The instance segmentation results of ships and berths include bounding box coordinates, classification probabilities, and instance masks. The instance masks include ship polygons and berth polygons. S4: Use the nearest distance solution method to obtain the nearest pixel distance between the ship polygon and the berth polygon; S5: Convert the nearest pixel distance between the ship polygon and the berth polygon into the actual nearest distance between the ship and the berth; S6: Using the Deep SORT tracking algorithm and the instance segmentation results, the pixel displacement during the ship's berthing process is obtained, thereby obtaining the ship's real-time ground speed during the berthing process. S7: Based on the actual closest distance and real-time ground speed, combined with the preset multi-level electronic early warning fence rules, generate berthing safety early warning information. The multi-level electronic early warning fence rules are designed according to the actual ship performance.

2. The method for UAV-assisted ship berthing situation monitoring and early warning based on the retinal segmentation mask task framework according to claim 1, characterized in that, The berthing segmentation model includes a preliminary instance segmentation model and a fine instance segmentation model; The preliminary instance segmentation model is used to perform preliminary instance detection and segmentation on the ship berthing image dataset to obtain initial instance segmentation features; The refined instance segmentation model is a model built based on the Retina Mask model architecture, used to refine the initial instance segmentation features to obtain instance segmentation results for ships and berths.

3. The method for UAV-cooperative ship berthing status monitoring and early warning based on the retinal segmentation mask task framework according to claim 2, characterized in that, The preliminary instance segmentation model includes: a backbone network, a neck network, and a detection head network; The backbone network is used to extract multi-scale ship feature maps from ship berthing images and transmit them to the neck network; The neck network is used to extract enhanced features with fused contextual information from multi-scale ship feature maps and transmit them to the detection head network; The detection head network is used to decode the enhanced features that fuse contextual information, and to perform bounding box regression, class confidence prediction and pixel-level mask generation operations for ship instances in parallel, thereby outputting initial instance segmentation features.

4. The method for UAV-assisted ship berthing status monitoring and early warning based on the retinal segmentation mask task framework according to claim 3, characterized in that, The backbone network includes a first convolutional block, a second convolutional block, a first convolutional extraction layer, a third convolutional block, a second convolutional extraction layer, a fourth convolutional block, a third convolutional extraction layer, a fifth convolutional block, a fourth convolutional extraction layer, a spatial pyramid module, and a mask prediction branch. The neck network includes a first upsampling layer, a first SE attention mechanism, a fifth convolutional extraction layer, a second upsampling layer, a second SE attention mechanism, a first edge extraction layer, a sixth convolutional block, a connection layer, a second edge extraction layer, a seventh convolutional block, a third SE attention mechanism, and a third edge extraction layer; The detection head network includes a first instance segmentation intersection detection module, a second instance segmentation intersection detection module, and a third instance segmentation intersection detection module; The first convolutional block is used to extract primary spatial features from the ship berthing image and transmit them to the second convolutional block. The second convolutional block is used to extract low-level features from the output features of the first convolutional block and transmit them to the first convolutional extraction layer; The first convolutional extraction layer is used to extract the first refined features from the output features of the second convolutional block and transmit them to the third convolutional block; The third convolutional block is used to extract deep features from the output features of the first convolutional extraction layer and transmit them to the second convolutional extraction layer; The second convolutional extraction layer is used to extract the refined features from the output features of the third convolutional block and transmit them to the fourth convolutional block and the first SE attention mechanism, respectively. The fourth convolutional block is used to extract the first abstract feature from the output features of the second convolutional extraction layer and transmit it to the third convolutional extraction layer; The third convolutional extraction layer is used to extract the refined features from the output features of the fourth convolutional block and transmit them to the fifth convolutional block and the second SE attention mechanism. The fifth convolutional block is used to extract the second abstract feature from the output feature of the third convolutional extraction layer and transmit it to the fourth convolutional extraction layer; The fourth convolutional extraction layer is used to extract the final backbone output features from the output features of the fifth convolutional block and transmit them to the spatial pyramid module. The spatial pyramid module is used to extract pooled features that incorporate multi-scale contextual information from the output features of the fourth convolutional extraction layer and transmit them to the mask prediction branch. The mask prediction branch is used to extract mask prototype features for predicting the fine boundaries of the object from the multi-scale pooling features output by the spatial pyramid module, and transmits them to the first upsampling layer and the third SE attention mechanism respectively. The first upsampling layer is used to upsample the output features of the mask prediction branch and output a first upsampled feature whose size matches the output features of the third convolution extraction layer; The second SE attention mechanism is used to reweight the output features of the third convolutional extraction layer and the first upsampled features through channels to enhance the target-related features and suppress background interference before transmitting them to the fifth convolutional extraction layer. The fifth convolutional extraction layer is used to extract the first convolutional features from the output features of the second SE attention mechanism and then transmit them to the second upsampling layer; The second upsampling layer is used to upsample the output features of the fifth convolutional extraction layer and output a second upsampled feature with a size that matches the output features of the second convolutional extraction layer. The first SE attention mechanism is used to perform channel reweighting on the output features of the second convolutional extraction layer and the second upsampled features to enhance the target-related features and suppress background interference before transmitting them to the first edge extraction layer; The first edge extraction layer is used to enhance the edge features of the output features of the first SE attention mechanism to obtain high-level features with clear edge information, and use them as the input features of the first instance segmentation intersection detection module; The first instance segmentation intersection detection module outputs preliminary target category, bounding box, and initial mask prediction based on the input features; The sixth convolutional block is used to perform deep feature extraction on the output features of the first edge extraction layer and output a high-dimensional abstract feature. The connection layer is used to concatenate the output features of the sixth convolutional block and the output features of the fifth convolutional extraction layer according to the channel dimension; The second edge extraction layer is used to perform edge-aware feature fusion on the output features of the connection layer to obtain context features that integrate deep semantics and detailed edge information, and use them as input features of the second instance segmentation intersection detection module; The second instance segmentation intersection detection module outputs further refined target categories, optimized bounding boxes, and mask predictions based on the input features; The seventh convolutional block is used to compress and optimize the output features of the second edge extraction layer, and output features that further improve feature resolution and target semantics; The third SE attention mechanism is used to reweight the output features of the seventh convolutional block and the output features of the mask prediction branch to enhance the target-related features and suppress background interference before transmitting them to the third edge extraction layer. The third edge extraction layer is used to refine the output features of the third SE attention mechanism to obtain the final features with accurate contour and detail information, and uses them as the input features of the third instance segmentation intersection detection module. The third instance segmentation intersection detection module outputs the final, high-confidence target category, bounding box, and instance segmentation mask based on the input features.

5. The method for UAV-cooperative ship berthing situation monitoring and early warning based on the retinal segmentation mask task framework according to claim 1, characterized in that, The specific steps for obtaining the nearest pixel distance between the ship polygon and the berth polygon using the nearest distance solution method include: Calculate the Euclidean distance between all point pairs in the ship polygon and berth polygon to construct the point set distance matrix. The formula for calculating the Euclidean distance is: In the formula, and For the first point in the polygonal point set of the ship i The point and the first j One point; and For the berth point in the polygonal point set of the berth i The point and the first j One point; Based on the point set distance matrix, obtain the index of the ship polygon point corresponding to the minimum Euclidean distance. and the index of the berth polygon points , is represented as: Based on the indices of the ship polygon points and the berth polygon points, the coordinates of the nearest point on the berth polygon and the coordinates of the nearest point on the berth polygon are obtained respectively. Calculate the distance between the nearest point on the ship polygon and the nearest point on the berth polygon to obtain the nearest pixel distance between the ship polygon and the berth polygon.

6. The method for UAV-assisted ship berthing situation monitoring and early warning based on the retinal segmentation mask task framework according to claim 5, characterized in that, The specific steps for converting the nearest pixel distance between the ship polygon and the berth polygon into the actual nearest distance between the ship and the berth include: Obtain the pixel lengths of the ship's length and width, as well as the actual lengths of the ship's length and width; The scaling factor is determined based on the pixel length of the ship's length and width, as well as the actual length and width of the ship. The nearest pixel distance between the ship polygon and the berth polygon is dynamically matched in real time based on the scaling factor to convert it into the actual nearest distance between the ship polygon and the berth polygon.

7. The method for UAV-assisted ship berthing situation monitoring and early warning based on the retinal segmentation mask task framework according to claim 6, characterized in that, The specific steps to obtain the pixel lengths of a ship's length and width include: Based on the instance segmentation results, obtain the mask point set, center point, width, height, and rotation angle; The length and width of the minimum area bounding rectangle of the ship are calculated using the minAreaRect function in the OpenCV library, along with the mask point set, center point, width and height, and rotation angle. The length and width of the minimum area bounding rectangle of the ship are then used as the pixel length of the ship's length and width.

8. The method for UAV-assisted ship berthing situation monitoring and early warning based on retinal segmentation mask task framework according to claim 1, characterized in that, The specific steps for obtaining the real-time ground velocity of a ship during berthing by using the Deep SORT tracking algorithm and combining the instance segmentation results include: Initialize the Deep SORT tracker, which performs Kalman filter prediction and data association based on the detection boxes and confidence scores of ships in the instance segmentation results, and outputs the tracking results; Based on the tracking results, the ship positions in the current frame image and the previous frame image under the same tracking ID are extracted, and then the pixel displacement is obtained; Determine the ship's heading based on the major axis of the ship's smallest circumscribed rectangle. Establish a ship coordinate system with the center of the ship's smallest circumscribed rectangle as the origin, where the longitudinal axis is parallel to the ship's heading and the transverse axis is perpendicular to the ship's heading. The pixel displacement is projected onto the ship's coordinate system, and through coordinate transformation, the pixel displacement is decomposed into displacement along the longitudinal axis of the ship and displacement along the transverse axis of the ship. Using pre-calibrated scaling factors, displacements along the longitudinal axis and transverse axis of the hull, the ship's position is calculated. The actual distance traveled in the berthing direction during the time interval; Based on the actual travel distance and The real-time ground speed of the ship during the berthing process is calculated by time interval calculation.

9. The method for UAV-assisted ship berthing situation monitoring and early warning based on the retinal segmentation mask task framework according to claim 8, characterized in that, In S7, the specific steps for designing multi-level electronic early warning fence rules based on actual ship performance include: Establish a core alert zone, a near-end alert zone, and a far-end alert zone, wherein: The core warning zone is defined by the actual boundary of the vessel. If the core warning zone is touched, a level one alarm will be triggered. The near-end warning zone is obtained by expanding the core warning zone based on a polygon vertex extrapolation algorithm. If the near-end warning zone is touched, a secondary alarm will be triggered. The remote warning zone is obtained by expanding the core warning zone based on a polygon vertex extrapolation algorithm. If the remote warning zone is touched, a level three alarm will be triggered. The expansion of the core warning zone based on the polygon vertex extrapolation algorithm includes: For each vertex on the ship polygon Calculate the vector formed by its adjacent points: ； Obtain the unit normal vector by orthogonalization: ； in, Representing vectors Counterclockwise orthogonal vectors; Extrapolate each vertex of the ship polygon to the target distance along the unit normal vector direction: ； in, D This is the pixel distance based on the outward expansion distance conversion.

10. A UAV-cooperative ship berthing situation monitoring and early warning system based on a retinal segmentation mask task framework, used to implement the method of claim 1, characterized in that, include: The monitoring and display module is used to acquire real-time or non-real-time video of ships berthing. The berthing status monitoring module is used to obtain the actual closest distance between the vessel and the berthing position and the berthing speed based on real-time or non-real-time vessel berthing video. The risk warning module is used to assess safety risks and issue alarms based on the actual closest distance and speed between the ship and the berth; The alarm log is used to record alarm events, including the time the alarm was triggered, the video frame index, the geofence category, and the confidence level. The recording, playback, and export module is used to play back key frames before and after the alarm and save the alarm footage as a video to a specified folder.

Images