A method for identifying offshore aquaculture net cages based on low-altitude images of unmanned aerial vehicles

Abstract

The present application relates to low-altitude image recognition technical field, specifically to a kind of offshore mariculture net cage identification method based on unmanned aerial vehicle low-altitude image, including the sequence image obtained by unmanned aerial vehicle shooting, output sparse three-dimensional point cloud and camera pose;Extract line segment feature, all line segment features extracted are uniformly back projected to the same horizontal reference surface, and stable line segment graph is fused to generate, and the quadrilateral of net cage frame is obtained;Spatial straight line fitting and net cage frame rigid body geometric consistency verification are carried out, and the candidate net cage frame quadrilateral that does not meet is removed, and the confirmed net cage frame set is obtained;Local feature is calculated respectively, and net cage boundary response graph is generated;Confirmed net cage frame is regarded as shape prior, and net cage boundary response graph is regarded as external energy term, energy function is constructed and solved using graph cut algorithm, and instance region is segmented to obtain.This application improves the integrity and stability of net cage frame extraction, and provides reliable geometric basis for subsequent net cage identification.

Description

Technical Field

[0001] This invention relates to the field of low-altitude image recognition technology, and in particular to a method for identifying marine aquaculture cages based on low-altitude images from unmanned aerial vehicles (UAVs). Background Technology

[0002] Marine aquaculture cages are crucial infrastructure for marine fisheries, and their quantity, location, and distribution are essential for marine area supervision, aquaculture scale statistics, and aquaculture facility inspections. Traditional survey methods rely primarily on manual patrols, shipborne observations, or manual interpretation of remote sensing images, which suffer from low efficiency, high costs, and susceptibility to factors such as sea conditions. With the development of UAV low-altitude remote sensing technology, using high-resolution sea surface images acquired by UAVs for automatic identification of marine aquaculture cages has become an important development direction for intelligent monitoring of marine aquaculture. Currently, existing UAV-based methods for identifying marine aquaculture cages mainly extract cage features through target detection, edge detection, rectangle detection, or semantic segmentation. However, factors such as sea waves, light reflection, floating debris obstruction, and changes in UAV attitude can easily cause breaks, gaps, or blurring at the edges of the cages, making it prone to missed detections and false detections when relying solely on two-dimensional image features, resulting in poor identification stability.

[0003] Furthermore, existing technologies typically perform line segment or rectangle detection independently on a single frame of imagery, failing to fully utilize the multi-view spatial geometric information generated by continuous drone photography. This makes it difficult to effectively integrate repeated observations of the same net cage frame, and it easily leads to misidentification of background structures such as sea surface ripples, ship edges, and floating bridges as net cage frames. Simultaneously, most existing region segmentation methods rely on single features such as grayscale, color, or texture, lacking comprehensive utilization of the net cage's geometric structure and the differences in texture between the inner and outer water surfaces. Under complex sea conditions, this can easily result in inaccurate boundary positioning, region adhesion, or over-segmentation, affecting the accuracy of net cage identification. Summary of the Invention

[0004] This invention provides a method for identifying marine aquaculture cages based on low-altitude UAV imagery. It can comprehensively utilize UAV image sequence, multi-view spatial geometric information, sparse 3D point cloud, and differences in water surface texture inside and outside the cage to verify the spatial geometric consistency of candidate cage frames. It also combines shape prior and boundary response to achieve accurate segmentation of individual cage instance regions, thereby improving the accuracy, stability, and robustness of marine aquaculture cage identification under complex sea conditions.

[0005] A method for identifying marine aquaculture cages based on low-altitude UAV imagery, comprising the following steps: S1: Acquire a sequence of images obtained by the UAV continuously shooting the target sea area at low altitude, and perform three-dimensional reconstruction on the sequence of images to output a sparse three-dimensional point cloud and the camera pose corresponding to each frame of the sequence of images. S2: Extract line segment features from each frame of the sequence image, use the camera pose to uniformly backproject all extracted line segment features onto the same horizontal reference plane, fuse them to generate a stable line segment map, and obtain multiple candidate cage frame quadrilaterals by closed quadrilateral detection in the stable line segment map. S3: Use the sparse three-dimensional point cloud to perform spatial line fitting and rigid body geometry consistency verification on each candidate wire mesh frame quadrilateral, and remove candidate wire mesh frame quadrilaterals that do not meet the preset rigid body constraints to obtain a confirmed wire mesh frame set. S4: For each confirmed cage frame in the confirmed cage frame set, delineate the internal water surface region and the external annular neighborhood, calculate the local features representing the water surface ripple texture respectively, and generate the cage boundary response map based on the difference between the internal and external local features. S5: Using the confirmed cage frame as the shape prior and the cage boundary response map as the external energy term, construct the energy function and solve it using the graph cut algorithm to segment and obtain the instance region of a single cage.

[0006] Optionally, S1 specifically includes: S11: Extract SIFT feature points frame by frame from the sequence image and perform feature matching. Use ratio test and random sampling consistency algorithm to remove mismatched point pairs to obtain a set of reliable matching point pairs. S12: Select the optimal image pair with the most matching point pairs and the baseline distance meets the preset threshold as the initial image pair, calculate and decompose the essential matrix based on epipolar geometry constraints to obtain the initial camera pose, and triangulate the matching point pairs to generate initial sparse 3D points. S13: With the goal of minimizing the reprojection error, bundle adjustment is used to jointly optimize the initial camera pose and the initial sparse 3D points to obtain the optimized initial reconstruction result. S14: Register the remaining images to the reconstruction results in descending order of the number of matches with the current reconstructed point cloud. Perform local bundle adjustment once for each registered image until all images are registered. Finally, perform global bundle adjustment to output the globally optimized sparse 3D point cloud and the camera pose corresponding to each image frame. The camera pose includes the 3D coordinates of the camera position and the camera attitude angle.

[0007] Optionally, S2 specifically includes: S21: The LSD line segment detection algorithm is used to extract initial line segments from each frame of the sequence image, and short line segments with lengths less than the minimum line segment length threshold are filtered out according to the preset minimum line segment length threshold to obtain the set of line segments to be projected for each frame image. S22: For each line segment to be projected, the pixel coordinates of the two endpoints are combined with the camera pose of the corresponding frame image to construct a spatial ray from the camera optical center through the corresponding endpoint. The intersection of the spatial ray and the horizontal reference plane is calculated to obtain the three-dimensional projection coordinates of the endpoint on the horizontal reference plane. The line segment to be projected is converted into a three-dimensional projection line segment on the horizontal reference plane. S23: Transform all the three-dimensional projection line segments corresponding to the frame images into the same horizontal reference plane coordinate system. Cluster them based on the direction angle of each line segment, the vertical distance between line segments, and the projection overlap of the endpoints. Merge multiple projection line segments that belong to the same real frame edge and overlap with each other into a candidate stable line segment. Use the number of original projection line segments included in the cluster as the confidence weight of the candidate stable line segment. Filter out isolated line segments with confidence weights lower than a preset threshold and output a stable line segment map composed of the remaining candidate stable line segments.

[0008] Optionally, in the stable line segment diagram, four candidate stable line segments are searched in a clockwise or counterclockwise direction. It is determined that the distance between the first and last endpoints of adjacent line segments is less than a preset closed distance threshold, and the four line segments as a whole satisfy the rectangular geometric constraints that the parallel error of opposite sides is less than a first angle threshold and the perpendicular error of adjacent sides is less than a second angle threshold. At the same time, the side length of the rectangle formed falls within the preset cage size range. All rectangles that meet the conditions are extracted as the candidate cage frame quadrilaterals.

[0009] Optionally, S3 specifically includes: S31: For each candidate quadrilateral of the wire mesh frame, perform a spatial neighborhood search on each of the four sides of the candidate quadrilateral of the wire mesh frame in sequence. Taking the two-dimensional projection line segment of the corresponding side on the horizontal reference plane as the center line, expand the first neighborhood threshold to both sides in the direction perpendicular to the center line, and at the same time expand the second neighborhood threshold in the elevation direction to construct a three-dimensional spatial neighborhood bounding box. The three-dimensional points in the sparse three-dimensional point cloud that fall into the bounding box are taken as the candidate fitting point set of the corresponding side. S32: The random sampling consensus algorithm is used to perform three-dimensional spatial line fitting on the candidate fitting point set of each edge. The line model with the most interior points is selected iteratively as the fitting line of the edge. The direction vector, spatial position and interior point ratio of the fitted interior point to the total number of points of the fitted line are recorded. If the number of candidate fitting point sets of a certain edge is less than the minimum number of points or the interior point ratio is lower than the preset ratio threshold, it is directly determined that the corresponding candidate cage frame quadrilateral does not meet the geometric consistency. S33: Perform joint verification of rigid body geometry consistency on the four spatial straight lines obtained by fitting the four sides respectively. The joint verification includes multiple constraints. S34: Eliminate candidate cage frame quadrilaterals that failed to fit in S32 or did not meet any of the constraints in S33, and output the candidate cage frame quadrilaterals that simultaneously satisfy the spatial straight line fitting and pass the joint verification of all rigid body geometric consistency as the confirmed cage frame set.

[0010] Optionally, the constraints specifically include: Coplanarity constraint: Calculate the average distance from all interior points on the four spatial lines to the common best-fit plane. If the average distance is greater than the preset coplanar distance threshold, it is determined that the coplanarity is not satisfied. Parallel and perpendicular constraints: Calculate the cosine deviation of the absolute value of the dot product of the direction vectors of opposite sides and the angle of 180°, and the cosine deviation of the absolute value of the dot product of the direction vectors of adjacent sides and the angle of 90°. If the angle of the direction vectors of opposite sides deviates from 180° and exceeds the first angle tolerance, or the angle of the direction vectors of adjacent sides deviates from 90° and exceeds the second angle tolerance, then it is determined that the parallel and perpendicular constraints are not satisfied. Consistency constraint for opposite side dimensions: Calculate the ratio of the lengths of the fitted lines of the two opposite sides. If the ratio exceeds the preset symmetry ratio range, it is determined that the dimension consistency is not met.

[0011] Optionally, S4 specifically includes: S41: For each confirmed cage frame in the confirmed cage frame set, taking the four sides of the confirmed cage frame as a reference, shrink the frame inward by a first preset width along the normal direction of each side to obtain an inner boundary line. The area enclosed by the inner boundary line is the inner water surface area. Expand the frame outward by a second preset width along the normal direction of each side to obtain an outer boundary line. The annular area enclosed between the outer boundary line and the frame side is the outer annular neighborhood. Both the first preset width and the second preset width are adaptively set according to the ground resolution of the current image. S42: Within the inner water surface region and the outer annular neighborhood, a fixed-size sliding window is used to traverse pixel by pixel, and a local feature vector for characterizing the water surface ripple texture is extracted for each window. S43: For each edge pixel on the frame edge, the local feature vector of the internal pixel in the internal water surface region corresponding to the spatial position of the edge pixel is compared with the local feature vector of the external pixel in the external annular neighborhood corresponding to the spatial position of the edge pixel. The difference measurement is calculated using weighted Euclidean distance to obtain the initial boundary response value of the edge pixel. S44: Normalize the initial boundary response values ​​of all edge pixel points to map the initial boundary response values ​​of all edge pixel points to a preset value range, which serves as the final boundary response intensity of each edge pixel point. Using the four edges of the confirmed cage frame as spatial indices, assign the final boundary response intensity of each edge pixel point according to the actual pixel coordinates of each edge pixel point in the image, thereby generating the cage boundary response map corresponding to the boundary position of the confirmed cage frame.

[0012] Optionally, the local feature vector specifically includes: Gray-scale statistical characteristics: mean, standard deviation, skewness, and kurtosis of pixel gray-scale values ​​within the window; Texture spectral features: After performing a two-dimensional discrete Fourier transform on the window, the radial spectral energy distribution and the angular spectral energy distribution are extracted, and the proportion of low-frequency energy and the proportion of high-frequency energy are used as spectral texture descriptors; Local binary pattern features: The pixels within the window are encoded using a rotation-invariant uniform local binary pattern to generate a local binary pattern histogram, and the entropy value and main peak ratio of the local binary pattern histogram are calculated. Gray-level co-occurrence matrix features: Calculate the gray-level co-occurrence matrix of pixels within the window in multiple directions, extract the contrast, correlation, energy and homogeneity parameters in each direction, and take the average value in all directions as the final texture descriptor.

[0013] Optionally, S5 specifically includes: S51: For each confirmed cage frame in the confirmed cage frame set, expand the confirmed cage frame outward by a preset expansion width based on the outer rectangle of the confirmed cage frame, delineate the local segmentation window corresponding to the confirmed cage frame, take all pixels in the local segmentation window as nodes of the graph cut algorithm, and set a binary label to be solved for each pixel. The first label value of the binary label indicates that the pixel belongs to the inner area of ​​the cage, and the second label value indicates that the pixel belongs to the outer area of ​​the cage. S52: Based on the confirmed cage frame, construct a shape prior term, assign a first prior probability value to all pixels in the inner area enclosed by the confirmed cage frame, assign a second prior probability value to all pixels in the outer area of ​​the frame, and the first prior probability value is greater than the second prior probability value, and the shape prior energy term is composed of the prior probability values ​​of each pixel. S53: Construct a boundary energy term based on the cage boundary response map. For each pair of adjacent pixels in the local segmentation window, calculate the boundary penalty weight between each pair of adjacent pixels according to the boundary response intensity of each pair of adjacent pixels at the corresponding position in the cage boundary response map. The higher the boundary response intensity, the smaller the boundary penalty weight; the lower the boundary response intensity, the larger the boundary penalty weight. S54: Based on the shape prior energy term and the boundary energy term, a weighted sum is performed using preset weight coefficients to construct the total energy function of the graph cut algorithm.

[0014] Optionally, S5 further includes using the minimum cut / maximum flow algorithm in graph cut algorithms to globally optimize the total energy function. Under the smooth constraint of maintaining the consistency of adjacent pixel labels, the optimal binary label allocation of each pixel that makes the total energy function reach the global minimum value is solved, and the connected region formed by all pixels with the first label value is used as the instance region of a single cage corresponding to the cage frame.

[0015] The beneficial effects of this invention are: This invention reconstructs 3D images from a sequence of images acquired at low altitude by a UAV to obtain sparse 3D point clouds and camera pose. Line segment features from multiple images are then uniformly back-projected onto the same horizontal reference plane and fused to generate a stable line segment map. This map is then combined with closed quadrilateral detection to obtain candidate cage frame quadrilaterals. Compared to traditional methods based on single-frame images, this invention utilizes multi-view geometric consistency to reduce the influence of factors such as sea waves, light reflection, and attitude changes, improving the completeness and stability of cage frame extraction and providing a reliable geometric basis for subsequent cage identification. This invention utilizes sparse 3D point clouds to perform spatial straight-line fitting on the quadrilaterals of candidate net cage frames. It then combines coplanarity constraints, parallelism and perpendicularity constraints, and opposite-side dimension consistency constraints for joint verification of rigid body geometric consistency. This upgrade from 2D geometric detection to 3D spatial geometric verification effectively eliminates pseudo-rectangular structures formed by floating debris and ship facilities, reducing the false detection rate. Simultaneously, by combining the confirmed net cage frame shape prior and utilizing the differences in water surface texture inside and outside the net cage to generate a net cage boundary response map, it achieves synergistic utilization of geometric constraints and texture information, improving the accuracy and robustness of net cage boundary identification under complex sea conditions. This invention constructs a priori energy term for the shape of a net cage frame, constructs a boundary energy term using the net cage boundary response map, and employs a graph cut algorithm for global energy optimization to achieve accurate segmentation of individual net cage instance regions. Compared to traditional segmentation methods, this invention comprehensively utilizes shape constraints, boundary responses, and pixel continuity to make the segmentation boundary more closely match the edge of the actual net cage, reducing missegmentation caused by sea surface ripples, reflections, and water disturbances, and improving the completeness of net cage instance segmentation, boundary positioning accuracy, and recognition stability under complex sea conditions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the execution steps of embodiment S3 of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. For some well-known technologies, those skilled in the art may also use other alternative methods to implement the invention. Moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0019] like Figures 1-2 As shown, a method for identifying marine aquaculture cages based on low-altitude UAV imagery includes the following steps: S1: Acquire a sequence of images obtained by the UAV continuously shooting the target sea area at low altitude, and perform three-dimensional reconstruction on the sequence of images to output a sparse three-dimensional point cloud and the camera pose corresponding to each frame of the sequence of images. S1 specifically includes: S11: Acquire a sequence of images captured by a drone at low altitude. Extract scale-invariant feature transform (SMT) feature points from each frame in chronological order of capture time and calculate the corresponding feature descriptors. For any adjacent images or two frames with overlapping regions, calculate the Euclidean distance between the feature descriptors to complete initial feature matching. To eliminate erroneous matches caused by repetitive textures, water reflections, and noise, a ratio test is first used to find the closest and second-closest matching points for each feature point. The matching is established when the ratio of the closest to the second-closest distance satisfies: ; If a matching pair is found, it is retained; otherwise, it is discarded.

[0020] in, The feature descriptor between the nearest neighbor matching points is the Euclidean distance. The feature descriptor between the second nearest neighbor matching points is the Euclidean distance. The threshold for the ratio test, The value is set between 0.70 and 0.85, preferably between 0.75 and 0.80, with 0.75 being the optimal value. The ratio test threshold is used to measure the degree of distinction between the nearest neighbor matching point and the second nearest neighbor matching point. When the ratio test threshold is too small, only matching points with very high descriptor discriminative power are retained. While this effectively reduces the false matching rate, it can easily lead to the incorrect removal of a large number of true matching points, resulting in an insufficient number of reliable matching points, which is detrimental to subsequent essential matrix solving and 3D reconstruction. When the ratio test threshold is too large, many matching points with close similarity are retained, especially in areas with sea surface ripples, light reflection, and repetitive textures. This can easily introduce a large number of false matching points, reducing the model estimation accuracy of the random sampling consensus algorithm and affecting the quality of sparse 3D point cloud reconstruction. Therefore, setting the ratio test threshold to 0.70–0.85 achieves a good balance between the number of matching points and matching reliability. This ensures that a sufficient number of reliable matching points in the UAV low-altitude imagery participate in subsequent 3D reconstruction, while effectively suppressing false matching caused by sea surface reflection, water ripples, and repetitive textures, thus improving the stability of sparse 3D point cloud and camera pose estimation. Furthermore, a ratio test threshold of 0.75 is preferred, as it achieves good matching accuracy and reconstruction results in most low-altitude imagery scenarios of marine aquaculture cages.

[0021] After completing the ratio test, the fundamental matrix is ​​estimated using the random sampling consensus algorithm for the retained matching point pairs. Based on the epipolar geometric constraints, it is determined whether each matching point pair belongs to the consistent model. Outer points that do not meet the model constraints are removed, and only the inner points are retained as a set of reliable matching point pairs, providing a stable feature correspondence for subsequent 3D reconstruction.

[0022] S12: Count the number of reliable matching point pairs for each image pair and calculate the baseline distance between the corresponding cameras. Among the image pairs that meet the baseline distance requirement of not less than a preset threshold, select the image pair with the most reliable matching point pairs as the initial image pair to ensure sufficient spatial disparity and stable geometric constraints. Subsequently, calculate the essential matrix based on the reliable matching point pairs in the initial image pair, and perform singular value decomposition on the essential matrix to recover the relative rotation matrix and translation vector between the two frames, thereby obtaining the initial camera pose.

[0023] The essential matrix satisfies: ; in, The essential matrix; The rotation matrix between cameras; Let the translation vector be between the cameras; Let be an antisymmetric matrix constructed from translation vectors.

[0024] After obtaining the initial camera pose, triangulation calculations are performed using the corresponding matching points in the two frames of images to recover the spatial point coordinates and obtain the initial sparse 3D points.

[0025] During triangulation, the spatial point coordinates are solved by minimizing the reprojection error of the spatial points. The objective is expressed as: ; in, The three-dimensional coordinates of a point in space; For the first The pixel coordinates of the matching point in the frame image; For the first The projection matrix corresponding to the frame image.

[0026] After triangulation of all matching points, initial sparse 3D points are generated, providing an initial model for subsequent incremental 3D reconstruction.

[0027] S13: The initial camera pose and initial sparse 3D points are used as joint optimization variables. The optimization objective is to minimize the sum of reprojection errors of all spatial points in each image. Bundle adjustment is used for nonlinear optimization to correct the spatial point coordinates and camera pose simultaneously.

[0028] The objective function is expressed as: ; in, The number of images participating in the optimization. For the number of spatial points, For visibility markers, when the first The _ spatial point can be _ ... The value is 1 when observing a frame image; otherwise, it is 0. For the first The actual observed pixel coordinates of the corresponding spatial point in the frame image. For the first Camera pose corresponding to a frame image For the first sparse 3D point coordinates The specific scheme for the camera projection function is as follows: First, based on the current reconstruction results, sparse 3D points observable in each frame of image are statistically analyzed, establishing a correspondence between 2D pixels and 3D spatial points. Only 2D-3D corresponding points with valid observation relationships are retained for optimization. Then, the current camera poses involved in optimization, including the 3D coordinates of the camera position, the camera attitude angle, and the coordinates of all sparse 3D points, are used as variables to be optimized. Each sparse 3D point is reprojected onto the corresponding image plane based on the current camera pose. The deviation between the projected pixel coordinates and the actual observed pixel coordinates in the image is calculated, and this deviation is used as the reprojection error of the corresponding observation point. Next, the reprojection errors of all valid observation points in all images are accumulated to form the overall reprojection error. The optimization objective is to minimize the overall reprojection error. A nonlinear least squares optimization method is used to iteratively update the camera pose parameters and the coordinates of each sparse 3D point. After each iteration, all sparse 3D points are reprojected onto each image based on the updated camera pose, the new reprojection error is calculated again, and it is determined whether the overall reprojection error has further decreased compared to the previous iteration. If the overall reprojection error can continue to decrease, the next round of parameter updates will continue. If the change in the overall reprojection error between two consecutive iterations is less than the preset convergence threshold, or the preset maximum number of iterations is reached, the optimization process will end. The optimized 3D coordinates of the camera position, camera attitude angle, and sparse 3D point coordinates will be output as the optimized initial reconstruction results, so that the projection position of each spatial point in each frame of image has a higher consistency with the actual observation position, thereby improving the overall accuracy of subsequent image registration and 3D reconstruction.

[0029] During the bundle adjustment iteration process, the three-dimensional coordinates of the camera position, the camera attitude angle, and the positions of each sparse three-dimensional point are updated simultaneously, so that the overall reprojection error gradually converges and finally the optimized initial reconstruction result is obtained.

[0030] S14: After obtaining the optimized initial reconstruction results, feature matching is performed between the remaining images that have not yet participated in the reconstruction and the current sparse 3D point cloud. The number of 2D-3D correspondences between each frame of remaining images and the current reconstructed point cloud is counted, and images to be registered are selected in descending order of the number of matches. For each frame of images to be registered, perspective is used based on the 2D-3D correspondences. The point algorithm estimates the camera pose corresponding to the current image frame and adds the newly recovered spatial point increments to the current reconstruction result. After each image frame is registered, a local bundle adjustment is performed on the newly added image and its related camera poses and sparse 3D points in the neighborhood to eliminate accumulated errors and improve local reconstruction accuracy.

[0031] Once all images have been registered, the camera poses and all sparse 3D points corresponding to all images are used as the overall optimization objects. Global bundle adjustment is then performed again to uniformly optimize the 3D coordinates of all camera positions, camera attitude angles, and sparse 3D point coordinates. Finally, the globally optimized sparse 3D point cloud and the camera pose corresponding to each frame in the image sequence are output. The camera pose includes the 3D coordinates of the camera position and the camera attitude angle, providing a unified spatial reference for subsequent unified back projection of line segment features and mesh recognition.

[0032] S2: Extract line segment features from each frame of the image sequence, use the camera pose to uniformly backproject all extracted line segment features onto the same horizontal reference plane, fuse them to generate a stable line segment map, and obtain multiple candidate cage frame quadrilaterals through closed quadrilateral detection in the stable line segment map. S2 specifically includes: S21: Process each frame of the image sequence using the LSD line segment detection algorithm to extract the initial line segments in the image; each initial line segment is represented by the pixel coordinates of its two endpoints, denoted as . and .

[0033] Then, the pixel length of each initial line segment in the image is calculated, and represented as: ; in, This is the initial line segment length; The pixel coordinates of the starting point of the line segment; These are the pixel coordinates of the endpoint of the line segment.

[0034] The calculated line segment length is compared with a preset minimum line segment length threshold. When the threshold is met... If the line segment is selected, it is retained as the line segment to be projected; otherwise, it is considered that the line segment mainly originates from sea surface ripples, wave patterns, or random noise, and it is filtered out.

[0035] in, The minimum line segment length threshold is set as follows: After completing the initial line segment screening, the set of line segments to be projected for each frame of image is obtained, providing input data for subsequent unified back-projection to the horizontal reference plane. The minimum line segment length threshold is preferably 20–80 pixels, further preferably 30–60 pixels, and optimally 40 pixels; this threshold is used to screen line segments with stable geometric meaning. If the threshold is too small, short line segments formed by sea surface ripples, waves, and noise are easily retained; if the threshold is too large, effective line segments at the edges of the actual cage frame are easily filtered out. Therefore, setting the minimum line segment length threshold within the above range can balance the retention rate of effective line segments and the noise suppression effect, improving the accuracy of subsequent line segment fusion and cage frame detection.

[0036] S22: For each line segment to be projected in each frame of the image, read the pixel coordinates of its two endpoints respectively, and combine them with the camera pose and camera parameters corresponding to the image to back-project the endpoint pixel coordinates into space, and construct a spatial ray that starts from the camera optical center and passes through the corresponding pixel point.

[0037] Space rays are represented as: ; in, Let be the coordinates of any point on the space ray; The three-dimensional coordinates of the camera's optical center; The ray direction vector; For ray scale parameters; The horizontal reference plane is then represented as: ; in, This is the normal vector of the horizontal reference plane; The plane parameters are for the horizontal reference plane.

[0038] Substitute the space ray into the equation of the horizontal reference surface to solve for the coordinates of the intersection point of the space ray and the horizontal reference surface. , represented as: ; in, This is the scale parameter corresponding to the intersection point of the ray and the horizontal reference plane.

[0039] Calculate the three-dimensional projection coordinates of the horizontal reference plane corresponding to the two endpoints of the line segment to be projected, and form the corresponding three-dimensional projection line segment according to the original endpoint connection relationship, so as to realize the spatial mapping of the line segment to be projected from the image coordinate system to the unified horizontal reference plane.

[0040] S23: Represent the three-dimensional projection line segments obtained from all frames of images in the same horizontal reference plane coordinate system, and calculate the direction angle of each three-dimensional projection line segment.

[0041] For any two three-dimensional projection line segments, the difference in their direction angles , represented as: ; in, These are the direction angles of the two three-dimensional projection line segments.

[0042] Subsequently, the perpendicular distance between the two 3D projection line segments is calculated. , represented as: ; in, For the equation parameters of the line corresponding to one of the projected line segments; Let be the coordinates of the endpoint of another projected line segment.

[0043] Simultaneously calculate the overlap of the endpoint projections of the two line segments. , represented as: ; in, The length of the overlapping portion of the two line segments; The total length covered by the two line segments.

[0044] When the directional angle difference, vertical distance, and endpoint projection overlap all meet the preset clustering conditions, the corresponding 3D projection line segments are divided into the same cluster, and all line segments within the cluster are merged to generate a candidate stable line segment.

[0045] The number of original projected line segments included in the cluster is further counted, and the confidence weights of candidate stable line segments are defined. , represented as: ; in, This represents the number of original projected line segments included within the cluster.

[0046] Finally, the confidence weights of the candidate stable line segments are compared with the preset confidence thresholds. Isolated line segments with confidence weights lower than the thresholds are filtered out, and the remaining candidate stable line segments are retained to form a stable line segment graph.

[0047] S24: In a stable line segment diagram, take any candidate stable line segment as the starting line segment, and search for three other candidate stable line segments that can be connected end to end in a clockwise or counterclockwise direction to form a closed quadrilateral.

[0048] First, calculate the distance between the endpoints of any two adjacent candidate stable line segments: ; in, The distance between the first and last endpoints of adjacent line segments; , These are the coordinates of the endpoints connecting two adjacent line segments.

[0049] Only retain those that satisfy: Closed candidate.

[0050] in, The closure distance threshold is preferably set between 0.20 and 0.80, more preferably between 0.30 and 0.50, and optimally between 0.40. In the image coordinate system, this can be converted to the corresponding pixel distance based on the ground resolution, typically 5 to 15 pixels. This threshold is used to determine whether the start and end points of adjacent candidate stable line segments belong to the same net cage frame vertex. If the threshold is too small, imaging errors, 3D reconstruction errors, and backprojection errors can easily cause the actual line segments to fail to close, increasing false negatives; if the threshold is too large, it can easily lead to incorrect connections of line segments with different structures, increasing false positives. Therefore, setting the closure distance threshold within the above range can balance the accuracy and robustness of closure detection, improving the detection effect of candidate net cage frame quadrilaterals.

[0051] Then, the angle difference between the directions of the two sets of opposite sides was calculated respectively: ; And the error of the included angle between adjacent sides deviating from 90 degrees: ; in, This is the error due to the parallelism of opposite sides. This refers to the perpendicularity error between adjacent sides. 、 、 These are the direction angles of the corresponding sides.

[0052] When satisfied and At that time, it is assumed that the closed figure satisfies the rectangular geometric constraint.

[0053] Further calculate the side lengths of the four sides of the rectangle and determine whether each side length is within the preset cage size range; for closed rectangles that simultaneously satisfy the closure constraint, rectangular geometric constraint and cage size constraint, determine them as candidate cage frame quadrilaterals to provide input for subsequent spatial rigid body geometric consistency verification using sparse 3D point clouds.

[0054] in, The first angle threshold; The first angle threshold is preferably 3°–10°, more preferably 4°–8°, and optimally 5°. This threshold is used to limit the parallelism error of opposite sides of the candidate wire mesh frame quadrilateral. If the threshold is too small, the real wire mesh frame affected by imaging errors may be misjudged as not meeting the rectangular constraint; if the threshold is too large, non-parallel line segments may be misidentified as wire mesh frames. Therefore, setting the first angle threshold within the above range can balance detection accuracy and robustness, improving the recognition accuracy of candidate wire mesh frame quadrilaterals. The second angle threshold is preferably 5°–15°, more preferably 6°–10°, and optimally 8°. This threshold is used to limit the degree to which adjacent sides of the candidate wire mesh frame quadrilateral deviate from right angles. If the threshold is too small, the real wire mesh frame with slight measurement errors may be misjudged as non-rectangular; if the threshold is too large, non-rectangular structures such as parallelograms may pass the detection. Therefore, setting the second angle threshold within the above range can balance geometric error tolerance and rectangular structure constraints, improving the reliability of wire mesh frame recognition.

[0055] S3: Use sparse 3D point cloud to perform spatial line fitting and rigid body geometry consistency verification on each candidate wire mesh frame quadrilateral, remove candidate wire mesh frame quadrilaterals that do not meet the preset rigid body constraints, and obtain the confirmed wire mesh frame set. S3 specifically includes: S31: For each candidate quadrilateral frame, read the two-dimensional projection line segments corresponding to its four sides in sequence, and perform a spatial neighborhood search in a unified horizontal reference coordinate system with each two-dimensional projection line segment as the center line.

[0056] First, the first neighborhood threshold is expanded to both sides along a direction perpendicular to the centerline, and the second neighborhood threshold is expanded upwards and downwards along the elevation direction, respectively, to construct a three-dimensional spatial bounding box around the two-dimensional projection line segment. For any three-dimensional point in the sparse three-dimensional point cloud, its perpendicular distance to the centerline is calculated. , represented as: ; in, The coordinates of the three-dimensional point to be determined; Let be the coordinates of any point on the center line; This is the direction vector of the centerline.

[0057] Simultaneously calculate the elevation deviation of the three-dimensional point relative to the horizontal reference plane: ; in, This refers to elevation deviation; Elevation of a three-dimensional point; The elevation of the horizontal reference plane where the centerline is located.

[0058] When three-dimensional points simultaneously satisfy and When the three-dimensional point falls within the bounding box of the corresponding three-dimensional spatial neighborhood, it is determined that the three-dimensional point falls within the bounding box of the neighborhood.

[0059] Finally, all sparse 3D points that fall within the bounding box of the corresponding 3D spatial neighborhood are used to form the candidate fitting point set for that edge, providing input for subsequent 3D spatial line fitting.

[0060] in, The threshold is the first neighborhood threshold; The first neighborhood threshold is set to 0.30–1.00, preferably 0.40–0.80, with an optimal value of 0.60. This threshold is used to determine the lateral search range of the cage frame edges. If the threshold is too small, it is easy to miss the 3D points corresponding to the actual frame edges; if the threshold is too large, it is easy to introduce irrelevant point clouds from the sea surface and surrounding structures. Therefore, setting the first neighborhood threshold to the above range can balance point cloud integrity and anti-interference ability, improving the accuracy of spatial line fitting. The second neighborhood threshold is set to 0.30–1.50, preferably 0.50–1.00, with an optimal value of 0.80. This threshold is used to determine the search range in the elevation direction. If the threshold is too small, it is easy to miss the actual frame points affected by wave fluctuations and reconstruction errors; if the threshold is too large, it is easy to introduce background points with large elevation differences. Therefore, setting the second neighborhood threshold to the above range can balance point cloud integrity and fitting accuracy, improving the reliability of subsequent geometric consistency verification.

[0061] S32: For the candidate fitting point set corresponding to each edge, the random sampling consensus algorithm is used to perform three-dimensional spatial line fitting.

[0062] Each time, two 3D points are randomly selected to establish a candidate spatial line, and the distances from all 3D points in the candidate fitting point set to this spatial line are calculated. , represented as: ; in, These are candidate fitting points; To fit any point on the straight line; To fit the direction vector of the straight line.

[0063] When the distance from a 3D point to the fitted line is less than a preset distance threshold, the 3D point is recorded as an interior point; otherwise, it is recorded as an exterior point. After multiple random iterations, the spatial line with the most interior points is selected as the fitted line for that edge, and the corresponding direction vector, spatial position, and number of interior points are recorded.

[0064] The preset distance threshold is set between 0.05 and 0.30, preferably between 0.10 and 0.20, and optimally between 0.15. This threshold is used to determine whether a 3D point belongs to the interior of the fitted line. If the threshold is too small, real frame points affected by 3D reconstruction errors and sea surface fluctuations are easily misclassified as outliers, resulting in an insufficient number of interior points. If the threshold is too large, stray points on the sea surface and other irrelevant points are easily included in the interior point set, reducing the accuracy of spatial line fitting. Therefore, setting the preset distance threshold within the above range can balance the retention rate of real frame points and the removal effect of outliers, improving the stability and accuracy of RANSAC spatial line fitting.

[0065] Further calculate the fitted inlier ratio: ; in, The ratio of interior points; This represents the number of inlier points in the fit; This represents the total number of points in the candidate fitted point set. If a certain edge satisfies or If the edge lacks sufficient reliable empty geometric information, it is directly determined that the corresponding candidate cage frame quadrilateral does not satisfy geometric consistency. The minimum number of points threshold, A preset ratio threshold is set, with a minimum number of points ranging from 15 to 50, preferably 20 to 40, and optimally 30. This threshold ensures sufficient point cloud support for the spatial line fitting. If the threshold is too small, the fitting result is easily affected by a small number of noise points, resulting in poor stability; if the threshold is too large, it is easy to misjudge sparse but real cage frame edges as fitting failures. Therefore, setting the minimum number of points within the above range balances fitting reliability and the retention rate of the real frame. A preset ratio threshold is set from 0.60 to 0.90, preferably 0.70 to 0.85, and optimally 0.80. This threshold is used to evaluate the consistency between the candidate fitting point set and the fitted line. If the threshold is too low, fitting results heavily influenced by noise are easily retained; if the threshold is too high, cage frame edges affected by reconstruction errors but still consistent with reality are easily discarded. Therefore, setting the preset ratio threshold within the above range effectively balances fitting accuracy and robustness, improving the reliability of subsequent rigid body geometry consistency verification.

[0066] S33, Perform joint verification of rigid body geometric consistency on the four spatial lines obtained by fitting the four sides respectively. The joint verification includes all the following constraints: (a) Coplanarity constraint: Specifically, all the fitted interior points corresponding to the four fitted space lines are uniformly included in the calculation of the common best-fit plane, and the corresponding common best-fit plane is obtained.

[0067] Then, the distances from all fitted interior points to the common best-fit plane are calculated. , represented as: ; in, , , , The parameters are the common best-fit plane parameters; To fit the coordinates of the interior points.

[0068] Further calculate the average distance: ; in, Average distance; This represents the total number of fitted interior points.

[0069] like If the condition is not met, then the coplanarity constraint is not satisfied.

[0070] in, The coplanar distance threshold is set between 0.05m and 0.30m, preferably between 0.08m and 0.20m, with 0.10m being the optimal value. This threshold is used to evaluate whether the four fitted spatial lines lie in the same plane. If the threshold is too small, the actual cage frame may be misjudged as non-coplanar due to 3D reconstruction errors and slight undulations in the sea surface. If the threshold is too large, line segments with significant spatial deviations may pass the coplanarity verification, reducing the accuracy of the geometric consistency determination. Therefore, setting the coplanar distance threshold within the above range can balance the tolerance of measurement errors with the authenticity of the cage frame's planar structure, improving the reliability of rigid body geometric consistency verification.

[0071] (b) Parallel and perpendicular constraints: Specifically, calculate the angle between the direction vectors of the two sets of opposite sides and the angle between the direction vectors of the two sets of adjacent sides.

[0072] Angle between opposite side direction vectors satisfy: ; in, , These are the direction vectors of the corresponding fitted lines.

[0073] If the angle between the direction vectors of the opposite side deviates If the angle exceeds the first tolerance, the parallel constraint is deemed not met. For adjacent sides, the included angle is calculated similarly; if the included angle of the adjacent side's direction vector deviates... If the angle exceeds the second tolerance, the vertical constraint is deemed not to be met.

[0074] The first angular tolerance is the allowable deviation for parallelism of opposite sides; the second angular tolerance is the allowable deviation for perpendicularity of adjacent sides.

[0075] (c) Consistent constraint on edge dimensions: Specifically, the effective length of each fitted line is calculated based on the projection range of the fitted interior points along the fitted line direction, and the ratio of the lengths of the two sets of opposite sides is calculated respectively. , represented as: ; in, , These are the fitted lengths for the two opposite sides, respectively.

[0076] like If the size consistency constraint is not met, then it is determined that the size consistency constraint is not satisfied.

[0077] in, , These are the lower and upper limits of the symmetrical proportion range, respectively.

[0078] Only when all of the above coplanarity constraints, parallelism and perpendicularity constraints and size consistency constraints are satisfied is the candidate mesh frame quadrilateral determined to pass the joint verification of rigid body geometric consistency.

[0079] S34. Eliminate candidate cage frame quadrilaterals that failed to fit in S32 or did not meet any of the constraints in S33. Output the candidate cage frame quadrilaterals that simultaneously satisfy the spatial straight line fitting and pass the joint verification of all rigid body geometric consistency as the confirmed cage frame set.

[0080] Specifically, the spatial line fitting results and rigid body geometric consistency joint verification results of each of the four sides of the candidate cage frame quadrilateral were checked one by one.

[0081] Candidate cage frame quadrilaterals that fail to fit any edge, have an insufficient number of candidate fitting points, have an interior point ratio lower than a preset ratio threshold, or fail any of the following checks: coplanarity constraint, parallel and perpendicular constraint, or edge size consistency constraint, will be removed and will not participate in subsequent cage boundary response analysis.

[0082] For a candidate cage frame quadrilateral that has completed spatial straight line fitting on all four sides and simultaneously satisfies coplanarity constraints, parallelism and perpendicularity constraints, and opposite side size consistency constraints, it is determined that it conforms to the rigid body geometric characteristics of marine aquaculture cage frames and is output to the confirmed cage frame set, providing a reliable geometric prior for subsequent internal water surface area analysis and instance segmentation.

[0083] S4: For each confirmed cage frame in the confirmed cage frame set, delineate the internal water surface region and the external annular neighborhood, calculate the local features representing the water surface ripple texture respectively, and generate the cage boundary response map based on the difference between the internal and external local features. S4 specifically includes: S41: For each confirmed cage frame in the confirmed cage frame set, read its four edge lines respectively, and calculate the normal direction corresponding to each edge line.

[0084] Based on the ground resolution of the current image, the preset actual width is converted into the corresponding pixel width. , represented as: ; in, Preset actual width; This represents the current ground resolution of the image.

[0085] Subsequently, the frame is translated inward by a first preset width along the normal direction of each edge line to form an internal boundary line. The area enclosed by the four internal boundary lines serves as the internal water surface area. Simultaneously, the frame is translated outward by a second preset width along the normal direction of each edge line to form an external boundary line. The closed annular area enclosed between the external boundary line and the original frame edge line serves as the external annular neighborhood.

[0086] The above method makes the internal water surface area and the external annular neighborhood correspond to the water area inside the cage and the water area outside the cage, respectively, providing spatial constraints for subsequent local texture feature extraction.

[0087] S42: A fixed-size sliding window is used to traverse the internal water surface area and the external annular neighborhood pixel by pixel. Each pixel is moved to extract the local texture features within the corresponding window and combine them to form the local feature vector of the corresponding position.

[0088] For grayscale statistical features, calculate the mean, standard deviation, skewness, and kurtosis of pixel grayscale values ​​within the window respectively; The mean gray level is expressed as: ; The formula for calculating standard deviation is: ; in, The grayscale mean; The standard deviation of grayscale; For the first Each pixel's grayscale value; This represents the number of pixels within the window.

[0089] For texture spectral features, a two-dimensional discrete Fourier transform is performed on the window, and the proportions of low-frequency energy and high-frequency energy are statistically analyzed. The proportion of low-frequency energy... , represented as: ; in, This refers to the spectral energy in the low-frequency region. This represents the total spectral energy.

[0090] For local binary pattern features, a rotation-invariant uniform local binary pattern is used to encode pixels within the window, the corresponding local binary pattern histogram is statistically analyzed, and the histogram entropy value is calculated. , represented as: ; in, For the first The probability of each code appearing; This represents the number of local binary mode categories. For gray-level co-occurrence matrix features, the gray-level co-occurrence matrices in multiple directions are calculated, and the contrast, correlation, energy, and homogeneity parameters corresponding to each direction are extracted. The average value of each direction is then calculated as the final texture descriptor.

[0091] Finally, one or more of the above texture features are combined to form a local feature vector corresponding to the position of the sliding window.

[0092] S43: For each edge pixel on the four edges of the confirmed cage frame, determine the corresponding pixel in the inner water surface area and the outer annular neighborhood according to the edge normal direction, and read the local feature vectors corresponding to the two locations.

[0093] When using weighted Euclidean distance to calculate the difference between two sets of local feature vectors, it is expressed as: ; in, These are local feature difference values; The dimension of the local features; For the first Dimensional features correspond to weights, These are the components corresponding to the internal local feature vectors; These are the components corresponding to the external local feature vectors.

[0094] Finally, the obtained local feature difference value is used as the initial boundary response value of the corresponding edge pixel. The larger the local feature difference, the more obvious the difference between the water surface texture inside and outside the frame, and the stronger the corresponding boundary response.

[0095] S44: Perform unified normalization on the initial boundary response values ​​corresponding to all edge pixels of the confirmed cage frame, expressed as: ; in, The final boundary response strength; This represents the initial boundary response value for the corresponding edge pixel. The minimum initial boundary response value among all edge pixels; It is the maximum initial boundary response value among all edge pixels.

[0096] After normalization, the final boundary response intensity is mapped to a preset numerical range, and values ​​are assigned point by point according to the actual pixel coordinates of each edge pixel in the original image.

[0097] Subsequently, using the four edges of the confirmed cage frame as spatial indices, the final boundary response intensity corresponding to all edge-lined pixels is written to the corresponding pixel positions, forming a continuous boundary response distribution. Finally, a cage boundary response map corresponding to the confirmed cage frame boundary positions is generated, providing external energy constraints for subsequent graph cut instance segmentation based on shape priors.

[0098] S5: Confirm the cage frame as the shape prior, use the cage boundary response map as the external energy term, construct the energy function and solve it using the graph cut algorithm to segment and obtain the instance region of a single cage.

[0099] S5 specifically includes: S51: For each confirmed cage frame in the confirmed cage frame set, first calculate its circumscribed rectangle, and then expand it outwards by a preset width in four directions based on this circumscribed rectangle to construct a local segmentation window that only covers the current cage and its adjacent areas, so as to reduce the influence of irrelevant areas on the subsequent segmentation results. Subsequently, all pixels within the local segmentation window are established as a node set in the graph cut algorithm, and the connection relationship between adjacent pixels is established. Each pixel node corresponds to the binary label to be solved.

[0100] Let the label corresponding to the pixel node be represented as: ; in, The binary label corresponding to the pixel node; This indicates the second tag value, meaning that the pixel belongs to the outer area of ​​the cage; This indicates the first label value, meaning that the pixel belongs to the area inside the cage.

[0101] After establishing all pixel nodes, the connection relationships between adjacent nodes, and the connection relationships between pixel nodes and source and sink nodes, a graph structure corresponding to the graph cut algorithm is formed, and the graph structure is used as the optimization object for solving the minimum cut / maximum flow problem in the graph cut algorithm.

[0102] S52: Based on the confirmed position of the cage frame, define all pixels within the area enclosed by the frame as the shape prior internal region, and define the area outside the frame as the shape prior external region.

[0103] For pixels located inside the frame, a higher first prior probability value is assigned; for pixels located outside the frame, a lower second prior probability value is assigned, making the area inside the frame more likely to be classified as a cage instance.

[0104] Furthermore, a shape prior energy term is constructed based on the prior probability values ​​corresponding to each pixel. , represented as: ; in, It is the set of all pixels within the local cropping window; represents the prior probability of the label corresponding to the pixel.

[0105] Since the first prior probability value is greater than the second prior probability value, the region inside the frame has less prior energy, which is beneficial for maintaining the consistency between the segmentation result and the confirmed cage frame.

[0106] S53: Based on the generated cage boundary response map, establish neighborhood relationships for all adjacent pixels within the local segmentation window, and read the boundary response intensity at the corresponding location.

[0107] Calculate the corresponding boundary penalty weight based on the boundary response strength. , represented as: ; in, This represents the boundary response intensity at the corresponding location; The response attenuation coefficient is set to a value between 1 and 10, preferably between 2 and 6, with 4 being the optimal value. This coefficient controls the influence of the boundary response intensity on the boundary penalty weight. When the attenuation coefficient is too small, the boundary penalty weight changes insignificantly, which is detrimental to highlighting the true cage boundary. When the attenuation coefficient is too large, the boundary penalty weight changes too drastically, making it susceptible to local noise. Therefore, setting the response attenuation coefficient within the above range balances the sensitivity of the boundary response with the stability of the segmentation results, improving the boundary localization accuracy of the graph cut algorithm.

[0108] When the boundary response strength is high, it indicates that the current location is more likely to be a real cage boundary, so the corresponding boundary penalty weight is small, making it easier to allow the graph cut boundary to pass; when the boundary response strength is low, the corresponding boundary penalty weight is large, which is beneficial to maintaining label consistency within the region.

[0109] Further construct the boundary energy term , represented as: ; in, It is a set of adjacent pixels; The label consistency function assigns a value of 1 when two pixels have different labels and 0 when they have the same label. The specific scheme for the label consistency function is as follows: First, establish pixel adjacency relationships within the local segmentation window, using either a four-neighborhood or eight-neighborhood approach, recording each pair of adjacent pixels as an adjacent pixel pair. Then, read the cage boundary response strength at the location of each adjacent pixel pair and calculate the corresponding boundary penalty weight based on the boundary response strength: a stronger boundary response indicates a higher probability of a real cage boundary at that location, assigning a smaller boundary penalty weight; a weaker boundary response indicates a higher probability of being within the same region, assigning a larger boundary penalty weight. Next, compare the binary labels of each pair of adjacent pixels. When two adjacent pixel labels are the same, the boundary energy of that adjacent pixel pair is zero; when two adjacent pixel labels are different, the corresponding boundary penalty weight is included in the boundary energy. This process is repeated for all adjacent pixel pairs within the local segmentation window, and the boundary penalty weights corresponding to all pixel pairs with different labels are accumulated to obtain the boundary energy term corresponding to the current label assignment result. During the graph cut solution process, the minimum cut / maximum flow algorithm continuously adjusts the binary labels of each pixel. When the cutting path passes through a location with a high boundary response, the corresponding penalty weight is smaller, resulting in less increase in total energy. Therefore, the cutting boundary is more likely to pass along the real mesh boundary. When the cutting path passes through a location with a low boundary response, the penalty weight is larger, increasing the total energy and suppressing arbitrary cutting within the texture continuity region. Ultimately, this ensures that the segmentation boundary is preferentially distributed in locations with a strong mesh boundary response, while maintaining the continuity of adjacent pixel labels inside and outside the mesh.

[0110] S54: Linearly combine the shape prior energy term and the boundary energy term according to the preset weight coefficients to establish the total energy function corresponding to the graph cut algorithm.

[0111] Total energy function , represented as: ; By adjusting the weight ratio of the two factors, the shape prior constraint and the boundary response constraint work together in the instance segmentation process. While ensuring the overall shape constraint of the cage, the actual boundary texture information is fully utilized to correct the segmentation results.

[0112] in, This is the shape prior energy term; For boundary energy terms; These are the shape prior weighting coefficients; This refers to the boundary energy weighting coefficient. Specifically, the shape prior energy term and the boundary energy term are first scaled to prevent either from having an excessively dominant effect on the total energy function due to different numerical ranges. Then, the shape prior costs corresponding to selecting either the inside or outside label of each pixel within the local segmentation window are accumulated to obtain the shape prior energy. Next, all adjacent pixel pairs are traversed; when adjacent pixel labels are different, the corresponding boundary penalty weight is included in the boundary energy; when labels are the same, no penalty is applied. Afterwards, the shape prior energy is multiplied by the shape prior weighting coefficient, and the boundary energy is multiplied by the boundary energy weighting coefficient, and the two are added together to form the total energy of the graph cut algorithm. The graph cut algorithm continuously reduces the total energy by adjusting the binary labels of each pixel: the shape prior term tends to label pixels inside the mesh frame as inside the mesh, and pixels outside the frame tend to be labeled as outside the mesh; the boundary energy term suppresses abrupt changes in the labels of adjacent pixels within a continuous texture region and allows the segmentation boundary to preferentially pass through locations with stronger mesh boundary responses. Finally, the minimum cut / maximum flow algorithm is used to obtain the pixel label allocation result with the minimum total energy. The shape prior weight coefficient is set to 0.5–2.0, preferably 0.8–1.5, and optimally 1.0; the boundary energy weight coefficient is preferably set to 0.5–3.0, preferably 1.0–2.0, and optimally 1.5. When the shape prior weight is too large, the segmentation result is easily confirmed by the excessive constraint of the cage frame; when the boundary energy weight is too large, the segmentation result is easily over-smoothed; the above values ​​can balance the preservation of the overall shape of the cage and the actual boundary positioning accuracy.

[0113] S55: Using the completed graph model as input, the minimum cut / maximum flow algorithm in the graph cut algorithm is used to globally optimize the total energy function. During the optimization process, under the smooth constraint of maintaining the consistency of adjacent pixel labels, the shape prior energy term and boundary energy term are combined to continuously adjust the binary labels corresponding to each pixel node so that the overall total energy function reaches the minimum value.

[0114] After completing the minimum cut calculation, the optimal binary label assignment results for all pixels within the local segmentation window are obtained. Subsequently, all pixels with the first label value are extracted, and connected component analysis is performed according to pixel connectivity. The largest connected component corresponding to the currently confirmed cage frame is determined as the single cage instance region corresponding to the currently confirmed cage frame, and the final cage instance segmentation result is output, providing accurate instance targets for subsequent marine aquaculture cage statistics, positioning, and area measurement.

[0115] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0116] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying marine aquaculture cages based on low-altitude UAV imagery, characterized in that, Includes the following steps: S1: Acquire a sequence of images obtained by the UAV continuously shooting the target sea area at low altitude, and perform three-dimensional reconstruction on the sequence of images to output a sparse three-dimensional point cloud and the camera pose corresponding to each frame of the sequence of images. S2: Extract line segment features from each frame of the sequence image, use the camera pose to uniformly backproject all extracted line segment features onto the same horizontal reference plane, fuse them to generate a stable line segment map, and obtain multiple candidate cage frame quadrilaterals by closed quadrilateral detection in the stable line segment map. S3: Use the sparse three-dimensional point cloud to perform spatial line fitting and rigid body geometry consistency verification on each candidate wire mesh frame quadrilateral, and remove candidate wire mesh frame quadrilaterals that do not meet the preset rigid body constraints to obtain a confirmed wire mesh frame set. S4: For each confirmed cage frame in the confirmed cage frame set, delineate the internal water surface region and the external annular neighborhood, calculate the local features representing the water surface ripple texture respectively, and generate the cage boundary response map based on the difference between the internal and external local features. S5: Using the confirmed cage frame as the shape prior and the cage boundary response map as the external energy term, construct the energy function and solve it using the graph cut algorithm to segment and obtain the instance region of a single cage.

2. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 1, characterized in that, S1 specifically includes: S11: Extract SIFT feature points frame by frame from the sequence image and perform feature matching. Use ratio test and random sampling consistency algorithm to remove mismatched point pairs to obtain a set of reliable matching point pairs. S12: Select the optimal image pair with the most matching point pairs and the baseline distance meets the preset threshold as the initial image pair, calculate and decompose the essential matrix based on epipolar geometry constraints to obtain the initial camera pose, and triangulate the matching point pairs to generate initial sparse 3D points. S13: With the goal of minimizing the reprojection error, bundle adjustment is used to jointly optimize the initial camera pose and the initial sparse 3D points to obtain the optimized initial reconstruction result. S14: Register the remaining images to the reconstruction results in descending order of the number of matches with the current reconstructed point cloud. Perform local bundle adjustment once for each registered image until all images are registered. Finally, perform global bundle adjustment to output the globally optimized sparse 3D point cloud and the camera pose corresponding to each image frame. The camera pose includes the 3D coordinates of the camera position and the camera attitude angle.

3. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 1, characterized in that, S2 specifically includes: S21: The LSD line segment detection algorithm is used to extract initial line segments from each frame of the sequence image, and short line segments with lengths less than the minimum line segment length threshold are filtered out according to the preset minimum line segment length threshold to obtain the set of line segments to be projected for each frame image. S22: For each line segment to be projected, the pixel coordinates of the two endpoints are combined with the camera pose of the corresponding frame image to construct a spatial ray from the camera optical center through the corresponding endpoint. The intersection of the spatial ray and the horizontal reference plane is calculated to obtain the three-dimensional projection coordinates of the endpoint on the horizontal reference plane. The line segment to be projected is converted into a three-dimensional projection line segment on the horizontal reference plane. S23: Transform all the three-dimensional projection line segments corresponding to the frame images into the same horizontal reference plane coordinate system. Cluster them based on the direction angle of each line segment, the vertical distance between line segments, and the projection overlap of the endpoints. Merge multiple projection line segments that belong to the same real frame edge and overlap with each other into a candidate stable line segment. Use the number of original projection line segments included in the cluster as the confidence weight of the candidate stable line segment. Filter out isolated line segments with confidence weights lower than a preset threshold and output a stable line segment map composed of the remaining candidate stable line segments.

4. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 3, characterized in that, In the stable line segment diagram, four candidate stable line segments are searched in a clockwise or counterclockwise direction. It is determined that the distance between the first and last endpoints of adjacent line segments is less than a preset closed distance threshold, and the four line segments as a whole satisfy the rectangular geometric constraints that the parallel error of opposite sides is less than the first angle threshold and the perpendicular error of adjacent sides is less than the second angle threshold. At the same time, the side length of the rectangle formed falls within the preset cage size range. All rectangles that meet the conditions are extracted as the candidate cage frame quadrilaterals.

5. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 1, characterized in that, S3 specifically includes: S31: For each candidate quadrilateral of the wire mesh frame, perform a spatial neighborhood search on each of the four sides of the candidate quadrilateral of the wire mesh frame in sequence. Taking the two-dimensional projection line segment of the corresponding side on the horizontal reference plane as the center line, expand the first neighborhood threshold to both sides in the direction perpendicular to the center line, and at the same time expand the second neighborhood threshold in the elevation direction to construct a three-dimensional spatial neighborhood bounding box. The three-dimensional points in the sparse three-dimensional point cloud that fall into the bounding box are taken as the candidate fitting point set of the corresponding side. S32: The random sampling consensus algorithm is used to perform three-dimensional spatial line fitting on the candidate fitting point set of each edge. The line model with the most interior points is selected iteratively as the fitting line of the edge. The direction vector, spatial position and interior point ratio of the fitted interior point to the total number of points of the fitted line are recorded. If the number of candidate fitting point sets of a certain edge is less than the minimum number of points or the interior point ratio is lower than the preset ratio threshold, it is directly determined that the corresponding candidate cage frame quadrilateral does not meet the geometric consistency. S33: Perform joint verification of rigid body geometry consistency on the four spatial straight lines obtained by fitting the four sides respectively. The joint verification includes multiple constraints. S34: Eliminate candidate cage frame quadrilaterals that failed to fit in S32 or did not meet any of the constraints in S33, and output the candidate cage frame quadrilaterals that simultaneously satisfy the spatial straight line fitting and pass the joint verification of all rigid body geometric consistency as the confirmed cage frame set.

6. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 5, characterized in that, The constraints specifically include: Coplanarity constraint: Calculate the average distance from all interior points on the four spatial lines to the common best-fit plane. If the average distance is greater than the preset coplanar distance threshold, it is determined that the coplanarity is not satisfied. Parallel and perpendicular constraints: Calculate the cosine deviation of the absolute value of the dot product of the direction vectors of opposite sides and the angle of 180°, and the cosine deviation of the absolute value of the dot product of the direction vectors of adjacent sides and the angle of 90°. If the angle of the direction vectors of opposite sides deviates from 180° and exceeds the first angle tolerance, or the angle of the direction vectors of adjacent sides deviates from 90° and exceeds the second angle tolerance, then it is determined that the parallel and perpendicular constraints are not satisfied. Consistency constraint for opposite side dimensions: Calculate the ratio of the lengths of the fitted lines of the two opposite sides. If the ratio exceeds the preset symmetry ratio range, it is determined that the dimension consistency is not met.

7. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 1, characterized in that, S4 specifically includes: S41: For each confirmed cage frame in the confirmed cage frame set, taking the four sides of the confirmed cage frame as a reference, shrink the frame inward by a first preset width along the normal direction of each side to obtain an inner boundary line. The area enclosed by the inner boundary line is the inner water surface area. Expand the frame outward by a second preset width along the normal direction of each side to obtain an outer boundary line. The annular area enclosed between the outer boundary line and the frame side is the outer annular neighborhood. Both the first preset width and the second preset width are adaptively set according to the ground resolution of the current image. S42: Within the inner water surface region and the outer annular neighborhood, a fixed-size sliding window is used to traverse pixel by pixel, and a local feature vector for characterizing the water surface ripple texture is extracted for each window. S43: For each edge pixel on the frame edge, the local feature vector of the internal pixel in the internal water surface region corresponding to the spatial position of the edge pixel is compared with the local feature vector of the external pixel in the external annular neighborhood corresponding to the spatial position of the edge pixel. The difference measurement is calculated using weighted Euclidean distance to obtain the initial boundary response value of the edge pixel. S44: Normalize the initial boundary response values ​​of all edge pixel points to map the initial boundary response values ​​of all edge pixel points to a preset value range, which serves as the final boundary response intensity of each edge pixel point. Using the four edges of the confirmed cage frame as spatial indices, assign the final boundary response intensity of each edge pixel point according to the actual pixel coordinates of each edge pixel point in the image, thereby generating the cage boundary response map corresponding to the boundary position of the confirmed cage frame.

8. The method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 7, characterized in that, The local feature vector specifically includes: Gray-scale statistical characteristics: mean, standard deviation, skewness, and kurtosis of pixel gray-scale values ​​within the window; Texture spectral features: After performing a two-dimensional discrete Fourier transform on the window, the radial spectral energy distribution and the angular spectral energy distribution are extracted, and the proportion of low-frequency energy and the proportion of high-frequency energy are used as spectral texture descriptors; Local binary pattern features: The pixels within the window are encoded using a rotation-invariant uniform local binary pattern to generate a local binary pattern histogram, and the entropy value and main peak ratio of the local binary pattern histogram are calculated. Gray-level co-occurrence matrix features: Calculate the gray-level co-occurrence matrix of pixels within the window in multiple directions, extract the contrast, correlation, energy and homogeneity parameters in each direction, and take the average value in all directions as the final texture descriptor.

9. A method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 1, characterized in that, S5 specifically includes: S51: For each confirmed cage frame in the confirmed cage frame set, expand the confirmed cage frame outward by a preset expansion width based on the outer rectangle of the confirmed cage frame, delineate the local segmentation window corresponding to the confirmed cage frame, take all pixels in the local segmentation window as nodes of the graph cut algorithm, and set a binary label to be solved for each pixel. The first label value of the binary label indicates that the pixel belongs to the inner area of ​​the cage, and the second label value indicates that the pixel belongs to the outer area of ​​the cage. S52: Based on the confirmed cage frame, construct a shape prior term, assign a first prior probability value to all pixels in the inner area enclosed by the confirmed cage frame, assign a second prior probability value to all pixels in the outer area of ​​the frame, and the first prior probability value is greater than the second prior probability value, and the shape prior energy term is composed of the prior probability values ​​of each pixel. S53: Construct a boundary energy term based on the cage boundary response map. For each pair of adjacent pixels in the local segmentation window, calculate the boundary penalty weight between each pair of adjacent pixels according to the boundary response intensity of each pair of adjacent pixels at the corresponding position in the cage boundary response map. The higher the boundary response intensity, the smaller the boundary penalty weight; the lower the boundary response intensity, the larger the boundary penalty weight. S54: Based on the shape prior energy term and the boundary energy term, a weighted sum is performed using preset weight coefficients to construct the total energy function of the graph cut algorithm.

10. A method for identifying marine aquaculture cages based on low-altitude UAV imagery according to claim 9, characterized in that, S5 further includes using the minimum cut / maximum flow algorithm in the graph cut algorithm to globally optimize the total energy function. Under the smooth constraint of maintaining the consistency of adjacent pixel labels, the optimal binary label allocation of each pixel that makes the total energy function reach the global minimum value is solved, and the connected region formed by all pixels with the first label value is used as the instance region of a single cage corresponding to the cage frame.

Images