Unmanned ship aquatic organism intelligent identification method and system based on deep learning

Abstract

The application belongs to the technical field of image processing, and particularly relates to an unmanned ship aquatic organism intelligent identification method and system based on deep learning, which comprises the following steps: generating candidate regions for underwater images of the unmanned ship by a deep learning target detection network; calculating information entropy and edge gradient mean value of a convolution feature response subgraph for each candidate region, normalizing and then weighting and fusing to obtain a crowdedness index; generating an inhibition factor for each candidate region through exponential decay, multiplying the inhibition factor by a basic intersection-over-union threshold to obtain an adaptive intersection-over-union threshold for the candidate region; and performing non-maximum suppression screening on the detection frame; obtaining an initial mask by using the detection frame, extracting a response extreme point as a seed, and when the mask is broken, recursively completing the mask to obtain a complete mask according to distance and gradient conditions. The application improves the detection accuracy and segmentation integrity of aquatic organisms in dense scenes by adaptively adjusting the non-maximum suppression threshold and recursively completing the broken region of the mask.

Description

Technical Field

[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to a method and system for intelligent identification of aquatic organisms on unmanned surface vessels based on deep learning. Background Technology

[0002] In the field of underwater target detection for unmanned surface vessels, a deep learning-based target detection network is typically used to generate a large number of candidate detection boxes from underwater images. Then, the non-maximum suppression (NMS) algorithm is used to eliminate redundant boxes. Finally, instance segmentation is performed on the remaining detection boxes to obtain the pixel-level contours of the target.

[0003] The NMS algorithm typically uses a fixed crossover ratio (IoU) threshold to determine whether two detection boxes belong to the same target. That is, when the IoU between the two boxes exceeds the threshold, the one with lower confidence is suppressed.

[0004] However, in high-density aquatic organism scenarios, such as dense schools of fish and shrimp, different individuals are spatially adjacent and their frames overlap significantly, resulting in a generally high IoU between the detection frames of different individuals.

[0005] Fixed-threshold NMS is difficult to effectively distinguish between repeated detections of different individuals and the same target. It is very easy to misclassify adjacent independent individuals as redundant and suppress them incorrectly, resulting in an increase in the false negative rate and a low count. At the same time, due to the constraints of the underwater optical environment, the instance segmentation mask often breaks or shrinks because the feature response is concentrated in the highlight center of the target, and it cannot completely outline the individual contour.

[0006] These shortcomings severely limit the detection accuracy and instance segmentation integrity of the unmanned vessel automatic identification system in real dense working conditions. There is a need for an identification method that can adaptively adjust the suppression strategy according to the degree of local congestion and repair incomplete masks. Summary of the Invention

[0007] To address the aforementioned technical problems, the present invention provides solutions in the following aspects.

[0008] In a first aspect, the present invention provides a deep learning-based intelligent identification method for aquatic organisms on unmanned surface vessels, characterized in that it includes:

[0009] Underwater images of unmanned vessels are acquired, candidate regions are generated through a deep learning object detection network, and the feature maps output from the intermediate convolutional layers of the deep learning object detection network are cropped and normalized according to the position of each candidate region to obtain the convolutional feature response sub-maps corresponding to each candidate region.

[0010] For each candidate region, the information entropy and mean edge gradient of its convolutional feature response sub-image are calculated, and normalization is performed based on the statistical extreme values ​​of the information entropy and mean edge gradient of all candidate regions in the current frame image. The normalized information entropy and mean edge gradient are then weighted and fused to obtain the crowding index.

[0011] An inhibition factor is generated by exponentially decaying the crowding index of each candidate region. This inhibition factor is then multiplied by the basic cross-union ratio threshold to obtain the adaptive cross-union ratio threshold.

[0012] Non-maximum suppression is performed based on the adaptive cross-union ratio threshold to filter out detection boxes;

[0013] The detection box is segmented into instances to obtain an initial mask, and the extreme points of the response in the convolutional feature response sub-image corresponding to the detection box are extracted as seed points.

[0014] Determine if the initial mask has any breaks: Take the non-mask pixels in the neighborhood of the mask boundary and the internal holes as the points to be filled, and recursively fill them according to the distance between each point to be filled and the seed point and the gradient conditions to obtain a complete mask.

[0015] Further, the information entropy is calculated as follows: the convolutional feature response values ​​of each pixel in the convolutional feature response sub-image are normalized to the interval between 0 and 1 to obtain the normalized feature probability of each pixel; the negative value of the sum of the normalized feature probability of each pixel and its logarithm is calculated as the information entropy.

[0016] Furthermore, the mean edge gradient is calculated as follows: the edge gradient magnitude of each pixel in the convolutional feature response sub-image is calculated using an edge detection operator; the arithmetic mean of the absolute values ​​of the edge gradient magnitudes of each pixel is calculated as the mean edge gradient.

[0017] Further, obtaining the congestion index includes: multiplying the normalized information entropy by a first weighting coefficient, multiplying the normalized mean edge gradient by a second weighting coefficient, and adding the weighted information entropy to the weighted mean edge gradient to obtain the congestion index.

[0018] Furthermore, the process of generating the inhibition factor through exponential decay includes: calculating the negative value of the product of the congestion index and a preset decay coefficient; and using the negative value as a power of the natural constant as the base, as the inhibition factor.

[0019] Further, the step of performing non-maximum suppression based on the adaptive cross-union ratio (CUI) threshold to filter out detection boxes includes: obtaining the confidence level of each candidate region and sorting the candidate regions from high to low confidence level; taking the candidate region with the highest confidence level as the current region and calculating the cross-union ratio (CUI) between the remaining candidate regions and the current region; taking the remaining candidate regions with CUI greater than the adaptive CUI threshold corresponding to the current region as redundancy suppression; repeating until all candidate regions have been processed, and taking the unsuppressed candidate regions as the detection boxes.

[0020] Further, extracting the extreme response points in the convolutional feature response sub-image corresponding to the detection box as seed points includes: inputting the detection box into a deep learning instance segmentation network to obtain a pixel-level initial mask; in the convolutional feature response sub-image corresponding to the detection box, selecting the pixel with the largest response value as the extreme response point, and using it as the segmentation seed point.

[0021] Furthermore, the recursive completion based on the distance and gradient conditions between each point to be completed and the seed point includes: for each point to be completed, if the pixel distance between the point and the seed point is less than a preset distance threshold, and the edge gradient magnitude of the point is greater than the mean edge gradient of the convolutional feature response sub-image, then the point is added to the mask; the judgment and addition process is repeated until no new points to be completed meet the conditions, and a complete mask is obtained.

[0022] Secondly, the present invention provides an intelligent identification system for aquatic organisms on unmanned vessels based on deep learning, which adopts the following technical solution: the system is implemented by a processor reading and executing computer program instructions in the memory to realize the above-mentioned intelligent identification method for aquatic organisms on unmanned vessels based on deep learning.

[0023] The embodiments of the present invention have at least the following beneficial effects:

[0024] The beneficial effects of this invention include the following aspects:

[0025] 1. By normalizing and weighting the information entropy of the convolutional feature response subgraph and the mean edge gradient, a crowding index for candidate regions is formed. This index reflects both the distribution of local feature responses and the complexity of boundary structures, so that the suppression strategy no longer relies solely on a fixed crossover ratio threshold. This helps to distinguish between locally crowded regions and ordinary regions in underwater dense individual scenarios.

[0026] 2. An inhibition factor is obtained by exponential decay based on the crowding index, and multiplied by the baseline cross-union ratio (CUP) threshold to form an adaptive CUP threshold for each candidate region. At the same time, the baseline CUP threshold can be adjusted within a limited range based on the historical detection capture rate. When the historical detection capture rate is lower than the set value, the baseline CUP threshold can be appropriately increased to reduce the risk of premature merging and suppression of densely adjacent individuals. When the capture rate meets the requirements, maintaining a relatively stable baseline threshold helps to achieve a more appropriate balance between detection box retention and redundancy removal.

[0027] 3. After obtaining the initial mask through instance segmentation, the response extreme points are used as seeds, and the mask breakage areas are recursively completed based on the distance between the point to be completed and the seed, as well as the edge gradient conditions. This completion process is consistent with the distance and gradient determination conditions defined in the claims, which helps to reduce mask breaks caused by occlusion, weak texture, or missing local responses, and allows the individual contour information of aquatic organisms to enter the subsequent counting or morphological analysis process more completely. Attached Figure Description

[0028] Figure 1 The schematic diagram illustrates the steps of the deep learning-based intelligent identification method and system for aquatic organisms on unmanned vessels in this invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] Step S1: Generate candidate regions from the underwater images of the unmanned vessel using a deep learning object detection network.

[0032] While the unmanned vessel is navigating on the water, it uses an onboard underwater high-definition camera to capture RGB color images at a preset frame rate. Each frame has a resolution of 1280×720 pixels, with the unit being pixels (px). The image resolution is determined by the implementer based on the camera parameters used in the specific implementation scenario.

[0033] Due to the absorption and scattering of light by water, underwater images generally exhibit low contrast, bluish-green color, and localized blurring.

[0034] The acquired image data is directly input into a deep learning object detection network deployed on the shipboard computing unit for processing in the form of a digital matrix.

[0035] This deep learning object detection network uses a two-stage object detection network, Faster R-CNN, which is a well-known technology and will not be described in detail here. Implementers can replace it with other object detection networks according to the specific implementation scenario.

[0036] The processing flow is as follows: using ResNet-50 as the backbone network to perform layer-by-layer convolution and pooling operations on the input image to obtain convolutional feature maps with different spatial resolutions; the region proposal network presets anchor boxes of various scales and aspect ratios at each spatial location of the feature map.

[0037] The dimensions are taken from empirical values ​​of 32×32 pixels, 64×64 pixels and 128×128 pixels, and the aspect ratios are taken from 1:1, 1:2 and 2:1, a total of 9 combinations. The dimensions and aspect ratios of the anchor frame are preset parameters, but can also be adjusted by the implementer according to the target size distribution.

[0038] A small convolutional network is used to output foreground and background binary classification scores and coordinate offsets relative to the anchor boxes for each anchor box, forming a region proposal.

[0039] The region proposal is input into the Region of Interest Align (RoIAlign) layer and uniformly mapped to a fixed-size feature of 7×7 pixels. Then, it is passed through a fully connected layer for classification scoring and bounding box refinement regression. The fixed size of 7×7 pixels is a preset empirical value that can be adjusted by the implementer according to the specific implementation scenario.

[0040] The final output includes all candidate regions in each frame of the image, with each candidate region described by a quintuple: , , , , .

[0041] in, and Let be the pixel coordinates of the center point of the candidate region in the image. The pixel width of the candidate region. The pixel height of the candidate region. , which is the confidence level that the candidate region belongs to the aquatic organism target, is dimensionless.

[0042] All candidate regions together constitute the candidate region set. If it is in a scene with a dense school of fish or shrimp, The number of candidate regions can typically reach hundreds to thousands.

[0043] While generating candidate regions, the intermediate convolutional layers of the backbone network are preserved. In this embodiment, the feature map output by the 4th convolutional block of ResNet-50 is used to extract the convolutional feature response sub-maps corresponding to each candidate region.

[0044] Step S2: For each candidate region, extract its convolutional feature response sub-image, calculate the information entropy and mean edge gradient of the sub-image and normalize them respectively, and then weight and fuse them to obtain the crowding index.

[0045] For candidate region set Each candidate region The bounding box coordinates on the original image are mapped onto the feature map output by the 4th convolutional block according to the scaling ratio of the ResNet-50 backbone network of the deep learning object detection network.

[0046] The region of interest is used to crop and align the data to a fixed size. subgraph ,in and The empirical value of 14 pixels is used. The unit is pixels and the symbol is px. This size is a preset parameter, but it can also be adjusted by the implementer according to the resolution of the feature map.

[0047] Subgraph The total number of spatial points is Each spatial location is one 3D feature vectors It is determined by the number of channels in the convolutional layer.

[0048] Because the spatial distribution uniformity of feature responses within a subgraph can reflect the aggregation state of aquatic biological targets within the candidate region, we first... The L2 norm of each spatial point is calculated in the channel dimension as the convolution response intensity of that point.

[0049] Then, the response intensities of all spatial points are summed and normalized to obtain the normalized feature probabilities of each spatial point. It is dimensionless. Therefore, the information entropy is calculated. The formula is:

[0050]

[0051] In the formula, For the first The feature response information entropy of each candidate region is expressed in bits, with the unit symbol being bit. is the total number of points in the subgraph space, which is a dimensionless integer; For the first The normalized feature probability of a spatial point, where the calculation of information entropy is a well-known technique.

[0052] when When the distribution tends to be uniform, An increase in the value of the feature response indicates a more balanced spatial distribution, reflecting a denser distribution of targets within the region and the absence of a dominant individual.

[0053] Because the intensity of the boundary gradient response generated by the interlacing of target edges can characterize the structural complexity within the region, this subgraph... Calculate the edge gradient magnitude for all spatial points and obtain the mean.

[0054] The specific steps are as follows: First, A single-channel image is obtained by averaging across the channel dimensions, and then the Sobel edge detection operator is used to calculate the edge detection for each spatial point. gradient magnitude The magnitude of this gradient is dimensionless. Therefore, the mean of the marginal gradient... The calculation formula is:

[0055]

[0056] In the formula, The mean gradient at the region's edge is dimensionless. For the first The edge gradient magnitude of each spatial point.

[0057] The larger the mean edge gradient, the more drastic the boundary changes in the region and the more complex the target contours.

[0058] Since the original numerical ranges of information entropy and the mean of marginal gradients differ significantly, direct weighted combination will lead to an imbalance in their contributions.

[0059] Therefore, the original information entropy and gradient mean of each candidate region are normalized so that both are mapped to... Interval.

[0060] The normalization method performs minimum-maximum normalization based on the statistical distribution of all candidate regions in the current frame image. That is, for each candidate region... Its normalized information entropy for:

[0061]

[0062] Normalized marginal gradient mean for:

[0063]

[0064] in, and These are the minimum and maximum values ​​of the original information entropy of all candidate regions in the image frame, respectively. and These are the minimum and maximum values ​​of the original gradient mean for all candidate regions, respectively. After normalization, and All values ​​are dimensionless. The normalization method used is the maximum and minimum value normalization method, which is a well-known technical means.

[0065] If the information entropy of all candidate regions in the current frame is equal, that is... Then the normalized information entropy of all candidate regions All are taken as 0; similarly, if Then the mean normalized edge gradient of all candidate regions All values ​​are set to 0.

[0066] Multiply the normalized information entropy by the first weighting coefficient. Multiply the normalized marginal gradient mean by the second weighting coefficient The crowding index of the candidate region is obtained by weighted fusion. The calculation formula is:

[0067]

[0068] In the formula, Contribute weights to the first weight coefficient, i.e., information entropy. The second weight coefficient, i.e. the mean of the edge gradient, contributes weight; both are configurable weight parameters of the weighted fusion crowding index.

[0069] In this embodiment Take an empirical value of 0.6. Taking an empirical value of 0.4, all values ​​are dimensionless, and Among them, the experience value can be adjusted by the implementer according to the specific implementation scenario.

[0070] and These are the normalized information entropy and the normalized marginal gradient mean, respectively.

[0071] Candidate region The crowding index, dimensionless, comprehensively represents the balance of local feature responses and the complexity of boundary structures. The larger the value, the more crowded the target distribution and the more significant the interlacing of contours within the candidate region.

[0072] Step S3: Generate inhibition factors based on the crowding index of each candidate region through exponential decay.

[0073] In images of dense schools of fish or shrimp underwater, candidate regions of different individuals often overlap significantly, and a fixed intersection and comparison threshold is insufficient to distinguish between real independent targets and redundant detections.

[0074] Therefore, using the already obtained crowding index of each candidate region, an inhibition factor is generated through exponential decay, and multiplied by the basic cross-union ratio threshold to calculate an adaptive cross-union ratio threshold for each candidate region.

[0075] First, define the candidate region. The congestion index is This index is dimensionless; the larger the value, the more crowded the area is with targets.

[0076] Will Input the exponential decay function: first set the middle value to negative. Then, using the natural constant as the base, and with To generate an inhibitory factor for the candidate region for exponential generation The calculation formula is:

[0077]

[0078] In the formula, Candidate region The inhibitory factor is dimensionless; The attenuation coefficient controls the steepness of the exponential curve. An empirical value of 0.7 is used. It is dimensionless and can be adjusted by the implementer according to the specific implementation scenario. For the natural constant An exponential function with base 0.

[0079] The effect of the exponential decay function is: when When it is large, For negative numbers with large absolute values, It tends to a minimum value near zero; when When the product is small, it approaches zero. Approaching 1.

[0080] This suppression effect manifests as follows: candidate regions with high crowding receive smaller suppression factors, which weakens the redundancy merging of them, thereby reducing the risk of false merging of high-density regions; regions with low crowding receive larger suppression factors, maintaining a strong redundancy removal capability.

[0081] Subsequently, a baseline cross-union ratio (CUI) threshold is set as the input parameter for subsequent adaptive calculations, denoted as . The empirical value of 0.5 is taken. It is dimensionless. This threshold reflects the crossover ratio (CUP) judgment scale under standard application environment. It can also be adjusted by the implementer based on the training set or the test statistics over the years.

[0082] For each candidate region to inhibit its factor Intersection over threshold with base Multiplying these yields the adaptive cross-union ratio (CUI) threshold, denoted as . The calculation formula is:

[0083]

[0084] In the formula, Candidate region The final adopted threshold for cross-union ratio (CUNR) is dimensionless. Because... and For fixed values, The range of values ​​is .

[0085] This threshold directly determines the criteria for retaining a candidate region as an independent detection box in subsequent non-maximum suppression, and its value adapts to the degree of crowding in the region.

[0086] At this point, each candidate region obtains an adaptive crossover ratio threshold calculated in real time from its local crowding features, which can then be used for nonmaximum suppression screening.

[0087] Furthermore, to support the adjustment of the historical detection capture rate of the basic intersection-union ratio threshold, the historical detection capture rate of the most recent 100 frames can be calculated once every 100 frames processed during continuous operation.

[0088] Historical detection capture rate can be obtained based on the ratio of the number of targets captured in high-confidence trajectory continuity, manually sampled and labeled data, or confirmed samples to the number of reference targets. Here, 100 represents the number of statistical frames. The value is set to 100, which can be adjusted by the implementer according to the specific implementation scenario.

[0089] When the historical detection capture rate is lower than the set value, such as 0.85, it indicates that there may be cases where dense individuals are falsely suppressed under the current fixed basic intersection-union ratio (IU) threshold. In this case, the basic IU threshold should be increased step by step in a preset increment of 0.05. Furthermore, the increased value does not exceed the preset upper limit of 0.7, in order to reduce the probability that adjacent independent individuals are judged as redundant boxes; when the historical detection capture rate is not lower than the set value, the current basic intersection-union ratio threshold is maintained or the value is reverted to the default value of 0.5 by a preset step size of 0.05. The preset step size and the upper limit value of 0.7 and 0.85 are empirical values ​​that can be adjusted by the implementer according to the specific implementation scenario.

[0090] The adjusted baseline crossover-union ratio threshold is used as the globally unified baseline threshold within the current work cycle. It is then multiplied by the suppression factor corresponding to each candidate region to obtain the final adaptive crossover-union ratio threshold adopted by each candidate region.

[0091] Step S4: Perform non-maximum suppression based on the adaptive cross-union ratio threshold to filter out detection boxes.

[0092] Each candidate region Each obtained an adaptive cross-union ratio threshold calculated from its local crowding features. This threshold provides a differentiated suppression criterion compared to the traditional fixed threshold.

[0093] Because the candidate regions corresponding to different individuals in dense aquatic organism images have significant spatial overlap, but their confidence level is low. The relative level of confidence can usually still reflect the detection quality, so all candidate regions are first prioritized according to confidence level to ensure that the most reliable candidate regions are selected and retained first.

[0094] The specific operation is as follows: Set the candidate regions All elements are ranked by confidence level The values ​​are sorted in descending order from highest to lowest to obtain an ordered list. The first item in this list is the candidate region with the highest confidence in the current frame image.

[0095] Because the core of nonmaximum suppression lies in iteratively using high-confidence candidate regions as a comparison benchmark to determine whether other low-confidence candidate regions constitute redundancy, the processing flow is based on an ordered list. Start with the head: Select The one with the highest confidence level that has not yet been processed is used as the current region. Treat it as a detection box for an independent target and from Middle shift-in output set .

[0096] Then, traverse Each remaining candidate region Calculate its relationship with the current region Spatial intersection and union ratio.

[0097] Intersection and Union Defined as the ratio of the intersection area to the union area of ​​the two bounding boxes, dimensionless, where the intersection area is the pixel area of ​​the overlapping region of the two rectangles, and the union area is the sum of the individual pixel areas of the two rectangles minus the intersection area. The unit for both areas is square pixels, and the unit symbol is [symbol missing]. The calculation formula is:

[0098]

[0099] In the formula, and Each is the current region and the region to be compared The bounding box pixel area, This represents the area intersection operation. Represent the area and perform calculations. The range of values ​​is The larger the value, the higher the degree of spatial overlap between the two candidate regions. The formula for calculating the intersection-union ratio is a well-known technique.

[0100] Calculated Then, it is compared with the adaptive intersection-union (IU) threshold corresponding to the current region. Compare them.

[0101] because It is based on An adaptive threshold is generated by the local environment's congestion index, so this comparison decision can make a personalized response to the physical density conditions of the area.

[0102] If and only if At that time, the judgment and Targeting the same aquatic organism, It is treated as redundancy suppression, i.e., from the ordered list It will be permanently removed from the list and will no longer participate in subsequent candidate region comparisons.

[0103] like Therefore, even if two candidate regions overlap spatially, they are determined to be highly likely to belong to different independent individuals. Retained in They are awaiting judgment in subsequent rounds.

[0104] In the current area and After all remaining candidate regions have completed one round of intersection-union comparison and redundancy removal, if If there are still candidate regions that have not been processed, the algorithm iterates to the next round: starting again from... The candidate region with the highest confidence level is selected as the new current region. Move it into the output set and with Repeat the above steps of crossover ratio calculation and threshold comparison for the remaining candidate regions.

[0105] This iterative process continues until an ordered list is reached. Until it is empty.

[0106] At this point, all candidate regions have been processed, and the output set has been generated. The bounding boxes contained therein are the detection boxes that are retained after adaptive nonmaximum suppression filtering.

[0107] For high-density, crowded areas, because The value is relatively large, and the inhibitory factor generated in step S3 The value approaches a very small value near zero, leading to an adaptive threshold. Significantly below the baseline threshold .

[0108] Therefore, in this step, even if the candidate regions of two adjacent independent targets have a high degree of overlap, their intersection-union ratio is unlikely to exceed the reduced threshold, thus avoiding erroneous suppression and effectively retaining them as two independent detection boxes.

[0109] In low-density and sparse areas, The value is small. Approaching 1, near It maintains a relatively strong redundancy suppression capability, ensuring accurate suppression of truly repetitive candidate regions.

[0110] Final output detection box set Each detection box retains its original confidence level. The corresponding adaptive threshold records are used as the spatial differentiation results obtained by independent screening in this step.

[0111] Step S5: Perform instance segmentation on the detection box to obtain an initial mask, and extract the extreme response points as seeds.

[0112] For the set of detection boxes after adaptive nonmaximum suppression filtering Each detection box in The RoIAlign features are then input into the mask prediction branch of the deep instance segmentation network, which is a well-known component of Mask R-CNN and will not be described further.

[0113] A class-independent binary initial mask is output through convolution and deconvolution operations. The mask size is taken as an empirical value of 28×28 pixels (px). This size is a preset parameter and can be adjusted by the implementer according to the resolution requirements. After bilinear interpolation upsampling to the size of the detection box in the original image, the initial mask corresponding to the detection box is obtained. .

[0114] Because the convolutional feature response intensity of the target within the detection box is not uniformly distributed, the point with the maximum response value is usually located at the location of the most significant local feature of the target.

[0115] Therefore, for each detection box From its extracted convolutional feature response subgraph The spatial point with the highest response intensity is extracted and used as the feature center point of the target instance within the detection box, denoted as the seed point. .

[0116] Seed point Using the sub-graph The location of the maximum value of the L2 norm of the eigenvectors of all spatial points is determined. Let the set of spatial points of the subgraph be denoted as... spatial point The eigenvector at that location is ,but:

[0117]

[0118] In the formula, Seed point in subgraph The grid coordinates in the image contain row and column coordinates, and the unit is pixels (px). The characteristic response intensity at that point is dimensionless; This is a spatial position operation to obtain the maximum value.

[0119] Mapping the sub-image coordinates back to the original image coordinates: Let the detection box... The coordinates of the top left corner in the original image are: Width is The height is subgraph The size is In this embodiment .

[0120] Seed points on the subgraph The coordinates mapped to the original image are: Furthermore, the mapped seed point coordinates serve as the reference point for calculating the planar Euclidean distance between the point to be completed and the seed point.

[0121] Step S6: When there are breaks in the mask, recursively complete the mask according to the distance between the point to be completed and the seed and the gradient conditions to obtain a complete mask.

[0122] The initial mask generated in step S5 In the detection frame, when multiple aquatic organisms are adjacent or partially occluded, the probability response output by the mask branch is often concentrated in the bright center area of ​​the individual, causing the mask boundary to break, shrink or have holes in the low response area of ​​the target contour.

[0123] These breaks manifest as the presence of unmasked pixels inside or at the edges of the masked area that are not marked as foreground pixels, even though such pixels should belong to the same biological entity in the actual target.

[0124] Initial mask The pixels in the 8-neighborhood of a boundary pixel that are not part of the current mask but are adjacent to it, as well as the non-mask pixels in the holes inside the mask, are collectively denoted as the set of points to be filled. .

[0125] Set of completion points Each point to be completed Calculate its mapping with the seed point obtained in step S5. Planar Euclidean distance The unit is pixels, and the unit symbol is px.

[0126] Because the seed point corresponds to the location with the strongest target feature response, the probability that its surrounding pixels belong to the same biological individual decreases with increasing distance. Therefore, a distance threshold is used. Limit the scope of completion.

[0127] An empirical value of 8 pixels is used, in the original image coordinate system, with the unit being pixels (px). This threshold is a preset parameter, but can also be adjusted by the implementer based on the average target size and the distribution of mask gaps.

[0128] If and only if When the point to be completed is considered to be within the effective extension range near the seed point, the distance condition is satisfied.

[0129] Points to be completed that meet the distance condition Further determine its edge gradient conditions.

[0130] First, add the points that need to be completed. The original image coordinates are transformed back to the sub-image through the geometric mapping of the detection box. The coordinates in the middle are denoted as ,in:

[0131]

[0132] here , for Pixel coordinates in the original image.

[0133] Then take the subgraph Edge gradient magnitude at the corresponding position and its subgraph Mean marginal gradient of all points in space Compare them.

[0134] Since the edge gradient response of the true target contour region is usually higher than that of the interior region where the response is flatter, the judgment formula is set as follows: .

[0135] When the inequality is true, it indicates the point to be completed. Located in the transition zone at the edge of the target, it has high structural recovery value and the gradient condition is satisfied.

[0136] For simultaneously satisfying and Points to be completed for both conditions Change its category from non-mask to mask, that is, let ,in Indicates the current recursive round, the initial round. The corresponding mask is .

[0137] After one round of traversal and completion, the mask area expands outwards or internal holes are partially filled, generating an updated mask. .

[0138] Because each expansion of the mask boundary will cause non-masked pixels that were originally adjacent to the newly added mask pixels to become new boundary adjacency points, therefore... Based on this, re-extract the set of points to be completed. And repeat the above distance judgment and gradient condition judgment.

[0139] The recursive process failed to find any points to be filled that satisfy both conditions in the current round, i.e. Terminates when empty.

[0140] The mask obtained at this point is the complete mask after the fracture has been repaired. Its boundary is consistent with the actual maximum response area of ​​the target contour, and the voids inside caused by occlusion or insufficient response have been recursively filled.

[0141] This invention also discloses a deep learning-based intelligent identification system for aquatic life on unmanned vessels, comprising a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the deep learning-based intelligent identification method for aquatic life on unmanned vessels of this invention. The system also includes other components well-known to those skilled in the art, such as a communication bus and communication interface; their configurations and functions are known in the art and will not be described further here.

[0142] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make several improvements and substitutions without departing from the technical principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based intelligent identification method for aquatic organisms on unmanned surface vessels, characterized in that, include: Underwater images of unmanned vessels are acquired, candidate regions are generated through a deep learning object detection network, and the feature maps output from the intermediate convolutional layers of the deep learning object detection network are cropped and normalized according to the position of each candidate region to obtain the convolutional feature response sub-maps corresponding to each candidate region. For each candidate region, the information entropy and mean edge gradient of its convolutional feature response sub-image are calculated, and normalization is performed based on the statistical extreme values ​​of the information entropy and mean edge gradient of all candidate regions in the current frame image. The normalized information entropy and mean edge gradient are then weighted and fused to obtain the crowding index. An inhibition factor is generated by exponentially decaying the crowding index of each candidate region. This inhibition factor is then multiplied by the basic cross-union ratio threshold to obtain the adaptive cross-union ratio threshold. Non-maximum suppression is performed based on the adaptive cross-union ratio threshold to filter out detection boxes; The detection box is segmented into instances to obtain an initial mask, and the extreme points of the response in the convolutional feature response sub-image corresponding to the detection box are extracted as seed points. Determine if the initial mask has any breaks: Take the non-mask pixels in the neighborhood of the mask boundary and the internal holes as the points to be filled, and recursively fill them according to the distance between each point to be filled and the seed point and the gradient conditions to obtain a complete mask.

2. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, The information entropy is calculated in the following way: The convolutional feature response values ​​of each pixel in the convolutional feature response sub-image are normalized to the range of 0 to 1 to obtain the normalized feature probability of each pixel. The negative value of the sum of the normalized feature probabilities of each pixel and their logarithmic products is used as the information entropy.

3. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, The mean edge gradient is calculated in the following way: The edge gradient magnitude of each pixel within the convolutional feature response sub-image is calculated using an edge detection operator. The arithmetic mean of the absolute values ​​of the edge gradient magnitudes of each pixel is calculated and used as the mean edge gradient.

4. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, The congestion index includes: Multiply the normalized information entropy by the first weighting coefficient, and multiply the normalized marginal gradient mean by the second weighting coefficient. The crowding index is obtained by adding the weighted information entropy to the weighted mean edge gradient.

5. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, The exponentially decaying inhibition factor includes: Calculate the negative value of the product of the congestion index and the preset attenuation coefficient; The inhibition factor is defined as a power of the natural constant, with the negative value as the base.

6. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, The step of performing non-maximum suppression based on the adaptive cross-union ratio threshold to filter out detection boxes includes: Obtain the confidence score of each candidate region and sort the candidate regions from high to low confidence score; The candidate region with the highest confidence is selected as the current region, and the intersection-union ratio of the remaining candidate regions with the current region is calculated. The remaining candidate regions with an intersection-to-union ratio greater than the adaptive intersection-to-union ratio threshold corresponding to the current region are treated as redundancy suppression. Repeat until all candidate regions have been processed, and use the unsuppressed candidate regions as the detection boxes.

7. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, Extracting the extreme response points from the convolutional feature response sub-image corresponding to the detection box as seed points includes: The detection box is input into a deep learning instance segmentation network to obtain a pixel-level initial mask; In the convolutional feature response sub-image corresponding to the detection box, the pixel with the largest response value is taken as the response extremum point, and it is used as the segmentation seed point.

8. The method for intelligent identification of aquatic organisms on unmanned vessels based on deep learning according to claim 1, characterized in that, The recursive completion based on the distance between each point to be completed and the seed point, and the gradient conditions, includes: For a point to be filled in, if the pixel distance between the point and the seed point is less than a preset distance threshold, and the magnitude of the edge gradient of the point is greater than the mean edge gradient of the convolutional feature response sub-image, then the point is added to the mask. Repeat the judgment and addition process until no new points to be filled meet the conditions, and a complete mask is obtained.

9. A deep learning-based intelligent identification system for aquatic organisms on unmanned surface vessels, characterized in that, It includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the deep learning-based intelligent identification method for aquatic organisms on unmanned vessels as described in any one of claims 1 to 8.

Images