Underwater netting attachment density monitoring method, device and equipment and storage medium

Abstract

The invention relates to the technical field of image recognition, and discloses an underwater netting attachment density monitoring method, device and equipment and a storage medium, and the method comprises the steps: obtaining a to-be-detected underwater netting image shot by an underwater robot; preprocessing the underwater netting image to be detected to obtain a preprocessed netting image; inputting the preprocessed netting image into a pre-trained netting attachment detection model for processing, identifying attachments on the underwater netting image to be detected, adding a bounding box to each attachment, and outputting a detected netting image; performing region division on the detected netting image to obtain a plurality of sub-regions; respectively calculating the local attachment density of each sub-region and the global attachment density of the detected netting image; and according to the local attachment density and / or the global attachment density, generating an attachment cleaning suggestion. According to the invention, automatic and precise monitoring of the underwater netting attachments is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular, to a method, device, equipment and storage medium for monitoring the density of underwater net cage attachments. Background Art

[0002] With the continuous development of modern fishery technology, the monitoring of net cage attachments has become an important part of ensuring the healthy operation of fishery equipment. The net cage is in a complex deep - water environment for a long time. In such an environment, there are many factors that are not conducive to the monitoring of net cage attachments. Since the net cage is in a deep - water environment, problems such as insufficient light and blurred vision increase the difficulty of automatically monitoring the net cage attachments.

[0003] Traditional manual inspection methods are difficult to meet the needs of large - scale monitoring of net cage attachments. On the one hand, manual inspection is inefficient, consuming a large amount of human and time costs, and it is difficult to complete a comprehensive inspection of a large - area net cage within a limited time. On the other hand, manual inspection is limited by human subjective judgment and physiological limits, and it is easy to miss inspections, making it difficult to ensure the accuracy and comprehensiveness of monitoring.

[0004] Therefore, there is an urgent need to develop an efficient and automated method for monitoring the density of underwater net cage attachments. Summary of the Invention

[0005] Based on this, in view of the problem that the existing monitoring of net cage attachments relies on manual inspection methods, a method, device, equipment and storage medium for monitoring the density of underwater net cage attachments are proposed.

[0006] The first aspect of the present invention provides a method for monitoring the density of underwater net cage attachments, the method comprising:

[0007] Obtaining an underwater net cage image to be detected captured by an underwater robot;

[0008] Pre - processing the underwater net cage image to be detected to obtain a pre - processed net cage image;

[0009] Inputting the pre - processed net cage image into a pre - trained net cage attachment detection model for processing, identifying the attachments on the underwater net cage image to be detected, adding a bounding box to each attachment, and outputting a detected net cage image;

[0010] Dividing the detected net cage image into regions to obtain a number of sub - regions;

[0011] Calculating the local attachment density of each sub - region and the global attachment density of the detected net cage image respectively;

[0012] Generating an attachment cleaning suggestion according to the local attachment density and / or the global attachment density.

[0013] Further, before the step of obtaining the underwater netting image to be detected captured by the underwater robot, the following steps are further included:

[0014] Making a training dataset for the netting attachment detection model;

[0015] Constructing the framework of the initial detection model and setting the initial model parameters; the initial detection model uses the YOLOv9 model as the basic model framework, and the basic model framework includes a backbone network, a neck network, and a detection head. The backbone network uses a neural network combining RepNCSPELAN and CSPNet architecture. Taking the training pictures as the input of the basic model framework, through each convolutional layer of the backbone network, extracting low-dimensional features and high-dimensional features of the damaged netting from the training pictures, and outputting damaged netting feature maps of different scales; through the neck network, performing feature fusion on the damaged netting feature maps of different scales output by the backbone network, and outputting a fused feature map; inputting the fused feature map into the detection head, and predicting the coordinates of each attachment by the detection head, and adding a bounding box to each attachment.

[0016] Training the initial detection model with the training dataset, updating the model parameters, and generating a trained netting attachment detection model.

[0017] Further, the step of constructing the framework of the initial detection model further includes:

[0018] Introducing a deformable attention module into the basic model framework. The step of introducing the deformable attention module into the basic model framework includes: selecting a target convolutional layer in multiple convolutional layers of the backbone network, inserting the deformable attention module after the target convolutional layer, inputting the attachment feature map output by the target convolutional layer into the deformable attention module for processing, and obtaining a receptive field adjustment feature map; performing feature fusion on the attachment feature map output by the target convolutional layer and the receptive field adjustment feature map, and then inputting it into the next convolutional layer of the target convolutional layer for processing.

[0019] Further, the step of respectively calculating the local attachment density of each sub-region and the global attachment density of the detected netting image includes:

[0020] Counting the number of bounding boxes in each sub-region;

[0021] Calculating the area of each sub-region;

[0022] Dividing the number of bounding boxes in each sub-region by the corresponding area of the sub-region to obtain the local attachment density of each sub-region;

[0023] Taking the average of the local attachment densities of each of the sub-regions to obtain the global attachment density.

[0024] Further, the step of calculating the area of each of the sub-regions includes:

[0025] Searching from the center of a sub-region towards the edge for a complete mesh, and taking the first complete mesh found as the calibration mesh;

[0026] Calculating the pixel ratio between the calibration mesh and the sub-region;

[0027] Dividing the actual area of a preset mesh by the pixel ratio to obtain the area of the sub-region.

[0028] Further, the step of generating an attachment cleaning suggestion based on the local attachment density and the global attachment density includes:

[0029] Judging whether the global attachment density exceeds a preset global density threshold;

[0030] If so, comparing the local attachment density of each sub-region with a preset multi-level local density threshold respectively to obtain the attachment rating of each sub-region;

[0031] Judging whether the attachment rating of each sub-region reaches a preset rating;

[0032] If so, generating a first attachment cleaning suggestion and writing the positions of the sub-regions that reach the preset rating into the attachment cleaning suggestion.

[0033] Further, the step of generating an attachment cleaning suggestion based on the local attachment density includes:

[0034] Comparing the local attachment density of each sub-region with a preset multi-level local density threshold respectively to obtain the attachment rating of each sub-region;

[0035] Judging whether the attachment rating of each sub-region reaches a preset rating;

[0036] If so, updating the number of local density exceeding regions by incrementing the count by 1;

[0037] Judging whether the updated number of local density exceeding regions exceeds a preset number threshold;

[0038] If so, generating a second attachment cleaning suggestion and writing the positions of the sub-regions that reach the preset rating into the attachment cleaning suggestion.

[0039] A second aspect of the present invention provides an underwater net attachment density monitoring device, the device includes:

[0040] An image acquisition module for acquiring an underwater netting image to be detected captured by an underwater robot;

[0041] An image preprocessing module for preprocessing the underwater netting image to be detected to obtain a preprocessed netting image;

[0042] An attachment recognition module for inputting the preprocessed netting image into a pre-trained netting attachment detection model for processing, recognizing the attachments on the underwater netting image to be detected, adding bounding boxes to each attachment, and outputting a detected netting image;

[0043] A region division module for dividing the detected netting image into several sub-regions;

[0044] An attachment density calculation module for calculating the local attachment density of each sub-region and the global attachment density of the detected netting image respectively;

[0045] A cleaning suggestion module for generating attachment cleaning suggestions according to the local attachment density and / or the global attachment density.

[0046] A third aspect of the present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor causes the processor to execute the steps of the above-mentioned underwater netting attachment density monitoring method.

[0047] A fourth aspect of the present invention provides a computer device, which includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, it causes the processor to execute the steps of the above-mentioned underwater netting attachment density monitoring method.

[0048] The underwater netting attachment density monitoring method, device, equipment and storage medium of the present invention realize the automatic and precise monitoring of underwater netting attachments by acquiring an underwater netting image to be detected captured by an underwater robot; preprocessing the underwater netting image to be detected to obtain a preprocessed netting image; inputting the preprocessed netting image into a pre-trained netting attachment detection model for processing, recognizing the attachments on the underwater netting image to be detected, adding bounding boxes to each attachment, and outputting a detected netting image; dividing the detected netting image into several sub-regions; calculating the local attachment density of each sub-region and the global attachment density of the detected netting image respectively; generating attachment cleaning suggestions according to the local attachment density and / or the global attachment density. Description of the Drawings

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0050] Among them:

[0051] Figure 1 It is an application environment diagram of the underwater netting attachment density monitoring method in an embodiment;

[0052] Figure 2 It is a flowchart of the underwater netting attachment density monitoring method in an embodiment;

[0053] Figure 3 It is a structural block diagram of the underwater netting attachment density monitoring device in an embodiment;

[0054] Figure 4 It is a structural block diagram of a computer device in an embodiment. Specific embodiments

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0056] Figure 1 It is an application environment diagram of the underwater netting attachment density monitoring method in an embodiment. Refer to Figure 1 , this underwater netting attachment density monitoring method is applied to an underwater netting attachment density monitoring system. The underwater netting attachment density monitoring system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 can specifically be a desktop terminal or a mobile terminal, and the mobile terminal can specifically be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers. The terminal 110 is used to execute the method for monitoring the underwater netting attachment density, and the server 120 is used to store the training data set, data of the netting attachment detection model, etc.

[0057] Such as Figure 2As shown, in one embodiment, a method for monitoring the density of attachments on an underwater net is provided. This method can be applied to both terminals and servers. In this embodiment, it is exemplified by being applied to a terminal. The method for monitoring the density of attachments on the underwater net specifically includes the following steps:

[0058] S1: Obtain an image of the underwater net to be detected captured by an underwater robot;

[0059] S2: Preprocess the image of the underwater net to be detected to obtain a preprocessed net image;

[0060] S3: Input the preprocessed net image into a pre-trained net attachment detection model for processing, identify the attachments on the image of the underwater net to be detected, add a bounding box to each attachment, and output a detected net image;

[0061] S4: Divide the detected net image into regions to obtain several sub-regions;

[0062] S5: Calculate the local attachment density of each sub-region and the global attachment density of the detected net image respectively;

[0063] S6: Generate an attachment cleaning suggestion based on the local attachment density and / or the global attachment density.

[0064] In this embodiment, in the above step S1, an underwater robot equipped with a high-definition camera device is used to move and capture in a preset underwater net area. The underwater robot can move through a preset path planning to ensure full coverage of the net area and avoid monitoring omissions. The above image of the underwater net to be detected is the image captured by the underwater robot, which can be an RGB image and includes the net and various objects that may be attached to it, such as algae, shellfish, etc.

[0065] In the above step S2, the above preprocessing includes denoising, enhancing contrast, color correction, etc. Specifically, a denoising algorithm (such as Gaussian filtering) is used to remove the noise points generated by the underwater environment interference in the image of the underwater net to be detected and reduce interference information; an image enhancement algorithm, such as histogram equalization, is used to enhance the contrast of the image of the underwater net to be detected and make the details in the image of the underwater net to be detected clearer; color correction is performed to adjust the color deviation caused by reasons such as underwater light refraction and restore the true color of the image of the underwater net to be detected. The quality of the preprocessed net image obtained through step S2 is higher, thereby improving the accuracy of monitoring.

[0066] In the above step S3, the above-mentioned netting attachment detection model is constructed based on a deep learning algorithm, such as a convolutional neural network. During the training phase of the model, a large number of underwater netting image samples with annotations are used to enable the model to learn the features of different attachments. When a preprocessed netting image is input, the model extracts and analyzes the image features through convolutional layers, pooling layers, etc., determines whether there are attachments in the image, and determines their positions and categories. Then, bounding boxes are added around each identified attachment, and a detected netting image with bounding box annotations is output.

[0067] In the above step S4, for region division of the detected netting image, a regular grid division method can be adopted, and the detected netting image is evenly divided into multiple rectangular sub-regions of the same size according to a certain size ratio; or a grid with different sizes can be used for division. For example, according to the netting depth identified from the image, a larger-size grid is used for the netting in the near distance, and a smaller-size grid is used for the netting in the far distance. In this step, the detected netting image is divided into several sub-regions, which is conducive to subsequent analysis of the attachment distribution in different sub-regions, understanding the aggregation degree of attachments in different parts of the netting, and thus providing more accurate attachment cleaning suggestions.

[0068] In the above step S5, when calculating the local attachment density, the number of bounding boxes (i.e., the number of attachments) in each sub-region can be counted, and then divided by the area of the sub-region to obtain the local density value. When calculating the global attachment density, the total number of attachments in the entire detected netting image is counted and divided by the total area of the image; or the average value of the local attachment densities of each sub-region can also be calculated.

[0069] In the above step S6, different density thresholds are set in advance according to empirical data or actual test results, such as low density, medium density, high density, etc. When the local density and / or global density exceeds the corresponding threshold, cleaning suggestions are generated according to the exceeding degree and regional distribution. The execution methods of the above-mentioned attachment cleaning suggestions include real-time alarms (such as sound or visual signals) and remote notifications (such as text messages or emails). In the remote notification, text or voice notifications can be generated. For example, when the local density in a certain region is too high, it is recommended to clean this region first; if the global density is on the high side, it is recommended to clean the netting comprehensively, etc.

[0070] Furthermore, by automatically recording the situations where the local attachment density and the global attachment density exceed the standard (including the exceeding period, the exceeding position, etc.), it can be used for subsequent analysis, such as automatically generating an optimized cleaning period and a resource allocation plan. For example, when the local density of a certain sub-region exceeds the set high-density threshold for 3 consecutive days, record its exceeding period as 3 days, and at the same time record the exceeding position, accurate to the specific sub-region position or number. By analyzing the historical data of exceeding the standard, an optimized cleaning period plan is automatically generated. If it is found that a certain area always frequently has the attachment density exceeding the standard in a specific season, the cleaning period can be shortened in advance before the arrival of this season to prevent the fishing net from being blocked; in terms of resource allocation, according to the exceeding situations of different regions, the cleaning personnel and equipment are reasonably allocated. For example, more cleaning boats and personnel are arranged for the areas with serious exceeding the standard.

[0071] Furthermore, based on the situations where the local attachment density and the global attachment density exceed the standard, the exceeding period can be analyzed, so as to dynamically adjust the monitoring frequency, reduce the calculation amount, improve the monitoring efficiency, reduce the risk of blockage of the underwater netting, and ensure the normal operation of aquaculture. Specifically, if the situation of exceeding the standard occurs multiple times within a certain period and the exceeding period is short, it indicates that the growth rate of the attachments on the netting is fast or the environmental factors cause the attachments to gather faster. At this time, the monitoring frequency is appropriately increased, from the original once a week to once every three days, so as to timely detect the density change. On the contrary, if the situation of exceeding the standard has not occurred for a long time and the exceeding period is long, the monitoring frequency can be reduced, from once every three days to once a week. By dynamically adjusting the monitoring frequency, while ensuring the monitoring effect, the unnecessary calculation amount is reduced, the monitoring efficiency is improved, the risk of blockage of the underwater netting is effectively reduced, and the normal operation of aquaculture is ensured.

[0072] The underwater netting attachment density monitoring method of this embodiment realizes the automatic and precise monitoring of the underwater netting attachments through the above steps.

[0073] In some specific embodiments, before step S1 of obtaining the image of the underwater netting to be detected taken by the underwater robot, it further includes:

[0074] S01: Making a training data set for the netting attachment detection model;

[0075] S02: Construct the framework of the initial detection model and set the initial model parameters. The initial detection model uses the YOLOv9 model as the basic model framework, and the basic model framework includes a backbone network, a neck network, and a detection head. The backbone network adopts a neural network with the RepNCSPELAN combined with the CSPNet architecture. Take the training images as the input of the basic model framework, and extract the low-dimensional and high-dimensional features of the damaged netting from the training images through each convolutional layer of the backbone network, and output damaged netting feature maps of different scales. Through the neck network, fuse the damaged netting feature maps of different scales output by the backbone network to output a fused feature map. Input the fused feature map into the detection head, and the detection head predicts the coordinates of each attachment, and adds a bounding box to each attachment.

[0076] S03: Train the initial detection model with the training dataset, update the model parameters, and generate a trained netting attachment detection model.

[0077] In this embodiment, in the above step S01, collect a large number of underwater netting images in different environments, including attachments in different waters, different lighting conditions, and different growth stages, etc. The sources of these images can be field shooting, historical data accumulation, etc. Then use a professional image annotation tool, such as Label Img, to manually annotate each attachment in the image, including its category (such as algae, shellfish, etc.) and precise location, draw a bounding box around the attachment and record relevant information. After the annotation is completed, divide the dataset into a training set, a validation set, and a test set according to a certain ratio (such as 7:2:1). The training set is used for model training, the validation set is used for adjusting model parameters, and the test set is used for evaluating model performance.

[0078] In the above step S02, the YOLOv9 model is used as the basic model framework, and the backbone network adopts a neural network with the RepNCSPELAN combined with the CSPNet architecture. During the model construction process, according to the design of the network structure, the parameters of each layer are determined, such as the convolutional kernel size, stride, etc. When the training images are input, the RepNCSPELAN architecture of the backbone network gradually extracts the low-dimensional features (such as simple edges, textures, etc.) and high-dimensional features (such as complex shapes, semantics, etc.) of the damaged net and attachments in the training images through different convolutional layers, and outputs damaged net feature maps of different scales. The CSPNet reduces redundant calculations (such as directly passing shallow features to deep layers) by partially cross-stage connecting different convolutional layers, improving the model inference speed and computational efficiency, and is particularly suitable for environments with limited resources (such as underwater monitoring). Then, the neck network receives these feature maps of different scales, integrates the features of different scales through operations such as upsampling, downsampling, and feature fusion, and outputs a fused feature map. Finally, the fused feature map is input into the detection head, and the detection head predicts the coordinates of each attachment according to the information of the feature map and adds a bounding box around each attachment.

[0079] In the above step S03, the training set is input into the initial detection model. The model calculates the difference between the prediction result and the true annotation according to the input image and annotation information, and usually uses a loss function (such as cross-entropy loss function, mean squared error loss function, etc.) to measure this difference. Then, through the backpropagation algorithm, the loss value is backpropagated to each layer of the model to adjust the parameters of the model (such as weights and biases), so that the prediction result of the model gradually approaches the true annotation. During the training process, this process is continuously iterated, and at the same time, the validation set is used to monitor the performance of the model to prevent overfitting. When the performance of the model on the validation set reaches certain metrics (such as accuracy, recall, etc.), the training is stopped, and a trained detection model for net attachments is obtained.

[0080] In some specific embodiments, step S02 of constructing the framework of the initial detection model further includes:

[0081] S021: Introduce a deformable attention module into the basic model framework. The steps of introducing a deformable attention module into the basic model framework include: selecting a target convolutional layer in multiple convolutional layers of the backbone network, inserting a deformable attention module after the target convolutional layer, inputting the attachment feature map output by the target convolutional layer into the deformable attention module for processing to obtain a receptive field adjustment feature map; after fusing the attachment feature map output by the target convolutional layer and the receptive field adjustment feature map, input it into the next convolutional layer of the target convolutional layer for processing.

[0082] In this embodiment, in the multi-layer convolutional layers of the backbone network, target convolutional layers are selected according to the stage of feature extraction and the characteristics of feature maps. For example, intermediate convolutional layers can be selected because the feature maps at this stage retain certain detailed information and possess some abstract semantic features; or the most suitable target convolutional layer can be determined through manual experience.

[0083] After determining the target convolutional layer, a deformable attention module is added after this target convolutional layer. The deformable attention module consists of a deformable convolution and an attention mechanism. The deformable convolution allows the sampling positions of the convolution kernel on the feature map to no longer be limited to regular grids, but can adaptively adjust the sampling positions according to the input features, thereby expanding the receptive field and capturing richer context information; the attention mechanism assigns different weights to features at different positions to highlight important features.

[0084] The attachment feature map output by the target convolutional layer is input into the deformable attention module. The deformable attention module first performs a convolution operation on the feature map using the deformable convolution to adjust the sampling positions to adapt to attachment features of different shapes and scales. Then, through the attention mechanism, the importance weights of features at each position are calculated, and the convolved feature map is weighted to obtain a receptive field adjusted feature map.

[0085] The original attachment feature map output by the target convolutional layer and the receptive field adjusted feature map obtained after being processed by the deformable attention module are fused. Common fusion methods include element-wise addition, concatenation, etc. For example, if the element-wise addition method is adopted, the element values at corresponding positions of the two feature maps are added to obtain the fused feature map. Then, the fused feature map is input into the next convolutional layer of the target convolutional layer to continue subsequent feature extraction and processing.

[0086] The deformable attention module of this embodiment can adaptively adjust the receptive field, enabling the model to better capture attachment features of different shapes, sizes, and postures. In the underwater environment, the shapes of fishing net attachments are diverse, and traditional fixed convolution kernels may not be able to fully capture these complex features. The deformable convolution of the deformable attention module can adjust the sampling positions according to the actual situation of the attachments, extract features more comprehensively, and improve the recognition accuracy of the model for various attachments.

[0087] In some specific embodiments, the steps S5 of calculating the local attachment density of each sub-region and the global attachment density of the detected netting image include:

[0088] S501: Count the number of bounding boxes in each sub-region;

[0089] S502: Calculate the area of each sub-region;

[0090] S503: Divide the number of bounding boxes in each sub-region by the area of the corresponding sub-region to obtain the local attachment density of each sub-region;

[0091] S504: Calculate the average of the local attachment densities of all sub-regions to obtain the global attachment density.

[0092] In this embodiment, in the above step S501, after dividing the detected netting image into several sub-regions, for each sub-region, count the bounding boxes one by one. Specifically, the pixel points in the sub-region can be traversed through image recognition, and it is determined whether it is a bounding box according to the characteristics of the bounding box (such as color, contour, etc.), and the bounding boxes are counted. For example, a pixel value range can be set. When the pixel value of a certain region is within this range, it is considered that this region belongs to the bounding box, and then the regions that meet the conditions are accumulated and counted.

[0093] In the above step S502, the method of image segmentation and pixel counting can be adopted, that is, count the number of pixels in the sub-region, and then convert the number of pixels into the actual area according to the scale of the image of each sub-region.

[0094] In the above step S503, after obtaining the number of bounding boxes and the area of the sub-region in each sub-region, perform a division operation on the two. For example, if the number of bounding boxes in a certain sub-region is A and the area of the sub-region is N, then the local attachment density of this sub-region is ρ = N / A. The local attachment density can intuitively reflect the density of the attachments in each sub-region. Calculating the density through the number of bounding boxes is beneficial to reducing the calculation amount and improving the statistical speed.

[0095] In the above step S504, add up the local attachment density values of all sub-regions calculated in step S503, and then divide by the total number of sub-regions. Suppose there are m sub-regions, and the local attachment densities of each sub-region are ρ1, ρ2, ρ3... ρ m , then the global attachment density is The global attachment density can reflect the average density of the attachments on the detected netting image as a whole.

[0096] In a specific embodiment, step S502 of calculating the area of each sub-region includes:

[0097] S5021: Search for complete meshes from the center of a sub-region to the edge, and take the first complete mesh found as the calibration mesh;

[0098] S5022: Calculate the pixel ratio between the calibration mesh and the sub-region;

[0099] S5023: Divide the actually preset mesh area by the pixel ratio to obtain the area of the sub-region.

[0100] In this embodiment, in the above step S5021, after the sub-region division is completed, for each sub-region, its central position is determined. Specifically, the central coordinates can be obtained by calculating the geometric center of the sub-region bounding box. Then, starting from this central position, a search is made towards the edge of the sub-region in a preset search manner (for example, expanding outwards in a spiral shape). During the search process, it is judged whether it is a complete mesh according to the characteristics of the mesh (such as regular shape, closed lines, etc.). Once the first mesh that meets the characteristics of a complete mesh is found, it is determined as the calibration mesh. For example, using edge detection and contour analysis in image processing algorithms, an area with a closed contour and a shape conforming to the mesh rules is identified and recognized as a complete mesh.

[0101] Due to the actual state of the underwater netting may have deformations, occlusions, etc., it is relatively difficult to directly calculate the area of the sub-region. Calculating the area of each sub-region accurately through 3D mapping will make the calculation complex. The calibration mesh is a relatively regular and easily recognizable element. Using it as a reference for calculating the area of the sub-region can reduce the error caused by the irregularity of the netting and relatively simply improve the accuracy of area calculation.

[0102] In the above step S5022, to calculate the pixel ratio between the calibration mesh and the sub-region, the number of pixels contained in the calibration mesh and the sub-region needs to be counted respectively.

[0103] In the above step S5023, the area of a calibration mesh is made equal to the actual area of the preset mesh, and the actual area can be obtained by measuring the netting in advance. Assume the actual area of the preset mesh is A eye , and the pixel ratio calculated in step S5022 is R, then the area A i of the sub-region = A eye / R.

[0104] This embodiment avoids the difficulty of directly measuring the area of the sub-region in the complex underwater environment through the relationship between the actual area of the preset mesh and the pixel ratio on the image. At the same time, since the calibration mesh is selected from within the sub-region, it can better reflect the characteristics of the netting where the sub-region is located, making the calculated area of the sub-region closer to the actual value.

[0105] In some specific embodiments, step S6 of generating an attachment cleaning suggestion according to the local attachment density and the global attachment density includes:

[0106] S601: Judge whether the global attachment density exceeds the preset global density threshold;

[0107] S602: If so, compare the local attachment density of each sub-region with a preset multi-level local density threshold to obtain the attachment rating of each sub-region.

[0108] S603: Determine whether the attachment rating of each sub-region reaches a preset rating.

[0109] S604: If so, generate a first attachment cleaning suggestion and write the location of the sub-regions that reach the preset rating into the attachment cleaning suggestion.

[0110] In this embodiment, in the above step S601, a global density threshold is set in advance according to experience or historical data, which reflects the upper limit of the acceptable overall attachment density of the fishing net. If the global attachment density does not exceed the threshold, it indicates that the overall condition of the fishing net is relatively good and no large-scale cleaning is required; if it exceeds the global density threshold, it indicates that the attachment problem of the overall fishing net is relatively serious. In some embodiments, a reminder for attachment cleaning can be directly given. In other embodiments, further analysis is carried out according to the subsequent steps to avoid waste of resources caused by excessive cleaning and to timely discover specific situations that need to be processed.

[0111] In the above step S602, when the global attachment density exceeds the preset global density threshold, further analysis of the sub-regions is started. A multi-level local density threshold is set in advance. For example, three levels can be set: low-level, medium-level, and high-level thresholds. The local attachment density of each sub-region is compared with these thresholds in turn. If the local attachment density is lower than the low-level threshold, the attachment rating of the sub-region is low; if it is between the low-level and medium-level thresholds, the rating is medium; if it is higher than the medium-level threshold, the rating is high.

[0112] In the above step S603, a rating standard is set in advance. For example, it is stipulated that only the sub-regions with a high rating reach the preset rating. For each sub-region, compare the attachment rating obtained in step S602 with the preset rating.

[0113] In the above step S604, when the attachment rating of a certain sub-region reaches the preset rating, a first attachment cleaning suggestion is generated. The content of the first attachment cleaning suggestion can include, for example, the cleaning method to be adopted and the time suggestion for cleaning. At the same time, the location information of the sub-regions that reach the preset rating (such as the coordinate range or number of the sub-region in the image) is written into the first attachment cleaning suggestion. The first attachment cleaning suggestion can be output in the form of a text file or a report for the convenience of staff to view and execute.

[0114] In some other specific embodiments, step S6 of generating an attachment cleaning suggestion according to the local attachment density includes:

[0115] S611: Compare the local attachment density of each sub-region with a preset multi-level local density threshold to obtain an attachment rating of each sub-region;

[0116] S612: Determine whether the attachment rating of each sub-area reaches a preset rating;

[0117] S613: If yes, then the number of local density exceeding standard areas is updated by adding 1 to the count;

[0118] S614: Determine whether the updated number of local density exceeding standard areas exceeds a preset number threshold;

[0119] S615: If yes, generate a second attachment cleaning suggestion, and write the position of the sub-region that reaches the preset rating into the attachment cleaning suggestion.

[0120] In this embodiment, in the above step S611, multiple levels of local density thresholds are set in advance based on experience or historical data, for example, three levels of thresholds are set: low, medium, and high. For each sub-region, the local attachment density calculated previously is compared with these preset multi-level thresholds. If the local attachment density is lower than the lowest threshold, the attachment rating of the sub-region is low; if it is between the lowest threshold and the middle threshold, the rating is medium; if it is higher than the middle threshold, the rating is high.

[0121] In the above step S612, a rating standard is preset, for example, only sub-regions with high ratings reach the preset rating. For each sub-region, the attachment rating obtained in step S611 is compared with the preset rating.

[0122] In the above step S613, a counter for recording the number of local density-exceeding areas is initialized, and the initial value is set to 0. When the attachment rating of a sub-area reaches a preset rating, the value of the counter is increased by 1. By counting the number of local density-exceeding areas, the distribution range of sub-areas with serious attachment problems in the entire net can be intuitively understood.

[0123] In the above step S614, a quantity threshold is pre-set, and the threshold can be determined comprehensively based on the overall scale of the net, historical experience, etc. When the number of areas exceeding the standard exceeds the threshold, it means that there are many areas with serious attachment problems on the net, which may have a greater impact on the normal function of the net and fishery production, and more effective cleaning actions need to be taken in time; if it does not exceed the threshold, targeted treatment can be carried out on individual areas exceeding the standard according to actual conditions.

[0124] In the above step S615, when the number of local density exceeding-standard regions exceeds a preset number threshold, a second attachment cleaning suggestion is generated. The content of the second attachment cleaning suggestion may include the cleaning method to be adopted, the time suggestion for cleaning, etc. At the same time, the location information of all sub-regions that reach the preset rating (such as the number of the sub-region, the coordinates in the fishing net, etc.) is written into the second attachment cleaning suggestion in detail. The second attachment cleaning suggestion can be output in the form of a document, a report or a notice for the convenience of relevant personnel to view and execute.

[0125] As Figure 3 shown, in one embodiment, an underwater fishing net attachment density monitoring device is provided. The device includes:

[0126] An image acquisition module 10 for acquiring an image of the underwater fishing net to be detected captured by an underwater robot;

[0127] An image preprocessing module 20 for preprocessing the image of the underwater fishing net to be detected to obtain a preprocessed fishing net image;

[0128] An attachment recognition module 30 for inputting the preprocessed fishing net image into a pre-trained fishing net attachment detection model for processing, recognizing the attachments on the image of the underwater fishing net to be detected, adding a bounding box to each attachment, and outputting a detected fishing net image;

[0129] A region division module 40 for dividing the detected fishing net image into several sub-regions;

[0130] An attachment density calculation module 50 for calculating the local attachment density of each sub-region and the global attachment density of the detected fishing net image respectively;

[0131] A cleaning suggestion module 60 for generating an attachment cleaning suggestion according to the local attachment density and / or the global attachment density.

[0132] In some specific embodiments, the underwater fishing net attachment density monitoring device further includes:

[0133] A training set production module for producing a training data set for the fishing net attachment detection model;

[0134] The model construction module is used to construct the framework of the initial detection model and set the initial model parameters; the initial detection model uses the YOLOv9 model as the basic model framework, and the basic model framework includes a backbone network, a neck network, and a detection head. The backbone network adopts a neural network with the RepNCSPELAN combined with the CSPNet architecture. The training images are used as the input of the basic model framework. Through the convolutional layers of the backbone network, the low-dimensional features and high-dimensional features of the damaged netting are extracted from the training images, and feature maps of the damaged netting at different scales are output; through the neck network, the feature maps of the damaged netting at different scales output by the backbone network are fused, and a fused feature map is output; the fused feature map is input into the detection head, and the detection head predicts the coordinates of each attachment, and adds a bounding box to each attachment.

[0135] The model training module is used to train the initial detection model through the training dataset, update the model parameters, and generate a trained netting attachment detection model.

[0136] In some specific embodiments, the model construction module includes:

[0137] The deformable attention unit is used to introduce a deformable attention module into the basic model framework. The steps of introducing a deformable attention module into the basic model framework include: selecting a target convolutional layer in multiple convolutional layers of the backbone network, inserting a deformable attention module after the target convolutional layer, inputting the attachment feature map output by the target convolutional layer into the deformable attention module for processing to obtain a receptive field adjustment feature map; after fusing the attachment feature map output by the target convolutional layer and the receptive field adjustment feature map, inputting it into the next convolutional layer of the target convolutional layer for processing.

[0138] In some specific embodiments, the attachment density calculation module 50 includes:

[0139] The bounding box statistics unit is used to count the number of bounding boxes in each sub-region;

[0140] The sub-region area calculation unit is used to calculate the area of each sub-region;

[0141] The local density calculation unit is used to divide the number of bounding boxes in each sub-region by the corresponding area of the sub-region to obtain the local attachment density of each sub-region;

[0142] The global density calculation unit is used to calculate the average value of the local attachment densities of each sub-region to obtain the global attachment density.

[0143] In some specific embodiments, the sub-region area calculation unit includes:

[0144] A calibration mesh search subunit for searching for a complete mesh from the center to the edge of a sub-region, and taking the first complete mesh found as the calibration mesh;

[0145] A ratio calculation subunit for calculating the pixel ratio of the calibration mesh to the sub-region;

[0146] A sub-region area calculation subunit for dividing the preset actual area of the mesh by the pixel ratio to obtain the area of the sub-region.

[0147] In some specific embodiments, the cleaning suggestion module 60 includes:

[0148] A first judgment unit for judging whether the global attachment density exceeds a preset global density threshold;

[0149] A first local density comparison unit for, if the preset global density threshold is exceeded, respectively comparing the local attachment density of each sub-region with a preset multi-level local density threshold to obtain the attachment rating of each sub-region;

[0150] A second judgment unit for judging whether the attachment rating of each sub-region reaches a preset rating;

[0151] A first suggestion generation unit for, if the preset rating is reached, generating a first attachment cleaning suggestion and writing the positions of the sub-regions that reach the preset rating into the attachment cleaning suggestion.

[0152] In some specific embodiments, the cleaning suggestion module 60 includes:

[0153] A second local density comparison unit for respectively comparing the local attachment density of each sub-region with a preset multi-level local density threshold to obtain the attachment rating of each sub-region;

[0154] A third judgment unit for judging whether the attachment rating of each sub-region reaches a preset rating;

[0155] A count update unit for, if the preset rating is reached, updating the number of local density exceeding regions by incrementing the count by 1;

[0156] A fourth judgment unit for judging whether the updated number of local density exceeding regions exceeds a preset number threshold;

[0157] A second suggestion generation unit for, if the preset number threshold is exceeded, generating a second attachment cleaning suggestion and writing the positions of the sub-regions that reach the preset rating into the attachment cleaning suggestion.

[0158] The underwater netting attachment density monitoring device according to the embodiments of the present invention realizes the automatic and precise monitoring of underwater netting attachments.

[0159] Figure 4 The internal structure diagram of a computer device in an embodiment is shown. The computer device can specifically be a terminal or a server. As Figure 4 shown, the computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and can also store a computer program. When the computer program is executed by the processor, the processor can implement the method for monitoring the density of underwater net clothing attachments. The internal memory can also store a computer program. When the computer program is executed by the processor, the processor can execute the method for monitoring the density of underwater net clothing attachments. Those skilled in the art can understand that Figure 4 the structure shown in

[0160] In one embodiment, a computer device is proposed, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the following steps:

[0161] Obtain the underwater net clothing image to be detected captured by the underwater robot;

[0162] Preprocess the underwater net clothing image to be detected to obtain a preprocessed net clothing image;

[0163] Input the preprocessed net clothing image into a pre-trained net clothing attachment detection model for processing, identify the attachments on the underwater net clothing image to be detected, add a bounding box to each attachment, and output a detected net clothing image;

[0164] Divide the detected net clothing image into regions to obtain several sub-regions;

[0165] Calculate the local attachment density of each sub-region and the global attachment density of the detected net clothing image respectively;

[0166] Generate an attachment cleaning suggestion according to the local attachment density and / or the global attachment density.

[0167] The underwater net clothing attachment density monitoring device in the embodiments of the present invention realizes the automatic and precise monitoring of underwater net clothing attachments.

[0168] In one embodiment, a computer-readable storage medium is proposed, storing a computer program. When the computer program is executed by the processor, the processor performs the following steps:

[0169] Obtain the underwater netting image to be detected captured by the underwater robot;

[0170] Preprocess the underwater netting image to be detected to obtain a preprocessed netting image;

[0171] Input the preprocessed netting image into a pre-trained netting attachment detection model for processing, identify the attachments on the underwater netting image to be detected, add bounding boxes to each attachment, and output a detected netting image;

[0172] Divide the detected netting image into regions to obtain a number of sub-regions;

[0173] Calculate the local attachment density of each sub-region and the global attachment density of the detected netting image respectively;

[0174] Generate attachment cleaning suggestions according to the local attachment density and / or the global attachment density.

[0175] The underwater netting attachment density monitoring device according to the embodiments of the present invention realizes the automatic and precise monitoring of underwater netting attachments.

[0176] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0177] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0178] The above-described embodiments merely represent several implementation manners of the present invention. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

Claims

1. An underwater netting attachment density monitoring method, characterized in that, The method includes: Obtaining an underwater netting image to be detected captured by an underwater robot; Preprocessing the underwater netting image to be detected to obtain a preprocessed netting image; Inputting the preprocessed netting image into a pre-trained netting attachment detection model for processing, identifying the attachments on the underwater netting image to be detected, adding bounding boxes to each of the attachments, and outputting a detected netting image; Dividing the detected netting image into several sub-regions; Calculating the local attachment density of each sub-region and the global attachment density of the detected netting image respectively; Generating an attachment cleaning suggestion according to the local attachment density and / or the global attachment density.

2. The underwater net clothing attachment density monitoring method according to claim 1, characterized in that, Before the step of obtaining the underwater netting image to be detected captured by the underwater robot, it further includes: Making a training dataset for the netting attachment detection model; Constructing the framework of an initial detection model and setting initial model parameters; the initial detection model uses the YOLOv9 model as the basic model framework, and the basic model framework includes a backbone network, a neck network, and a detection head. The backbone network uses a neural network with the RepNCSPELAN combined with the CSPNet architecture. Taking the training pictures as the input of the basic model framework, extracting the low-dimensional features and high-dimensional features of the damaged netting from the training pictures through each convolutional layer of the backbone network, and outputting damaged netting feature maps of different scales; performing feature fusion on the damaged netting feature maps of different scales output by the backbone network through the neck network to output a fused feature map; inputting the fused feature map into the detection head, and predicting the coordinates of each attachment by the detection head, and adding bounding boxes to each of the attachments; Training the initial detection model with the training dataset, updating the model parameters, and generating a trained netting attachment detection model.

3. The underwater netting attachment density monitoring method according to claim 2, characterized in that, The step of constructing the framework of the initial detection model further includes: Introducing a deformable attention module into the basic model framework. The step of introducing the deformable attention module into the basic model framework includes: selecting a target convolutional layer in multiple convolutional layers of the backbone network, inserting the deformable attention module after the target convolutional layer, inputting the attachment feature map output by the target convolutional layer into the deformable attention module for processing to obtain a receptive field adjusted feature map; performing feature fusion on the attachment feature map output by the target convolutional layer and the receptive field adjusted feature map, and then inputting the result into the next convolutional layer of the target convolutional layer for processing.

4. The underwater netting attachment density monitoring method according to claim 1, characterized in that, The step of calculating the local attachment density of each sub-region and the global attachment density of the detected netting image respectively includes: Counting the number of bounding boxes in each sub-region; Calculating the area of each sub-region; Dividing the number of bounding boxes in each sub-region by the corresponding area of the sub-region to obtain the local attachment density of each sub-region; Calculating the average value of the local attachment densities of each sub-region to obtain the global attachment density.

5. The underwater net clothing attachment density monitoring method according to claim 4, characterized in that, The step of calculating the area of each sub-region includes: Search for a complete mesh from the center to the edge of a said sub-region, and use the first complete mesh found as the calibration mesh; Calculate the pixel ratio between the said calibration mesh and the said sub-region; Divide the actually preset mesh area by the said pixel ratio to obtain the area of the said sub-region.

6. The underwater net clothing attachment density monitoring method according to claim 1, wherein The step of generating an attachment cleaning suggestion according to the said local attachment density and the said global attachment density includes: Judge whether the said global attachment density exceeds a preset global density threshold; If so, compare the local attachment density of each said sub-region with a preset multi-level local density threshold respectively to obtain the attachment rating of each sub-region; Judge whether the attachment rating of each sub-region reaches a preset rating; If so, generate a first attachment cleaning suggestion and write the positions of the sub-regions that reach the preset rating into the said attachment cleaning suggestion.

7. The underwater netting attachment density monitoring method according to claim 1, characterized in that, The step of generating an attachment cleaning suggestion according to the said local attachment density includes: Compare the local attachment density of each said sub-region with a preset multi-level local density threshold respectively to obtain the attachment rating of each sub-region; Judge whether the attachment rating of each sub-region reaches a preset rating; If so, update the number of locally density-exceeded areas by incrementing the count by 1; Judge whether the updated number of locally density-exceeded areas exceeds a preset number threshold; If so, generate a second attachment cleaning suggestion and write the positions of the sub-regions that reach the preset rating into the said attachment cleaning suggestion.

8. An underwater net attachment density monitoring device, characterized in that, The said device includes: An image acquisition module, configured to acquire an underwater netting image to be detected captured by an underwater robot; An image preprocessing module, configured to preprocess the said underwater netting image to be detected to obtain a preprocessed netting image; An attachment recognition module, configured to input the said preprocessed netting image into a pre-trained netting attachment detection model for processing, recognize the attachments on the said underwater netting image to be detected, add a bounding box to each said attachment, and output a detected netting image; A region division module, configured to divide the said detected netting image into regions to obtain a number of sub-regions; An attachment density calculation module, configured to calculate the local attachment density of each sub-region and the global attachment density of the said detected netting image respectively; A cleaning suggestion module, configured to generate an attachment cleaning suggestion according to the said local attachment density and / or the said global attachment density.

9. A computer-readable storage medium, characterized in that, Stores a computer program, which when executed by a processor causes the processor to execute the steps of the underwater netting attachment density monitoring method according to any one of claims 1 to 7.

10. A computer device, characterized in that, The said device includes a memory and a processor, the memory stores a computer program, which when executed by the processor causes the processor to execute the steps of the underwater netting attachment density monitoring method according to any one of claims 1 to 7.

Images