A deep learning-based net attachment detection method and system

Abstract

The present application relates to a kind of net clothes adherend detection method and system based on deep learning, belong to and underwater detection field.Method includes: obtaining net clothes video and making detection dataset;Image in detection dataset is preprocessed, and obtain the fusion image dataset after pre-processing;Based on the improved YOLOv5 target detection model and DeepLabv3+ semantic segmentation model constructs adherend detection model;Using fusion image dataset, adherend detection model is trained, and using the adherend detection model trained is carried out net clothes adherend detection, and the segmentation image of adherend segmentation is output;The adhesion degree of adherend segmented in segmentation image is counted, and the segmentation image of adhesion degree greater than adhesion degree threshold is regarded as adherend identification image and is output.Adopting the net clothes adherend detection method and system provided by the present application, non-contact real-time detection of net clothes adherend can be realized, and the speed and accuracy of net clothes adherend detection are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater detection, in particular to a net clothing adherend detection method and system based on deep learning. BACKGROUND

[0002] Due to the problems of water body eutrophication and over-dense breeding area in shallow beaches, the focus of aquaculture industry gradually shifts to the development of deep sea net cage culture. Net cage is an important device for isolating the breeding environment from the outside sea, and the state of the net clothing system is the most critical. Due to the influence of sea waves, natural erosion and other factors, biological adhesion phenomenon occurs in the net clothing, which will affect the exchange and update of water resources inside and outside the net cage to varying degrees, thereby reducing the oxygen content of the cultured aquatic animals, which is not conducive to the growth of fish, and even the deterioration of water quality leads to fish diseases, so it is urgent to detect the net clothing condition in real time.

[0003] At present, the main method for detecting net clothing adherends is completed by manual work, but due to the complex and changeable environment of deep sea net cage culture, it belongs to open sea area, and there will be many uncontrollable factors, so a non-contact detection method is needed to provide a new idea for unmanned inspection of net clothing of net cage. SUMMARY

[0004] The purpose of the present application is to provide a net clothing adherend detection method and system based on deep learning, to realize non-contact real-time detection of net clothing adherends and improve the detection speed and accuracy of net clothing adherends.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A net clothing adherend detection method based on deep learning, comprising:

[0007] Obtaining net clothing video and making a detection data set;

[0008] Preprocessing the images in the detection data set to obtain a preprocessed fusion image data set;

[0009] Constructing an adherend detection model based on the improved YOLOv5 target detection model and DeepLabv3+ semantic segmentation model;

[0010] Training the adherend detection model using the preprocessed fusion image data set to generate a trained adherend detection model;

[0011] Using the trained adherend detection model to detect net clothing adherends and output a segmentation image in which the adherends are segmented;

[0012] The adhesion degree of the adhesion object segmented in the segmented image is counted, and the segmented image with the adhesion degree greater than an adhesion degree threshold is output as an adhesion object recognition image.

[0013] Optionally, the net video is acquired and a detection data set is made, specifically including:

[0014] The target net is photographed by an underwater robot to acquire the net video;

[0015] The net video is frame-extracted and converted into multiple images;

[0016] The multiple images are labeled by using a data labeling software, and the multiple images with labels constitute the detection data set.

[0017] Optionally, the images in the detection data set are preprocessed to obtain a preprocessed fusion image data set, specifically including:

[0018] For each image in the detection data set, the image is copied to obtain a first image and a second image;

[0019] The first image is subjected to white point detection and white point adjustment by using a dynamic threshold white balance algorithm, and then is corrected by using an automatic color balance fast algorithm to obtain a corrected image;

[0020] The second image is divided into a low-frequency part and a high-frequency part by using an adaptive contrast enhancement algorithm, the high-frequency part representing details is enhanced, and a reconstructed image is obtained, and then the reconstructed image is filtered by using a fast bilateral filtering algorithm to obtain a filtered image;

[0021] The corrected image and the filtered image are respectively subjected to wavelet transform to establish a wavelet pyramid decomposition layer of the image;

[0022] Each decomposition layer is subjected to fusion processing from high to low, and different frequency components on each decomposition layer are subjected to fusion processing by using different fusion rules to obtain a fused wavelet pyramid;

[0023] The fused wavelet pyramid is subjected to inverse wavelet transform, and a reconstructed image obtained is taken as a fusion image; and multiple fusion images constitute a preprocessed fusion image data set.

[0024] Optionally, the adhesion object detection model is constructed based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model, specifically including:

[0025] An effective channel attention module is introduced to improve the network structure of the YOLOv5 network, and the five-parameter regression method is used to improve the annotation method of the data target detection box of the YOLOv5 network, resulting in an improved YOLOv5 target detection model.

[0026] The output of the improved YOLOv5 object detection model was used as the input of the DeepLabv3+ semantic segmentation model to construct an attachment detection model.

[0027] Optionally, the step of using the trained attachment detection model to detect netting attachments and outputting a segmented image of the attachments specifically includes:

[0028] Acquire the currently captured image of the target mesh and perform preprocessing to obtain the preprocessed target fusion image;

[0029] The target fusion image is input into the trained improved YOLOv5 target detection model for target detection. If both fish non-attached object labels and attached object labels are detected simultaneously, the non-attached object label bounding box regions are removed from the target fusion image, and then the trained Deeplabv3+ semantic segmentation model is used for segmentation to output a segmented image with the attached object. If only the attached object label is detected, the target fusion image is directly input into the trained Deeplabv3+ semantic segmentation model for segmentation to output a segmented image with the attached object.

[0030] Optionally, the step of statistically analyzing the adhesion degree of the segmented attachments in the segmented image specifically includes:

[0031] The area of ​​the attached objects segmented from the segmented image and the remaining area after removing the non-attached object label box regions from the segmented image are calculated.

[0032] The ratio of the area of ​​the attached material to the remaining area is calculated as the degree of adhesion.

[0033] Optionally, the method for detecting attachments to the mesh fabric further includes:

[0034] A soft filter pruning method was used to prune the trained attachment detection model to obtain a compressed model.

[0035] The compressed model was ported to an embedded platform, and the embedded platform was mounted on an underwater robot to perform real-time detection of netting attachments.

[0036] A deep learning-based system for detecting netting attachments includes:

[0037] The data acquisition module is used to acquire mesh clothing videos and create detection datasets;

[0038] The data preprocessing module is configured to preprocess images in the detection data set to obtain a preprocessed fusion image data set.

[0039] The model establishing module is configured to establish the adherend detection model based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model.

[0040] The model training module is configured to train the adherend detection model by using the preprocessed fusion image data set to generate a trained adherend detection model.

[0041] The adherend detection module is configured to perform netting adherend detection by using the trained adherend detection model and output a segmentation image in which the adherend is segmented.

[0042] The adherend recognition module is configured to count the adhering degree of the adherend segmented in the segmentation image and output a segmentation image in which the adhering degree is greater than an adhering degree threshold as an adherend recognition image.

[0043] Optionally, the data preprocessing module specifically includes:

[0044] The image copying unit is configured to copy each image in the detection data set to obtain a first image and a second image.

[0045] The image balancing and correction unit is configured to perform white point detection and white point adjustment on the first image by using a dynamic threshold white balancing algorithm and then perform correction by using an automatic color equalization fast algorithm to obtain a corrected image.

[0046] The image enhancement and filtering unit is configured to divide the second image into a low-frequency part and a high-frequency part by using an adaptive contrast enhancement algorithm, enhance the high-frequency part representing details, recombine to obtain an enhanced image, and then perform filtering on the enhanced image by using a fast bilateral filtering algorithm to obtain a filtered image.

[0047] The wavelet transformation unit is configured to respectively perform wavelet transformation on the corrected image and the filtered image to establish a wavelet pyramid decomposition layer of the image.

[0048] The fusion processing unit is configured to perform fusion processing on each decomposition layer from high to low, and different frequency components on each decomposition layer are fused by using different fusion rules to obtain a fused wavelet pyramid.

[0049] The wavelet inverse transformation unit is configured to perform wavelet inverse transformation on the fused wavelet pyramid to obtain a reconstructed image as a fusion image, and a plurality of fusion images constitute the preprocessed fusion image data set.

[0050] Optionally, the model establishing module specifically comprises:

[0051] The YOLOv5 network improvement unit is configured to introduce an effective channel attention module to improve the network structure of the YOLOv5 network, and utilize a five-parameter regression method to improve the labeling mode of the data target detection frame of the YOLOv5 network, so as to obtain an improved YOLOv5 target detection model.

[0052] The adhering object detection model construction unit is configured to take the output of the improved YOLOv5 target detection model as the input of a DeepLabv3+ semantic segmentation model, and construct an adhering object detection model.

[0053] According to the embodiments of the present application, the following technical effects are provided.

[0054] The present application provides a deep learning-based netting adhering object detection method and system, which comprises the following steps: acquiring netting videos and preparing a detection dataset; pre-processing the images in the detection dataset to obtain a pre-processed fusion image dataset; constructing an adhering object detection model based on an improved YOLOv5 target detection model and a DeepLabv3+ semantic segmentation model; training the adhering object detection model using the pre-processed fusion image dataset to generate a trained adhering object detection model; performing netting adhering object detection using the trained adhering object detection model and outputting a segmentation image in which the adhering objects are segmented; and counting the adhering degree of the segmented adhering objects in the segmentation image and outputting the segmentation image with an adhering degree greater than an adhering degree threshold as an adhering object recognition image. The deep learning-based netting adhering object detection method and system can realize non-contact real-time detection of netting adhering objects and improve the detection speed and accuracy of netting adhering objects. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 The flowchart of the deep learning-based netting adhering object detection method of the present application is shown in FIG. 1.

[0057] Figure 2 The schematic diagram of the data pre-processing process in the netting adhering object detection method of the present application is shown in FIG. 2.

[0058] Figure 3A structure comparison diagram of the YOLOv5 target detection model before and after improvement in the webbing attachment detection method of the present application;

[0059] Figure 4 A structure diagram of the effective channel attention module in the improved YOLOv5 target detection model of the present application;

[0060] Figure 5 A principle diagram of the five-parameter regression method for adding an angle parameter in the improved YOLOv5 target detection model of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0062] The purpose of the present application is to provide a webbing attachment detection method and system based on deep learning, to realize non-contact real-time detection of webbing attachments and improve the detection speed and accuracy of webbing attachments.

[0063] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0064] Figure 1 A flowchart of the webbing attachment detection method based on deep learning of the present application. Referring to Figure 1 , the webbing attachment detection method based on deep learning of the present application comprises:

[0065] Step 1: Obtain webbing video and make detection data set.

[0066] Before detecting webbing attachments, the present application first needs to collect webbing video and make data set. The step 1 of obtaining webbing video and making detection data set specifically comprises:

[0067] Step 1.1: Shoot the target webbing by underwater robot to obtain webbing video; the underwater camera carried by the underwater robot can be a monocular underwater camera, and can also be other types of cameras;

[0068] Step 1.2: Frame extraction and conversion are performed on the webbing video to convert it into multiple images;

[0069] ​Step 1.3: label multiple images using data labeling software, and the multiple images with labels constitute the detection data set; the software for making the data set can be label me data labeling software, and can also be other types of labeling software.

[0070] In one specific embodiment, the detection data set can be labeled using the roLabelImg labeling software to rotate the target frame, and is divided into two types of labels: fish non-attached objects and attached objects; the segmentation data set can be labeled using the labelme labeling software to points, and is divided into one type of label: attached objects. The detection data set is used as the input of the improved YOLOv5 target detection model for training, and the segmentation data set is used as the input of the Deeplabv3+ semantic segmentation model for training.

[0071] Step 2: pre-process the images in the detection data set to obtain a pre-processed fusion image data set.

[0072] After the netting video is shot and the data set is made, the data set is further pre-processed as follows: color correction is performed through a dynamic threshold white balance algorithm and an automatic color balance algorithm, contrast is enhanced and noise is removed through an adaptive contrast enhancement algorithm and a fast bilateral filtering processing, and a final fusion image is obtained by using a fusion algorithm. The dynamic threshold white balance algorithm solves the white drift phenomenon caused by different color temperatures, restores the white objects, and is divided into two parts: white point detection and white point adjustment. The automatic color balance fast algorithm adjusts the contrast of the image by calculating the relative light and dark relationship between the target point and the surrounding pixel points to correct the final pixel value, and realizes the color constancy and brightness constancy of the human eye. The principle of the adaptive contrast enhancement algorithm is to divide an image into low-frequency and high-frequency two parts, and the target is to enhance the high-frequency part representing the details, multiply the high-frequency part by a certain gain value, and then reorganize to obtain an enhanced image. The fast bilateral filtering algorithm can achieve the effect of keeping the edge and reducing the noise, and like other filtering principles, it uses a weighted average method to represent the intensity of a certain pixel with a weighted average of the brightness values of the surrounding pixels, and the weighted average is based on Gaussian distribution. Image fusion refers to the fusion of different images of the same target scene to make the quality of the image better and contain more useful information. The image fusion method based on wavelet transform can maximize the preservation of the characteristics of the two images.

[0073] Figure 2 For the schematic diagram of the data pre-processing process in the netting attached object detection method of the application, refer to Figure 2 , the step 2 pre-processes the images in the detection data set to obtain a pre-processed fusion image data set, specifically including:

[0074] Step 2.1: For each image in the detection dataset, copy the image to obtain a first image and a second image.

[0075] Step 2.2: Use the dynamic threshold white balance algorithm to detect and adjust white points in the first image, and then use the automatic color equalization fast algorithm to correct it, so as to obtain the corrected image.

[0076] The process of using the dynamic threshold white balance algorithm to detect and adjust white points in the first image is as follows.

[0077] First, take the first image from... Color space conversion to Color space. The first image is divided into blocks according to aspect ratio, and color space is calculated separately for each block. and average and And then according to and Calculate separately and variance and ,in:

[0078]

[0079]

[0080] In color space Represents the brightness signal. Represents the red chromaticity component. Represents the blue chromaticity component; therefore, in the formula... Represents the pixels in the first image The redness value, Represents the pixels in the first image The blueness value; The number of pixels in each block region.

[0081] Set the brightness matrix of the white reference point And a discriminant expression is given to determine which points belong to the white reference points, where This is a symbolic function. The discriminant expression used in this invention is as follows:

[0082]

[0083]

[0084] If it meets the above discrimination expression, it is used as a reference white point, and that pixel is... the luminance value of the pixel point is assigned to the luminance matrix ; if the pixel point does not satisfy the judging expression, the value of the luminance matrix is . .

[0085] For the pixel which has been preliminarily judged as the white reference point, the pixel luminance value (Y) is arranged from high to low according to the Y, R and B components, and the white points of the top 10% of the Y, R and B components are selected as the final white reference points, and the minimum value of the white points is selected as the minimum value of the Y, R and B components. ; otherwise, .

[0086] The Y, R and B components of the first image are multiplied by the Y, R and B components of the second image respectively to obtain Y', R' and B', and the average values of Y', R' and B' are calculated respectively.

[0087] The gain of each channel in the Y'RGB color space is calculated according to the following formula: , wherein Ymax is the maximum value of the Y component in the whole image in the Y'RGB color space.

[0088]

[0089] Then, the final color value of each channel is calculated according to the following formula to adjust the original first image, wherein Y, R and B are the values in the original color space:

[0090]

[0091] The value of the first image is adjusted to Y', R' and B', i.e. the white point detection and white point adjustment of the first image by the dynamic threshold white balance algorithm are completed.

[0092] ​​​​​​​​​​​​​​​​​​​​Automatic Color Enhancement (ACE) considers the spatial relationship between color and brightness in an image, performing adaptive filtering of local characteristics to adjust brightness, color, and contrast in images with local and nonlinear features, while satisfying the gray world theory and white spot assumptions. Due to its high complexity, ACE is not suitable for images with a large number of pixels. The image needs to be executed. The computation time required for sequential nonlinear mapping increases with the image size. Therefore, a fast automatic color balance algorithm has been developed to address the limitations of conventional automatic color balance algorithms. The process of image correction using this fast automatic color balance algorithm is as follows.

[0093] The first image, processed by the dynamic threshold white balance algorithm, is normalized by removing the largest and smallest 0.5% of pixel values, and then linearly stretched to [0,1] to obtain the image. Define a fast algorithm implementation function for automatic color equalization, and calculate the weight parameter matrix based on the convolution template radius (radius) in the function; process the image... After scaling, it is enhanced using a fast automatic color equalization algorithm, and the result is then scaled up to match the image size. Images with the same dimensions are obtained. , Images obtained by conventional automatic color equalization algorithms They are very close. An enhanced implementation function for a fast single-channel automatic color equalization algorithm is defined. Its principle is that since the automatic color equalization algorithm performs differential comparison operations across the entire image domain, the comparison with nearby neighboring pixels constitutes the image... The detailed information, compared with distant pixels, constitutes the image. Global background information, image and images Since the global background information is the same, only the detailed information needs to be updated in the image. Add images to the basis The difference result of the middle neighbor pixels, and subtracted from the image The difference between the nearest neighbor pixels is the final output image. .definition Three-channel enhancement function for images The image is then enhanced separately for each of the three channels to obtain the corrected image.

[0094] Step 2.3: The second image is divided into low-frequency and high-frequency parts using an adaptive contrast enhancement algorithm. The high-frequency part, which represents details, is enhanced and recombined to obtain an enhanced image. Then, the enhanced image is filtered using a fast bilateral filtering algorithm to obtain a filtered image.

[0095] Image is composed of low frequency part and high frequency part, low frequency part can be obtained by low pass filter of image, and high frequency part can be obtained by subtracting low frequency part from original image. The object of the application using adaptive contrast enhancement algorithm is to enhance high frequency part representing details, that is, multiplying high frequency part by a coefficient, and then reorganizing to obtain enhanced image. The core of adaptive contrast enhancement algorithm is the calculation of high frequency part gain coefficient (CG).

[0096] Suppose the pixel point in the second image is represented as , then in the area with as the center and window size of , its local mean and variance can be represented as:

[0097]

[0098]

[0099] In the above formula, represents the pixel value of the point with coordinates , is the local mean of the area with the point as the center and window size of , and the corresponding is the local variance, is the standard deviation of local image, all are positive integers.

[0100] The adaptive contrast enhancement algorithm is used to enhance the second image, and the specific formula is as follows:

[0101]

[0102]

[0103] In the above formula, is the enhanced pixel value, is the value of high frequency part gain coefficient, represents the pixel value of the pixel point . Generally, CG is always greater than 1, so the high frequency part can be enhanced. is the global mean, is a coefficient parameter, generally taking a decimal number greater than 0 and less than 1.

[0104] The output result after processing by the adaptive contrast enhancement algorithm is image , and the enhanced image is the input of the fast bilateral filtering algorithm.

[0105] The weight of bilateral filtering not only considers the Euclidean distance of pixels, but also considers the radiation difference in the pixel range domain, and both weights are considered when calculating the center pixel, and the bilateral filtering operation is performed according to the formula, wherein the enhanced image The input image is denoted as the following formula The output image is the filtered image:

[0106]

[0107]

[0108] wherein , represent the physical coordinates of the pixel points, is the gray value of the neighborhood pixel point is the gray value of the center pixel point is the gray value of the center pixel point and are the gray proximity function and the spatial proximity function, is the distance standard deviation based on the Gaussian function, is the gray standard deviation based on the Gaussian function. is the spatial domain kernel, is the value domain kernel, wherein represents the Euclidean distance, represents the absolute value, is the range of bilateral filtering, is the sum of the product of the spatial weight and the gray weight. Step 2.4: respectively wavelet transform the corrected image and the filtered image, and establish the wavelet pyramid decomposition layer of the image.

[0109] Image fusion can provide more abundant, more accurate and more reliable detailed information which other levels of fusion processing do not have, and is beneficial to further analysis, processing and understanding of the image. Therefore, the corrected image and the filtered image are respectively wavelet transformed in the present application, the wavelet pyramid decomposition of the image is established, and each decomposition layer is output.

[0110] Step 2.5: respectively perform fusion processing on each decomposition layer from high to low, different frequency components on each decomposition layer are fused by using different fusion rules, and a fused wavelet pyramid is obtained.

[0111]

[0112] ​Wherein, the fusion rule of the low-frequency component is to calculate the local variance of all pixel points, and divide the sum of the variances of all points by the sum of all points to obtain the weight of the two images, and the value of each pixel point of the fusion image is the weighted average value of the corresponding pixel points of the two images.

[0113] The fusion rule of the high-frequency component is to perform edge extraction on all pixel points, calculate the local variance of each pixel point of the edge image to obtain a variance image, and the value of each pixel point of the fusion image is the value of the pixel point of the image with a larger value in the corresponding pixel point of the variance image.

[0114] Step 2.6: performing wavelet inverse transformation on the fused wavelet pyramid to obtain a reconstructed image as a fusion image; and a plurality of fusion images constitute a pre-processed fusion image data set.

[0115] Step 3: constructing an adherent detection model based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model.

[0116] The present application introduces an effective channel attention module to improve the network structure of the YOLOv5 network, and uses a five-parameter regression method to improve the labeling method of the data target detection frame of the YOLOv5 network, to obtain an improved YOLOv5 target detection model. Based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model, an adherent detection model is constructed, the output of the improved YOLOv5 target detection model is taken as the input of the DeepLabv3+ semantic segmentation model, the target categories not belonging to the adherent are obtained through the target detection model, and the detection accuracy of the semantic segmentation task is improved as prior information.

[0117] It should be understood that the starting point of the improvement of the YOLOv5 target detection model is to suppress useless information and increase the feature extraction capability of the model, the feature extraction network of YOLOv5 is improved by adding an effective channel attention (ECA) module, the complexity of the model is reduced by increasing local cross-channel interaction and channel sharing parameters, and the efficiency of learning attention is improved.

[0118] In view of the limitation of the target detection frame for fish and other non-adherent target detection, a large amount of redundant regions will be generated, which will reduce the accuracy of the semantic segmentation task after removal, the five-parameter regression method is introduced, the angle parameter is added to rotate the target detection frame, the rotating target detection is realized, the generation of redundant regions is reduced, and better effect is achieved in underwater target dense scenes.

[0119] The front and rear structures of the feature extraction network of YOLOv5 improved by adding an effective channel attention ECA module are as shown in Figure 3 The original YOLOv5 model is composed of Backbone, Neck and Prediction, wherein CBL is a convolution block composed of Conv, Batch Normalization (BN) and Leaky Relu three network layers. The Conv layer is a convolution layer, which processes the input image using multiple different convolution kernels to obtain different response feature maps. The BN layer is a batch normalization layer, which is used before the activation function and after the convolution layer. The mean and variance of all batch data are calculated, and then the pixel value is subtracted from the mean and divided by the variance for normalization, while an offset factor and a scale change factor are added to control the normalized value. The Leaky Relu function is a variant of the Relu function, which solves the problem that the neuron does not learn when the Relu function enters the negative interval. The BottleneckCSP in Backbone is used to extract the deep semantic information of the image, and the BottleneckCSP in Neck is used to fuse feature maps of different scales to enrich semantic information.

[0120] The improvement of the original YOLOv5 model in the application is that an effective channel attention module (Attention) is embedded into the CBL module of Backbone and the residual block of BottleneckCSP to suppress useless information, increase the feature extraction capability of the model, and enhance the expression effect of image features. Figure 3

[0121] Further, according to the structure of the effective channel attention module as shown in Figure 4 Figure 4 In the formula, C represents a feature map channel set that needs to be weighted, H and W represent two-dimensional features of each channel, GAP represents a global average pooling operation, represents an activation function, and k is the size of the convolution kernel.

[0122] The effective channel attention module first uses a global average pooling operation to map the feature map of each channel to a single variable, that is, the feature map is changed from a [h, w, c] matrix to a [1, 1, c] vector; an adaptive one-dimensional convolution kernel size kernel_size (set k to 5) is calculated; kernel_size is used in one-dimensional convolution to obtain the weight of each channel of the feature map; the normalized weight and the original input feature map are multiplied by channel to generate a weighted feature map.

[0123] ​​The effective channel attention module does not reduce the dimensionality of the channels after global average pooling. By considering the local cross-channel interaction information of each channel and its k neighbors, it avoids dimensionality reduction and effectively captures the cross-channel interaction information. Its channel weights... The calculation formula is:

[0124]

[0125] in for For k-domain channels, the ECA module can be effectively implemented using a fast one-dimensional convolution kernel of size k, where the kernel size k represents the coverage of local cross-channel interactions. It is the ReLU activation function. For shared parameters. This represents the feature representation of the channel after global average pooling. express The first 1×1 convolution within the There are several variables. To avoid manually tuning k through cross-validation, k can be determined adaptively, where the interaction coverage and channel dimension are related. Proportional, this linear relationship can be extended to a non-linear relationship, channel dimension. The relationship between the size of the one-dimensional convolution kernel k and the kernel size k is as follows:

[0126]

[0127]

[0128] Where the function Indicates taking away The most recent odd number, Set to 2, Set to 1.

[0129] Furthermore, the principle of using the five-parameter regression method to improve the annotation method of the data target detection box in the YOLOv5 network is as follows: Figure 5 As shown, the five parameters refer to This invention increases the angle parameter. This improves the calculation method for object detection bounding boxes. Among other things, This indicates the coordinates of the center point corresponding to the target; This represents the width and height of the target detection bounding box; These are angular biases (i.e., the angle parameters added in this invention); these five parameters are encapsulated into a vector, which is used to calculate the corresponding loss with the true value.

[0130] This invention increases the angle parameter , to rotate the target detection frame, realize the rotation target detection, and reduce the generation of redundant regions.

[0131] Step 4: training the attachment detection model using the pre-processed fusion image dataset to generate a trained attachment detection model.

[0132] When training the model, the input of the improved YOLOv5 target detection model is the fusion image dataset composed of the fusion images output after step 2 preprocessing; the output is two types of target detection results: fish non-attachment label and attachment label. When the image does not detect fish non-attachment label and attachment label, it means that there is neither fish nor attachment in the region, and no subsequent processing is needed; when the image detects fish non-attachment label, the region is excluded from the whole image (to improve the accuracy of subsequent attachment segmentation) and subsequent judgment is performed; when the image detects the attachment label, the attachment is segmented by the Deeplabv3+ semantic segmentation model.

[0133] The input of the DeepLabv3+ semantic segmentation model is the image containing the attachment label detected by the improved YOLOv5 target detection model, and the output is the segmentation image of the segmented attachment.

[0134] Step 5: using the trained attachment detection model to detect net attachment, output the segmentation image of the segmented attachment.

[0135] When using the trained attachment detection model to detect net attachment, first, the target net image currently taken should be obtained and pre-processed to obtain the pre-processed target fusion image or target fusion image dataset; the target fusion image or its dataset is input into the trained improved YOLOv5 target detection model for target detection, if neither fish non-attachment label nor attachment label is detected, no processing is performed; if fish non-attachment label and attachment label are detected at the same time, the non-attachment label frame region should be excluded from the target fusion image before segmentation by the trained Deeplabv3+ semantic segmentation model, and the segmentation image of the segmented attachment is output; if only the attachment label is detected, the target fusion image is directly input into the trained Deeplabv3+ semantic segmentation model for segmentation, and the segmentation image of the segmented attachment is output.

[0136] Step 6: statistics the attachment degree of the segmented attachment in the segmentation image, and output the segmentation image with the attachment degree greater than the attachment degree threshold as the attachment recognition image.

[0137] The present application detects the attached object segmented in the image by the trained attached object detection model, obtains the attached area and the attached degree by counting the pixel points, sets the attached degree threshold value to be processed, and outputs the attached object greater than the attached degree threshold value as the detection result.

[0138] Specifically, the attached object area of the attached object segmented in the segmented image and the remaining area after excluding the non-attached object label frame area from the segmented image are counted; the ratio of the attached object area to the remaining area is calculated as the attached degree. The attached degree threshold value is set, and it is judged whether the attached degree exceeds the attached degree threshold value. If the attached degree exceeds the attached degree threshold value, the attached object recognition image exceeding the threshold value is output, and the area and the attached degree thereof are labeled; otherwise, the detection is ended.

[0139] In actual application, the attached object detection model can also be compressed and transplanted to an embedded platform, and carried on an underwater robot carrier to realize real-time detection of the net clothes attached object. Since the deep learning target detection model requires large computing and storage resources, when the model is deployed on a device with insufficient computing resources or strict delay requirements, real-time detection cannot be realized, and there is a problem of low detection efficiency, so the model needs to be compressed first; after the model is compressed, the model will become smaller, and in the case that the model performance is not significantly reduced, the training and inference of the model can be accelerated.

[0140] Specifically, the present application adopts a soft filter pruning (SFP) method to prune the trained attached object detection model to obtain a compressed model; the compressed model is transplanted to an embedded platform, and the embedded platform is carried on an underwater robot to perform real-time detection of the net clothes attached object. During detection, a video stream of the target to be detected is obtained, the video stream is taken as an input parameter of the attached object detection model, and real-time detection of the attached object is performed.

[0141] It should be understood that the SFP method sets the filter parameters to be pruned to zero when performing the filter pruning operation each time, the filter is not removed from the network, and the full network form continues the training of the next epoch. The SFP performs pruning operation at the end of each epoch, trains an epoch after pruning is completed, and then continues pruning. The SFP method can make the learning ability of the network more powerful, improve the effect of the model, and the specific steps are as follows:

[0142] (1) The first loop is an epoch, and pruning operation is performed after each training of an epoch;

[0143] (2) The second loop is to traverse each layer of the network, and calculate the L2 norm of each convolution kernel of each layer;

[0144] (3) Pruning probability and the number of convolution kernels of the layer The multiplication represents the number of convolution kernels pruned in a network layer, and the L2 norm of the convolution kernel is the lowest The convolution kernel is pruned by setting the value of the convolution kernel to 0, thereby completing the pruning operation of the convolution kernel of the layer;

[0145] (4) As the epoch is iterated continuously, the final model obtained will contain some convolution kernels with a value of 0, and the final pruning model is obtained by removing these convolution kernels, which is transplanted to an embedded platform as a compression model.

[0146] The net clothes adhering object detection method based on deep learning adds an ECA module to the YOLOv5 target detection model to improve the feature extraction network, enhances the expression effect of image features, introduces a five-parameter regression method, and rotates the target detection frame by increasing the angle parameter , to realize rotating target detection and reduce the generation of redundant areas; the adhering object detection model is constructed based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model, the target categories not belonging to the adhering object are obtained through the target detection model as prior information to improve the detection accuracy of the semantic segmentation task, and the detection accuracy of the adhering object detection model is improved. The adhering object detection model is used to detect and segment the adhering object in the image, the number of pixel points is counted to obtain the adhering area and the adhering degree, and the adhering degree threshold value to be processed is set, the adhering object greater than the threshold value is output as the detection result; finally, the adhering object detection model is compressed and transplanted to an embedded platform, and is mounted on an underwater robot carrier, and can realize real-time detection of net clothes adhering objects.

[0147] Based on the method provided by the application, the application also provides a net clothes adhering object detection system based on deep learning, comprising:

[0148] A data acquisition module is configured to acquire net clothes videos and make detection data sets;

[0149] A data preprocessing module is configured to preprocess images in the detection data set to obtain a preprocessed fusion image data set;

[0150] A model establishing module is configured to construct an adhering object detection model based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model;

[0151] A model training module is configured to train the adhering object detection model using the preprocessed fusion image data set to generate a trained adhering object detection model;

[0152] An attachment detection module is configured to perform net attachment detection by using the trained attachment detection model, and output a segmentation image in which attachments are segmented.

[0153] An attachment recognition module is configured to count attachment degrees of the attachments segmented in the segmentation image, and output a segmentation image in which the attachment degree is greater than an attachment degree threshold as an attachment recognition image.

[0154] The data preprocessing module specifically includes:

[0155] An image copying unit is configured to copy an image to obtain a first image and a second image for each image in the detection dataset.

[0156] An image balancing and correction unit is configured to perform white point detection and white point adjustment on the first image by using a dynamic threshold white balancing algorithm, and then perform correction by using an automatic color equalization fast algorithm to obtain a corrected image.

[0157] An image enhancement and filtering unit is configured to divide the second image into a low-frequency part and a high-frequency part by using an adaptive contrast enhancement algorithm, enhance the high-frequency part representing details, recombine to obtain an enhanced image, and then perform filtering on the enhanced image by using a fast bilateral filtering algorithm to obtain a filtered image.

[0158] A wavelet transformation unit is configured to respectively perform wavelet transformation on the corrected image and the filtered image to establish a wavelet pyramid decomposition layer of the image.

[0159] A fusion processing unit is configured to perform fusion processing from high to low on each decomposition layer, and different frequency components on each decomposition layer are fused by using different fusion rules to obtain a fused wavelet pyramid.

[0160] A wavelet inverse transformation unit is configured to perform wavelet inverse transformation on the fused wavelet pyramid, and the obtained reconstructed image is taken as a fused image. A plurality of fused images constitute a preprocessed fused image dataset.

[0161] The model establishing module specifically includes:

[0162] A YOLOv5 network improvement unit is configured to introduce an effective channel attention module to improve the network structure of the YOLOv5 network, and use a five-parameter regression method to improve the labeling mode of the data target detection frame of the YOLOv5 network to obtain an improved YOLOv5 target detection model.

[0163] The adhering object detection model construction unit constructs the adhering object detection model by taking the output of the improved YOLOv5 target detection model as the input of the DeepLabv3+ semantic segmentation model.

[0164] Compared with the traditional manual detection method, the net-clothing adhering object detection method and system based on deep learning has high detection efficiency and low risk, can realize all-around and multi-angle monitoring of the underwater environment, and provides a new idea for unmanned inspection of net-clothing. Compared with the traditional image preprocessing method, the fusion algorithm based on wavelet transform is used in the present application, so that the image is not only color corrected, but also contrast enhanced and noise removed, improving the image accuracy. Compared with the segmentation network in deep learning, the improved YOLOv5 target detection model is used in the present application to extract non-adhering objects and remove their detection frame position information, which is transmitted into the DeepLabv3+ semantic segmentation model as prior information, improving the accuracy of adhering object recognition and segmentation. Compared with the traditional adhering object detection method, the present application fully considers the problem of large-area adhesion in actual situations, counts the number of pixels to obtain the adhesion area and adhesion degree, sets the adhesion degree threshold value to be processed, and outputs the adhering object greater than the threshold value as the detection result, improving the accuracy of net-clothing adhering object detection. Compared with the traditional detection model, the adhering object detection model is compressed and transplanted to the embedded platform and carried on the underwater robot carrier, realizing real-time detection of net-clothing adhering objects.

[0165] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0166] The principles and implementation modes of the present application are described by using specific examples in this paper. The above description of the embodiments is only used to help understand the method of the present application and its core idea. For those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A deep learning-based net attachment detection method, characterized by, The application relates to a method for detecting and identifying netting attachments, and belongs to the technical field of netting attachment detection and identification. The method comprises the following steps: acquiring netting video and preparing a detection data set; preprocessing images in the detection data set to obtain a preprocessed fusion image data set; constructing an attachment detection model based on an improved YOLOv5 target detection model and a DeepLabv3+ semantic segmentation model; specifically, introducing an effective channel attention module to improve the network structure of the YOLOv5 network, and using a five-parameter regression method to improve the labeling mode of the data target detection frame of the YOLOv5 network to obtain the improved YOLOv5 target detection model; taking the output of the improved YOLOv5 target detection model as the input of the DeepLabv3+ semantic segmentation model to construct the attachment detection model; training the attachment detection model by using the preprocessed fusion image data set to generate a trained attachment detection model; using the trained attachment detection model to detect netting attachments and output a segmentation image in which the attachments are segmented out; 2. The web clothing attachment detection method according to claim 1, characterized by, statistically analyzing the attachment degree of the attachments segmented out in the segmentation image, and outputting the segmentation image with an attachment degree greater than an attachment degree threshold as an attachment identification image. The method for acquiring netting video and preparing a detection data set specifically comprises the following steps: photographing target netting by using an underwater robot to acquire netting video; frame extracting and converting the netting video into multiple images; 3. The web attachment detection method according to claim 1, characterized by, using a data labeling software to label the multiple images, and taking the labeled multiple images as the detection data set. The method for preprocessing images in the detection data set to obtain a preprocessed fusion image data set specifically comprises the following steps: for each image in the detection data set, duplicating the image to obtain a first image and a second image; using a dynamic threshold white balance algorithm to detect and adjust the white point of the first image, and then using an automatic color equalization fast algorithm to correct the first image to obtain a corrected image; using a self-adaptive contrast enhancement algorithm to divide the second image into a low-frequency part and a high-frequency part, enhancing the high-frequency part representing details, recombining to obtain an enhanced image, and then using a fast bilateral filtering algorithm to filter the enhanced image to obtain a filtered image; respectively performing wavelet transform on the corrected image and the filtered image to establish a wavelet pyramid decomposition layer of the image; respectively performing fusion processing on each decomposition layer from high to low, and using different fusion rules to perform fusion processing on different frequency components on each decomposition layer to obtain a fused wavelet pyramid; 4. The web attachment detection method according to claim 1, characterized by, performing wavelet inverse transform on the fused wavelet pyramid to obtain a reconstructed image as a fusion image; and multiple fusion images constitute the preprocessed fusion image data set. The method for using the trained attachment detection model to detect netting attachments and output a segmentation image in which the attachments are segmented out specifically comprises the following steps: acquiring a target netting image currently photographed and preprocessing the target netting image to obtain a preprocessed target fusion image; inputting the target fusion image into the trained improved YOLOv5 target detection model for target detection, if fish non-attachment label and attachment label are simultaneously detected, removing the non-attachment label frame region from the target fusion image, and then performing segmentation on the target fusion image by using the trained Deeplabv3+ semantic segmentation model to output a segmentation image in which the attachment is segmented; if only the attachment label is detected, directly inputting the target fusion image into the trained Deeplabv3+ semantic segmentation model for segmentation to output the segmentation image in which the attachment is segmented.

5. The web attachment detection method according to claim 1, characterized by, The attachment degree of the attachment segmented from the segmentation image is counted, and specifically includes: counting the attachment area of the attachment segmented from the segmentation image and the remaining area after removing the non-attachment label frame region from the segmentation image; calculating the ratio of the attachment area to the remaining area as the attachment degree.

6. The web attachment detection method according to claim 1, characterized by, Further comprising: using a soft filter pruning method to prune the trained attachment detection model to obtain a compressed model; transplanting the compressed model to an embedded platform, and carrying the embedded platform to the underwater robot to detect the netting attachment in real time.

7. A deep learning-based net attachment detection system, characterized by, Comprise: a data acquisition module for acquiring netting video and making detection data set; a data preprocessing module for preprocessing the images in the detection data set to obtain a preprocessed fusion image data set; a model establishing module for establishing an attachment detection model based on the improved YOLOv5 target detection model and the DeepLabv3+ semantic segmentation model; the model establishing module specifically comprises: a YOLOv5 network improvement unit for introducing an effective channel attention module to improve the network structure of the YOLOv5 network, and using a five-parameter regression method to improve the labeling method of the data target detection frame of the YOLOv5 network to obtain an improved YOLOv5 target detection model; an attachment detection model establishing unit for taking the output of the improved YOLOv5 target detection model as the input of the DeepLabv3+ semantic segmentation model to establish an attachment detection model; a model training module for training the attachment detection model using the preprocessed fusion image data set to generate a trained attachment detection model; an attachment detection module for detecting the netting attachment using the trained attachment detection model to output a segmentation image in which the attachment is segmented; an attachment recognition module for counting the attachment degree of the attachment segmented from the segmentation image, and outputting the segmentation image with the attachment degree greater than an attachment degree threshold as an attachment recognition image.

8. The web clothing attachment detection system of claim 7, wherein The data preprocessing module specifically comprises: an image copying unit for copying each image in the detection data set to obtain a first image and a second image; an image balancing and correction unit for detecting and adjusting the white point of the first image by using a dynamic threshold white balance algorithm, and then correcting the first image by using an automatic color equalization fast algorithm to obtain a corrected image; an image enhancement and filtering unit, configured to divide the second image into a low frequency part and a high frequency part by using an adaptive contrast enhancement algorithm, enhance the high frequency part representing details, recombine to obtain an enhanced image, and then filter the enhanced image by using a fast bilateral filtering algorithm to obtain a filtered image; a wavelet transform unit, configured to respectively perform wavelet transform on the corrected image and the filtered image to establish wavelet pyramid decomposition layers of the images; a fusion processing unit, configured to perform fusion processing on each decomposition layer from high to low, and perform fusion processing on different frequency components on each decomposition layer by using different fusion rules to obtain a fused wavelet pyramid; a wavelet inverse transform unit, configured to perform wavelet inverse transform on the fused wavelet pyramid to obtain a reconstructed image as a fused image; and a plurality of the fused images constitute a preprocessed fused image dataset.

Images