An underwater target detection method based on multi-scale feature learning

Abstract

This invention discloses a method for underwater target detection based on multi-scale feature learning. First, a dual-backbone network is used to extract features, including depth and texture features. These features are then jointly processed, and a coordinate attention mechanism is used to better focus on position information, enabling more accurate target detection. This method increases the number of prediction heads to four, enabling better detection of small targets and reducing the resource usage of the network model, making the model suitable for embedded devices with limited hardware. Compared to other advanced target detectors, this method further improves underwater target detection accuracy, enabling effective target detection with excellent accuracy and real-time performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to an underwater target detection method based on multi-scale feature learning. Background Art

[0002] The marine environment is a complex system. Because the underwater environment differs significantly from the terrestrial environment, several remote sensing techniques, including acoustics, magnetics, and 3D shallow seismic, have achieved excellent performance in the marine domain. With the development of computer vision, using computer vision to explore the ocean has become a new avenue. As a fundamental task in computer vision, object detection based on optical imaging has become a research hotspot in the marine field. In recent years, many researchers have begun to study underwater object detection based on optical imaging and have achieved significant results. Currently, underwater object detection has many applications in the marine environment, including marine ecosystem research, marine population estimation, marine species conservation, offshore fisheries, underwater unexploded ordnance detection, underwater archaeology, and many other potential applications, providing an effective approach for the development of marine resources. Although object detectors based on deep convolutional neural networks (DCNNs) have performed well on general-purpose datasets in recent years, their direct application to underwater scenarios is limited due to the slow speed and large model size of large networks. Because underwater scenes are more complex than terrestrial scenes, images obtained by underwater camera equipment are small and generally of low quality. For one thing, these images often suffer from high noise, low visibility, blurred edges, low contrast, and color casts.

[0003] Furthermore, while object detection technology has achieved good performance in general datasets, image quality is generally poor due to low visibility and color deviation in complex underwater environments. For underwater target images in complex scenes, such as small targets, the performance of the commonly used YOLOv5 is unsatisfactory. The YOLOv5 model still has significant room for improvement. Furthermore, issues such as small objects result in limited extractable information, making it difficult to achieve satisfactory results. YOLOv5's original feature extraction network has 53 layers, resulting in a large number of parameters and computational overhead. Furthermore, due to limitations in convolutional sampling methods, the handcrafted convolutional network used in CSPDarknet53 is insensitive to objects of varying scales. Its ability to handle geometric variations in features is relatively limited, requiring extensive image training to improve the network's generalization capabilities. In practical applications, YOLOv5 is prone to missed detections and false detections if the network encounters elements not present in the dataset, thus compromising object detection results. In deep learning-based underwater object detection research, most studies focus on using deep networks, resulting in insufficiently rich feature extraction and limited detection accuracy, especially for small objects. Therefore, the application of object detection technology in marine environments requires improved detection performance. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention proposes an underwater target detection method based on multi-scale feature learning.

[0005] The technical solution of the present invention is: an underwater target detection method based on multi-scale feature learning, the specific steps are as follows:

[0006] Step 1: Image preprocessing,

[0007] For underwater target detection, a data-independent enhancement method, namely the Mixup method, is used to construct virtual samples, increase sample robustness, and enhance the image. During the training process, the Mosaic method is used to process the data set. By randomly reading four images and performing operations such as scaling, flipping, and cropping, the images are spliced ​​into a single image as training data to increase data diversity and enrich the background of the detected objects.

[0008] Step 2: Extraction of deep features.

[0009] The structure of the CSPDarknet53 network is improved, and the feature extraction network is replaced with ESNet. A dual-backbone network is used to extract richer features, and the SE module is added to the ES block.

[0010] The channels are replaced by those in ShuffleNetV2 to provide information exchange. When stride = 2, depthwise convolution and pointwise convolution are added to fuse information from different channels. When stride = 1, the Ghost module is introduced in the ES block to improve its performance.

[0011] The ES block in ESNet uses depthwise separable convolution, which divides the convolution into two parts: depthwise (DW) and pointwise (PW) to extract feature maps. The ES block with stride 1 uses Ghost block.

[0012] Step 3: Texture feature extraction,

[0013] To extract texture information of the target, the Gray-Level Co-occurrence Matrix (GLCM) is used to describe the correlation between adjacent pixels in the local area, reflecting the comprehensive information of the image grayscale direction, interval and change amplitude.

[0014] The RGB image is divided into a 3×3 grid. For each grid area, the multi-channel image is converted to a grayscale image. Histogram equalization is used to adjust contrast. The resulting grayscale is then compressed to reduce the amount of subsequent computation. Using statistical parameters calculated by CLCM evolution, we further analyze the texture classification characteristics caused by the recurrence of specific grayscale structures. Contrast, energy, and inverse variance are used to represent the texture characteristics.

[0015] The contrast expression is as follows:

[0016]

[0017] Where i represents the i-th row of the grayscale matrix, j represents the j-th column of the grayscale matrix, P(i,j) represents the probability of grayscale j appearing with grayscale starting at point i, and Con represents the required contrast.

[0018] The expression of angular second moment (energy) is as follows:

[0019]

[0020] Among them, P(i,j) represents the probability of gray level j appearing with gray level i as the starting point, and Asm represents the desired angular second moment.

[0021] The inverse variance expression is as follows:

[0022]

[0023] Where d represents the spatial distance, θ represents the direction, H represents the inverse variance, P(i,j|d,θ) represents the probability of grayscale j appearing with grayscale starting at point i given the spatial distance d and direction θ, and N represents the maximum number of rows (columns) of the grayscale matrix.

[0024] By examining different texture patterns in four directions: horizontal, vertical, and diagonal, that is, the directions of the gray-level co-occurrence matrix are 0°, 45°, 90°, and 135° respectively, based on the gray-level co-occurrence matrix in each direction, three types of rich texture feature parameters, namely contrast, second-order moment, and inverse variance, are extracted, and the mean of all image features is calculated and normalized:

[0025]

[0026] Among them, x is the eigenvalue to be calculated, x max is the largest eigenvalue in the matrix, x min is the smallest eigenvalue in the matrix, x * is the normalized eigenvalue.

[0027] While focusing on the local grid texture features, the classifier compares the texture features between grid regions and finally outputs 1×108 features to represent the image texture features.

[0028] Step 4: Combine the depth features and texture features to process them together.

[0029] The extracted texture features and depth features are combined to allow the network to obtain richer feature information. A coordinate attention mechanism is used after feature extraction. A new coordinate attention block is added to the last layer of the feature extraction network. Coordinate Attention (CA) is used to embed position information into channel attention. The coordinate attention block decomposes channel attention into two one-dimensional feature encoding processes that aggregate features in different directions. The resulting feature maps are then individually encoded to form a pair of direction-aware and position-sensitive feature maps.

[0030] Step 5: Predict the feature map and get the result.

[0031] The prediction module transfers the feature information through upsampling and fuses it into a feature map. After obtaining the feature map, prediction is performed to obtain the final result. The prediction module contains 4 YOLO heads. The YOLO head of the lower layer applies the feature information of the YOLO head of the upper layer, upsamples it, and then splices the feature map to obtain the feature map of this layer for prediction.

[0032] Beneficial effects of the present invention: The present invention discloses an underwater target detection method based on multi-scale feature learning. First, feature extraction is performed through a dual-backbone network, and depth features and texture features are extracted. The depth features and texture features are jointly processed. Through the coordinate attention mechanism, the position information is better focused, so as to more accurately detect the target. The method of the present invention increases the number of prediction heads to 4, which can better detect small targets and reduce the resource usage of the network model, making the model suitable for embedded devices with limited hardware conditions. Compared with other advanced target detectors, the detection accuracy of underwater targets is further improved, and targets can be effectively detected with good detection accuracy and real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of an underwater target detection method based on multi-scale feature learning of the present invention.

[0034] Figure 2 This is an overall structural diagram of an underwater target detection method based on multi-scale feature learning according to the present invention.

[0035] Figure 3 This is a structural diagram of ESNet according to an embodiment of the present invention.

[0036] Figure 4 This is a structural diagram of the ES block according to an embodiment of the present invention.

[0037] Figure 5 This is a structural diagram of a texture feature extraction module according to an embodiment of the present invention.

[0038] Figure 6 This is a coordinate attention structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The method of the present invention is further described below with reference to the accompanying drawings and examples.

[0040] like Figure 1 As shown in FIG, a flow chart of an underwater target detection method based on multi-scale feature learning of the present invention is shown, and the specific steps are as follows:

[0041] like Figure 2 As shown in the figure, the overall structure of the underwater target detection method based on multi-scale feature learning of the present invention is based on the YOLOv5 detection algorithm, and consists of a multi-scale feature extraction network, an attention mechanism based on coordinate attention blocks, and a YOLOv5 target detector. The CBL module consists of convolution operations, batch processing, and activation functions. The multi-scale feature extraction part includes a depth feature extraction part and a texture feature extraction part. Figure 3 A structural diagram of the deep feature extraction part is given in the figure. Since the dataset is generally extracted from the captured video stream, there may be continuous target images. The input of the deep network selects two continuous adjacent images I1 and I2, extracts image features respectively, and then performs feature splicing processing to obtain the output after feature extraction.

[0042] Step 1: Image preprocessing,

[0043] Due to underwater depth, light and other conditions, some images in the dataset have low quality, mainly manifested as blurred targets, low contrast, or target overlap and occlusion problems, which have a great impact on detection accuracy.

[0044] For underwater target detection, a data-independent enhancement method, namely the Mixup method, is used to construct virtual samples, increase sample robustness, and enhance the image. During the training process, the Mosaic method is used to process the data set. By randomly reading four images and performing operations such as scaling, flipping, and cropping, the images are spliced ​​into a single image as training data to increase data diversity and enrich the background of the detected objects.

[0045] Step 2: Deep feature extraction,

[0046] The structure of the CSPDarknet53 network is improved, and the feature extraction network is replaced by ESNet. A dual-backbone network is used to extract richer features. Since the size of the input image is 416×416×3, the sizes of the three feature maps output after a single ESNet extracts features are 384, 192, and 96 respectively. The specific structure of the ES block is as follows: Figure 4 As shown in the figure, the SE module is added to the ES block. SE is an excellent operation for improving feature expression capabilities.

[0047] The design of the SE module is based on MobileNetV3, with two activation functions: Sigmoid and H-Sigmoid. Channels are permuted to those in ShuffleNetV2 to facilitate information exchange, but this can lead to loss of fused features. To address this, when stride = 2, depthwise convolution and pointwise convolution are added to fuse information from different channels. The Ghost module in GhostNet can generate more features with fewer parameters and improve the model's learning ability. Therefore, when stride = 1, the Ghost module is introduced in the ES block to improve its performance.

[0048] Currently, most convolution operations are point-by-point convolutions for dimensionality reduction. The ES block in ESNet uses depthwise separable convolution, splitting the convolution into depthwise (DW) and pointwise (PW) components to extract feature maps. The ES block with stride 1 uses the Ghost block, which combines linear operations with ordinary convolution. This linearly transforms redundant feature maps from the generated ordinary convolution feature map to obtain similar feature maps, creating a high-dimensional convolution effect and reducing model parameters and computational complexity. Finally, the output is the image depth features S1, S2, and S3.

[0049] Step 3: Texture feature extraction,

[0050] Texture feature extraction module such as Figure 5 As shown in the figure, the texture information of the target is extracted, and the Gray-Level Co-occurrence Matrix (GLCM) is used to describe the correlation between adjacent pixels in the local area, reflecting the comprehensive information of the image grayscale direction, interval and change amplitude.

[0051] The RGB image is divided into a 3×3 grid. For each grid area, the multi-channel image is converted to a grayscale image. Histogram equalization is used to adjust the contrast. The grayscale of the adjusted result is then compressed to reduce the amount of subsequent computation. Due to the large feature dimension of the statistical gray-level co-occurrence matrix, it is difficult for the subsequent classifier to distinguish texture information based on it. The statistical parameters calculated by the CLCM evolution are used to further analyze the texture classification characteristics caused by the recurrence of specific grayscale structures. The following parameters are selected to express the texture characteristics:

[0052] Contrast: This measures how the values ​​of the matrix are distributed and how much local variation there is in the image. It reflects the clarity of the image and the depth of the texture grooves. The deeper the texture grooves, the greater the contrast and the clearer the effect. Conversely, a smaller contrast value means shallow grooves and a blurred effect. The expression is as follows:

[0053]

[0054] Where i represents the i-th row of the grayscale matrix, j represents the j-th column of the grayscale matrix, P(i,j) represents the probability of grayscale j appearing with grayscale starting at point i, and Con represents the required contrast.

[0055] Angular Second Moment (Energy): Angular Second Moment (ASM) is an indicator that reflects the uniformity of image grayscale distribution and texture coarseness. A more uniform grayscale distribution in a local area indicates a larger ASM value. The expression is as follows:

[0056]

[0057] Among them, P(i,j) represents the probability of gray level j appearing with gray level i as the starting point, and Asm represents the desired angular second moment.

[0058] Inverse variance: Inverse variance reflects the magnitude of local changes in image texture. If the image texture is relatively uniform in different areas and changes slowly, the inverse variance will be larger, otherwise it will be smaller.

[0059]

[0060] Where d represents the spatial distance, θ represents the direction, and H represents the desired inverse variance. P(i,j|d,θ) represents the probability of grayscale level j occurring with grayscale starting at point i given the spatial distance d and direction θ. N represents the maximum number of rows (columns) in the grayscale matrix.

[0061] By examining different texture patterns in four directions: horizontal, vertical, and diagonal, that is, the directions of the gray-level co-occurrence matrix are 0°, 45°, 90°, and 135° respectively, based on the gray-level co-occurrence matrix in each direction, three types of rich texture feature parameters, namely contrast, second-order moment, and inverse variance, are extracted, and the mean of all image features is calculated and normalized:

[0062]

[0063] Among them, x is the eigenvalue to be calculated, x max is the largest eigenvalue in the matrix, x min is the smallest eigenvalue in the matrix, x * is the normalized eigenvalue.

[0064] While focusing on the local grid texture features, the classifier compares the texture features between grid regions and finally outputs 1×108 features T to represent the image texture features.

[0065] Step 4: Fusion processing of depth features and texture features.

[0066] Image texture information is extracted from images I1 and I2 respectively, and then fused with the extracted depth features to allow the network to obtain richer feature information. Finally, the feature extraction network outputs features C3, C4, and C5. It can be expressed as:

[0067] C3=concat(S11+T1,S21+T2)

[0068] C4=concat(S12,S22)

[0069] C5=concat(S13,S23)

[0070] Among them, T1 represents the texture feature information extracted from the first image, T2 represents the texture feature information extracted from the second image, and Sij(i∈(1,2),j∈(1,2,3)) represents the jth extracted depth feature of the i-th image.

[0071] After the size of the feature map is reduced, the coordinate attention mechanism is used. Although channel attention can significantly improve the performance of the model, it usually ignores the position information, which is very important for generating spatial selective attention. In order to enable the model to extract more useful features, a new coordinate attention block is added to the last layer of the feature extraction network, such as Figure 6As shown in the figure, coordinate attention (CA) is used to embed position information into channel attention, allowing the network to focus on a larger area. Different from the channel attention that converts the input into a single feature vector through two-dimensional global pooling, the coordinate attention block decomposes the channel attention into two one-dimensional feature encoding processes, which aggregate features in different directions. Then, the generated feature maps are encoded separately to form a pair of direction-aware and position-sensitive feature maps, which can be complementarily applied to the input feature map to enhance the representation of the object of interest.

[0072] Step 5: Predict the feature map and get the result.

[0073] The prediction module transfers and fuses feature information into a feature map through upsampling, performs prediction on the obtained feature map, and then obtains the final result. The prediction module contains four YOLO heads. The lower-layer YOLO head applies the feature information of the upper-layer YOLO head, upsamples it, and then splices the feature map to obtain the feature map of this layer for prediction. Compared with the original three-layer prediction of YOLOv5, the improvement in this embodiment is that a YOLO head specifically for detecting small target samples is added to the prediction module. That is, the current YOLO head has four dimensions: 96×96, 48×48, 24×24, and 12×12, which is beneficial for detecting small underwater targets.

[0074] Since there is a big difference between the real box size and the default size of the original algorithm during the data annotation process, and the prior box size has an impact on the network detection speed and accuracy, so choosing a priori box size that is more compatible with the experimental model can not only speed up the convergence speed during training, but also be more accurate for target positioning. Therefore, the anchor box size needs to be redesigned, and after the network is improved, the prior box needs to be changed from the previous default of 9 to 12, and assigned to 4 different scale detection layers. The original YOLOv5 uses the K-Means clustering algorithm, but this algorithm has certain drawbacks. The k-means clustering algorithm may have large differences in clustering results after selecting different initial centroids. In view of the drawbacks of K-means in selecting the initial centroid, K-Means++ is used instead of K-Means to solve this problem. The algorithm process is as follows:

[0075] First, a sample is randomly selected from the data sample as the initial centroid of the cluster. Then the shortest distance D(x) between other samples in the data set and the current cluster center is calculated. Then the probability P(x) of the sample being selected as the next cluster center is calculated. The sample with the highest probability is the next cluster center. The formula for calculating P(x) is as follows:

[0076]

[0077] Where x represents a sample, X represents the set of all samples, and P(x) represents the probability of a sample being selected as the next cluster center.

[0078] The distance and probability calculations are repeated for each cluster center selected until K cluster centers are selected. Because a new layer of scale detection is added, the number of cluster centers K is set to 12. Then, using the K-Means++ clustering algorithm, 12 sets of prior boxes of varying sizes are obtained, also known as anchor box sizes. The smallest feature map has the largest receptive field, so a large anchor point is used, suitable for detecting larger objects. The largest feature map has a smaller receptive field, so the smallest anchor point is used for detecting small objects.

[0079] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for underwater target detection based on multi-scale feature learning, the specific steps are as follows: Step 1: Image preprocessing, For underwater target detection, a data-independent enhancement method, the Mixup method, is used to construct virtual samples, increase sample robustness, and enhance images. During training, the Mosaic method is used to process the dataset. Four images are randomly read, scaled, flipped, and cropped, and then stitched together into a single image as training data, increasing data diversity and enriching the background of the detected objects. Step 2: Deep feature extraction, The structure of the CSPDarknet53 network was improved by replacing the feature extraction network with ESNet. A dual-backbone network was used to extract richer features. The dual-backbone network processes two consecutive adjacent images separately and concatenates the extracted features. The SE module was added to the ES block. Channels are replaced with those in ShuffleNetV2 to provide information exchange. When stride = 2, depthwise convolution and pointwise convolution are added to fuse information from different channels. When stride = 1, the Ghost module is introduced into the ES block to improve its performance. The ES block in ESNet uses depthwise separable convolution, which divides the convolution into depthwise and pointwise parts to extract feature maps. The ES block with stride 1 uses Ghost block. Step 3: Texture feature extraction, To extract texture information of the target, the gray level co-occurrence matrix GLCM is used to describe the correlation between adjacent pixels in the local area, reflecting the comprehensive information of the image gray level direction, interval and change amplitude; The RGB image is divided into a 3×3 grid. For each grid area, the multi-channel image is converted into a grayscale image. Histogram equalization is used to adjust the contrast. The grayscale of the adjusted result is then compressed. The statistical parameters calculated by CLCM evolution are used to further analyze the texture classification characteristics caused by the recurrence of specific grayscale structures. Contrast, energy, and inverse variance are used to express the texture characteristics. The contrast expression is as follows: in, i represents the i-th row of the grayscale matrix, j represents the j-th column of the grayscale matrix, P(i,j) represents the probability of grayscale j appearing with i as the starting point, and Con represents the required contrast; The expression of the angular second moment is as follows: Among them, P(i,j) represents the probability of gray level j appearing with gray level i as the starting point, and Asm represents the desired angular second moment; The inverse variance expression is as follows: Where d represents the spatial distance, θ represents the direction, H represents the inverse variance, P(i,j|d,θ) represents the probability of grayscale j appearing with grayscale starting from point i given the spatial distance d and direction θ, and N represents the dimension of the grayscale matrix. Different texture patterns are examined in four directions: horizontal, vertical, and diagonal. That is, the directions of the gray-level co-occurrence matrix are 0°, 45°, 90°, and 135°. Based on the gray-level co-occurrence matrix in each direction, three types of rich texture feature parameters, namely contrast, second-order moment, and inverse variance, are extracted. The mean of all image features is calculated and normalized: Among them, x is the eigenvalue to be calculated, x max is the largest eigenvalue in the matrix, x min is the smallest eigenvalue in the matrix, x * is the normalized eigenvalue; While focusing on the local grid texture features, the classifier also compares the texture features between grid regions. Finally, it outputs 1×108 features to represent the image texture features. Step 4: Combine the depth features and texture features to process them together. The extracted texture features and depth features are combined, and a coordinate attention mechanism is used after feature extraction. A new coordinate attention block is added to the last layer of the feature extraction network. The coordinate attention is used to embed position information into the channel attention. The coordinate attention block decomposes the channel attention into two one-dimensional feature encoding processes. These processes aggregate features in different directions. Then, the generated feature maps are encoded separately to form a pair of direction-aware and position-sensitive feature maps. Step 5: Predict the feature map and get the result. The prediction module transfers the feature information through upsampling and fuses it into a feature map. After obtaining the feature map, prediction is performed to obtain the final result. The prediction module contains 4 YOLO heads. The YOLO head of the lower layer applies the feature information of the YOLO head of the upper layer, upsamples it, and then splices the feature map to obtain the feature map of this layer for prediction.

Images