Marine target detection method and device based on image enhancement and feature fusion
Through the method of image enhancement and self-attention and convolutional features fusion, the feature fusion problem in complex environments in marine target detection is solved, and the robustness and accuracy of the detection are improved.
Patent Information
- Application Number
- CN202411786664.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-04-04
AI Technical Summary
The existing maritime target detection technology has problems such as unclear feature fusion methods and poor model detection results in complex marine environments.
Improve the quality of offshore image data through image enhancement preprocessing technology, and design a deep feature fusion module based on self-attention and convolution, which is fused into the YOLO-SACFusion model to improve feature extraction and fusion capabilities.
It improves the robustness and accuracy of maritime target detection, and can more accurately identify and locate maritime targets, especially in complex environments.
Smart Images
Figure CN119762964A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target detection, and in particular to a method and device for detecting marine targets based on image enhancement and feature fusion. Background Art
[0002] The main task of maritime target detection is to locate and identify targets such as ships in the ocean. Its development helps to timely detect and respond to maritime incidents, improve the efficiency and accuracy of maritime monitoring, and is of great significance to maritime safety management. It is a key technology of maritime computer vision. However, the task of maritime target detection faces a series of problems, mainly involving two aspects: maritime image data and existing model design.
[0003] In terms of marine image data, the complexity of the marine environment and the particularity of image acquisition conditions lead to low quality of directly collected marine image data. For example, the target blur caused by bad weather conditions such as sea fog and low light conditions such as night, the image noise obtained by near-infrared acquisition, small-scale targets, etc., will have an adverse effect on model training and reasoning. In terms of the design of existing target detection models, the current attention mechanism of deep learning has been widely used in the field of computer vision and is considered to be an effective means to improve the adaptability of deep models to complex environments. Among them, the self-attention mechanism has become a basic module for building modern deep learning models like convolution. However, there are still technical problems in the application of self-attention mechanism in marine target detection. First, the fusion method of self-attention features and convolution features is not clear; second, how to combine the above feature fusion method with advanced models to detect marine targets is also under study. Problems in both data acquisition and model design make the existing marine target detection technology still have room for improvement. Summary of the invention
[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a method and device for marine target detection based on image enhancement and feature fusion. The present invention improves the robustness and accuracy of marine target detection by image enhancement and self-attention and convolution feature fusion.
[0005] To solve the above problems, the technical solution of the present invention is:
[0006] A method for detecting marine targets based on image enhancement and feature fusion comprises the following steps:
[0007] Build a high-quality marine image dataset;
[0008] Design a deep feature fusion module based on self-attention and convolution;
[0009] Construct a marine target detection model based on deep feature fusion module;
[0010] Partitioning high-quality marine imagery datasets;
[0011] Using the divided high-quality marine image dataset to train, verify and test the marine target detection model;
[0012] Use the trained and tested marine target detection model to perform marine target detection.
[0013] Preferably, the step of constructing a high-quality maritime image dataset specifically includes: obtaining a high-quality maritime image dataset by performing frame extraction, deduplication and image enhancement preprocessing on a public database Singapore maritime video dataset, wherein the image enhancement preprocessing process includes:
[0014] In order to reduce the poor image quality caused by sea fog, the dark channel prior algorithm is used to dehaze the image;
[0015] To improve the illumination level of the image, an adaptive histogram equalization method is used to improve the image contrast;
[0016] In order to further improve the image quality, the denoising method based on non-local means filtering is used.
[0017] Preferably, in the step of designing a deep feature fusion module based on self-attention and convolution, the deep feature fusion module SACFusion is composed of a feature extraction submodule based on self-attention and convolution and a feature fusion submodule based on selective channel attention in series, so as to meet the requirements of feature extraction and fusion for marine target detection.
[0018] Preferably, the steps of designing a deep feature fusion module SACFusion based on self-attention and convolution specifically include:
[0019] Constructing a feature extraction submodule of the deep feature fusion module, wherein the feature extraction submodule is composed of a self-attention module and a convolution module in parallel;
[0020] A feature fusion submodule of the deep feature fusion module is constructed, wherein the feature fusion submodule is composed of an additive fusion module, a global average pooling module, a three-layer fully connected network with batch normalization, and a channel-weighted fusion module in series.
[0021] Preferably, the step of constructing a marine target detection model based on a deep feature fusion module specifically includes:
[0022] A marine target detection model YOLO-SACFusion based on a deep feature fusion module is constructed. The backbone network of the YOLO-SACFusion model is improved based on the cross-stage local Darknet. In order to enhance the feature expression capability of marine targets, a deep feature fusion module SACFusion module is introduced. The SACFusion module is inserted into the deepest feature extraction stage of the cross-stage local Darknet.
[0023] The neck network of the YOLO-SACFusion model is improved based on the path aggregation network PAN_FPN with feature pyramid, and the SACFusion modules are inserted after the first C2f layer of PAN-FPN and before the three decoupled prediction networks respectively;
[0024] The head network of the YOLO-SACFusion model adopts three decoupled prediction networks, each of which consists of two parallel detection branches, which are used to predict the target bounding box position, category and confidence.
[0025] Preferably, the step of dividing the high-quality marine image data set specifically includes: dividing the high-quality marine image data set into a training set, a validation set and a test set, with a division ratio of 7:2:1.
[0026] Preferably, the step of using the divided high-quality marine image data set to train, verify and test the marine target detection model specifically includes:
[0027] Initialize the parameters of the marine target detection model, input the training set into the model, and the model outputs the position, category label and confidence of the detected marine target.
[0028] Choose loss function and optimizer;
[0029] Update, validate the model and save the optimal model;
[0030] The marine target detection model is tested using the test set.
[0031] Furthermore, the present invention also provides a marine target detection device based on image enhancement and feature fusion, the device comprising a processor and a memory for storing executable instructions of the processor, the processor being configured to execute the marine target detection method based on image enhancement and feature fusion as described above by executing the executable instructions.
[0032] Compared with the prior art, the present invention introduces an image enhancement preprocessing process in terms of marine image data. Image enhancement steps such as dark channel prior algorithm, adaptive histogram equalization, and non-local mean filtering can improve the clarity and contrast of the image, thereby providing high-quality data for model training and reasoning; in terms of marine target detection models, a self-attention and convolution feature selective fusion module based on channel attention is designed, and it is integrated into the backbone network and neck network of the YOLOv8 model to obtain a marine target detection model, so that the features of the model at key stages are enhanced, so that marine targets can be more accurately identified and located in complex environments. The present invention improves the robustness and accuracy of marine target detection through image enhancement and self-attention and convolution feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0034] Figure 1 A flowchart of a method for detecting marine targets based on image enhancement and feature fusion provided by an embodiment of the present invention;
[0035] Figure 2 Some sample images of the marine image dataset;
[0036] Figure 3 Some sample images of high-quality marine image dataset;
[0037] Figure 4 It is a structural diagram of the deep feature fusion module SACFusion based on self-attention and convolution of the present invention;
[0038] Figure 5 This is a structural diagram of the marine target detection model YOLO-SACFusion based on the deep feature fusion module of the present invention. DETAILED DESCRIPTION
[0039] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0040] Specifically, the present invention provides a method for detecting marine targets based on image enhancement and feature fusion, such as Figure 1 As shown, the method comprises the following steps:
[0041] S1: Build a high-quality marine image dataset;
[0042] Specifically, the dataset is obtained by preprocessing the Singapore maritime video dataset in the public database by extracting frames, removing duplicates and enhancing images. Figure 2 As shown, some samples in the marine image dataset are shown.
[0043] Each image in the marine image dataset is enhanced to ensure that low-quality images are enhanced. The image enhancement preprocessing process includes the following steps:
[0044] S11: To reduce the poor image quality caused by sea fog, the dark channel prior algorithm is used to defog the image;
[0045] The parameters used are as follows: guided filter radius is 81, regularization parameter is 0.001, minimum transmittance ratio is 0.95, and maximum transmittance ratio is 0.80.
[0046] S12: In order to improve the illumination level of the image, an adaptive histogram equalization method is used to improve the image contrast;
[0047] The parameters adopted are as follows: contrast limit is 2, grid size is 8×8.
[0048] S13: To further improve the image quality, a denoising process is performed based on a non-local mean filtering method.
[0049] The parameters used are as follows: filter window size is 2, image detail retention size is 2, template window size is 7, and search window size is 21.
[0050] like Figure 3 As shown, some samples of the high-quality marine image dataset are shown.
[0051] S2: Design a deep feature fusion module based on self-attention and convolution;
[0052] Specifically, a deep feature fusion module SACFusion based on self-attention and convolution is designed. The SACFusion module is composed of a feature extraction submodule based on self-attention and convolution and a feature fusion submodule based on selective channel attention in series, which is used to meet the requirements of feature extraction and fusion for marine target detection.
[0053] like Figure 4As shown in the figure, Projection represents the projection module shared by the self-attention module and the convolution module, ConvAggregation represents the convolution aggregation module, Self-Attention Aggregation represents the self-attention module aggregation module, Fully ConnectedNetwork represents the fully connected layer, Shift&Sum represents the translation and addition operation, Concatenation represents the splicing operation, Fusion represents the fusion module, Global Average Pooling represents the global average pooling, Conv 1×1 represents the 1×1 convolution, Batch Normalization represents the batch normalization, ReLU represents the ReLU activation function, Sigmoid represents the Sigmoid activation function, F represents the input feature, and F a represents the output features after the self-attention aggregation module, F c Represents the output features of the convolutional aggregation module, F f Indicates F a and F c Add the features after initial fusion, Represents the output features after the SACFusion module.
[0054] The method for designing a deep feature fusion module SACFusion based on self-attention and convolution includes:
[0055] S21: construct a feature extraction submodule of the SACFusion module, wherein the feature extraction submodule is composed of a self-attention module and a convolution module in parallel;
[0056] Specifically, the self-attention operation is decomposed into projection and self-attention convergence, and the convolution operation is decomposed into projection and convolution convergence. The above two projections use a shared projection module. The shared projection module projects the input feature F using three 1×1 convolutions to generate intermediate features. Then, the above intermediate features are reorganized to obtain N parts, forming a 3×N group of feature sets of size H×W×(C / N), which are shared by the self-attention path and the convolution path.
[0057] The self-attention convergence module refers to: performing N self-attention operations on the intermediate features generated by the shared projection module on the self-attention path. Specifically, first, each set of features is used as the query matrix Q, the key matrix K, and the value matrix V for self-attention operation SA(.), and finally, the connection Concat(.) is performed to obtain the self-attention output. The operation of the self-attention convergence module can be expressed as: F a =Concat((SA(Proj(F)))), where Proj(.) represents the operation of the shared projection module and SA(.) refers to Where d is the dimension of the query matrix.
[0058] The convolutional convergence module refers to: on the convolution path, N convolution operations with a kernel size of k×k are performed on the intermediate features generated by the shared projection module. Specifically, first, k are generated through a lightweight fully connected layer FC(.) 2 ×N groups of feature sets of size H×W×(C / N), then perform shift and sum operations on the generated feature sets, and finally, perform concat(.) to complete the convolution. The operation of the convolutional aggregation module can be expressed as: c =Concat(Shift&Sum(FC(Proj(F)))), where the Shift&Sum(.) operation is implemented by a depthwise convolution with a learnable convolution kernel.
[0059] S22: Construct a feature fusion submodule of the SACFusion module, wherein the feature fusion submodule is composed of a fusion module, a global average pooling module, a three-layer fully connected network with batch normalization, and a channel-weighted fusion module in series.
[0060] Specifically, the output F of the self-attention module a And the output F of the convolution module c , the fusion module performs initial fusion by element-by-element summation: F f =F a +F c Then, the global average pooling module GAP(.) calculates the channel global information s∈R of the initial fusion result F C :s=GAP(F f ). Then, through a three-layer fully connected network with batch normalization, the input dimension of the network is C. To improve the computational efficiency, the number of hidden layer neurons is reduced to Cr, where r is the reduction rate. The hidden layer has a batch normalization layer BN(.) function, the activation function of the hidden layer is ReLU(.) function, the output dimension is restored to C, and the output layer activation function is Sigmoid(.) function. The calculation of the three-layer fully connected network can be expressed as:
[0061] w=Sigmoid(W2ReLU(BN(W1s))), where W1∈R C×Cr , W2∈R Cr×C is the weight of the three-layer fully connected network. Finally, the channel weighted fusion module generates the weight w based on the three-layer fully connected network to the output F of the self-attention module. a And the output F of the convolution module c To perform selective fusion: in, Indicates channel-wise multiplication. It is the channel-weighted fusion result, that is, the output of the SACFusion module.
[0062] S3: Build a marine target detection model based on deep feature fusion module;
[0063] Specifically, a marine target detection model YOLO-SACFusion based on the SACFusion module is constructed. The model uses the cross-stage local Darknet improved based on the SACFusion module in step S2 as the backbone network, the path aggregation network with feature pyramid improved based on the SACFusion module in step S2 as the neck network, and uses three decoupled prediction networks as the head network.
[0064] like Figure 5 As shown, Input represents input data, Backbone represents the backbone network of the detection model, Neck represents the neck network of the detection model, Head represents the head network of the detection model, Conv represents the convolution module, C2f represents the C2f module, SACFusion represents the SACFusion module designed by the present invention, Concat represents the concatenation operation, Upsample represents the upsampling operation, DecoupledDetector represents the decoupled detection network, Output represents the output data, and the specific method of constructing the marine target detection model YOLO-SACFusion based on the SACFusion module includes:
[0065] S31: The backbone network of the YOLO-SACFusion model is improved based on the cross-stage local Darknet (CSPDarknet);
[0066] Based on the classic Darknet architecture, CSPDarknet effectively decomposes the feature flow by partially stacking across stages, reducing the model's computational complexity while maintaining good feature extraction capabilities. In order to further enhance the feature expression capabilities of maritime targets, the SACFusion module is introduced. This module is inserted into the deep feature extraction stage of CSPDarknet, taking the output of the spatial feature pyramid module as input, and its output provides input for the feature fusion of the neck network, ensuring that the features extracted in the final stage have stronger discriminability.
[0067] S32: The neck network of the YOLO-SACFusion model is improved based on the path aggregation network PAN-FPN with feature pyramid;
[0068] In order to improve the multi-level and multi-scale feature fusion capabilities, the YOLO-SACFusion model uses PAN-FPN in the neck network. The feature pyramid network (FPN) helps to fuse multi-level features, while the path aggregation network (PAN) further optimizes the information flow, ensuring that high-level semantic information can be passed down and effectively combined with low-level detail features. In order to further enhance the feature aggregation capability of the neck network, self-attention and convolution feature fusion modules are added to the key layers to enhance the features. Specifically, first, the SACFusion module is inserted after the first C2f layer of PAN-FPN, so that the output of the SACFusion module is the input of the second upsampling module. Then, the SACFusion modules are inserted before the three decoupled prediction networks respectively, so that the output of the SACFusion module is the input of the detection head. This operation can make the fusion of features of different scales more accurate and enhance the robustness of the model to small targets and complex backgrounds at sea.
[0069] S33: The head network of the YOLO-SACFusion model adopts three decoupled prediction networks, each of which is composed of two parallel detection branches, which are used to predict the target bounding box position, category and confidence.
[0070] This decoupled design helps each task to be optimized independently, which can improve detection accuracy. In the YOLO-SACFusion model, each decoupled detection network takes the output of a SACFusion module as input.
[0071] S4: partitioning high-quality marine image datasets;
[0072] The high-quality marine image dataset constructed in step S1 is preliminarily divided into a training set, a validation set, and a test set, with a specific division ratio of approximately 7:2:1. During the division, in order to ensure the consistency of data distribution and improve the generalization ability of the model, each set is required to cover different situations as much as possible, including: the location of video shooting, weather and lighting conditions, and imaging conditions.
[0073] S5: using the divided high-quality marine image dataset to train, verify and test the marine object detection model;
[0074] Specifically, the method of training, verifying and testing the YOLO-SACFusion model includes:
[0075] S51: Initialize the YOLO-SACFusion model;
[0076] Initialize the parameters of the YOLO-SACFusion model, input the training set into the model, and the model output results are the location, category label and confidence of the detected marine target.
[0077] S52: Select loss function and optimizer;
[0078] The category classification loss adopts the multi-category cross entropy loss function, the bounding box regression loss adopts the distribution focus loss function and the complete intersection-over-union loss function, and the optimizer adopts the adaptive moment estimation optimizer Adam.
[0079] S53: update, verify the model and save the optimal model;
[0080] In each round of training, the model parameters are first updated through the optimizer, and then the current model performance is evaluated with the validation set, and the YOLO-SACFusion model with the best performance is saved. The cosine annealing learning rate scheduler is used to dynamically adjust the learning rate during the training process to ensure that the model converges stably in the later stages of training.
[0081] S54: Test the YOLO-SACFusion model using the test set.
[0082] S6: Use the trained and tested marine target detection model to perform marine target detection.
[0083] Furthermore, the present invention also provides a marine target detection device based on image enhancement and feature fusion, the device comprising a processor and a memory for storing executable instructions of the processor, the processor being configured to execute the marine target detection method based on image enhancement and feature fusion as described above by executing the executable instructions.
[0084] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.
Claims
1. A method for detecting marine targets based on image enhancement and feature fusion, characterized in that: The method comprises the following steps: Build a high-quality marine image dataset; Design a deep feature fusion module based on self-attention and convolution; Construct a marine target detection model based on deep feature fusion module; Partitioning high-quality marine imagery datasets; Using the divided high-quality marine image dataset to train, verify and test the marine target detection model; Use the trained and tested marine target detection model to perform marine target detection.
2. The method for detecting marine targets based on image enhancement and feature fusion according to claim 1, characterized in that: The step of constructing a high-quality maritime image dataset specifically includes: obtaining a high-quality maritime image dataset by performing frame extraction, deduplication and image enhancement preprocessing on a public database Singapore maritime video dataset, and the image enhancement preprocessing process includes: In order to reduce the poor image quality caused by sea fog, the dark channel prior algorithm is used to dehaze the image; To improve the illumination level of the image, an adaptive histogram equalization method is used to improve the image contrast; In order to further improve the image quality, the denoising method based on non-local means filtering is used.
3. The method for detecting marine targets based on image enhancement and feature fusion according to claim 1, characterized in that: In the step of designing a deep feature fusion module based on self-attention and convolution, the deep feature fusion module SACFusion is composed of a feature extraction submodule based on self-attention and convolution and a feature fusion submodule based on selective channel attention in series, so as to meet the requirements of feature extraction and fusion for marine target detection.
4. The method for detecting marine targets based on image enhancement and feature fusion according to claim 3, characterized in that: The steps of designing the deep feature fusion module SACFusion based on self-attention and convolution specifically include: Constructing a feature extraction submodule of the deep feature fusion module, wherein the feature extraction submodule is composed of a self-attention module and a convolution module in parallel; A feature fusion submodule of the deep feature fusion module is constructed, wherein the feature fusion submodule is composed of an additive fusion module, a global average pooling module, a three-layer fully connected network with batch normalization, and a channel-weighted fusion module in series.
5. The method for detecting marine targets based on image enhancement and feature fusion according to claim 1, characterized in that: The step of constructing a marine target detection model based on a deep feature fusion module specifically includes: A marine target detection model YOLO-SACFusion based on a deep feature fusion module is constructed. The backbone network of the YOLO-SACFusion model is improved based on the cross-stage local Darknet. In order to enhance the feature expression capability of marine targets, a deep feature fusion module SACFusion module is introduced. The SACFusion module is inserted into the deepest feature extraction stage of the cross-stage local Darknet. The neck network of the YOLO-SACFusion model is improved based on the path aggregation network PAN_FPN with feature pyramid, and the SACFusion modules are inserted after the first C2f layer of PAN-FPN and before the three decoupled prediction networks respectively; The head network of the YOLO-SACFusion model adopts three decoupled prediction networks, each of which consists of two parallel detection branches, which are used to predict the target bounding box position, category and confidence.
6. The method for detecting marine targets based on image enhancement and feature fusion according to claim 1, characterized in that: The step of dividing the high-quality marine image data set specifically includes: dividing the high-quality marine image data set into a training set, a validation set and a test set, with a division ratio of 7:2:
1.
7. The method for detecting marine targets based on image enhancement and feature fusion according to claim 1, characterized in that: The steps of using the divided high-quality marine image data set to train, verify and test the marine target detection model specifically include: Initialize the parameters of the marine target detection model, train the model with the training set, and output the model results as the position, category label and confidence of the detected marine target. Choose loss function and optimizer; Update, validate the model and save the optimal model; The marine target detection model is tested using the test set.
8. A marine target detection device based on image enhancement and feature fusion, characterized in that: The device includes a processor and a memory for storing executable instructions of the processor, and the processor is configured to perform the marine target detection method based on image enhancement and feature fusion as described in any one of claims 1 to 7 by executing the executable instructions.