Ship image target detection method and system in complex environment

Abstract

A method and system for detecting ship images in complex environments involves preprocessing raw ship images through acquisition in an offline phase. A deep neural network for ship target detection is constructed based on the FDDEA algorithm and the AASST strategy, and the network is trained using the preprocessed images as training samples. In the online phase, test images or actual ship images are input into the trained deep neural network to obtain predicted frames and classifications of ship targets. This method significantly improves the accuracy of ship target detection in complex environments through a ship image feature denoising and expansion coding algorithm and an adaptive ship target positive and negative sample allocation strategy.

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, in particular to a method and system for detecting ship image targets in complex environments. Background Art

[0002] Ship image target detection plays a vital role in tasks such as intelligence acquisition and sea surface monitoring. While general target detection methods based on deep neural networks have achieved significant breakthroughs in ship image target detection, these general target detection algorithms often fall short of practical accuracy requirements due to issues such as poor image quality, large variations in ship shapes, small target scales, and severe environmental occlusion and noise interference in ship images captured in complex environments. Therefore, developing a ship image target detection method and system for these complex environments remains an important research direction. Summary of the Invention

[0003] Aiming at the problems of small ship target scale, noise interference, environmental redundancy and large shape differences in existing ship target detection in complex environments, the present invention proposes a ship image target detection method and system in complex environments. Based on the Feature Denoising Dilated Encoding Algorithm for Ship Image (FDDEA) and the Adaptive Positive and Negative Sample Allocation Strategy for Ship Object (AASST), the accuracy of ship target detection in complex environments is effectively improved.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a ship image target detection method in a complex environment. In the offline stage, the original ship image is preprocessed by acquisition; a deep neural network for ship target detection is constructed based on the FDDEA algorithm and the AASST strategy, and the preprocessed image is used as a training sample to train the ship target detection network; in the online stage, a test image or an actual ship image is input into the trained ship target detection deep neural network to obtain a prediction frame and classification of the ship target.

[0006] The ship target detection deep neural network includes: a backbone network based on the Resnet framework, a feature pyramid module (FPN), a feature enhancement and denoising module based on FDDEA, and a detection head based on the AASST strategy, wherein: the backbone network extracts feature maps from the preprocessed image; the FPN performs a top-down secondary fusion based on feature maps of different levels to obtain a fused feature map; the feature enhancement and denoising module performs feature enhancement extraction and feature denoising on the fused feature map; the detection head performs detection on each layer of feature map and obtains the final prediction box.

[0007] The feature enhancement and denoising module based on FDDEA includes: a feature expansion encoder and a feature denoiser, wherein: the feature expansion encoder performs expansion coding according to the output information of FPN using a residual network and a dilated convolution to obtain an expanded coding feature map with rich semantic information; the feature denoiser performs feature denoising processing according to the expanded coding feature map using a soft threshold function based on a threshold attention mechanism to obtain a denoised expanded coding feature map; finally, the denoised expanded coding feature map is input into a detection head for target detection.

[0008] The AASST-based strategy refers to: after filtering out positive samples using an IoU threshold containing a shape adaptive attenuation factor, the positive samples are weighted dynamically based on the distance between the sample center point and the true box center point, and then a loss function consisting of classification loss and regression loss is calculated.

[0009] The IoU threshold T iou =α sa *(m g +v g ), where: adaptive attenuation factor Ar is the average aspect ratio of the real box, ω is the attenuation weight, ε max is the extreme value of aspect ratio, m g is the mean IoU between the candidate box and the real box, v g is the mean IoU between the candidate box and the real box, I i,j is the IoU value between the i-th candidate box and the j-th real box.

[0010] Preferably, the IoU threshold is fixedly set to 8 when the aspect ratio is greater than the extreme aspect ratio value.

[0011] The present invention relates to a system for implementing the above-mentioned method, comprising: an image processing unit, a ship image prediction unit and an output unit, wherein: the ship image processing unit performs cropping and scaling processing on the original ship image; the ship image prediction unit generates a prediction frame according to the preprocessed image based on a ship target detection deep neural network; and the output unit marks the position information and classification information of the ship target in the original ship image according to the prediction frame.

[0012] Technical Effects

[0013] The feature enhancement and denoising module based on FDDEA in the present invention solves the problems of poor image quality, large ship shape differences, small target scale, severe environmental occlusion and noise interference in ship target detection in complex environments through cascaded dilated convolution, and the detection head based on AASST redefines the IoU threshold for positive and negative sample selection by considering the shape information of the target, and introduces a shape adaptive attenuation factor. By enriching the feature semantics of small targets, the detection effect of ship targets with large shape differences and small-scale targets is significantly improved; the noise in the feature map is removed by a soft threshold function based on the threshold attention mechanism, which solves the problems of environmental occlusion and signal noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the present invention;

[0015] Figure 2 Schematic diagram of image feature denoising expansion coding algorithm;

[0016] Figure 3 This is the architecture diagram of the threshold attention mechanism of the present invention. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, this embodiment relates to a method for detecting ship targets in a complex environment, comprising the following steps:

[0018] Step 1: Preprocess the original ship images and enhance the dataset to improve training results and algorithm model robustness, including:

[0019] Step 1.1: Randomly scale the ship image while keeping the aspect ratio unchanged.

[0020] Step 1.2: Perform random horizontal flips on the scaled image.

[0021] Step 1.3: For the flipped image, randomly transform the hue, saturation, and brightness (i.e., HSV transform). All relevant parameters are obtained by taking a random natural number between [0,100].

[0022] Step 1.4: Perform padding on the HSV transformed image and set the padding size_divider to 32. This completes the image preprocessing.

[0023] Step 2: If Figure 2 As shown in the figure, after constructing the deep neural network for ship target detection based on the FDDEA algorithm and the AASST strategy, the images preprocessed in step 1 are used as training samples to train the ship target detection network, specifically including:

[0024] Step 2.1: Input the preprocessed image into the backbone network ResNet and extract features at different levels through residual convolution.

[0025] Step 2.2: The FPN first uses BottomUp to obtain feature maps of different sizes output by different backbone stages. Then, using TopDown, it starts with the topmost feature map, upsamples it, and adds it to the feature map of the next level. Through BottomUp and TopDown, the feature maps of each layer contain rich semantic information and can detect objects of different scales. The FPN network provides two capabilities: feature fusion and divide-and-conquer. Feature fusion is manifested by TopDown, which ensures that even the lowest feature map contains the semantic information extracted by the highest layer. Divide-and-conquer is manifested by the fact that in the final detection process, objects of different scales are detected on different feature maps.

[0026] In order to give full play to its divide-and-conquer capability, the present invention introduces a feature enhancement and denoising module based on the FDDEA algorithm after FPN.

[0027] Step 2.3: Use the feature enhancement and denoising module based on the FDDEA algorithm to enhance and denoise the feature map. The specific steps include:

[0028] Step 2.3.1: First, based on the feature map {P2, P3, P4, P5} output by FPN, upsample P2 to obtain P1, upsample P5 to obtain P6, and use the final {P1, P2P3, P4, P5, P6} as the input of the input feature expansion encoder.

[0029] Step 2.3.2: The feature expansion encoder is based on the residual network and dilated convolution. For each layer of the input feature map, it first performs a 1x1 convolution plus a ReLU activation function, then a 3x3 dilated convolution plus a ReLU activation function, and then another 1x1 convolution plus a ReLU activation function. Finally, the output of the convolution module is added to the original input to obtain the output of the residual expansion module, specifically: Where: x is the input of the residual expansion module, ConvRelu (1×1) Indicates a regular convolution with a convolution kernel of 1×1 and a Relu activation function. r represents the expansion rate of the dilated convolution. DilatedConvRelu (3×3) (x, r) represents a dilated convolution with a kernel of 3×3 and a dilation rate of r on the input x, followed by a ReLU activation function. O represents the output of the residual dilation module.

[0030] Through the residual expansion module, it is ensured that the feature semantic information obtained after each expansion encoding is not lower than the input feature semantic information, and the multi-scale information of the feature map is retained, further improving the algorithm's detection effect on small targets.

[0031] Step 2.3.3: If Figure 3 As shown in the figure, the denoising module obtains an adaptive threshold t based on the output of the residual expansion module through the threshold attention mechanism, and then uses the soft threshold function to perform feature denoising based on the threshold t, specifically:

[0032] Where: x is the input feature, y is the output feature, and t is the threshold; the soft threshold function sets the values ​​within a certain threshold to 0 and makes the values ​​outside the threshold move towards 0;

[0033] Step 2.3.4: Repeat steps 2.2.2 and 2.2.3 four times, and use gradually increasing dilation rates, generally set to (2, 3, 4, 5), to obtain the denoised dilated coding feature map.

[0034] Step 2.4: Use the AASST-based detection head to define positive and negative samples and perform loss regression, including:

[0035] Step 2.4.1: Use the denoised dilated coded feature map as the input of the detection head. First, define the positive and negative samples, including:

[0036] Step 2.4.1.1: Select the candidate box based on the center distance between the anchor and the true box, specifically: where w i Indicates the width of the i-th gt box, h i Indicates the length of the i-th gt box, x i ,y i Represents the coordinates of the center point of the positive sample, x j ,y j Indicates the center coordinates of the candidate box, d ij The closer the center point of the anchor is to the center point of the ground truth box, the higher the detection quality. This is because the closer the center point of the ground truth box is, the closer the features that the candidate box can describe are to the target. Therefore, the top K candidate boxes are first selected based on the center distance between the anchor and the ground truth box.

[0037] Step 2.4.1.2: Use the sum of the mean and standard deviation of the IoU between the candidate box and the real box as the initial threshold for defining the IoU of positive and negative samples, specifically: T iou =m g +v g ,in:

[0038] Where: m g represents the mean IoU between the candidate box and the real box, v g Represents the mean IoU between the candidate box and the real box, I i,j Represents the IoU value between the i-th candidate box and the j-th real box.

[0039] Step 2.4.1.3: Calculate the shape adaptive attenuation factor. First, count the target aspect ratio information. For the real box corresponding to the candidate box, calculate the average aspect ratio Ar of all real boxes. Based on the average Ar, the adaptive attenuation factor Where: Ar is the average aspect ratio of the real frame, ω represents the attenuation weight, which is generally set to 16 according to the empirical value, and ε max Indicates the extreme value of the aspect ratio. When the aspect ratio is greater than this value, the threshold will no longer decrease. It is generally set to 8 according to experience.

[0040] Step 2.4.1.4: Substitute the calculated shape adaptive attenuation factor α sa Substitute into the formula in step 2.3.1.2 to obtain the final positive and negative sample selection IoU threshold T iou =α sa *(m g +v g ).

[0041] Step 2.4.1.5: Based on the calculated IoU threshold, the candidate boxes with an IoU greater than the threshold are defined as positive samples, and the other candidate boxes are defined as negative samples.

[0042] Step 2.4.2: Assign dynamic sample weights to the defined positive samples based on the distance between the sample center point and the center point of the true box Where: w i Indicates the width of the i-th gt box, h i Indicates the length of the i-th gt box, x i ,y i Represents the coordinates of the center point of the positive sample, x j ,y j Indicates the center coordinates of the candidate box, w dy represents the sample weight.

[0043] Step 2.4.3: Calculate the loss function, which consists of classification loss and regression loss. Use the loss function to perform stochastic gradient descent on the network, gradually optimizing the network and improving the detection effect.

[0044] Step 3: Input the test image or actual ship image into the trained ship target detection network to obtain the predicted box and classification of the ship target.

[0045] In the specific environment settings of Ubuntu 18.04 server, python 3.6.8, and pytorch 1.3.1, the ablation experiment of the FDDEA algorithm and the AASST algorithm proposed in this invention is conducted on the public ship dataset HRSC. The experimental results are shown as follows:

[0046]

[0047] Compared with existing technologies, this method improves the detection of small-scale and noisy targets through a feature enhancement and denoising module based on the FDDEA algorithm, resulting in a 5.94 increase in mAP compared to the method without the FDDEA algorithm. By introducing a detection head based on the AASST strategy, it addresses the low detection accuracy caused by large target shape variations, improving the mAP by 13.01 compared to the method without the AASST strategy. A deep neural network for ship target detection using both the FDDEA and AASST strategies can solve the problem of ship target detection in complex environments, achieving a mAP of 89.78, an improvement of 14.67 compared to 75.11 without the FDDEA and AASST strategies.

[0048] In summary, the present invention solves the problem of insufficient feature extraction of small-scale targets of ship targets by constructing an expansion encoder after the FPN structure, and uses the expansion encoder to enrich the feature semantics and improve the detection effect of small targets; by designing a soft threshold function based on the threshold attention mechanism, the noise and environmental redundant information in the feature map are eliminated, the problems of noise interference and environmental redundancy in ship images are solved, and the detection effect of the model is improved; by statistically analyzing the shape information of the target and the IoU information between the candidate box and the real box, the threshold for defining positive and negative samples is dynamically set, thereby balancing the positive and negative samples and improving the detection accuracy of the model.

[0049] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principles and purpose of the present invention. The scope of protection of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. All implementation schemes within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for detecting ship images in complex environments, characterized in that: In the offline stage, the original ship images are preprocessed by acquisition; a deep neural network for ship target detection is constructed based on the FDDEA algorithm and AASST strategy, and the preprocessed images are used as training samples to train the ship target detection network; In the online stage, the test image or the actual ship image is input into the trained ship target detection deep neural network to obtain the predicted box and classification of the ship target; The ship target detection deep neural network includes: a backbone network based on the Resnet framework, a feature pyramid module, a feature enhancement and denoising module based on FDDEA, and a detection head based on the AASST strategy, wherein: the backbone network extracts feature maps from the preprocessed image; the FPN performs a top-down secondary fusion based on feature maps of different levels to obtain a fused feature map; the feature enhancement and denoising module performs feature enhancement extraction and feature denoising on the fused feature map; the detection head performs detection on each layer of feature map and obtains the final prediction box; After constructing the deep neural network for ship target detection based on the FDDEA algorithm and the AASST strategy, the preprocessed images are used as training samples to train the ship target detection network, specifically including: Step 2.1: Input the preprocessed image into the backbone network ResNet and extract features at different levels through residual convolution; Step 2.2: Use BottomUp to obtain feature maps of different sizes output by different stages of the backbone. Then, use TopDown to start from the top-level feature map, upsample it, and add it to the next-level feature map. Step 2.3: Use the feature enhancement and denoising module based on the FDDEA algorithm to enhance and denoise the feature map. The specific steps include: Step 2.3.1: First, based on the feature maps {P2, P3, P4, P5} output by FPN, upsample P2 to obtain P1, upsample P5 to obtain P6, and use the final {P1, P2, P3, P4, P5, P6} as the input of the input feature expansion encoder; Step 2.3.2: The feature expansion encoder is based on a residual network and dilated convolution. For each layer of the input feature map, it first performs a 1x1 convolution plus a Relu activation function, then a 3x3 dilated convolution plus a Relu activation function, and then another 1x1 convolution plus a Relu activation function. Finally, the output of the convolution module is added to the original input to obtain the output of the residual expansion module, specifically: ,in: is the input of the residual expansion module, To perform a regular convolution with a convolution kernel of 1 × 1 and a RELU activation function, r is the expansion rate of the void convolution, is to perform a dilated convolution with a kernel of 3 × 3 and a dilation rate of r on the input x and a RELU activation function, where O is the output of the residual dilation module; Step 2.3.3: The denoising module obtains an adaptive threshold t based on the output of the residual expansion module through the threshold attention mechanism, and then uses the soft threshold function to perform feature denoising based on the threshold t. Specifically: , where: 𝑥 is the input feature, 𝑦 is the output feature, and t is the threshold; the soft threshold function sets the values ​​within a certain threshold to 0 and makes the values ​​outside the threshold move towards 0; Step 2.3.4: Repeat steps 2.2.2 and 2.2.3 four times, using gradually increasing dilation rates to obtain the denoised dilated coded feature map; Step 2.4: Use the AASST-based detection head to define positive and negative samples and perform loss regression, including: Step 2.4.1: Use the denoised dilated coded feature map as the input of the detection head to define positive and negative samples, including: Step 2.4.1.1: Select the candidate box based on the center distance between the anchor and the true box, specifically: ,in is the width of the i-th gt box, is the length of the i-th gt box, , is the coordinate of the center point of the positive sample, , is the center coordinate of the candidate box, is the center distance; Anchor The closer the center point of the anchor is to the center point of the true box, the higher the detection quality can be provided. This is because the closer to the center point of the true box, the closer the features that the candidate box can describe are to the target. Therefore, the TopK candidate boxes are first selected based on the center distance between the anchor and the true box; Step 2.4.1.2: Use the sum of the mean and standard deviation of the IoU between the candidate box and the true box as the initial threshold for defining the IoU of positive and negative samples, specifically: ,in: ,in: is the mean IoU between the candidate box and the real box, is the mean IoU between the candidate box and the real box, is the IoU value between the i-th candidate box and the j-th real box; Step 2.4.1.3: Calculate the shape adaptive attenuation factor; first count the target aspect ratio information; for the real box corresponding to the candidate box, calculate the aspect ratio mean Ar of all real boxes, based on the mean Ar, the adaptive attenuation factor , where: Ar is the average aspect ratio of the real frame, is the decay weight, is the extreme value of aspect ratio; Step 2.4.1.4: Apply the calculated shape adaptive attenuation factor Substitute into step 2.4.1.2 to get the final IoU threshold for positive and negative sample selection ; Step 2.4.1.5: Based on the calculated IoU threshold, the candidate boxes with an IoU greater than the threshold are defined as positive samples, and the other candidate boxes are defined as negative samples; Step 2.4.2: Assign dynamic sample weights to the defined positive samples based on the distance between the sample center point and the center point of the true box ,in: is the width of the i-th gt box, is the length of the i-th gt box, , is the coordinate of the center point of the positive sample, , is the center coordinate of the candidate box, is the sample weight; Step 2.4.3: Calculate the loss function, which consists of classification loss and regression loss; perform stochastic gradient descent on the network through the loss function to gradually optimize the network and improve the detection effect.

2. The method for detecting ship images in complex environments according to claim 1, wherein: The feature enhancement and denoising module based on FDDEA includes: a feature expansion encoder and a feature denoiser, wherein: the feature expansion encoder performs expansion coding according to the output information of FPN using a residual network and a dilated convolution to obtain an expanded coding feature map with rich semantic information; the feature denoiser performs feature denoising processing according to the expanded coding feature map using a soft threshold function based on a threshold attention mechanism to obtain a denoised expanded coding feature map; finally, the denoised expanded coding feature map is input into a detection head for target detection.

3. The method for detecting ship images in complex environments according to claim 1, wherein: The AASST-based strategy refers to: after filtering out positive samples using an IoU threshold containing a shape adaptive attenuation factor, the positive samples are weighted dynamically based on the distance between the sample center point and the true box center point, and then a loss function consisting of classification loss and regression loss is calculated.

4. The method for detecting ship images in complex environments according to claim 3, wherein: The IoU threshold , where: Adaptive attenuation factor , , Ar is the average aspect ratio of the real frame, is the decay weight, is the extreme value of aspect ratio, is the mean IoU between the candidate box and the real box, is the mean IoU between the candidate box and the real box, is the IoU value between the i-th candidate box and the j-th real box.

5. The method for detecting ship images in complex environments according to claim 4, wherein: When the aspect ratio is greater than the extreme value, the IoU threshold is fixed to 8.

6. The method for detecting ship images in a complex environment according to any one of claims 1 to 5, wherein: include: Step 1: Preprocess the original ship images and enhance the dataset to improve training results and algorithm model robustness, including: Step 1.1: Randomly scale the ship image while keeping the aspect ratio unchanged; Step 1.2: Randomly flip the scaled image horizontally; Step 1.3: Randomly transform the hue, saturation, and brightness of the flipped image. Step 1.4: Fill the image after HSV transformation; Step 2: After building a deep neural network for ship target detection based on the FDDEA algorithm and AASST strategy, the images preprocessed in step 1 are used as training samples to train the ship target detection network; Step 3: Input the test image or actual ship image into the trained ship target detection network to obtain the predicted box and classification of the ship target.

7. A system for implementing the method for detecting ship images in complex environments as recited in any one of claims 1 to 6, characterized in that: include: An image processing unit, a ship image prediction unit and an output unit, wherein: the ship image processing unit crops and scales the original ship image; the ship image prediction unit generates a prediction frame based on the preprocessed image based on the ship target detection deep neural network; the output unit marks the position information and classification information of the ship target in the original ship image according to the prediction frame.

Images