Multi-modal marine target detection method for improving weighting loss

Abstract

The invention discloses a multi-modal maritime target detection method for improving weighted loss, which comprises the following steps: firstly, constructing a target detection model which comprises a feature learning module and a target learning module; the target learning module comprises a segmentation head and a detection head. During training, a target segmentation loss function is calculated based on a segmentation mask label and a segmentation mask predicted by a segmentation head, and network parameters of the segmentation head and a feature learning module are optimized. Meanwhile, a target detection loss function is calculated based on a segmentation mask predicted by a segmentation head, a detection frame predicted by a detection head and a real detection frame, and network parameters of the detection head and a feature learning module are optimized. According to the method, on the premise of not additionally marking segmentation labels, through a multi-task learning mode, the model is promoted to pay attention to a difficult area with large loss, the characteristics with higher generalization are learned, the optimization process of the loss is stabilized, and the target detection effect is effectively improved.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection, and in particular relates to a method for detecting targets at sea. Background Art

[0002] The detection of maritime targets is of great significance for the safe navigation of ships and marine environment monitoring. Multimodal image target detection can provide richer and more reliable environmental perception capabilities by fusing information from different sensors or data sources, such as visible light images and infrared images, and significantly improves the accuracy and robustness of target detection, thereby achieving stable target monitoring in complex and changing environments. Therefore, multimodal image target detection technology for maritime targets in remote sensing images is gradually becoming a research frontier.

[0003] Although some progress has been made in the research, in terms of target detection loss, existing methods usually use intersection-over-union to calculate target detection loss. This loss function does not consider the impact of the prediction difficulty and importance of different positions in the detection frame on the training process, making the training process unstable and restricting further performance improvement. A few methods (such as YOLOP) also consider introducing the target segmentation task into the target detection task for joint learning to improve the target detection effect, but this method requires annotating mask labels that match the hull shape to construct the traditional segmentation task, so it requires a lot of manpower and material resources. Summary of the invention

[0004] The present invention proposes a multimodal marine target detection method with improved weighted loss, the purpose of which is to solve the problem that the existing loss function does not consider the prediction difficulty and importance of different positions in the detection frame, resulting in an unstable training process and poor detection performance, and to solve the problem of how to use the information learned from segmentation to assist in improving the target detection effect without marking the segmentation labels that match the hull shape.

[0005] The technical solution of the present invention is as follows: a multi-modal marine target detection method with improved weighted loss, comprising the following steps: Step 1: construct a target detection model, wherein the target detection model includes a feature learning module for learning fusion features from input visible light ship images and infrared ship images and a target learning module for obtaining detection results based on the fusion features; The target learning module includes a segmentation head and a detection head, wherein the segmentation head obtains a segmentation mask for representing the target position based on the fusion features, and the detection head obtains a detection frame for representing the target position based on the fusion features; Step 2, training the target detection model; the training samples include visible light ship images and infrared ship images of the same target, and also include a real detection frame for indicating the target position and a segmentation mask label corresponding to the real detection frame; during training, the network parameters of the segmentation head and the feature learning module are optimized by the target segmentation loss function calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head, and the target detection loss function is calculated based on the segmentation mask predicted by the segmentation head, the detection frame predicted by the detection head, and the real detection frame to optimize the network parameters of the detection head and the feature learning module; Step 3: Input the visible light ship image and infrared ship image to be detected into the trained target detection model, and use the detection frame output by the detection head as the detection result.

[0006] As a further improvement of the improved weighted loss multimodal maritime target detection method: the feature learning module includes a backbone network and a feature fusion network; the backbone network contains two weight-sharing backbone networks, one backbone network is used to extract visible light feature maps from visible light ship images, and the other backbone network is used to extract infrared feature maps from infrared ship images.

[0007] As a further improvement of the multimodal marine target detection method with improved weighted loss: the feature fusion network includes a neck network, which is used to fuse the visible light feature map and the infrared feature map to obtain the fused feature.

[0008] As a further improvement of the multimodal marine target detection method with improved weighted loss: the backbone network, neck network and detection head respectively adopt the structures of the backbone network, neck network and detection head in Yolov8.

[0009] As a further improvement of the multimodal marine target detection method with improved weighted loss: the fusion feature is a multi-scale feature map.

[0010] As a further improvement of the multimodal marine target detection method with improved weighted loss: the fusion feature includes feature maps of three scales: P3, P4, and P5; In the segmentation head, feature map P5 is first upsampled to obtain feature map P5u, and then feature map P5u is fused with feature map P4 and then convolved and upsampled to obtain feature map P45. Feature map P45 is then fused with feature map P3 and then convolved to obtain feature map P345. Feature map P345 is then convolved, transposed several times, and convolved several times to obtain the segmentation mask.

[0011] As a further improvement of the multimodal marine target detection method with improved weighted loss, the target segmentation loss function is calculated as follows: Let the segmentation mask label The segmentation label corresponding to the pixel is , the segmentation mask predicted by the segmentation head The probability value corresponding to the pixel is , both the segmentation mask label and the segmentation mask have pixels, then the target segmentation loss function for: .

[0012] As a further improvement of the multimodal marine target detection method with improved weighted loss, the target detection loss function is calculated as follows: ; In the formula, is the real detection frame And the predicted detection box The number of pixels after the intersection is taken, Represents the predicted detection box The number of pixels minus the true detection box And the predicted detection box Take the difference after the number of pixels after the intersection, is the detection box predicted by the segmentation head Remove the real detection frame And the predicted detection box The area remaining after the intersection area The average loss of prediction is and They are respectively the real detection boxes And the predicted detection box The number of pixels.

[0013] As a further improvement of the multimodal maritime target detection method with improved weighted loss, The calculation method is: ; in, For Region The number of pixels, is the number of pixels of the segmentation mask predicted by the segmentation head for the input image, is the segmentation mask The probability value of pixels; flag bit The value of is as follows: if the segmentation mask pixels belong to the region 1 if yes, 0 otherwise.

[0014] As a further improvement of the multi-modal marine target detection method with improved weighted loss, the specific process of step 2 is to repeat the following steps several times: Step 2-1, take several training samples, input the visible light ship image and infrared ship image into the target detection model, and obtain the segmentation mask predicted by the segmentation head and the detection frame predicted by the detection head; Step 2-2, calculate the target segmentation loss function of each training sample in this batch, and then update the network parameters of the segmentation head and feature learning module based on the average value of the target segmentation loss function of this batch; Step 2-3: Calculate the target detection loss function for each training sample in this batch, and then update the network parameters of the detection head and feature learning module based on the average value of the target detection loss function of this batch.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. During training, the present invention calculates the target detection loss function based on the segmentation mask predicted by the segmentation head, the detection frame predicted by the detection head, and the real detection frame to optimize the network parameters of the detection head and the feature learning module. The target detection loss function introduces the predicted value in the segmentation mask through the average loss of non-overlapping areas, thereby prompting the model to focus on difficult areas with large losses and stabilize the loss optimization process, effectively improving the target detection effect.

[0016] 2. The segmentation head designed in the present invention is used to enable the model to fully learn image features through multi-task learning. The segmentation mask label is obtained through the real detection frame instead of the traditional mask label that completely matches the target edge, which greatly reduces the segmentation label annotation cost.

[0017] 3. The YOLOv8 based on the present invention has the advantages of fast detection speed and stable performance, and can improve the speed and performance of multimodal ship target detection.

[0018] 4. The segmentation head uses a multi-scale segmentation network, which can effectively fuse feature maps of different scales to obtain prediction results. Among them, low-level feature maps have higher resolution and can capture more detailed information, which helps to improve the detection performance of small targets; high-level feature maps have stronger semantic information, which helps the model understand the global context of the image. By fusing features of different scales, the model can better handle targets of different sizes, thereby improving the generalization ability of the model, and ultimately more effectively assisting the detection head to improve the target detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the architecture of the target detection model of the present invention; Figure 2 This is a schematic diagram of the architecture of the segmentation head. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0021] A multi-modal marine target detection method with improved weighted loss, comprising the following steps: Step 1: Build a target detection model.

[0022] like Figure 1 The target detection model includes a feature learning module for learning fusion features from input visible light ship images and infrared ship images and a target learning module for obtaining detection results based on the fusion features.

[0023] The feature learning module includes a backbone network and a feature fusion network. The backbone network includes two weight-sharing backbone networks, one backbone network is used to extract a visible light feature map from a visible light ship image, and the other backbone network is used to extract an infrared feature map from an infrared ship image. In this embodiment, both the visible light feature map and the infrared feature map include three feature maps D3-D5, whose sizes are 80×80×256, 40×40×512, and 20×20×512, respectively.

[0024] The feature fusion network includes a neck network, which is used to fuse the visible light feature map and the infrared feature map to obtain the fused feature. Furthermore, the fused feature is a multi-scale feature map, including feature maps of three scales, P3, P4, and P5, with sizes of 80×80×256, 40×40×512, and 20×20×512, respectively.

[0025] In this embodiment, the backbone network and the neck network respectively adopt the structures of the backbone network and the neck network in Yolov8.

[0026] The target learning module includes a segmentation head and a detection head. The segmentation head obtains a segmentation mask for representing the target position based on the fusion features, and the detection head obtains a detection frame for representing the target position based on the fusion features.

[0027] like Figure 2In the segmentation head, the multi-scale feature maps are first fused, and then the segmentation mask is obtained: the feature map P5 is first upsampled to obtain the feature map P5u, and the feature map P5u has the same size as the feature map P4. Then the feature map P5u and the feature map P4 are fused on the channel and then successively convolved and upsampled to obtain the feature map P45. The convolution is used to reduce the channel, and the upsampling is used to convert the feature map to the same size as the feature map P3. Then the feature map P45 and the feature map P3 are fused and then successively convolved to obtain the feature map P345. The feature map P345 is then successively convolved once and transposed three times to obtain a feature map with the same size as the original input image (640×640×256). After that, it is successively convolved 1x1, 3x3, and 1x1 to obtain a segmentation mask of size 640×640×3.

[0028] The detection head adopts the structure of the detection head in Yolov8.

[0029] Step 2: training the target detection model.

[0030] First, we construct a dataset: an optoelectronic platform with integrated visible light and infrared cameras is installed on a drone. We collect infrared and visible light images of the same ship target, crop both images to 640×640×3, and manually annotate the position labels of the ship target in the form of an oblique frame (x1, y1, x2, y2) to obtain the true detection frame. x1 and y1 are the x and y coordinates of the upper left corner of the true detection frame, and x2 and y2 are the x and y coordinates of the lower right corner of the true detection frame. Then, we construct a segmentation mask label based on the true detection frame. The segmentation mask value of the area within the true detection frame is set to 1, and the other areas are set to 0. The final sample contains visible light ship images and infrared ship images of the same target, as well as a true detection frame used to represent the target position and a segmentation mask label corresponding to the true detection frame, which can be expressed as: <visible light ship image, infrared ship image, true detection frame, segmentation mask label>.

[0031] Then, the samples in the data set are randomly divided into training samples and test samples in a ratio of 7:3 to construct the training set and test set.

[0032] During training, the target segmentation loss function is calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head to optimize the network parameters of the segmentation head and the feature learning module. At the same time, the target detection loss function is calculated based on the segmentation mask predicted by the segmentation head, the detection box predicted by the detection head and the real detection box to optimize the network parameters of the detection head and the feature learning module.

[0033] Specifically, set the learning rate to 0.0002, the number of training times to 500, the batch size to e=64, and perform the following steps 500 times: Step 2-1: Take e training samples, input the visible light ship image and infrared ship image into the target detection model, and obtain the segmentation mask predicted by the segmentation head and the detection box predicted by the detection head.

[0034] Step 2-2: Calculate the target segmentation loss function for each training sample in this batch, and then update the network parameters of the segmentation head and feature learning module based on the average value of the target segmentation loss function of this batch.

[0035] The target segmentation loss function is calculated as: Let the segmentation mask label The label corresponding to the pixel is , the segmentation mask predicted by the segmentation head The probability value corresponding to the pixel is , both the segmentation mask label and the segmentation mask have pixels, then the target segmentation loss function for: .

[0036] Step 2-3: Calculate the target detection loss function for each training sample in this batch, and then update the network parameters of the detection head and feature learning module based on the average value of the target detection loss function of this batch.

[0037] The target detection loss function is calculated as: ; In the formula, is the real detection frame And the predicted detection box Take the number of pixels after the intersection (i.e. area), Represents the predicted detection box The number of pixels minus the true detection box And the predicted detection box Take the difference after the number of pixels after the intersection, is the detection box predicted by the segmentation head Remove the real detection frame And the predicted detection box The area remaining after the intersection area The average loss of prediction is and They are respectively the real detection boxes And the predicted detection box The number of pixels.

[0038] Further, The calculation method is: ; in, For Region The number of pixels, is the number of pixels of the segmentation mask predicted by the segmentation head for the input image, is the segmentation mask The probability value of pixels; flag bit The value of is as follows: if the segmentation mask pixels belong to the region 1 if yes, 0 otherwise.

[0039] Step 3: Input the visible light ship image and infrared ship image to be detected into the trained target detection model, and use the detection frame output by the detection head as the detection result.

[0040] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A multi-modal marine target detection method with improved weighted loss, characterized in that the steps include: Step 1: construct a target detection model, wherein the target detection model includes a feature learning module for learning fusion features from input visible light ship images and infrared ship images and a target learning module for obtaining detection results based on the fusion features; The target learning module includes a segmentation head and a detection head, wherein the segmentation head obtains a segmentation mask for representing the target position based on the fusion features, and the detection head obtains a detection frame for representing the target position based on the fusion features; Step 2, training the target detection model; the training samples include visible light ship images and infrared ship images of the same target, and also include a real detection frame for indicating the target position and a segmentation mask label corresponding to the real detection frame; during training, the network parameters of the segmentation head and the feature learning module are optimized by the target segmentation loss function calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head, and the target detection loss function is calculated based on the segmentation mask predicted by the segmentation head, the detection frame predicted by the detection head, and the real detection frame to optimize the network parameters of the detection head and the feature learning module; Step 3: Input the visible light ship image and infrared ship image to be detected into the trained target detection model, and use the detection frame output by the detection head as the detection result.

2. The multi-modal marine target detection method with improved weighted loss as claimed in claim 1, characterized in that: The feature learning module includes a backbone network and a feature fusion network; the backbone network includes two weight-sharing backbone networks, one backbone network is used to extract visible light feature maps from visible light ship images, and the other backbone network is used to extract infrared feature maps from infrared ship images.

3. The multi-modal marine target detection method with improved weighted loss as claimed in claim 2, characterized in that: The feature fusion network includes a neck network, which is used to fuse the visible light feature map and the infrared feature map to obtain the fused feature.

4. The multi-modal marine target detection method with improved weighted loss as claimed in claim 3, characterized in that: The backbone network, neck network and detection head respectively adopt the structures of the backbone network, neck network and detection head in Yolov8.

5. The multi-modal marine target detection method with improved weighted loss as claimed in claim 1, characterized in that: The fused feature is a multi-scale feature map.

6. The multi-modal marine target detection method with improved weighted loss as claimed in claim 5, characterized in that: The fusion features include feature maps of three scales: P3, P4, and P5; In the segmentation head, feature map P5 is first upsampled to obtain feature map P5u, and then feature map P5u is fused with feature map P4 and then convolved and upsampled to obtain feature map P45. Feature map P45 is then fused with feature map P3 and then convolved to obtain feature map P345. Feature map P345 is then convolved, transposed several times, and convolved several times to obtain the segmentation mask.

7. The multi-modal marine target detection method with improved weighted loss as claimed in claim 1, characterized in that: The target segmentation loss function is calculated as: Let the segmentation mask label The segmentation label corresponding to the pixel is , the segmentation mask predicted by the segmentation head The probability value corresponding to the pixel is , both the segmentation mask label and the segmentation mask have pixels, then the target segmentation loss function for: 。 8. The multi-modal marine target detection method with improved weighted loss as claimed in claim 1, characterized in that: The target detection loss function is calculated as: ； In the formula, is the real detection frame And the predicted detection box The number of pixels after the intersection is taken, Represents the predicted detection box The number of pixels minus the true detection box And the predicted detection box Take the difference after the number of pixels after the intersection, is the detection box predicted by the segmentation head Remove the real detection frame And the predicted detection box The area remaining after the intersection area The average loss of prediction is and They are respectively the real detection boxes And the predicted detection box The number of pixels.

9. The multi-modal marine target detection method with improved weighted loss as claimed in claim 8, characterized in that: The calculation method is: ； in, For Region The number of pixels, is the number of pixels of the segmentation mask predicted by the segmentation head for the input image, is the segmentation mask The probability value of pixels; flag bit The value of is as follows: if the segmentation mask pixels belong to the region 1 if the value is true, 0 otherwise.

10. The multi-modal marine target detection method with improved weighted loss according to any one of claims 1 to 9, characterized in that: The specific process of step 2 is to repeat the following steps several times: Step 2-1, take several training samples, input the visible light ship image and infrared ship image into the target detection model, and obtain the segmentation mask predicted by the segmentation head and the detection frame predicted by the detection head; Step 2-2, calculate the target segmentation loss function of each training sample in this batch, and then update the network parameters of the segmentation head and feature learning module based on the average value of the target segmentation loss function of this batch; Step 2-3: Calculate the target detection loss function for each training sample in this batch, and then update the network parameters of the detection head and feature learning module based on the average value of the target detection loss function of this batch.

Images