A multimodal maritime target detection method with improved weighted loss

Abstract

The present invention discloses a multimodal marine target detection method with improved weighted loss. First, a target detection model is constructed, which includes a feature learning module and a target learning module; the target learning module includes a segmentation head and a detection head. During training, the target segmentation loss function is calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head to optimize the network parameters of the segmentation head and the feature learning module. At the same time, the target detection loss function is calculated based on the segmentation mask predicted by the segmentation head, the detection frame predicted by the detection head, and the real detection frame to optimize the network parameters of the detection head and the feature learning module. The present invention can, without the need for additional segmentation labeling, encourage the model to focus on difficult areas with large losses, learn features with stronger generalization, and stabilize the loss optimization process, thereby effectively improving the target detection effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a method for detecting targets at sea. Background Art

[0002] Detecting maritime targets is crucial for safe navigation and marine environmental monitoring. Multimodal image target detection, by fusing information from different sensors or data sources, such as visible light and infrared images, provides richer and more reliable environmental perception capabilities, significantly improving the accuracy and robustness of target detection and enabling stable target monitoring in complex and changing environments. Therefore, multimodal image target detection technology for maritime targets in remote sensing images is becoming a leading research topic.

[0003] Despite some progress, existing methods typically use the intersection-over-union (IoU) loss function to calculate object detection loss. This loss function fails to consider the impact of the difficulty and importance of predictions at different locations within the detection frame on the training process, resulting in unstable training and hindering further performance improvements. A few methods, such as YOLOP, have considered integrating object segmentation with object detection for joint learning, thereby improving object detection performance. However, this approach requires labeling masks that match the ship's shape to construct the traditional segmentation task, which consumes significant manpower and resources. Summary of the Invention

[0004] The present invention proposes a multimodal maritime target detection method with an improved weighted loss. Its purpose is to solve the problem that the existing loss function does not consider the prediction difficulty and importance of different positions in the detection frame, resulting in an unstable training process and poor detection performance. It also solves the problem of how to use the information learned from segmentation to assist in improving the target detection effect without annotating the segmentation labels that match the hull shape.

[0005] The technical solution of the present invention is as follows: a multimodal marine target detection method with improved weighted loss, comprising the following steps:

[0006] Step 1: Construct a target detection model, which includes a feature learning module for learning fusion features from input visible light ship images and infrared ship images, and a target learning module for obtaining detection results based on the fusion features;

[0007] The target learning module includes a segmentation head and a detection head, wherein the segmentation head obtains a segmentation mask for representing the target position based on the fusion features, and the detection head obtains a detection frame for representing the target position based on the fusion features;

[0008] Step 2: training the target detection model; the training samples include visible light ship images and infrared ship images of the same target, as well as a true detection frame for indicating the target position and a segmentation mask label corresponding to the true detection frame; during training, the network parameters of the segmentation head and the feature learning module are optimized by using a target segmentation loss function calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head. At the same time, the network parameters of the detection head and the feature learning module are optimized by using a target detection loss function calculated based on the segmentation mask predicted by the segmentation head, the detection frame predicted by the detection head, and the true detection frame;

[0009] Step 3: Input the visible light ship image and infrared ship image to be detected into the trained target detection model, and use the detection box output by the detection head as the detection result.

[0010] As a further improvement of the multimodal maritime target detection method with improved weighted loss: the feature learning module includes a backbone network and a feature fusion network; the backbone network includes two weight-sharing backbone networks, one backbone network is used to extract visible light feature maps from visible light ship images, and the other backbone network is used to extract infrared feature maps from infrared ship images.

[0011] As a further improvement of the multimodal maritime target detection method with improved weighted loss: the feature fusion network includes a neck network, which is used to fuse the visible light feature map and the infrared feature map to obtain the fused feature.

[0012] As a further improvement of the multimodal maritime target detection method with improved weighted loss: the backbone network, neck network and detection head respectively adopt the structures of the backbone network, neck network and detection head in Yolov8.

[0013] As a further improvement of the multimodal maritime target detection method with improved weighted loss: the fusion feature is a multi-scale feature map.

[0014] As a further improvement of the multimodal maritime target detection method with improved weighted loss, the fusion feature includes feature maps at three scales: P3, P4, and P5;

[0015] In the segmentation head, the feature map P5 is first upsampled to obtain the feature map P5u, and then the feature map P5u is fused with the feature map P4 and then subjected to convolution and upsampling in sequence to obtain the feature map P45. Then, the feature map P45 is fused with the feature map P3 and then subjected to convolution in sequence to obtain the feature map P345. The feature map P345 is then subjected to convolution, several convolution transpositions and several convolutions in sequence to obtain the segmentation mask.

[0016] As a further improvement of the multimodal maritime target detection method with improved weighted loss, the target segmentation loss function is calculated as follows:

[0017] Let the segmentation mask label The segmentation label corresponding to the pixel is , the segmentation mask predicted by the segmentation head The probability value corresponding to the pixel is , both the segmentation mask label and the segmentation mask have pixels, then the target segmentation loss function for:

[0018] .

[0019] As a further improvement of the multimodal maritime target detection method with improved weighted loss, the target detection loss function is calculated as follows:

[0020] ;

[0021] Where, is the real detection box and the predicted detection box The number of pixels after the intersection, Represents the predicted detection box The number of pixels minus the true detection box and the predicted detection box Take the difference after the number of pixels after the intersection, The segmentation head is the predicted detection box Remove the real detection frame and the predicted detection box The area remaining after the intersection area The average loss of predictions, and They are respectively the real detection boxes and the predicted detection box The number of pixels.

[0022] As a further improvement of the multimodal maritime target detection method with improved weighted loss, The calculation method is:

[0023] ;

[0024] in, For the region The number of pixels, is the number of pixels of the segmentation mask predicted by the segmentation head for the input image, The first segmentation mask The probability value of pixels; flag bit The value of is: if the segmentation mask pixels belong to the region 1 if yes, 0 otherwise.

[0025] As a further improvement of the multimodal maritime target detection method with improved weighted loss, the specific process of step 2 is to repeat the following steps several times:

[0026] Step 2-1: Take several training samples and input the visible light ship image and infrared ship image into the target detection model to obtain the segmentation mask predicted by the segmentation head and the detection box predicted by the detection head;

[0027] Step 2-2: Calculate the target segmentation loss function for each training sample in this batch, and then update the network parameters of the segmentation head and feature learning module based on the average value of the target segmentation loss function of this batch;

[0028] Step 2-3: Calculate the target detection loss function for each training sample in this batch, and then update the network parameters of the detection head and feature learning module based on the average value of the target detection loss function of this batch.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. During training, the present invention calculates the target detection loss function based on the segmentation mask predicted by the segmentation head, the detection box predicted by the detection head, and the real detection box to optimize the network parameters of the detection head and the feature learning module. The target detection loss function introduces the predicted value in the segmentation mask through the average loss of non-overlapping areas, thereby prompting the model to focus on difficult areas with large losses, and stabilizes the loss optimization process, effectively improving the target detection effect.

[0031] 2. The segmentation head designed in the present invention enables the model to fully learn image features through multi-task learning. Its segmentation mask label is obtained through the real detection frame, rather than the traditional mask label that completely matches the target edge, which greatly reduces the segmentation label annotation cost.

[0032] 3. The YOLOv8 based on this invention has the advantages of fast detection speed and stable performance, and can improve the speed and performance of multimodal ship target detection.

[0033] 4. The segmentation head utilizes a multi-scale segmentation network, effectively fusing feature maps of different scales to generate predictions. Low-level feature maps have higher resolution and capture more detailed information, helping to improve detection performance for small objects. High-level feature maps contain stronger semantic information, helping the model understand the global context of the image. By fusing features at different scales, the model can better handle objects of varying sizes, thereby improving its generalization capabilities and ultimately more effectively assisting the detection head in improving object detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the architecture of the target detection model of the present invention;

[0035] Figure 2 This is a schematic diagram of the architecture of the segmentation head. DETAILED DESCRIPTION

[0036] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only a part of the embodiments of the present invention, rather than all the embodiments.

[0037] A multimodal marine target detection method with improved weighted loss, comprising the following steps:

[0038] Step 1: Build a target detection model.

[0039] like Figure 1 The target detection model includes a feature learning module for learning fusion features from input visible light ship images and infrared ship images and a target learning module for obtaining detection results based on the fusion features.

[0040] The feature learning module includes a backbone network and a feature fusion network. The backbone network includes two weight-sharing backbone networks: one for extracting visible light feature maps from visible light ship images, and the other for extracting infrared feature maps from infrared ship images. In this embodiment, both the visible light feature maps and the infrared feature maps include three feature maps, D3-D5, with sizes of 80×80×256, 40×40×512, and 20×20×512, respectively.

[0041] The feature fusion network includes a neck network, which is used to fuse the visible light feature map and the infrared feature map to obtain the fused feature. Furthermore, the fused feature is a multi-scale feature map, including feature maps at three scales: P3, P4, and P5, with sizes of 80×80×256, 40×40×512, and 20×20×512, respectively.

[0042] In this embodiment, the backbone network and the neck network respectively adopt the structures of the backbone network and the neck network in Yolov8.

[0043] The target learning module includes a segmentation head and a detection head. The segmentation head obtains a segmentation mask for representing the target position based on the fusion features, and the detection head obtains a detection frame for representing the target position based on the fusion features.

[0044] like Figure 2In the segmentation head, multi-scale feature maps are first fused to obtain a segmentation mask: Feature map P5 is first upsampled to obtain feature map P5u, which has the same size as feature map P4. Feature map P5u is then fused with feature map P4 on the channel and then subjected to convolution and upsampling to obtain feature map P45. Convolution is used to reduce the number of channels, and upsampling is used to convert the feature map to the same size as feature map P3. Feature map P45 is then fused with feature map P3 and then subjected to convolution to obtain feature map P345. Feature map P345 then undergoes one convolution and three transposition convolutions to obtain a feature map of the same size as the original input image (640×640×256). It then undergoes 1x1 convolution, 3x3 convolution, and 1x1 convolution to obtain a segmentation mask of size 640×640×3.

[0045] The detection head adopts the structure of the detection head in Yolov8.

[0046] Step 2: Train the target detection model.

[0047] First, we constructed a dataset: an optoelectronic platform equipped with visible and infrared cameras was installed on a drone. We captured infrared and visible light images of the same ship target. Both images were cropped and resized to 640×640×3. The ship target's location was manually annotated with an oblique box (x1, y1, x2, y2), where x1 and y1 represent the x and y coordinates of the upper left corner of the ground truth box, and x2 and y2 represent the x and y coordinates of the lower right corner. We then constructed a segmentation mask label based on the ground truth box. The segmentation mask value for the region within the ground truth box was set to 1, and 0 for all other regions. The final sample contains both visible light and infrared ship images of the same target, as well as the ground truth box representing the target's location and the segmentation mask label corresponding to the ground truth box. This can be expressed as: <visible light ship image, infrared ship image, ground truth box, segmentation mask label>.

[0048] Then, the samples in the dataset are randomly divided into training samples and test samples in a ratio of 7:3 to construct the training set and test set.

[0049] During training, the target segmentation loss function is calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head to optimize the network parameters of the segmentation head and feature learning module. At the same time, the target detection loss function is calculated based on the segmentation mask predicted by the segmentation head, the detection box predicted by the detection head and the real detection box to optimize the network parameters of the detection head and feature learning module.

[0050] Specifically, set the learning rate to 0.0002, the number of training epochs to 500, the batch size to e=64, and perform the following steps 500 times:

[0051] Step 2-1: Take e training samples, input the visible light ship image and infrared ship image into the target detection model, and obtain the segmentation mask predicted by the segmentation head and the detection box predicted by the detection head.

[0052] Step 2-2: Calculate the target segmentation loss function for each training sample in this batch, and then update the network parameters of the segmentation head and feature learning module based on the average value of the target segmentation loss function of this batch.

[0053] The target segmentation loss function is calculated as:

[0054] Let the segmentation mask label The label corresponding to the pixel is , the segmentation mask predicted by the segmentation head The probability value corresponding to the pixel is , both the segmentation mask label and the segmentation mask have pixels, then the target segmentation loss function for:

[0055] .

[0056] Step 2-3: Calculate the target detection loss function for each training sample in this batch, and then update the network parameters of the detection head and feature learning module based on the average value of the target detection loss function of this batch.

[0057] The target detection loss function is calculated as:

[0058] ;

[0059] Where, is the real detection box and the predicted detection box Take the number of pixels after the intersection (i.e. area), Represents the predicted detection box The number of pixels minus the true detection box and the predicted detection box Take the difference after the number of pixels after the intersection, The segmentation head is the predicted detection box Remove the real detection frame and the predicted detection box The area remaining after the intersection area The average loss of predictions, and They are respectively the real detection boxes and the predicted detection box The number of pixels.

[0060] Furthermore, The calculation method is:

[0061] ;

[0062] in, For the region The number of pixels, is the number of pixels of the segmentation mask predicted by the segmentation head for the input image, The first segmentation mask The probability value of pixels; flag bit The value of is: if the segmentation mask pixels belong to the region 1 if yes, 0 otherwise.

[0063] Step 3: Input the visible light ship image and infrared ship image to be detected into the trained target detection model, and use the detection box output by the detection head as the detection result.

[0064] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. The scope of the present invention is defined by the claims rather than the foregoing description.

Claims

1. A multimodal marine target detection method with improved weighted loss, characterized in that the steps include: Step 1: Construct a target detection model, which includes a feature learning module for learning fusion features from input visible light ship images and infrared ship images, and a target learning module for obtaining detection results based on the fusion features; The target learning module includes a segmentation head and a detection head, wherein the segmentation head obtains a segmentation mask for representing the target position based on the fusion features, and the detection head obtains a detection frame for representing the target position based on the fusion features; Step 2: training the target detection model; the training samples include visible light ship images and infrared ship images of the same target, as well as a true detection frame for indicating the target position and a segmentation mask label corresponding to the true detection frame; during training, the network parameters of the segmentation head and the feature learning module are optimized by using a target segmentation loss function calculated based on the segmentation mask label and the segmentation mask predicted by the segmentation head. At the same time, the network parameters of the detection head and the feature learning module are optimized by using a target detection loss function calculated based on the segmentation mask predicted by the segmentation head, the detection frame predicted by the detection head, and the true detection frame; The target detection loss function is calculated as: ； Where, is the real detection box and the predicted detection box The number of pixels after the intersection, Represents the predicted detection box The number of pixels minus the true detection box and the predicted detection box Take the difference after the number of pixels after the intersection, The segmentation head is the predicted detection box Remove the real detection frame and the predicted detection box The area remaining after the intersection area The average loss of predictions, and They are respectively the real detection boxes and the predicted detection box The number of pixels; The calculation method is: ； in, For the region The number of pixels, is the number of pixels of the segmentation mask predicted by the segmentation head for the input image, The first segmentation mask The probability value of pixels; flag bit The value of is: if the segmentation mask pixels belong to the region If yes, it is 1, otherwise it is 0; Step 3: Input the visible light ship image and infrared ship image to be detected into the trained target detection model, and use the detection box output by the detection head as the detection result.

2. The multimodal maritime target detection method with improved weighted loss according to claim 1, wherein: The feature learning module includes a backbone network and a feature fusion network; the backbone network includes two weight-sharing backbone networks, one backbone network is used to extract visible light feature maps from visible light ship images, and the other backbone network is used to extract infrared feature maps from infrared ship images.

3. The multimodal maritime target detection method with improved weighted loss according to claim 2, wherein: The feature fusion network includes a neck network, which is used to fuse the visible light feature map and the infrared feature map to obtain the fused feature.

4. The multimodal maritime target detection method with improved weighted loss according to claim 3, wherein: The backbone network, neck network and detection head respectively adopt the structures of the backbone network, neck network and detection head in Yolov8.

5. The multimodal maritime target detection method with improved weighted loss according to claim 1, wherein: The fusion feature is a multi-scale feature map.

6. The multimodal maritime target detection method with improved weighted loss according to claim 5, characterized in that: The fusion features include feature maps at three scales: P3, P4, and P5; In the segmentation head, the feature map P5 is first upsampled to obtain the feature map P5u, and then the feature map P5u is fused with the feature map P4 and then subjected to convolution and upsampling in sequence to obtain the feature map P45. Then, the feature map P45 is fused with the feature map P3 and then subjected to convolution in sequence to obtain the feature map P345. The feature map P345 is then subjected to convolution, several convolution transpositions and several convolutions in sequence to obtain the segmentation mask.

7. The multimodal maritime target detection method with improved weighted loss according to claim 1, wherein: The target segmentation loss function is calculated as: Let the segmentation mask label The segmentation label corresponding to the pixel is , the segmentation mask predicted by the segmentation head The probability value corresponding to the pixel is , both the segmentation mask label and the segmentation mask have pixels, then the target segmentation loss function for: 。 8. The multimodal maritime target detection method with improved weighted loss according to any one of claims 1 to 7, characterized in that: The specific process of step 2 is to repeat the following steps several times: Step 2-1: Take several training samples and input the visible light ship image and infrared ship image into the target detection model to obtain the segmentation mask predicted by the segmentation head and the detection box predicted by the detection head; Step 2-2: Calculate the target segmentation loss function for each training sample in this batch, and then update the network parameters of the segmentation head and feature learning module based on the average value of the target segmentation loss function of this batch; Step 2-3: Calculate the target detection loss function for each training sample in this batch, and then update the network parameters of the detection head and feature learning module based on the average value of the target detection loss function of this batch.

Images