A maritime target detection method based on dual-mode image fusion

Abstract

This invention discloses a maritime target detection method based on dual-modal image fusion. It constructs a three-level architecture model including a dual-modal preprocessing module, a lightweight detection module, and a decision-level fusion module. The method involves: preprocessing infrared and visible light images using the dual-modal preprocessing module; inputting the resulting infrared and visible light branches into the lightweight detection module; employing a dual-stream feature extraction network; and inputting the extracted features into a YOLOv8 detection head to achieve separate detection of the infrared and visible light images. A decision-level fusion module is then constructed, which uses a three-level decision-level fusion framework. Modal weights are dynamically allocated based on sea state parameters. The target detection results and corresponding weights are input into an improved D-S evidence theory. An adaptive discount factor is used to reconstruct the probability distribution and match detection boxes, ultimately outputting the fused detection result. This invention proposes a more efficient and robust dual-modal image fusion detection method, effectively improving the performance of maritime target detection.

Description

Technical Field

[0001] This invention relates to the field of maritime multi-source target situation fusion technology, and in particular to a maritime target detection method based on dual-mode image fusion. Background Technology

[0002] With the increasing development of marine resources and maritime traffic, maritime target detection has become an important technological requirement. Traditional maritime target detection methods typically rely on a single sensor, such as optical or infrared imaging, but each sensor has its limitations. Optical images are easily affected by weather conditions (such as fog, rain, and snow) or water surface reflection in complex sea conditions, leading to a decrease in target recognition accuracy. While infrared imaging has certain advantages at night or in low visibility conditions, it has high requirements for the thermal radiation characteristics of the target and is easily affected by sea surface fluctuations and background temperature changes, resulting in unstable detection results.

[0003] To overcome the limitations of single sensors, dual-modal image fusion technology has been increasingly applied in recent years. By fusing infrared and visible light images, the advantages of both can be effectively combined: optical images provide high-resolution spatial information, while infrared images can provide reliable target information in low-visibility or nighttime environments. Existing methods can be broadly classified into three categories: Image-layer fusion methods: These methods primarily perform weighted fusion or transformation processing on infrared and visible light images at the pixel or image level to obtain a fused image containing dual-modal information. Then, traditional detection algorithms or deep learning models are used for target detection. Their advantage lies in their intuitiveness and ease of implementation, but they are prone to information redundancy or loss of detail, resulting in limited detection accuracy.

[0004] Feature-based fusion methods: These methods utilize deep learning models to extract features from infrared and visible light images separately, and then achieve fusion through feature concatenation, attention mechanisms, or cross-modal alignment, which can preserve complementary information between the two modalities to some extent. However, due to the significant differences in feature distribution between different modalities, feature mismatch or information conflict can easily occur during the fusion process, thus affecting the detection performance.

[0005] Decision-layer-based fusion methods typically perform target detection independently on infrared and visible light images, and then fuse them using a weighted or decisional strategy at the result layer. While this improves detection robustness to some extent, it is highly dependent on the detection accuracy of each modality, and the overall performance is significantly limited when the performance of a particular modality deteriorates.

[0006] Despite some progress in bimodal detection methods, numerous challenges remain in the complex context of maritime environments. These include adaptively balancing the importance of different modalities during fusion, mitigating the impact of resolution differences and imaging noise, and designing lightweight networks to meet the demands of real-time maritime detection. Therefore, it is still necessary to propose a more efficient and robust bimodal image fusion detection method to further improve the performance of maritime target detection.

[0007] In summary, the present invention aims to solve the following specific problems: ① Insufficient cross-modal feature extraction: Traditional methods have bottlenecks in feature extraction from infrared and visible light images, and cannot simultaneously take into account the target saliency information in infrared and the details and texture information in visible light, resulting in insufficient feature expression after fusion and affecting detection accuracy.

[0008] ② Insufficient robustness under complex sea conditions: Existing deep learning models are prone to performance degradation when faced with wave reflections, sea fog, distant small targets, and changes in day and night lighting. They also have insufficient generalization ability and are difficult to adapt to real and complex marine environments.

[0009] ③ Unresolved conflicts in decision-level fusion: Traditional decision-level fusion often directly weights or votes on the detection results of the two modalities without considering the differences and conflicts between class confidences, which can easily lead to false alarms or missed detections, especially in scenarios where the target edge is blurry or the signal-to-noise ratio is low.

[0010] Therefore, this patent aims to propose an efficient dual-modal image fusion detection method to achieve full expression of cross-modal features, robust detection under complex sea conditions, and a more reasonable decision-level fusion mechanism, thereby improving the overall detection accuracy and robustness. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a method for detecting maritime targets based on dual-mode image fusion, which addresses the shortcomings of the existing technology.

[0012] The technical solution adopted by this invention to solve its technical problem is: This invention provides a method for detecting maritime targets based on dual-modal image fusion, constructing a three-level architecture model including a dual-modal preprocessing module, a lightweight detection module, and a decision-level fusion module. The method includes the following steps: Step 1: Acquire infrared and visible light images of the maritime target and input them into the dual-modal preprocessing module. For the infrared image, an adaptive gamma transform enhancement strategy is adopted, dynamically adjusting the gamma coefficient based on the local signal-to-noise ratio of the infrared image to enhance the contrast between the target and wave clutter. For the visible light image, during the model training phase, data augmentation methods including random scaling, cropping, and rotation are used to make the model more adaptable to targets of different scales. During the model inference phase, the visible light image is directly input, resulting in preprocessed infrared and visible light branches. Step 2: In the lightweight detection module, a dual-stream feature extraction network is adopted. A multi-scale feature pyramid network is constructed in the visible light branch, and an adaptive receptive field convolution module is integrated. By dynamically adjusting the convolution kernel dilation rate, accurate capture of ship targets at different scales is achieved. In the infrared branch, an attention-guided residual module is designed and embedded. Combined with the channel-space dual-domain attention mechanism, the saliency features of the target's thermal radiation area are enhanced, and wave clutter interference is suppressed. Both branches use the lightweight MobileNetV3 as the backbone network. The parameter amount is compressed through the channel pruning strategy, and each is input into the YOLOv8 detection head to achieve separate detection of infrared and visible light images. Step 3: In the decision-level fusion module, a three-level decision-level fusion framework is constructed, including: converting the relative coordinates of the center point of the output detection box to the absolute coordinates in the image; matching different modal decisions based on the intersection-union ratio between the infrared and visible light detection boxes; establishing a modal confidence evaluation system and dynamically adjusting the confidence weights of infrared and visible light through sea state parameters; improving the DS evidence theory synthesis rules, introducing an adaptive discount factor to handle highly conflicting evidence, and improving decision reliability through probability quality function reconstruction; and outputting the final maritime target detection results.

[0013] Furthermore, the method of employing an adaptive gamma transform enhancement strategy in step 1 of the present invention specifically includes: The infrared image is segmented locally, and the signal-to-noise ratio (SNR) of each region is calculated. The gamma coefficient is dynamically adjusted based on the local SNR to change the brightness distribution of the local image. When the local SNR is below the threshold, a gamma coefficient greater than 1 is used to darken the overall image and enhance the contrast between the target area and wave clutter. In areas where the SNR is above the threshold, a gamma coefficient less than or equal to 1 is selected.

[0014] Furthermore, the specific formula for the adaptive gamma transform enhancement strategy employed in step 1 of this invention is as follows: Suppose there are N pixels in a local region, and their grayscale value is I. i If i = 1, 2, ..., N, then the local mean and standard deviation are respectively:

[0015]

[0016] The local signal-to-noise ratio at a linear scale is defined as:

[0017] For an image pixel value I normalized to [0,1], the formula for the gamma transform is:

[0018] The γ value is adaptively set based on the local signal-to-noise ratio, I out These are the pixel values ​​of the image after gamma transformation.

[0019] Furthermore, in the lightweight detection module of step 2 of the present invention, the specific detection method for the infrared branch includes: The feature map of the input infrared branch is as follows:

[0020] Where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively; Global average pooling is performed on the feature map F extracted by the convolutional layer:

[0021] Obtain the channel description vector z∈R B×C Channel attention weights are generated using a two-layer fully connected network.

[0022] Where W1 and W2 are the weight matrices of the fully connected layer, and σ is the activation function that weights the original feature maps:

[0023] Where ⊙ represents the feature map F after element-wise multiplication and channel attention. ch It will amplify important channels related to the target's thermal radiation; In the spatial attention part, two spatial description maps are generated by performing averaging and max pooling operations on the channel-weighted feature maps:

[0024]

[0025] After concatenation, the data is input into a convolutional layer, and then passed through a sigmoid activation function to obtain the attention weights for each pixel location.

[0026]

[0027] The spatial attention weights are fused with the channel-weighted features:

[0028] Finally, by embedding the attention-guided residual module AG-ResBlock, the feature F, which has undergone dual attention enhancement of channel and space, is... att Perform residual connections with input features:

[0029] Among them, F out X represents the infrared features that are input into the YOLOv8 detector head for detection.

[0030] Furthermore, in the lightweight detection module of step 2 of the present invention, the specific detection method for visible light branches includes: The Multi-Scale Feature Pyramid Network (FPN) is used to construct a feature hierarchy from low resolution to high resolution. Let the input visible light feature map extracted by the FPN be:

[0031] Where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively; An adaptive receptive field convolution module, ARF-Conv, is integrated on top of FPN. It employs multiple convolution branches with different dilation rates and flexibly changes the receptive field size by dynamically adjusting the dilation rate of the convolution kernel, thereby extracting both local details and global contextual information simultaneously. The ARF-Conv module performs multi-branch convolutions on the input feature map in parallel. Each branch corresponds to a different dilation rate. For the i-th branch, its output is represented as:

[0032] Where K represents the convolution kernel parameter, Indicating the expansion rate The following convolution operation is performed on the input X; the outputs of each branch are aggregated, and the aggregated features are obtained by summation:

[0033] To adaptively fuse features from each branch, the module utilizes a weight generation mechanism guided by global information; global average pooling is performed on the aggregated feature U to obtain the channel description vector:

[0034] Input s into a two-layer fully connected network, activate it using ReLU, and then normalize it using Softmax to obtain the dynamic weights of each branch:

[0035] Where W1 and W2 are the weight matrices of the fully connected layer, the outputs of each branch are dynamically weighted and fused through a weight generation mechanism guided by global information to obtain the final feature representation:

[0036] in, a i The weights of the i-th branch are represented by ⊙, and element-wise multiplication is represented by ⊙. The fused output is further processed by batch normalization and ReLU activation function to form high-quality, fine-grained features that are finally input into the YOLOv8 detector head.

[0037] Where BN() represents batch normalization.

[0038] Furthermore, in the decision-level fusion module of step 3 of the present invention, the method for matching based on the intersection-union ratio specifically includes: For the output detection box of each modality, its center point is represented by relative coordinates; the relative coordinates ( x rel , y rel Convert to absolute coordinates in the image ( x abs , y abs When the image width and height are W and H respectively, the conversion formula is:

[0039] After transformation, the bounding boxes for each modality are all located in the same coordinate system; matching is performed based on the Intersection over Union (IoU) between the bounding boxes of different modalities; the IoU calculation formula is:

[0040] Here, B1 and B2 represent detection boxes from different modalities. When the IoU exceeds a set threshold, the two detection boxes are considered to have matched the same target.

[0041] Furthermore, in the decision-level fusion module of step 3 of the present invention, the method for establishing a confidence evaluation system specifically includes: By establishing a modal reliability evaluation system and dynamically adjusting the detection light weights of infrared and visible light according to sea state parameters, the position of the fused detection frame is calculated using a weighted method as follows:

[0042] Among them, B fuseB represents the position of the merged detection box. ir Indicates the position of the infrared detection frame, w ir B represents the infrared detection weight. vis Indicates the position of the visible light detection frame, w vis Indicates the visible light detection weight; In the category and confidence fusion stage, for the same matched target, infrared and visible light respectively provide preliminary category judgments and confidence scores p. ir With p vis Weighted fusion; using modal weights dynamically adjusted by sea state parameters, the weighted fusion confidence level is calculated:

[0043] Wherein, weight w ir and w vis It is dynamically calculated based on real-time sea state parameters using the function f(visibility, wave height).

[0044] Furthermore, in the decision-level fusion module of step 3 of the present invention, the implementation method of the improved DS evidence theory synthesis rule specifically includes: Assuming the probability distributions of the two modalities are m1 and m2, when there is a conflict in the detection results, the evidence with low confidence is discounted using an adaptive discount factor α:

[0045] The discount factor α is dynamically adjusted based on the current sea state and detection confidence level. The corrected evidence m'1 and m'2 are synthesized using the DS combination rule, and the fusion probability of target A is calculated using orthogonal sum rule-based evidence synthesis:

[0046] The degree of conflict K is defined as:

[0047] Here, B and C represent candidate sets in the trust distributions of two information sources for different target categories or hypotheses.

[0048] This invention provides a maritime target detection system based on dual-mode image fusion, comprising: Memory, used to store executable computer programs; The processor, when executing an executable computer program stored in memory, implements the aforementioned maritime target detection method based on dual-mode image fusion.

[0049] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for detecting maritime targets based on dual-mode image fusion.

[0050] The beneficial effects of this invention are: (1) Improved detection performance: In the scenario of dense fog at sea (visibility <50m), compared with the traditional single-mode detection method, the present invention improves the average detection accuracy (mAP@0.5) from 63.2% to 87.5%.

[0051] (2) Enhanced environmental adaptability: Through the dual-modal synergistic enhancement mechanism, the detection stability under complex sea conditions such as strong light reflection and wave interference is improved, and the false alarm rate is greatly reduced.

[0052] (3) Breakthrough in reliability integration: The use of spatiotemporal constraint evidence theory greatly improves the consistency of dual-modal detection results and effectively solves the class conflict problem in traditional methods. Attached Figure Description

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is an overall architecture diagram of the method according to an embodiment of the present invention; Figure 2 This is a diagram of the lightweight dual-stream detection network structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of decision matching according to an embodiment of the present invention; Figure 4 This is a diagram illustrating the decision-level fusion effect of marine infrared and visible light images according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] Example 1 This invention constructs a three-tier architecture system consisting of a "dual-modal preprocessing module - lightweight detection module - decision-level fusion," as follows: Figure 1 The diagram shows the overall architecture of a dual-mode image fusion method for maritime target detection. The left side shows the infrared and visible light input data streams, the middle section shows the dual-stream detection network structure, and the right side shows the decision fusion module and output results. Specifically, it includes: (1) Dual-modal preprocessing module For infrared images, an adaptive gamma transform enhancement strategy is employed, dynamically adjusting the gamma coefficients based on the local signal-to-noise ratio of the infrared image to enhance the contrast between the target and wave clutter. For visible light images, during the model training phase, data augmentation techniques such as random scaling, cropping, and rotation are used to enhance the model's adaptability to targets of different scales. Visible light images are directly input during the model inference phase.

[0056] (2) Lightweight testing module like Figure 2 As shown, a dual-stream feature extraction network was developed. A multi-scale feature pyramid network was constructed in the visible light branch, integrating an adaptive receptive field convolutional module (ARF-Conv). By dynamically adjusting the convolutional kernel dilation rate (1×1 to 5×5), accurate capture of ship targets at different scales was achieved. In the infrared branch, an attention-guided residual module (AG-ResBlock) was designed and embedded, combined with a channel-spatial dual-domain attention mechanism, to focus on enhancing the saliency features of the target's thermal radiation region and suppressing wave clutter interference. The original YOLOv8 backbone network was replaced with the more lightweight MobileNetV3 network, and the number of parameters was compressed by 42% through a channel pruning strategy, while retaining the key feature extraction capability, enabling separate detection of infrared and visible light images.

[0057] (3) Decision-level integration A three-level decision-level fusion framework is constructed: ① The relative coordinates of the center point of the output detection box are converted into absolute coordinates in the image. Then, the matching of different modal decisions is achieved based on the intersection-union ratio between different modal detection boxes; ② A modal confidence evaluation system is established, and the infrared / visible light confidence weights are dynamically adjusted through sea state parameters (visibility, wave height); ③ The DS evidence theory synthesis rules are improved, an adaptive discount factor is introduced to handle highly conflicting evidence, and the decision reliability is improved through probability quality function reconstruction.

[0058] Example 2 Based on Example 1, this invention provides specific model and algorithm design methods and formulas. Figure 1 This is an overall flowchart of a maritime target detection method based on dual-mode image fusion.

[0059] First, infrared and visible light images are processed separately by a ship target detection network to obtain detection results. During the detection result fusion stage, the bounding boxes undergo coordinate transformation, and then the bounding boxes are matched to determine whether fusion is necessary. During the fusion process, modal weights are dynamically allocated based on sea state parameters (such as visibility and wave height); for example, when visibility is <50m, the infrared weight can be set to 0.85. Subsequently, the target detection results and their corresponding weights are input into an improved DS evidence theory. The probability distribution is reconstructed using an adaptive discount factor, and the bounding boxes are matched. Finally, the fused detection results (target category, confidence level, bounding box) are output, effectively resolving the conflict between infrared and visible light detection models in category prediction.

[0060] Figure 2 This paper presents the architecture of a lightweight dual-stream detection network. The network employs a dual-stream design, processing visible light and infrared images separately. The visible light branch extracts multi-scale features using a Feature Pyramid Network (FPN) and enhances small target detection capabilities with Adaptive Receptive Field Convolution (ARF-Conv). The infrared branch strengthens target thermal radiation features and suppresses background clutter through Attention-Guided Residual Modules (AG-ResBlock). Both branches use a lightweight MobileNetV3 backbone network, optimizing computational efficiency through channel pruning. Each branch then inputs its own data into a YOLOv8 detection head, generating independent detection results and achieving efficient and accurate detection of images with different modalities.

[0061] I. Dual-modal preprocessing module For infrared images, this invention employs a gamma transform-based enhancement strategy. This strategy first performs local image segmentation and calculates the signal-to-noise ratio (SNR) for each region. The gamma coefficient is then dynamically adjusted based on the local SNR, thereby altering the brightness distribution of the local image. When the local SNR is low, a larger gamma coefficient (>1) is used to darken the overall image, thus enhancing the contrast between the target area and wave clutter. Conversely, in areas with high SNR, a smaller gamma coefficient (≈1 or <1) is selected to avoid over-enhancement. This adaptive adjustment effectively improves the distinction between the target and background in infrared images, providing clearer feature information for subsequent detection.

[0062] In image processing, the local signal-to-noise ratio (SNR) is typically defined by first calculating the mean and standard deviation of pixels within a local region. Let there be N pixels in the local region, with grayscale values ​​Ii (i=1,2,…,N). Then the local mean and standard deviation are as follows:

[0063]

[0064] Based on this, the local signal-to-noise ratio at a linear scale can be defined as:

[0065] For an image pixel value I (normalized to [0,1]), the formula for the gamma transform is:

[0066] The γ value is adaptively set based on the local signal-to-noise ratio.

[0067] II. Lightweight Detection Module In the infrared branch, to effectively highlight the thermal radiation characteristics of the target and reduce wave clutter interference, this patent designs an attention-guided residual module (AG-ResBlock). This module combines attention mechanisms in both channel and spatial dimensions. Its core idea is to assign different weights to different features, thereby making the network pay more attention to areas with strong thermal radiation signals.

[0068] Specifically, the input infrared feature map is first subjected to multiple convolutional operations, combined with batch normalization and ReLU activation function for preliminary feature extraction. Next, in the channel attention part, global average pooling is performed on each channel to extract global semantic information, and then a small fully connected network is used to generate weights for each channel. These weights reflect the importance of different channels in describing the thermal radiation characteristics of the target. By weighting the original features, the thermal information related to the target can be effectively amplified, while useless features related to wave clutter are suppressed.

[0069] Assume the input infrared feature map is:

[0070] Where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0071] First, global average pooling is performed on the feature map F extracted by the convolutional layer:

[0072] Obtain the channel description vector z∈R B×C Next, channel attention weights are generated using a two-layer fully connected network:

[0073] Where W1 and W2 are the weight matrices of the fully connected layer, and σ is the activation function (using Sigmoid). Finally, the original feature maps are weighted:

[0074] Where ⊙ represents element-wise multiplication, and the feature map F after channel attention is performed. ch It will amplify important channels related to the target's thermal radiation.

[0075] In the spatial attention part, two spatial description maps are generated by performing averaging and max pooling operations on the channel-weighted feature maps:

[0076] These two types of information are concatenated and then input into a convolutional layer, followed by a sigmoid activation function to obtain the attention weights for each pixel location:

[0077] Finally, the spatial attention weights are fused with the channel-weighted features:

[0078] This step can focus on areas in the image with significant thermal radiation signals and assign lower weights to areas with noise or background interference (such as waves), thereby further improving the discriminative power of the features.

[0079] Finally, AG-ResBlock performs a residual connection between the features enhanced by both channel and spatial attention and the input features:

[0080] This not only helps alleviate the vanishing gradient problem in deep networks, but also preserves the effective information in the original input. Through this design, the module is more accurate in extracting salient features of the target's thermal radiation region in infrared images, and to a certain extent filters out interference caused by background clutter such as waves, significantly improving the robustness and accuracy of the detection model under complex sea conditions.

[0081] In the visible light branch, a multi-scale feature pyramid network (FPN) is first used to construct a feature hierarchy from low to high resolution, enabling the network to capture the target's appearance at different scales. Let the input feature map extracted by the FPN be:

[0082] Where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively.

[0083] To further enhance the detection capability for small-scale ship targets, an Adaptive Receptive Field Convolutional Module (ARF-Conv) was integrated on top of the FPN. This module employs multiple convolutional branches with different dilation rates, ranging from 1×1 to 5×5. By dynamically adjusting the dilation rate of the convolutional kernel, the receptive field size can be flexibly changed, thereby simultaneously extracting local details and global contextual information.

[0084] The ARF-Conv module performs multi-branch convolutions on the input feature map in parallel, with each branch corresponding to a different dilation rate. This allows different branches to focus on capturing feature information at different scales. For the i-th branch, its output can be represented as:

[0085] Where K represents the convolution kernel parameter, Indicating the expansion rate Next, perform a convolution operation on the input X. Then, aggregate the outputs of each branch; this can be done by summing to obtain the aggregated features.

[0086] To adaptively fuse features from different branches, the module utilizes a weight generation mechanism guided by global information. First, global average pooling is performed on the aggregated feature U to obtain the channel description vector:

[0087] Subsequently, s is input into a two-layer fully connected network, activated by ReLU, and then normalized by Softmax to obtain the dynamic weights of each branch:

[0088] W1 and W2 are the weight matrices of the fully connected layer. Through a weight generation mechanism guided by global information, the outputs of each branch are dynamically weighted and fused to obtain the final feature representation.

[0089] Where ai represents the weight corresponding to the i-th branch (after appropriate dimensional expansion to match Fi), and ⊙ represents element-wise multiplication. The fused output is usually further processed by batch normalization and ReLU activation function to form high-quality, fine-grained features that are finally input into the YOLOv8 detector head.

[0090] This design enables the ARF-Conv module to adaptively adjust the receptive field of each branch according to the image content. Whether it is subtle local features or macroscopic global information, it can be fully reflected in the final fused feature map, thereby achieving accurate capture of ship targets at different scales.

[0091] The organic integration of FPN and ARF-Conv modules makes the visible light branch more flexible and efficient in multi-scale feature extraction. FPN is responsible for constructing hierarchical feature representations, while ARF-Conv achieves cross-scale information fusion between feature layers by dynamically adjusting the convolution kernel dilation rate, improving the detection head's responsiveness to targets at various scales. This design effectively enhances the network's robustness to targets at different scales, providing high-quality, fine-grained feature inputs for subsequent YOLOv8 detection heads, thereby improving overall detection accuracy.

[0092] III. Decision-level Fusion Module The purpose of constructing a three-level decision-making fusion framework is to fully integrate infrared and visible light detection results to improve the reliability of decision-making.

[0093] 1. Detection box intersection-union decision matching First, the YOLOv8 detector head outputs bounding boxes. For each modality's output bounding box, its center point is typically represented by relative coordinates. When converting the relative coordinates (xrel, yrel) to absolute coordinates in the image, assuming the image width and height are W and H respectively, the conversion formula is:

[0094] After transformation, the detection boxes for each modality are all located in the same coordinate system, facilitating subsequent matching. Next, matching is performed based on the Intersection over Union (IoU) ratio between the detection boxes of different modalities. The IoU calculation formula is:

[0095] B1 and B2 represent detection boxes from different modalities. When the IoU exceeds a set threshold (0.5), the two detection boxes are considered to have matched the same target, thus achieving matching of decisions from different modalities.

[0096] like Figure 3 As shown, decision matching is performed based on different crossover ratios of detection boxes.

[0097] Two modal detection boxes correspond to each other, as shown in Figure 1 and 1'. The same target is detected in both modal images, and the IOU of the detection boxes is high. The two detection boxes can be fused to obtain 1f in b. Detection boxes of one modality correspond to multiple detection boxes of another modality, as shown in Figure 2 and 2', 3'. In this case, it is necessary to calculate the IOU between 2 and 2', and 2 and 3' respectively. Detection box 2' with the higher IOU is selected and fused with 2 to obtain 2f in (b). Detection box 3' is retained in the fused image to become 3f. Detection boxes of one modality do not correspond to detection boxes of another modality, as shown in Figure 4'. 4' is retained in the fused image to become 4f.

[0098] 2. Confidence-based decision fusion After matching is completed, the matched detection boxes are fused for decision processing. At this point, by establishing a modal reliability evaluation system and dynamically adjusting the detection light weights of infrared and visible light based on sea state parameters (visibility, wave height), the positions of the fused detection boxes can be calculated using a weighted method as follows:

[0099] In this way, the detection frame fusion not only integrates the positional information of the two modes, but also reflects the reliability of each mode under different sea conditions.

[0100] In the detection category and confidence fusion stage, for the same matched target, infrared and visible light provide preliminary category judgments and confidence scores (PIR and PVIs) weighted fusion, respectively. First, the weighted fusion confidence score is calculated using the modal weights dynamically adjusted by sea state parameters.

[0101] The weights wir and wvis are dynamically calculated based on real-time sea state parameters using the function f(visibility, wave height). For example, when visibility is below 50 meters, wir is set to 0.85 and wvis to 0.15; when visibility is good, the weights can be adjusted accordingly.

[0102] 3. Improve the rules for combining evidence in the DS theory. However, due to potential significant conflicts in category determination among different modalities, direct weighting may lead to decision bias. Therefore, an improved DS evidence theory synthesis rule is introduced. The specific process is as follows: assuming the probability distributions of the two modalities are m1 and m2, when there is a high degree of conflict in the detection results, an adaptive discount factor α is used to discount and correct the low-confidence evidence.

[0103] The discount factor α is dynamically adjusted based on the current sea state and detection confidence level, so that the side with lower weight is appropriately weakened when there is significant conflict. The modified evidence m'1 and m'2 are synthesized using the DS combination rule. The fusion probability of target A is calculated using orthogonal sum rule-based evidence synthesis:

[0104] The degree of conflict K is defined as:

[0105] B and C represent candidate sets in the trust distributions of two information sources for different target categories or hypotheses. This reconstruction method based on probability quality functions effectively reduces the negative impact of highly conflicting evidence on the final decision, thereby improving the reliability of the fusion decision.

[0106] In summary, this three-level decision-level fusion framework sequentially completes the following: converting the relative coordinates of the center points of each modal detection box to absolute coordinates and achieving matching through IoU; dynamically allocating infrared / visible light confidence weights based on sea state parameters; and reconstructing the probability distribution through an adaptive discount factor using an improved DS evidence theory, ultimately achieving high-precision and robust target detection decision-making. Figure 4 The method proposed in this patent was used to detect infrared and visible light images of ships.

[0107] Figure 4 It showcases infrared and visible light images of the sea in typical scenarios, as well as the effect of decision-level fusion of the two.

[0108] Infrared image (first column): Infrared thermal imaging technology can capture the differences in thermal radiation of objects. At sea, it can highlight the difference in thermal characteristics between targets such as ships and the surrounding seawater. However, it may be affected by sea surface clutter, thermal noise, etc., and the image will show a specific gray-scale thermal distribution, which is used to identify the thermal signal characteristics of the target.

[0109] Visible light images (second column): Based on visible light images, details such as texture, color and shape of ships, sea surface, etc. can be presented, and the appearance of the target can be clearly displayed. However, in low light, fog and other environments, insufficient light or environmental interference may affect the clear identification of the target.

[0110] Decision-level fusion image (third column): This involves fusing the detection decision results from infrared and visible light images. By combining the thermal feature advantages of infrared images with the detailed texture advantages of visible light images, it is possible to identify maritime targets more accurately and comprehensively. This reduces the false detections and missed detections that may occur when using single-modal images (infrared or visible light), improves the accuracy and reliability of target detection, and provides better visual evidence for applications such as maritime target monitoring and identification.

[0111] from Figure 4 It can be seen that the method proposed in this patent can effectively detect ship targets simultaneously in infrared and visible light images, and achieve fusion at the decision level, avoiding the instability of detection results under a single modality. To further verify the effectiveness of the method, experimental comparisons were conducted on a publicly available maritime dual-modality image dataset, and the results show that: Detection accuracy: Under single-modality conditions, the average accuracy (mAP) of infrared images is 72.5%, and that of visible light images is 76.3%; however, through the dual-modality fusion method of this approach, the overall detection accuracy is improved to 83.7%.

[0112] Recall: The individual detection recall rates for infrared and visible light images were 70.2% and 73.8%, respectively, which increased to 82.4% after fusion, significantly reducing missed detections.

[0113] Improved robustness: The fusion method can still stably detect ship targets in complex scenarios such as low light, wave reflection, and distant ships, proving the model's adaptability under different environmental conditions.

[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0115] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for detecting maritime targets based on dual-mode image fusion, characterized in that, A three-level architecture model is constructed, comprising a dual-modal preprocessing module, a lightweight detection module, and a decision-level fusion module. This method includes the following steps: Step 1: Acquire infrared and visible light images of the maritime target and input them into the dual-modal preprocessing module. For the infrared image, an adaptive gamma transform enhancement strategy is adopted, dynamically adjusting the gamma coefficient based on the local signal-to-noise ratio of the infrared image to enhance the contrast between the target and wave clutter. For the visible light image, during the model training phase, data augmentation methods including random scaling, cropping, and rotation are used to make the model more adaptable to targets of different scales. During the model inference phase, the visible light image is directly input, resulting in preprocessed infrared and visible light branches. Step 2: In the lightweight detection module, a dual-stream feature extraction network is adopted. A multi-scale feature pyramid network is constructed in the visible light branch, and an adaptive receptive field convolution module is integrated. By dynamically adjusting the convolution kernel dilation rate, accurate capture of ship targets at different scales is achieved. In the infrared branch, an attention-guided residual module is designed and embedded. Combined with the channel-space dual-domain attention mechanism, the saliency features of the target's thermal radiation area are enhanced, and wave clutter interference is suppressed. Both branches use the lightweight MobileNetV3 as the backbone network. The parameter amount is compressed through the channel pruning strategy, and each is input into the YOLOv8 detection head to achieve separate detection of infrared and visible light images. Step 3: In the decision-level fusion module, a three-level decision-level fusion framework is constructed, including: converting the relative coordinates of the center point of the output detection box to the absolute coordinates in the image; matching different modal decisions based on the intersection-union ratio between the infrared and visible light detection boxes; establishing a modal confidence evaluation system and dynamically adjusting the confidence weights of infrared and visible light through sea state parameters; improving the DS evidence theory synthesis rules, introducing an adaptive discount factor to handle highly conflicting evidence, and improving decision reliability through probability quality function reconstruction; and outputting the final maritime target detection results.

2. The maritime target detection method based on dual-mode image fusion according to claim 1, characterized in that, The method of using the adaptive gamma transform enhancement strategy in step 1 specifically includes: The infrared image is segmented locally, and the signal-to-noise ratio (SNR) of each region is calculated. The gamma coefficient is dynamically adjusted based on the local SNR to change the brightness distribution of the local image. When the local SNR is below the threshold, a gamma coefficient greater than 1 is used to darken the overall image and enhance the contrast between the target area and wave clutter. In areas where the SNR is above the threshold, a gamma coefficient less than or equal to 1 is selected.

3. The maritime target detection method based on dual-mode image fusion according to claim 2, characterized in that, The specific formula for the adaptive gamma transform enhancement strategy used in step 1 is as follows: Let there be N pixels in a local region, whose gray values are I i , i = 1, 2, …, N, then the local average and standard deviation are respectively: The local signal-to-noise ratio at a linear scale is defined as: For an image pixel value I normalized to [0,1], the formula for the gamma transform is: wherein the value of γ is adaptively set according to the local signal-to-noise ratio, I out is the pixel value of the image after gamma transformation.

4. The maritime target detection method based on dual-mode image fusion according to claim 1, characterized in that, In the lightweight detection module of step 2, the specific detection method for the infrared branch includes: The feature map of the input infrared branch is as follows: Where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively; Global average pooling is performed on the feature map F extracted by the convolutional layer: A channel description vector z e R B×C ; generate channel attention weights using a two-layer fully connected network: Where W1 and W2 are the weight matrices of the fully connected layer, and σ is the activation function that weights the original feature maps: Where ⊙ represents the feature map F after element-wise multiplication and channel attention. ch It will amplify important channels related to the target's thermal radiation; In the spatial attention part, two spatial description maps are generated by performing averaging and max pooling operations on the channel-weighted feature maps: After concatenation, the data is input into a convolutional layer, and then passed through a sigmoid activation function to obtain the attention weights for each pixel location. The spatial attention weights are fused with the channel-weighted features: Finally, by embedding the attention-guided residual module AG-ResBlock, the feature F, which has undergone dual attention enhancement of channel and space, is... att Perform residual connections with input features: Among them, F out X represents the infrared features that are input into the YOLOv8 detector head for detection.

5. The maritime target detection method based on dual-mode image fusion according to claim 1, characterized in that, In the lightweight detection module of step 2, the specific detection method for visible light branches includes: The Multi-Scale Feature Pyramid Network (FPN) is used to construct a feature hierarchy from low resolution to high resolution. Let the input visible light feature map extracted by the FPN be: Where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map, respectively; An adaptive receptive field convolution module, ARF-Conv, is integrated on top of FPN. It employs multiple convolution branches with different dilation rates and flexibly changes the receptive field size by dynamically adjusting the dilation rate of the convolution kernel, thereby extracting both local details and global contextual information simultaneously. The ARF-Conv module performs multi-branch convolutions on the input feature map in parallel. Each branch corresponds to a different dilation rate. For the i-th branch, its output is represented as: Where K represents the convolution kernel parameter, Indicating the expansion rate The following convolution operation is performed on the input X; the outputs of each branch are aggregated, and the aggregated features are obtained by summation: To adaptively fuse features from each branch, the module utilizes a weight generation mechanism guided by global information; global average pooling is performed on the aggregated feature U to obtain the channel description vector: Input s into a two-layer fully connected network, activate it using ReLU, and then normalize it using Softmax to obtain the dynamic weights of each branch: Where W1 and W2 are the weight matrices of the fully connected layer, the outputs of each branch are dynamically weighted and fused through a weight generation mechanism guided by global information to obtain the final feature representation: in, a i The weights of the i-th branch are represented by ⊙, and element-wise multiplication is represented by ⊙. The fused output is further processed by batch normalization and ReLU activation function to form high-quality, fine-grained features that are finally input into the YOLOv8 detector head. Where BN() represents batch normalization.

6. The maritime target detection method based on dual-mode image fusion according to claim 1, characterized in that, In the decision-level fusion module of step 3, the matching method based on the intersection-union ratio specifically includes: For the output detection box of each modality, its center point is represented by relative coordinates; the relative coordinates ( x rel , y rel Convert to absolute coordinates in the image ( x abs , y abs When the image width and height are W and H respectively, the conversion formula is: After transformation, the bounding boxes for each modality are all located in the same coordinate system; matching is performed based on the Intersection over Union (IoU) between the bounding boxes of different modalities; the IoU calculation formula is: Here, B1 and B2 represent detection boxes from different modalities. When the IoU exceeds a set threshold, the two detection boxes are considered to have matched the same target.

7. The maritime target detection method based on dual-mode image fusion according to claim 1, characterized in that, In the decision-level fusion module of step 3, the method for establishing the confidence evaluation system specifically includes: By establishing a modal reliability evaluation system and dynamically adjusting the detection light weights of infrared and visible light according to sea state parameters, the position of the fused detection frame is calculated using a weighted method as follows: Among them, B fuse B represents the position of the merged detection box. ir Indicates the position of the infrared detection frame, w ir B represents the infrared detection weight. vis Indicates the position of the visible light detection frame, w vis Indicates the visible light detection weight; In the category and confidence fusion stage, for the same matched target, infrared and visible light respectively provide preliminary category judgments and confidence scores p. ir With p vis Weighted fusion; using modal weights dynamically adjusted by sea state parameters, the weighted fusion confidence level is calculated: Wherein, weight w ir and w vis It is dynamically calculated based on real-time sea state parameters using the function f(visibility, wave height).

8. The maritime target detection method based on dual-mode image fusion according to claim 7, characterized in that, In the decision-level fusion module of step 3, the implementation method of improving the DS evidence theory synthesis rule specifically includes: Assuming the probability distributions of the two modalities are m1 and m2, when there is a conflict in the detection results, the evidence with low confidence is discounted using an adaptive discount factor α: The discount factor α is dynamically adjusted based on the current sea state and detection confidence level. The corrected evidence m'1 and m'2 are synthesized using the DS combination rule, and the fusion probability of target A is calculated using orthogonal sum rule-based evidence synthesis: The conflict degree K is defined as: Here, B and C represent candidate sets in the trust distributions of two information sources for different target categories or hypotheses.

9. A maritime target detection system based on dual-mode image fusion, characterized in that, include: Memory, used to store executable computer programs; A processor, when executing an executable computer program stored in a memory, implements the maritime target detection method based on dual-mode image fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the maritime target detection method based on dual-mode image fusion as described in any one of claims 1 to 8.

Images