A multi-modal ship image fusion detection method for complex marine environment

Abstract

The application belongs to the technical field of ship target detection, and relates to a multi-modal ship image fusion detection method for complex marine environments, which comprises the following steps: step S1, a step of double-mode data construction and preprocessing; step S2, a step of double-flow feature extraction; step S3, a step of processing by a modal conditional local kernel modulation module; step S4, a step of processing by a scale perception residual interactive attention module; step S5, a step of environment perception modal weight self-adaptive adjustment; and step S6, a step of detection head and loss design. The application proposes an image fusion method with cross-modal alignment, multi-scale semantic compensation and environment self-adaptive weight adjustment capability to effectively improve the detection precision and robustness of ship targets in complex sea conditions, in view of the defects that cross-modal registration errors are not fully processed, multi-scale information interaction is insufficient, and modal contribution cannot be self-adaptively adjusted in different environments in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of ship target detection technology, specifically relating to a multimodal ship image fusion detection method for complex marine environments. Background Technology

[0002] In recent years, my country's shipping industry has developed rapidly. Ship target detection, as a key technology for intelligent ship perception, is crucial for maritime traffic, national defense, and search and rescue. Currently, detection mainly relies on visible light images, which are rich in detail but greatly affected by lighting conditions, making them prone to missed detections at night or in inclement weather. Infrared images, relying on thermal radiation imaging, are highly resistant to light interference, but their detail information is relatively weak. A single modality cannot adapt to all time-of-day scenarios; therefore, the fusion of visible light and infrared images has become an inevitable choice.

[0003] The significant variations in ship target size and the complex and ever-changing sea environment increase the difficulty of ship detection. Existing visible-infrared image fusion strategies have obvious limitations: pixel-level fusion is prone to introducing noise; decision-level fusion cannot fully utilize cross-modal information; while feature-level fusion is effective, it still faces problems such as detail misalignment and semantic conflicts caused by insufficient handling of cross-modal registration errors, difficulty in identifying small targets at long distances due to insufficient multi-scale information interaction, and overall performance degradation due to the inability to adaptively adjust modal contributions under different environments.

[0004] In view of this, it is very necessary to provide a multimodal ship image fusion detection method for complex marine environments to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing multimodal image fusion technologies, such as insufficient handling of cross-modal registration errors, inadequate multi-scale information interaction, and inability to adaptively adjust modal contributions under different environments. This invention provides a method for ship target detection using visible light and infrared image fusion in complex marine environments, thereby solving the aforementioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multimodal ship image fusion detection method for complex marine environments includes the following steps: Step S1, the dual-modal data acquisition and preprocessing step, in which: Acquire visible light and infrared images, preprocess the acquired visible light and infrared images, and perform data augmentation processing on the preprocessed visible light and infrared images; A ship target detection network is constructed, which includes a two-stream feature extraction network, a modality-conditional local kernel modulation module, a scale-aware residual interactive attention module, an environment-aware modality weight adaptive module, and a multi-scale detection head module. Step S2, the two-stream feature extraction step, in which: The data-augmented visible light image is input into the dual-stream feature extraction network to obtain multi-scale features of the visible light image at three scales; the data-augmented infrared image is input into the dual-stream feature extraction network to obtain multi-scale features of the infrared image at three scales. Step S3, the modality-conditional local kernel modulation module processing step, in which: The multi-scale features of visible light images and infrared images are processed by the Modal Conditional Local Kernel Modulation (MCLKM) module to perform local spatial alignment, adaptive feature modulation and feature fusion to obtain multi-scale fused features; Step S4, the step of the scale-aware residual interactive attention module processing, in which: The multi-scale fusion features are enhanced by cross-scale semantic compensation and edge enhancement through the Scale-Aware Residual Interactive Attention Module (SRCA). Step S5, the adaptive adjustment of environmental perception modal weights, in which: The Environment-Aware Modal Weight Adaptive Module (EAMW) extracts environment-aware weight coefficients that reflect the environmental state from the input image; the environment-aware weight coefficients are used to weight and adjust the multi-scale fusion features enhanced by the SRCA module to achieve environment-driven feature response adaptation and obtain multi-scale detection features after environment adjustment. Step S6, the step of designing the detection head and loss, in which: The environmentally adjusted multi-scale detection features are input into the multi-scale detection head module to complete the detection of ship targets.

[0007] Preferably, step S1 specifically includes: Acquire strictly spatially registered visible light images With infrared images As dual-modal input data, the collected image data covers a variety of complex marine environments, including day-night variations, fog and haze, strong light reflection, and sea wave disturbances, to enhance the robustness and generalization ability of the algorithm in real-world application scenarios.

[0008] Since the imaging mechanisms of visible light and infrared light are significantly different, targeted preprocessing is performed on the two modal images to improve the effectiveness of subsequent feature extraction: low-light enhancement and dehazing are applied to the visible light image to alleviate the impact of insufficient light and atmospheric scattering on the appearance of the target; histogram stretching and contrast enhancement are applied to the infrared image to enhance the thermal radiation difference between the ship target and the background.

[0009] After completing the above preprocessing operations, data augmentation operations such as geometric transformation, random cropping, rotation, and brightness perturbation are simultaneously applied to the preprocessed visible light image and infrared image to construct diverse training samples: , in, This indicates a data augmentation operation. Represents a visible light image. Represents an infrared image. This represents the data-enhanced visible light image. This represents the augmented infrared image.

[0010] Finally, the resolutions of the augmented visible light and infrared images are uniformly scaled to the input resolution required by the two-stream feature extraction network and then normalized. The normalized visible light and infrared images are used as input data for the two-stream feature extraction network.

[0011] This step can achieve the following technical effects: Data augmentation and preprocessing can enhance the adaptability of ship detection models to different lighting, weather, and sea conditions, reduce the dependence of ship detection models on specific imaging conditions, reduce the risk of overfitting, and improve their generalization performance and detection stability in real and complex marine environments.

[0012] Preferably, step S2 specifically includes: Data-enhanced visible light images With infrared images The data are input into a dual-stream feature extraction network for feature extraction, constructing visible light feature extraction branches and infrared feature extraction branches. Both branches employ convolutional neural networks to extract discriminative multi-level semantic features from their respective input data, expressed as: ,

[0013] in, This indicates a feature extraction operation. This represents the multi-level semantic features of the visible light image after feature extraction. This represents the multi-level semantic features of the infrared image after feature extraction.

[0014] During feature extraction, both branches employ consistent spatial resolution and feature level processing to ensure that features from different modalities at the same scale can be aligned. A two-stream feature extraction network is used to extract features from data-enhanced visible light images. The output shows multi-scale features of visible light images at three scales, derived from data-enhanced infrared images. The output shows multi-scale features of infrared images at three different scales.

[0015] This step can achieve the following technical effects: By using a dual-stream feature extraction network as the foundational structure for ship target detection, the visible light and infrared images maintain modal independence during the feature extraction stage, avoiding feature interference problems caused by early fusion. This dual-stream structure can also fully exploit the complementary information of the two modalities while balancing detection accuracy and computational efficiency, providing stable and reliable input features for subsequent cross-modal deep fusion, thereby improving the accuracy and robustness of ship target detection in complex marine environments.

[0016] Preferably, step S3 specifically includes: The multi-scale features of visible light images at three different scales constitute the multi-scale feature set of visible light images. The multi-scale features of infrared images at three scales constitute the multi-scale feature set of infrared images. In the multi-scale feature sets of visible light images and infrared images, the multi-scale features of visible light images and infrared images of the same scale are spatially divided into several local regions to obtain the local regions of visible light features. and infrared feature local areas , The visible light and infrared feature regions are input into the migration predictor in the modal conditional local kernel modulation module. The migration predictor predicts the cross-modal space migration within the local receptive field:

[0017] in, Indicates the cross-modal space offset. This represents the prediction function of the offset predictor. Local alignment is achieved by spatially transforming the local infrared feature region based on the cross-modal spatial offset.

[0018] in, This represents a spatial transformation operation. This represents the infrared features after local alignment. This process can compensate for local registration errors caused by sensor differences or changes in imaging conditions.

[0019] Locally aligned infrared features Local regions with visible light characteristics The concatenation and input of the dynamic kernel generation network in the modal conditional local kernel modulation module generates the conditional modulation weights. :

[0020] in, Represents a dynamic kernel generation network. This represents the splicing operation. Conditional modulation weights are used. Modulate the convolution kernel (or feature channels):

[0021] in, The symbol represents the modulated convolution kernel (or feature channel), and ⊙ represents element-wise multiplication. This represents the original convolutional kernel (or original feature channel) to be modulated.

[0022] After the above modulation, feature fusion is performed through the scale-adaptive gating structure in the modality-conditional local kernel modulation module, ultimately yielding multi-scale fused features. :

[0023] in, This represents the convolution operation. It indicates the degree of cross-modal compensation, which is dynamically adjusted by the response strength of the current scale feature (low response strength for small targets or weak features, and high response strength for large targets or clear features).

[0024] Multi-scale feature sets Each scale in the dataset is processed by the MCLKM module to obtain a multi-scale fused feature set. .

[0025] This step can achieve the following technical effects: This step, through the MCLKM module, enables the modeling of spatial misalignment and semantic differences between visible and infrared modes at the local region level, and allows for conditional modulation weights. This invention achieves adaptive modeling of the contributions of different modal information, enabling stronger cross-modal enhancement for small-scale targets while maintaining overall structural stability for large-scale targets. Compared to existing direct stitching or fixed-weighted fusion methods, this application utilizes the MCLKM module to achieve adaptive cross-modal compensation for local regions, reducing the impact of registration errors and background noise on detection results, and particularly improving the detection capability of long-distance, small-scale ship targets.

[0026] Preferably, step S4 specifically includes: The multi-scale fusion feature set output by the MCLKM module In the input scale-aware residual interactive attention module, cross-scale feature alignment is performed on the input multi-scale fused features through upsampling or downsampling operations, thereby ensuring the consistency of cross-scale features in spatial dimensions.

[0027] By utilizing the intermediate features obtained from cross-scale feature alignment, a cross-scale residual difference relationship is constructed to guide the semantic information of high-scale features to compensate for the details of low-scale features, resulting in multi-scale fusion feature semantic residuals and multi-scale fusion feature detail residuals.

[0028] To enhance the structural representation of ship targets and add edge enhancement processing, the Sobel operator (Sobel edge detection operator) is used to extract edge prior information from mid-scale features, and then fused with the semantic residuals and detail residuals of multi-scale fusion features to form structural enhancement features.

[0029] Construct a cross-scale bidirectional attention structure and generate high-scale attention weights and low-scale attention weights based on the structure enhancement features.

[0030] By combining high-scale attention weights and low-scale attention weights, the intermediate features obtained by cross-scale feature alignment are weighted and then fused with the mid-scale features through residual injection to obtain the enhanced mid-scale fused features.

[0031] The above cross-scale semantic compensation and edge enhancement operations are performed on scales P3, P4, and P5 to obtain multi-scale fusion features enhanced by the SRCA module at each scale. , , .Will , , Merged, denoted as the multi-scale fusion feature set enhanced by the SRCA module. .

[0032] This step can achieve the following technical effects: This step utilizes the SRCA module to perform residual modeling of semantic differences between features at different scales and achieves full interaction of multi-scale information through a cross-scale bidirectional attention structure. Compared to the FPN method (Feature Pyramid Network), which only performs unidirectional information transmission, the SRCA module can simultaneously enhance the semantic expressive power of low-scale features and the detail perception capability of high-scale features, and improve the clarity of ship target outlines by combining edge prior information, thereby improving the overall detection accuracy of multi-scale ship targets.

[0033] Preferably, step S5 specifically includes: An Environment-Aware Modal Weight Adaptive Module (EAMW) is employed to model the environmental state of the current scene and adaptively adjust the response intensity of features during subsequent detection processes. This module does not directly participate in the re-fusion of visible light and infrared features, but rather acts as an environment-aware control unit to adjust the overall response of the fused features, thereby avoiding irreversible modal splitting of features that have already undergone scale fusion.

[0034] The EAMW module extracts statistical features reflecting the environmental state, including illumination, from the input image data or the multi-level semantic features output by the dual-stream feature extraction network. Contrast Infrared energy intensity And environmental characteristics that characterize complex weather conditions such as fog, haze, rain, snow, or strong reflection. The statistical features reflecting the environmental state described above are extracted and represented using global average pooling, statistical analysis, or lightweight convolutional coding networks, and then combined to form an environmental feature vector. :

[0035] Environmental feature vectors Input Environment Coding Network Predicting environmental perception weight coefficients :

[0036] Weight It is used to characterize the credibility and effectiveness of fused features under the current environmental conditions, and participates in subsequent feature response control as a global adjustment factor.

[0037] Multi-scale fusion feature set enhanced by SRCA module Weighted adjustment is performed to obtain multi-scale detection features after environmental adjustment. :

[0038] This step can achieve the following technical effects: The Environment-Aware Modal Weight Adaptive (EAMW) module can model the impact of complex environmental factors such as lighting conditions, contrast distribution, infrared energy intensity, and fog, nighttime, or strong reflection on the reliability of feature representation at the overall feature level. It adaptively adjusts the response intensity of fused features through environment-aware weights, enabling the detection network to automatically highlight more discriminative feature information under different environmental conditions while suppressing invalid or noisy feature responses severely affected by environmental interference. Compared to existing detection methods that use fixed thresholds or uniform feature response intensity, this application utilizes the EAMW module to achieve dynamic modeling of feature reliability in complex marine environments, reducing the impact of lighting changes, weather interference, and strong sea surface reflection on the stability of detection results, especially improving the reliability and robustness of ship target detection under nighttime, low-contrast, and adverse weather conditions.

[0039] Preferably, step S6 specifically includes: Multi-scale detection features after environmental adjustment The improved YOLO detection head is used to detect ship targets at different scales. The detection head includes a bounding box regression branch and a class prediction branch, which output the ship target's location, class, and confidence level. A ship target detection loss function is calculated based on the improved YOLO detection method, and the parameters of the entire ship target detection network are optimized through backpropagation to learn the spatial location and class information of the ship targets.

[0040] Furthermore, to further improve detection accuracy and stability, the ship target detection network combines multi-scale feature output results and employs various constraint loss functions, including modality consistency constraint loss function, scale consistency supervision loss function, and ship edge preservation loss function, to jointly optimize the modality conditional local kernel modulation module and the scale-aware residual interactive attention module. This enables the ship target detection network to further enhance its cross-modal feature representation ability and environmental adaptability while learning target location information.

[0041] The training process of the ship target detection network involves repeating the steps of dual-stream feature extraction, modality conditional local kernel modulation module processing, scale-aware residual interactive attention module processing, environment-aware modal weight adaptive adjustment, and detection head and loss design, until the training is completed.

[0042] This step can achieve the following technical effects: By employing a multi-scale detection head module and multiple constraint loss functions, the bounding box regression, category prediction, and cross-modal consistency of ship detection can be jointly optimized. This allows the ship detection model to improve positioning accuracy and classification reliability while maintaining detection speed. By constraining scale consistency and ship edge information, the deviation between multi-scale prediction results is reduced, thereby improving the positioning accuracy of ship target boundaries.

[0043] Furthermore, this invention also provides a multimodal ship image fusion detection system for complex marine environments, comprising: The dual-modal data acquisition and preprocessing unit contains: Acquire visible light and infrared images, preprocess the acquired visible light and infrared images, and perform data augmentation processing on the preprocessed visible light and infrared images; A ship target detection network is constructed, which includes a two-stream feature extraction network, a modality-conditional local kernel modulation module, a scale-aware residual interactive attention module, an environment-aware modality weight adaptive module, and a multi-scale detection head module. A two-stream feature extraction unit, in which: The data-augmented visible light image is input into the dual-stream feature extraction network to obtain multi-scale features of the visible light image at three scales; the data-augmented infrared image is input into the dual-stream feature extraction network to obtain multi-scale features of the infrared image at three scales. The modality-conditional local kernel modulation module processing unit, in which: The multi-scale features of visible light images and infrared images are processed by the Modal Conditional Local Kernel Modulation (MCLKM) module to perform local spatial alignment, adaptive feature modulation and feature fusion to obtain multi-scale fused features; The scale-aware residual interactive attention module processing unit, in which: The multi-scale fusion features are enhanced by cross-scale semantic compensation and edge enhancement through the Scale-Aware Residual Interactive Attention Module (SRCA). The environmental perception modal weight adaptive adjustment unit, in which: The Environment-Aware Modal Weight Adaptive Module (EAMW) extracts environment-aware weight coefficients that reflect the environmental state from the input image; the environment-aware weight coefficients are used to weight and adjust the multi-scale fusion features enhanced by the SRCA module to achieve environment-driven feature response adaptation and obtain multi-scale detection features after environment adjustment. The detection head and loss design unit, in which: The environmentally adjusted multi-scale detection features are input into the multi-scale detection head module to complete the detection of ship targets.

[0044] Preferably, the dual-modal data construction and preprocessing unit specifically includes: Acquire strictly spatially registered visible light images With infrared images As dual-modal input data, the collected data covers a variety of complex marine environments, including diurnal variations, haze, strong light reflection, and sea surface wave disturbances, to enhance the robustness and generalization ability of the algorithm in real-world application scenarios.

[0045] Since the imaging mechanisms of visible light and infrared light are significantly different, targeted preprocessing is performed on the two modal images to improve the effectiveness of subsequent feature extraction: low-light enhancement and dehazing are applied to the visible light image to alleviate the impact of insufficient light and atmospheric scattering on the appearance of the target; histogram stretching and contrast enhancement are applied to the infrared image to enhance the thermal radiation difference between the ship target and the background.

[0046] After completing the above preprocessing operations, data augmentation operations such as geometric transformation, random cropping, rotation, and brightness perturbation are simultaneously applied to the preprocessed visible light image and infrared image to construct diverse training samples: , in, This indicates a data augmentation operation. Represents a visible light image. Represents an infrared image. This represents the data-enhanced visible light image. This represents the augmented infrared image.

[0047] Finally, the resolutions of the augmented visible light and infrared images are uniformly scaled to the input resolution required by the two-stream feature extraction network and then normalized. The normalized visible light and infrared images are used as input data for the two-stream feature extraction network.

[0048] This unit can achieve the following technical effects: Data augmentation and preprocessing can enhance the adaptability of ship detection models to different lighting, weather, and sea conditions, reduce the dependence of ship detection models on specific imaging conditions, reduce the risk of overfitting, and improve their generalization performance and detection stability in real and complex marine environments.

[0049] Preferably, the dual-stream feature extraction unit specifically includes: Data-enhanced visible light images With infrared images The data are input into a dual-stream feature extraction network for feature extraction, constructing visible light feature extraction branches and infrared feature extraction branches. Both branches employ convolutional neural networks, extracting discriminative multi-level semantic features from their respective input data, expressed as: ,

[0050] in, This indicates a feature extraction operation. This represents the multi-level semantic features of the visible light image after feature extraction. This represents the multi-level semantic features of the infrared image after feature extraction.

[0051] During feature extraction, both branches employ consistent spatial resolution and feature level processing to ensure that features from different modalities at the same scale can be aligned. A two-stream feature extraction network is used to extract features from data-enhanced visible light images. The output shows multi-scale features of visible light images at three scales, derived from data-enhanced infrared images. The output shows multi-scale features of infrared images at three different scales.

[0052] This unit can achieve the following technical effects: By using a dual-stream feature extraction network as the foundational structure for ship target detection, the visible light and infrared images maintain modal independence during the feature extraction stage, avoiding feature interference problems caused by early fusion. This dual-stream structure can also fully exploit the complementary information of the two modalities while balancing detection accuracy and computational efficiency, providing stable and reliable input features for subsequent cross-modal deep fusion, thereby improving the accuracy and robustness of ship target detection in complex marine environments.

[0053] Preferably, the modal conditional local kernel modulation module processing unit specifically includes: The multi-scale features of visible light images at three different scales constitute the multi-scale feature set of visible light images. The multi-scale features of infrared images at three scales constitute the multi-scale feature set of infrared images. In the multi-scale feature sets of visible light images and infrared images, the multi-scale features of visible light images and infrared images of the same scale are spatially divided into several local regions to obtain the local regions of visible light features. and infrared feature local areas , The visible light and infrared feature regions are input into the migration predictor in the modal conditional local kernel modulation module. The migration predictor predicts the cross-modal space migration within the local receptive field:

[0054] in, Indicates the cross-modal space offset. This represents the prediction function of the offset predictor. Local alignment is achieved by spatially transforming the local infrared feature region based on the cross-modal spatial offset.

[0055] in, This represents a spatial transformation operation. This represents the infrared features after local alignment. This process can compensate for local registration errors caused by sensor differences or changes in imaging conditions.

[0056] Locally aligned infrared features Local regions with visible light characteristics The concatenation and input of the dynamic kernel generation network in the modal conditional local kernel modulation module generates the conditional modulation weights. :

[0057] in, Represents a dynamic kernel generation network. This represents the splicing operation. Conditional modulation weights are used. Modulate the convolution kernel (or feature channels):

[0058] in, The symbol represents the modulated convolution kernel (or feature channel), and ⊙ represents element-wise multiplication. This represents the original convolutional kernel (or original feature channel) to be modulated.

[0059] After the above modulation, feature fusion is performed through the scale-adaptive gating structure in the modality-conditional local kernel modulation module, ultimately yielding multi-scale fused features. :

[0060] in, This represents the convolution operation. It indicates the degree of cross-modal compensation, which is dynamically adjusted by the response strength of the current scale feature (low response strength for small targets or weak features, and high response strength for large targets or clear features).

[0061] Multi-scale feature sets Each scale in the dataset is processed by the MCLKM module to obtain a multi-scale fused feature set. .

[0062] This unit can achieve the following technical effects: This unit, through the MCLKM module, can model the spatial misalignment and semantic differences between visible light modes and infrared modes at the local region level, and can conditionally modulate weights. This invention achieves adaptive modeling of the contributions of different modal information, enabling stronger cross-modal enhancement for small-scale targets while maintaining overall structural stability for large-scale targets. Compared to existing direct stitching or fixed-weighted fusion methods, this application utilizes the MCLKM module to achieve adaptive cross-modal compensation for local regions, reducing the impact of registration errors and background noise on detection results, and particularly improving the detection capability of long-distance, small-scale ship targets.

[0063] Preferably, the scale-aware residual interactive attention module processing unit specifically includes: The multi-scale fusion feature set output by the MCLKM module In the input scale-aware residual interactive attention module, upsampling or downsampling operations are used to align the multi-scale fused features across scales, thereby ensuring the consistency of features across scales in spatial dimensions.

[0064] By utilizing the intermediate features obtained from cross-scale feature alignment, a cross-scale residual difference relationship is constructed to guide the semantic information of high-scale features to compensate for the details of low-scale features, resulting in multi-scale fusion feature semantic residuals and multi-scale fusion feature detail residuals.

[0065] To enhance the structural representation of ship targets and add edge enhancement processing, the Sobel operator (Sobel edge detection operator) is used to extract edge prior information from mid-scale features, and then fused with the semantic residuals and detail residuals of multi-scale fusion features to form structural enhancement features.

[0066] Construct a cross-scale bidirectional attention structure and generate high-scale attention weights and low-scale attention weights based on the structure enhancement features.

[0067] By combining high-scale attention weights and low-scale attention weights, the intermediate features obtained by cross-scale feature alignment are weighted and then fused with the mid-scale features through residual injection to obtain the enhanced mid-scale fused features.

[0068] The above-mentioned cross-scale semantic compensation and edge enhancement operations were performed on scales P3, P4, and P5, and the multi-scale fusion features enhanced by the SRCA module could be obtained at each scale. , , .Will , , Merged, denoted as the multi-scale fusion feature set enhanced by the SRCA module. .

[0069] This unit can achieve the following technical effects: This unit uses the SRCA module to perform residual modeling of semantic differences between features at different scales and achieves full interaction of multi-scale information through a cross-scale bidirectional attention structure. Compared to the FPN method, which only performs unidirectional information transmission, the SRCA module can simultaneously enhance the semantic expressiveness of low-scale features and the detail perception capability of high-scale features, and improve the clarity of ship target outlines by combining edge prior information, thereby improving the overall detection accuracy of multi-scale ship targets.

[0070] Preferably, the environmental perception modal weight adaptive adjustment unit specifically includes: An Environment-Aware Modal Weight Adaptive Module (EAMW) is employed to model the environmental state of the current scene and adaptively adjust the response intensity of features during subsequent detection processes. This module does not directly participate in the re-fusion of visible light and infrared features, but rather acts as an environment-aware control unit to adjust the overall response of the fused features, thereby avoiding irreversible modal splitting of features that have already undergone scale fusion.

[0071] The EAMW module extracts statistical features reflecting the environmental state, including illumination, from the input image data or the multi-level semantic features output by the dual-stream feature extraction network. Contrast Infrared energy intensity And environmental characteristics that characterize complex weather conditions such as fog, haze, rain, snow, or strong reflection. The statistical features reflecting the environmental state described above are extracted and represented using global average pooling, statistical analysis, or lightweight convolutional coding networks, and then combined to form an environmental feature vector. :

[0072] Environmental feature vectors Input Environment Coding Network Predicting environmental perception weight coefficients :

[0073] Weight It is used to characterize the credibility and effectiveness of fused features under the current environmental conditions, and participates in subsequent feature response control as a global adjustment factor.

[0074] Multi-scale fusion feature set enhanced by SRCA module Weighted adjustment is performed to obtain multi-scale detection features after environmental adjustment. :

[0075] This unit can achieve the following technical effects: The Environment-Aware Modal Weight Adaptive (EAMW) module can model the impact of complex environmental factors such as lighting conditions, contrast distribution, infrared energy intensity, and fog, nighttime, or strong reflection on the reliability of feature representation at the overall feature level. It adaptively adjusts the response intensity of fused features through environment-aware weights, enabling the detection network to automatically highlight more discriminative feature information under different environmental conditions while suppressing invalid or noisy feature responses severely affected by environmental interference. Compared to existing detection methods that use fixed thresholds or uniform feature response intensity, this application utilizes the EAMW module to achieve dynamic modeling of feature reliability in complex marine environments, reducing the impact of lighting changes, weather interference, and strong sea surface reflection on the stability of detection results, especially improving the reliability and robustness of ship target detection under nighttime, low-contrast, and adverse weather conditions.

[0076] Preferably, the detection head and loss design unit specifically include: Multi-scale detection features after environmental adjustment The improved YOLO detection head is used to detect ship targets at different scales. The detection head includes a bounding box regression branch and a class prediction branch, which output the ship target's location, class, and confidence level. A ship target detection loss function is calculated based on the improved YOLO detection method, and the parameters of the entire ship target detection network are optimized through backpropagation to learn the spatial location and class information of the ship targets.

[0077] Furthermore, to further improve detection accuracy and stability, the ship target detection network combines multi-scale feature output results and employs various constraint loss functions, including modality consistency constraint loss function, scale consistency supervision loss function, and ship edge preservation loss function, to jointly optimize the modality conditional local kernel modulation module and the scale-aware residual interactive attention module. This enables the ship target detection network to further enhance its cross-modal feature representation ability and environmental adaptability while learning target location information.

[0078] The training process of the ship target detection network involves repeating the steps of dual-stream feature extraction, modality conditional local kernel modulation module processing, scale-aware residual interactive attention module processing, environment-aware modal weight adaptive adjustment, and detection head and loss design, until the training is completed.

[0079] This unit can achieve the following technical effects: By employing a multi-scale detection head module and multiple constraint loss functions, the bounding box regression, category prediction, and cross-modal consistency of ship detection can be jointly optimized. This allows the ship detection model to improve positioning accuracy and classification reliability while maintaining detection speed. By constraining scale consistency and ship edge information, the deviation between multi-scale prediction results is reduced, thereby improving the positioning accuracy of ship target boundaries.

[0080] The beneficial effects of this invention lie in its ability to fully integrate the complementary advantages of visible light and infrared images, achieving high-precision and robust ship target detection in complex marine environments. Through the Modal Conditional Local Kernel Modulation (MCLKM) module, registration errors between visible light and infrared images are dynamically compensated at the local level, resolving detail misalignment and semantic conflicts in cross-modal feature fusion and improving feature alignment accuracy. The Scale-Aware Residual Interactive Attention (SRCA) mechanism enables semantic interaction and detail enhancement between multi-scale features, improving the recognition capability and boundary positioning accuracy of long-distance, small-scale ship targets. The Environment-Aware Modal Weight Adaptive (EAMW) module adaptively adjusts the fusion feature response intensity based on illumination, contrast, and weather conditions, enhancing the stability and robustness of ship detection in complex marine environments and ensuring the algorithm's detection stability and generalization ability under all-weather and complex sea conditions. This invention improves the detection capability of multi-scale ship targets without significantly increasing computational complexity, demonstrating good engineering practicality and application promotion value.

[0081] Therefore, it is evident that the present invention has outstanding substantive features and significant progress compared with the prior art, and the beneficial effects of its implementation are also obvious. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0083] Figure 1 This is a flowchart of a multimodal ship image fusion detection method for complex marine environments provided by the present invention.

[0084] Figure 2 This is a schematic diagram of a multimodal ship image fusion detection system for complex marine environments provided by the present invention.

[0085] Figure 3 This is a general framework diagram of a multimodal ship image fusion detection method for complex marine environments provided by the present invention.

[0086] Figure 4 This is a schematic diagram of the Modal Conditioned Local Kernel Modulation Module (MCLKM) provided by the present invention.

[0087] Figure 5 This is a schematic diagram of the scale-aware residual interactive attention module (SRCA) provided by the present invention.

[0088] Figure 6 This is a structural diagram of the Environment-Aware Modal Weight Adaptive Module (EAMW) provided by the present invention.

[0089] The system comprises: 1-Dual-modal data construction and preprocessing unit; 2-Dual-stream feature extraction unit; 3-Modal conditional local kernel modulation module processing unit; 4-Scale-aware residual interactive attention module processing unit; 5-Environment-aware modal weight adaptive adjustment unit; and 6-Detection head and loss design unit. Detailed Implementation

[0090] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following implementation methods.

[0091] Example 1: like Figure 1 and Figure 3 As shown in the figure, this embodiment provides a multimodal ship image fusion detection method for complex marine environments, which includes the following steps: Step S1, the dual-modal data acquisition and preprocessing step, in which: Acquire visible light and infrared images, preprocess the acquired visible light and infrared images, and perform data augmentation processing on the preprocessed visible light and infrared images; A ship target detection network is constructed, which includes a two-stream feature extraction network, a modality-conditional local kernel modulation module, a scale-aware residual interactive attention module, an environment-aware modality weight adaptive module, and a multi-scale detection head module. Step S1 specifically includes: Acquire strictly spatially registered visible light images With infrared images As dual-modal input data, the collected data covers a variety of complex marine environments, including diurnal variations, haze, strong light reflection, and sea surface wave disturbances, to enhance the robustness and generalization ability of the algorithm in real-world application scenarios.

[0092] Since the imaging mechanisms of visible light and infrared light are significantly different, targeted preprocessing is performed on the two modal images to improve the effectiveness of subsequent feature extraction: low-light enhancement and dehazing are applied to the visible light image to alleviate the impact of insufficient light and atmospheric scattering on the appearance of the target; histogram stretching and contrast enhancement are applied to the infrared image to enhance the thermal radiation difference between the ship target and the background.

[0093] After completing the above preprocessing operations, data augmentation operations such as geometric transformation, random cropping, rotation, and brightness perturbation are simultaneously applied to the preprocessed visible light image and infrared image to construct diverse training samples: , in, This indicates a data augmentation operation. Represents a visible light image. Represents an infrared image. This represents the data-enhanced visible light image. This represents the augmented infrared image.

[0094] Finally, the resolutions of the augmented visible light and infrared images are uniformly scaled to the input resolution required by the two-stream feature extraction network and then normalized. The normalized visible light and infrared images are used as input data for the two-stream feature extraction network.

[0095] This step can achieve the following technical effects: Data augmentation and preprocessing can enhance the adaptability of ship detection models to different lighting, weather, and sea conditions, reduce the dependence of ship detection models on specific imaging conditions, reduce the risk of overfitting, and improve their generalization performance and detection stability in real and complex marine environments.

[0096] Step S2, the two-stream feature extraction step, in which: The data-augmented visible light image is input into the dual-stream feature extraction network to obtain multi-scale features of the visible light image at three scales; the data-augmented infrared image is input into the dual-stream feature extraction network to obtain multi-scale features of the infrared image at three scales. Step S2 specifically includes: Data-enhanced visible light images With infrared images The data are input into a dual-stream feature extraction network for feature extraction, constructing visible light feature extraction branches and infrared feature extraction branches. Both branches employ convolutional neural networks, extracting discriminative multi-level semantic features from their respective input data, expressed as: ,

[0097] in, This indicates a feature extraction operation. This represents the multi-level semantic features of the visible light image after feature extraction. This represents the multi-level semantic features of the infrared image after feature extraction.

[0098] During feature extraction, both branches employ consistent spatial resolution and feature level processing to ensure that features from different modalities at the same scale can be aligned. A two-stream feature extraction network is used to extract features from data-enhanced visible light images. The output shows multi-scale features of visible light images at three scales, derived from data-enhanced infrared images. The output shows multi-scale features of infrared images at three different scales.

[0099] This step can achieve the following technical effects: By using a dual-stream feature extraction network as the foundational structure for ship target detection, the visible light and infrared images maintain modal independence during the feature extraction stage, avoiding feature interference problems caused by early fusion. This dual-stream structure can also fully exploit the complementary information of the two modalities while balancing detection accuracy and computational efficiency, providing stable and reliable input features for subsequent cross-modal deep fusion, thereby improving the accuracy and robustness of ship target detection in complex marine environments.

[0100] Step S3, the modality-conditional local kernel modulation module processing step, in which: Multi-scale features of visible light images and multi-scale features of infrared images are combined using methods such as... Figure 4 The Modal Conditioned Local Kernel Modulation (MCLKM) module shown performs local spatial alignment, adaptive feature modulation, and feature fusion to obtain multi-scale fused features. Step S3 specifically includes: The multi-scale features of visible light images at three different scales constitute the multi-scale feature set of visible light images. The multi-scale features of infrared images at three scales constitute the multi-scale feature set of infrared images. In the multi-scale feature sets of visible light images and infrared images, the multi-scale features of visible light images and infrared images of the same scale are spatially divided into several local regions to obtain the local regions of visible light features. and infrared feature local areas , The visible light and infrared feature regions are input into the migration predictor in the modal conditional local kernel modulation module. The migration predictor predicts the cross-modal space migration within the local receptive field:

[0101] in, Indicates the cross-modal space offset. This represents the prediction function of the offset predictor. Local alignment is achieved by spatially transforming the local infrared feature region based on the cross-modal spatial offset.

[0102] in, This represents a spatial transformation operation. This represents the infrared features after local alignment. This process can compensate for local registration errors caused by sensor differences or changes in imaging conditions.

[0103] Locally aligned infrared features Local regions with visible light characteristics The concatenation and input of the dynamic kernel generation network in the modal conditional local kernel modulation module generates the conditional modulation weights. :

[0104] in, Represents a dynamic kernel generation network. This represents the splicing operation. Conditional modulation weights are used. Modulate the convolution kernel (or feature channels):

[0105] in, The symbol represents the modulated convolution kernel (or feature channel), and ⊙ represents element-wise multiplication. This represents the original convolutional kernel (or original feature channel) to be modulated.

[0106] After the above modulation, feature fusion is performed through the scale-adaptive gating structure in the modality-conditional local kernel modulation module, ultimately yielding multi-scale fused features. :

[0107] in, This represents the convolution operation. It indicates the degree of cross-modal compensation, which is dynamically adjusted by the response strength of the current scale feature (low response strength for small targets or weak features, and high response strength for large targets or clear features).

[0108] Multi-scale feature sets Each scale in the dataset is processed by the MCLKM module to obtain a multi-scale fused feature set. .

[0109] This step can achieve the following technical effects: This step, through the MCLKM module, enables the modeling of spatial misalignment and semantic differences between visible and infrared modes at the local region level, and allows for conditional modulation weights. This invention achieves adaptive modeling of the contributions of different modal information, enabling stronger cross-modal enhancement for small-scale targets while maintaining overall structural stability for large-scale targets. Compared to existing direct stitching or fixed-weighted fusion methods, this application utilizes the MCLKM module to achieve adaptive cross-modal compensation for local regions, reducing the impact of registration errors and background noise on detection results, and particularly improving the detection capability of long-distance, small-scale ship targets.

[0110] Step S4, the step of the scale-aware residual interactive attention module processing, in which: Through such Figure 5 The scale-aware residual interactive attention module (SRCA) shown performs cross-scale semantic compensation and edge enhancement on the multi-scale fusion features to obtain the multi-scale fusion features enhanced by the SRCA module. Step S4 specifically includes: The multi-scale fusion feature set output by the MCLKM module In the input scale-aware residual interactive attention module, upsampling or downsampling operations are used to align the input multi-scale fused features across scales, thus ensuring the consistency of features across spatial dimensions. Taking P4 scale as an example: high-scale features... Aligned to via upsampling Scale, to obtain the first alignment feature :

[0111] low-scale features Aligned to via downsampling Scale, to obtain the second alignment feature :

[0112] This ensures the consistency of cross-scale features across spatial dimensions.

[0113] By utilizing the first and second alignment features, a cross-scale residual difference relationship is constructed to guide the compensation of high-scale feature semantic information to low-scale feature details:

[0114] in, Indicates mesoscale features. This represents the semantic residual of the multi-scale fusion features from P5 scale to P4 scale. This represents the feature detail residual from the multi-scale fusion from P4 scale to P3 scale.

[0115] To enhance the structural representation of ship targets and incorporate edge enhancement processing, the Sobel operator (Sobel edge detection operator) is used to extract prior edge information from mesoscale features.

[0116] and fused with multi-scale feature semantic residuals and multi-scale fusion feature detail residual Fusion, forming structural enhancement features :

[0117] Construct a cross-scale bidirectional attention structure and enhance features based on structure. Generate high-scale attention weights and low-scale attention weights :

[0118] in, This represents a cross-scale bidirectional attention function.

[0119] The first and second alignment features are weighted by combining high-scale and low-scale attention weights, and then compared with the mesoscale features. The enhanced mesoscale fused features are obtained by fusion through residual injection. :

[0120] The above-mentioned cross-scale semantic compensation and edge enhancement operations were performed on scales P3, P4, and P5, and the multi-scale fusion features enhanced by the SRCA module could be obtained at each scale. , , .Will , , Merged, denoted as the multi-scale fusion feature set enhanced by the SRCA module. .

[0121] This step can achieve the following technical effects: This step utilizes the SRCA module to perform residual modeling of semantic differences between features at different scales and achieves full interaction of multi-scale information through a cross-scale bidirectional attention structure. Compared to the FPN method (Feature Pyramid Network), which only performs unidirectional information transmission, the SRCA module can simultaneously enhance the semantic expressive power of low-scale features and the detail perception capability of high-scale features, and improve the clarity of ship target outlines by combining edge prior information, thereby improving the overall detection accuracy of multi-scale ship targets.

[0122] Step S5, the adaptive adjustment of environmental perception modal weights, in which: like Figure 6 The Environment-Aware Modal Weight Adaptive Module (EAMW) shown dynamically adjusts the response intensity of the multi-scale fusion features enhanced by the SRCA module based on environmental conditions such as illumination intensity, contrast distribution, and weather conditions in the input image, thus obtaining multi-scale detection features that have been adaptively adjusted by the environment. Step S5 specifically includes: An Environment-Aware Modal Weight Adaptive Module (EAMW) is employed to model the environmental state of the current scene and adaptively adjust the response intensity of features during subsequent detection processes. This module does not directly participate in the re-fusion of visible light and infrared features, but rather acts as an environment-aware control unit to adjust the overall response of the fused features, thereby avoiding irreversible modal splitting of features that have already undergone scale fusion.

[0123] The EAMW module extracts statistical features reflecting the environmental state, including illumination, from the input image data or the multi-level semantic features output by the dual-stream feature extraction network. Contrast Infrared energy intensity And environmental characteristics that characterize complex weather conditions such as fog, haze, rain, snow, or strong reflection. The statistical features reflecting the environmental state described above are extracted and represented using global average pooling, statistical analysis, or lightweight convolutional coding networks, and then combined to form an environmental feature vector. :

[0124] Environmental feature vectors Input Environment Coding Network Predicting environmental perception weight coefficients :

[0125] Weight It is used to characterize the credibility and effectiveness of fused features under the current environmental conditions, and participates in subsequent feature response control as a global adjustment factor.

[0126] Enhanced multi-scale fusion feature set output by the SRCA module Weighted adjustment is performed to obtain multi-scale detection features after environmental adjustment. :

[0127] This step can achieve the following technical effects: The Environment-Aware Modal Weight Adaptive (EAMW) module can model the impact of complex environmental factors such as lighting conditions, contrast distribution, infrared energy intensity, and fog, nighttime, or strong reflection on the reliability of feature representation at the overall feature level. It adaptively adjusts the response intensity of fused features through environment-aware weights, enabling the detection network to automatically highlight more discriminative feature information under different environmental conditions while suppressing invalid or noisy feature responses severely affected by environmental interference. Compared to existing detection methods that use fixed thresholds or uniform feature response intensity, this application utilizes the EAMW module to achieve dynamic modeling of feature reliability in complex marine environments, reducing the impact of lighting changes, weather interference, and strong sea surface reflection on the stability of detection results, especially improving the reliability and robustness of ship target detection under nighttime, low-contrast, and adverse weather conditions.

[0128] Step S6, the step of designing the detection head and loss, in which: The environmentally adjusted multi-scale detection features are input into the multi-scale detection head module to complete the detection of ship targets.

[0129] Step S6 specifically includes: Multi-scale detection features after environmental adjustment The improved YOLO detection head is used to detect ship targets at different scales. The detection head includes a bounding box regression branch and a class prediction branch, which output the ship target's location, class, and confidence level. A ship target detection loss function is calculated based on the improved YOLO detection method, and the parameters of the entire ship target detection network are optimized through backpropagation to learn the spatial location and class information of the ship targets.

[0130] Furthermore, to further improve detection accuracy and stability, the ship target detection network combines multi-scale feature output results and employs various constraint loss functions, including modality consistency constraint loss function, scale consistency supervision loss function, and ship edge preservation loss function, to jointly optimize the modality conditional local kernel modulation module and the scale-aware residual interactive attention module. This enables the ship target detection network to further enhance its cross-modal feature representation ability and environmental adaptability while learning target location information.

[0131] The training process of the ship target detection network involves repeating the steps of dual-stream feature extraction, modality conditional local kernel modulation module processing, scale-aware residual interactive attention module processing, environment-aware modal weight adaptive adjustment, and detection head and loss design, until the training is completed.

[0132] This step can achieve the following technical effects: By employing a multi-scale detection head module and multiple constraint loss functions, the bounding box regression, category prediction, and cross-modal consistency of ship detection can be jointly optimized. This allows the ship detection model to improve positioning accuracy and classification reliability while maintaining detection speed. By constraining scale consistency and ship edge information, the deviation between multi-scale prediction results is reduced, thereby improving the positioning accuracy of ship target boundaries.

[0133] Example 2: like Figure 2 As shown in the figure, this embodiment provides a multimodal ship image fusion detection system for complex marine environments, including: Dual-modal data acquisition and preprocessing unit 1, in which: Acquire visible light and infrared images, preprocess the acquired visible light and infrared images, and perform data augmentation processing on the preprocessed visible light and infrared images; A ship target detection network is constructed, which includes a two-stream feature extraction network, a modality-conditional local kernel modulation module, a scale-aware residual interactive attention module, an environment-aware modality weight adaptive module, and a multi-scale detection head module. The dual-modal data construction and preprocessing unit 1 specifically includes: Acquire strictly spatially registered visible light images With infrared images As dual-modal input data, the collected data covers a variety of complex marine environments, including diurnal variations, haze, strong light reflection, and sea surface wave disturbances, to enhance the robustness and generalization ability of the algorithm in real-world application scenarios.

[0134] Since the imaging mechanisms of visible light and infrared light are significantly different, targeted preprocessing is performed on the two modal images to improve the effectiveness of subsequent feature extraction: low-light enhancement and dehazing are applied to the visible light image to alleviate the impact of insufficient light and atmospheric scattering on the appearance of the target; histogram stretching and contrast enhancement are applied to the infrared image to enhance the thermal radiation difference between the ship target and the background.

[0135] After completing the above preprocessing operations, data augmentation operations such as geometric transformation, random cropping, rotation, and brightness perturbation are simultaneously applied to the preprocessed visible light image and infrared image to construct diverse training samples: , in, This indicates a data augmentation operation. Represents a visible light image. Represents an infrared image. This represents the data-enhanced visible light image. This represents the augmented infrared image.

[0136] Finally, the resolutions of the augmented visible light and infrared images are uniformly scaled to the input resolution required by the two-stream feature extraction network and then normalized. The normalized visible light and infrared images are used as input data for the two-stream feature extraction network.

[0137] This unit can achieve the following technical effects: Data augmentation and preprocessing can enhance the adaptability of ship detection models to different lighting, weather, and sea conditions, reduce the dependence of ship detection models on specific imaging conditions, reduce the risk of overfitting, and improve their generalization performance and detection stability in real and complex marine environments.

[0138] Dual-stream feature extraction unit 2, in which: The data-augmented visible light image is input into the dual-stream feature extraction network to obtain multi-scale features of the visible light image at three scales; the data-augmented infrared image is input into the dual-stream feature extraction network to obtain multi-scale features of the infrared image at three scales. The dual-stream feature extraction unit 2 specifically includes: Data-enhanced visible light images With infrared images The data are input into a dual-stream feature extraction network for feature extraction, constructing visible light feature extraction branches and infrared feature extraction branches. Both branches employ convolutional neural networks, extracting discriminative multi-level semantic features from their respective input data, expressed as: ,

[0139] in, This indicates a feature extraction operation. This represents the multi-level semantic features of the visible light image after feature extraction. This represents the multi-level semantic features of the infrared image after feature extraction.

[0140] During feature extraction, both branches employ consistent spatial resolution and feature level processing to ensure that features from different modalities at the same scale can be aligned. A two-stream feature extraction network is used to extract features from data-enhanced visible light images. The output shows multi-scale features of visible light images at three scales, derived from data-enhanced infrared images. The output shows multi-scale features of infrared images at three different scales.

[0141] This unit can achieve the following technical effects: By using a dual-stream feature extraction network as the foundational structure for ship target detection, the visible light and infrared images maintain modal independence during the feature extraction stage, avoiding feature interference problems caused by early fusion. This dual-stream structure can also fully exploit the complementary information of the two modalities while balancing detection accuracy and computational efficiency, providing stable and reliable input features for subsequent cross-modal deep fusion, thereby improving the accuracy and robustness of ship target detection in complex marine environments.

[0142] Modal conditional local kernel modulation module processing unit 3, in which: The multi-scale features of visible light images and infrared images are processed by the Modal Conditional Local Kernel Modulation (MCLKM) module to perform local spatial alignment, adaptive feature modulation and feature fusion to obtain multi-scale fused features; The modal conditional local kernel modulation module processing unit 3 specifically includes: The multi-scale features of visible light images at three different scales constitute the multi-scale feature set of visible light images. The multi-scale features of infrared images at three scales constitute the multi-scale feature set of infrared images. In the multi-scale feature sets of visible light images and infrared images, the multi-scale features of visible light images and infrared images of the same scale are spatially divided into several local regions to obtain the local regions of visible light features. and infrared feature local areas , The visible light and infrared feature regions are input into the migration predictor in the modal conditional local kernel modulation module. The migration predictor predicts the cross-modal space migration within the local receptive field:

[0143] in, Indicates the cross-modal space offset. This represents the prediction function of the offset predictor. Local alignment is achieved by spatially transforming the local infrared feature region based on the cross-modal spatial offset.

[0144] in, This represents a spatial transformation operation. This represents the infrared features after local alignment. This process can compensate for local registration errors caused by sensor differences or changes in imaging conditions.

[0145] Locally aligned infrared features Local regions with visible light characteristics The concatenation and input of the dynamic kernel generation network in the modal conditional local kernel modulation module generates the conditional modulation weights. :

[0146] in, Represents a dynamic kernel generation network. This represents the splicing operation. Conditional modulation weights are used. Modulate the convolution kernel (or feature channels):

[0147] in, The symbol represents the modulated convolution kernel (or feature channel), and ⊙ represents element-wise multiplication. This represents the original convolutional kernel (or original feature channel) to be modulated.

[0148] After the above modulation, feature fusion is performed through the scale-adaptive gating structure in the modality-conditional local kernel modulation module, ultimately yielding multi-scale fused features. :

[0149] in, This represents the convolution operation. It indicates the degree of cross-modal compensation, which is dynamically adjusted by the response strength of the current scale feature (low response strength for small targets or weak features, and high response strength for large targets or clear features).

[0150] Multi-scale feature sets Each scale in the dataset is processed by the MCLKM module to obtain a multi-scale fused feature set. .

[0151] This unit can achieve the following technical effects: This unit, through the MCLKM module, can model the spatial misalignment and semantic differences between visible light modes and infrared modes at the local region level, and can conditionally modulate weights. This invention achieves adaptive modeling of the contributions of different modal information, enabling stronger cross-modal enhancement for small-scale targets while maintaining overall structural stability for large-scale targets. Compared to existing direct stitching or fixed-weighted fusion methods, this application utilizes the MCLKM module to achieve adaptive cross-modal compensation for local regions, reducing the impact of registration errors and background noise on detection results, and particularly improving the detection capability of long-distance, small-scale ship targets.

[0152] Scale-aware residual interactive attention module processing unit 4, in which: The multi-scale fusion features are enhanced by cross-scale semantic compensation and edge enhancement through the Scale-Aware Residual Interactive Attention Module (SRCA). The scale-aware residual interactive attention module processing unit 4 specifically includes: The multi-scale fusion feature set output by the MCLKM module In the input scale-aware residual interactive attention module, upsampling or downsampling operations are used to align the input multi-scale fused features across scales, thus ensuring the consistency of features across spatial dimensions. Taking P4 scale as an example: high-scale features... Aligned to via upsampling Scale, to obtain the first alignment feature :

[0153] low-scale features Aligned to via downsampling Scale, to obtain the second alignment feature :

[0154] This ensures the consistency of cross-scale features across spatial dimensions.

[0155] By utilizing the first and second alignment features, a cross-scale residual difference relationship is constructed to guide the compensation of high-scale feature semantic information to low-scale feature details:

[0156] in, Indicates mesoscale features. This represents the semantic residual of the multi-scale fusion features from P5 scale to P4 scale. This represents the feature detail residual from the multi-scale fusion from P4 scale to P3 scale.

[0157] To enhance the structural representation of ship targets and incorporate edge enhancement processing, the Sobel operator (Sobel edge detection operator) is used to extract prior edge information from mesoscale features.

[0158] and fused with multi-scale feature semantic residuals and multi-scale fusion feature detail residual Fusion, forming structural enhancement features :

[0159] Construct a cross-scale bidirectional attention structure and enhance features based on structure. Generate high-scale attention weights and low-scale attention weights :

[0160] in, This represents a cross-scale bidirectional attention function.

[0161] The first and second alignment features are weighted by combining high-scale and low-scale attention weights, and then compared with the mesoscale features. The enhanced mesoscale fused features are obtained by fusion through residual injection. :

[0162] The above-mentioned cross-scale semantic compensation and edge enhancement operations were performed on scales P3, P4, and P5, and the multi-scale fusion features enhanced by the SRCA module could be obtained at each scale. , , .Will , , Merged, denoted as the multi-scale fusion feature set enhanced by the SRCA module. .

[0163] This unit can achieve the following technical effects: This unit uses the SRCA module to perform residual modeling of semantic differences between features at different scales and achieves full interaction of multi-scale information through a cross-scale bidirectional attention structure. Compared to the FPN method, which only performs unidirectional information transmission, the SRCA module can simultaneously enhance the semantic expressiveness of low-scale features and the detail perception capability of high-scale features, and improve the clarity of ship target outlines by combining edge prior information, thereby improving the overall detection accuracy of multi-scale ship targets.

[0164] Environmental perception modal weight adaptive adjustment unit 5, in which: The Environment-Aware Modal Weight Adaptive Module (EAMW) dynamically adjusts the response intensity of the multi-scale fusion features enhanced by the SRCA module based on environmental conditions such as illumination intensity, contrast distribution, and weather conditions in the input image, thus obtaining multi-scale detection features that have been adaptively adjusted by the environment. The aforementioned environmental perception modal weight adaptive adjustment unit 5 specifically includes: An Environment-Aware Modal Weight Adaptive Module (EAMW) is employed to model the environmental state of the current scene and adaptively adjust the response intensity of features during subsequent detection processes. This module does not directly participate in the re-fusion of visible light and infrared features, but rather acts as an environment-aware control unit to adjust the overall response of the fused features, thereby avoiding irreversible modal splitting of features that have already undergone scale fusion.

[0165] The EAMW module extracts statistical features reflecting the environmental state, including illumination, from the input image data or the multi-level semantic features output by the dual-stream feature extraction network. Contrast Infrared energy intensity And environmental characteristics that characterize complex weather conditions such as fog, haze, rain, snow, or strong reflection. The statistical features reflecting the environmental state described above are extracted and represented using global average pooling, statistical analysis, or lightweight convolutional coding networks, and then combined to form an environmental feature vector. :

[0166] Environmental feature vectors Input Environment Coding Network Predicting environmental perception weight coefficients :

[0167] Weight It is used to characterize the credibility and effectiveness of fused features under the current environmental conditions, and participates in subsequent feature response control as a global adjustment factor.

[0168] Multi-scale fusion feature set enhanced by SRCA module Weighted adjustment is performed to obtain multi-scale detection features after environmental adjustment. :

[0169] This unit can achieve the following technical effects: The Environment-Aware Modal Weight Adaptive (EAMW) module can model the impact of complex environmental factors such as lighting conditions, contrast distribution, infrared energy intensity, and fog, nighttime, or strong reflection on the reliability of feature representation at the overall feature level. It adaptively adjusts the response intensity of fused features through environment-aware weights, enabling the detection network to automatically highlight more discriminative feature information under different environmental conditions while suppressing invalid or noisy feature responses severely affected by environmental interference. Compared to existing detection methods that use fixed thresholds or uniform feature response intensity, this application utilizes the EAMW module to achieve dynamic modeling of feature reliability in complex marine environments, reducing the impact of lighting changes, weather interference, and strong sea surface reflection on the stability of detection results, especially improving the reliability and robustness of ship target detection under nighttime, low-contrast, and adverse weather conditions.

[0170] Detection head and loss design unit 6, in which: The environmentally adjusted multi-scale detection features are input into the multi-scale detection head module to complete the detection of ship targets.

[0171] The detection head and loss design unit 6 specifically include: Multi-scale detection features after environmental adjustment The improved YOLO detection head is used to detect ship targets at different scales. The detection head includes a bounding box regression branch and a class prediction branch, which output the ship target's location, class, and confidence level. A ship target detection loss function is calculated based on the improved YOLO detection method, and the parameters of the entire ship target detection network are optimized through backpropagation to learn the spatial location and class information of the ship targets.

[0172] Furthermore, to further improve detection accuracy and stability, the ship target detection network combines multi-scale feature output results and employs various constraint loss functions, including modality consistency constraint loss function, scale consistency supervision loss function, and ship edge preservation loss function, to jointly optimize the modality conditional local kernel modulation module and the scale-aware residual interactive attention module. This enables the ship target detection network to further enhance its cross-modal feature representation ability and environmental adaptability while learning target location information.

[0173] The training process of the ship target detection network involves repeating the steps of dual-stream feature extraction, modality conditional local kernel modulation module processing, scale-aware residual interactive attention module processing, environment-aware modal weight adaptive adjustment, and detection head and loss design, until the training is completed.

[0174] This unit can achieve the following technical effects: By employing a multi-scale detection head module and multiple constraint loss functions, the bounding box regression, category prediction, and cross-modal consistency of ship detection can be jointly optimized. This allows the ship detection model to improve positioning accuracy and classification reliability while maintaining detection speed. By constraining scale consistency and ship edge information, the deviation between multi-scale prediction results is reduced, thereby improving the positioning accuracy of ship target boundaries.

[0175] The above-disclosed embodiments are merely preferred embodiments of the present invention, but the present invention is not limited thereto. Any non-creative variations that can be conceived by those skilled in the art, as well as any improvements and modifications made without departing from the principles of the present invention, should fall within the protection scope of the present invention.

Claims

1. A multimodal ship image fusion detection method for complex marine environments, characterized in that, Includes the following steps: Step S1, the dual-modal data acquisition and preprocessing step, in which: Acquire visible light and infrared images, preprocess the acquired visible light and infrared images, and perform data augmentation processing on the preprocessed visible light and infrared images; A ship target detection network is constructed, which includes a two-stream feature extraction network, a modality-conditional local kernel modulation module, a scale-aware residual interactive attention module, an environment-aware modality weight adaptive module, and a multi-scale detection head module. Step S2, the two-stream feature extraction step, in which: The data-augmented visible light image is input into the dual-stream feature extraction network to obtain multi-scale features of the visible light image at three scales; the data-augmented infrared image is input into the dual-stream feature extraction network to obtain multi-scale features of the infrared image at three scales. Step S3, the modality-conditional local kernel modulation module processing step, in which: The multi-scale features of visible light images and infrared images are processed by the modality-conditional local kernel modulation module to perform local spatial alignment, adaptive feature modulation and feature fusion to obtain multi-scale fused features. Step S4, the step of the scale-aware residual interactive attention module processing, in which: By performing cross-scale semantic compensation and edge enhancement on the multi-scale fusion features through the scale-aware residual interactive attention module, the multi-scale fusion features enhanced by the scale-aware residual interactive attention module are obtained. Step S5, the adaptive adjustment of environmental perception modal weights, in which: The environment-aware modal weight adaptive module dynamically adjusts the response intensity of the multi-scale fusion features enhanced by the scale-aware residual interactive attention module based on environmental conditions such as light intensity, contrast distribution, and weather conditions in the input image, thus obtaining multi-scale detection features that have been adaptively adjusted by the environment. Step S6, the step of designing the detection head and loss, in which: The environmentally adjusted multi-scale detection features are input into the multi-scale detection head module to complete the detection of ship targets.

2. The multimodal ship image fusion detection method for complex marine environments according to claim 1, characterized in that, Step S1 specifically includes: Acquire strictly spatially registered visible light images With infrared images As dual-modal input data, the collected image data covers a variety of complex marine environments, including diurnal variations, haze, strong light reflection, and sea surface wave disturbances. Preprocessing is performed on visible light images with low-light enhancement and dehazing; preprocessing is performed on infrared images with histogram stretching and contrast enhancement. Data augmentation operations that simultaneously apply geometric transformations, random cropping, rotation, and brightness perturbation to preprocessed visible light and infrared images: , in, This indicates a data augmentation operation. Represents a visible light image. Represents an infrared image. This represents the data-enhanced visible light image. This represents the augmented infrared image. The resolutions of the augmented visible light and infrared images are uniformly scaled to the input resolution required by the dual-stream feature extraction network and then normalized.

3. The multimodal ship image fusion detection method for complex marine environments according to claim 1, characterized in that, Step S2 specifically includes: Data-enhanced visible light images With infrared images The samples are fed into a dual-stream feature extraction network for feature extraction, and visible light feature extraction branch and infrared feature extraction branch are constructed. Both the visible light feature extraction branch and the infrared feature extraction branch use convolutional neural networks to extract discriminative multi-level semantic features from their respective input data. The expressions are as follows: , in, This indicates a feature extraction operation. This represents the multi-level semantic features of the visible light image after feature extraction. This represents the multi-level semantic features of the infrared image after feature extraction; Using a two-stream feature extraction network, from data-enhanced visible light images The output shows multi-scale features of visible light images at three scales, derived from data-enhanced infrared images. The output shows multi-scale features of infrared images at three different scales.

4. The multimodal ship image fusion detection method for complex marine environments according to claim 3, characterized in that, The feature extraction process in step S2 also includes: The visible light feature extraction branch and the infrared feature extraction branch have the same spatial resolution processing and feature level processing.

5. The multimodal ship image fusion detection method for complex marine environments according to claim 1, characterized in that, Step S3 specifically includes: The multi-scale features of visible light images at three different scales constitute the multi-scale feature set of visible light images. The multi-scale features of infrared images at three scales constitute the multi-scale feature set of infrared images. ; In the multi-scale feature sets of visible light images and infrared images, the multi-scale features of visible light images and infrared images of the same scale are divided into several local regions in spatial dimension to obtain the local regions of visible light features. and infrared feature local areas , ; The visible light feature local region and the infrared feature local region are input into the offset predictor in the modal conditional local kernel modulation module; The migration predictor predicts cross-modal space migrations within the local receptive field: in, Indicates the cross-modal space offset. This represents the offset predictor's prediction function; it performs spatial transformation on local infrared feature regions based on cross-modal spatial offsets to achieve local alignment. in, This represents a spatial transformation operation. Indicates the infrared features after local alignment; Locally aligned infrared features Local regions with visible light characteristics The data is then spliced ​​and input into the dynamic kernel generation network in the modal conditional local kernel modulation module. The dynamic kernel generation network generates conditional modulation weights. : in, Represents a dynamic kernel generation network. Indicates a splicing operation; Utilizing conditional modulation weights Modulate the convolution kernel: in, This represents the modulated convolution kernel, and ⊙ represents element-wise multiplication. This represents the original convolutional kernel to be modulated; Feature fusion is performed using a scale-adaptive gating structure in the modality-conditional local kernel modulation module, ultimately yielding multi-scale fused features. : in, This represents the convolution operation. It indicates the degree of cross-modal compensation, which is dynamically adjusted by the response intensity of the current scale feature; Multi-scale feature sets Each scale in the dataset is processed by the modality-conditional local kernel modulation module to obtain a multi-scale fused feature set. .

6. The multimodal ship image fusion detection method for complex marine environments according to claim 1, characterized in that, Step S4 specifically includes: The multi-scale fusion feature set output by the modality-conditional local kernel modulation module In the input scale-aware residual interactive attention module, cross-scale feature alignment is performed on the input multi-scale fusion features through upsampling or downsampling operations; By utilizing the intermediate features obtained from cross-scale feature alignment, a cross-scale residual difference relationship is constructed to guide the semantic information of high-scale features to compensate for the details of low-scale features, thereby obtaining the semantic residual of multi-scale fusion features and the detail residual of multi-scale fusion features. To enhance edge processing, the Sobel operator is used to extract edge prior information from mid-scale features and fuse it with the semantic residuals of multi-scale fusion features and the detail residuals of multi-scale fusion features to form structural enhancement features. Construct a cross-scale bidirectional attention structure and generate high-scale attention weights and low-scale attention weights based on the structure enhancement features; By combining high-scale attention weights and low-scale attention weights, the intermediate features obtained by cross-scale feature alignment are weighted and fused with the mid-scale features through residual injection to obtain the enhanced mid-scale fused features. The above-mentioned cross-scale semantic compensation and edge enhancement operations are performed on scales P3, P4, and P5, resulting in multi-scale fusion features enhanced by the scale-aware residual interactive attention module at each scale. , , ; Will , , The merged feature set is denoted as the multi-scale fusion feature set enhanced by the scale-aware residual interactive attention module. .

7. The multimodal ship image fusion detection method for complex marine environments according to claim 1, characterized in that, Step S5 specifically includes: The environmental perception modality weight adaptive module extracts statistical features reflecting the environmental state from the input image data or the multi-level semantic features output by the dual-stream feature extraction network. Statistical features reflecting environmental conditions are extracted and represented using global average pooling, statistical analysis, or lightweight convolutional coding networks, and then combined to form an environmental feature vector. ; Environmental feature vectors Input Environment Coding Network Predicting environmental perception weight coefficients : Multi-scale fusion feature set enhanced by scale-aware residual interactive attention module Weighted adjustment is performed to obtain multi-scale detection features after environmental adjustment. : 。 8. The multimodal ship image fusion detection method for complex marine environments according to claim 7, characterized in that, The statistical characteristics reflecting the environmental state in step S5 include: Illuminance, contrast, infrared energy intensity, and environmental characteristics that characterize complex weather conditions such as fog, rain, snow, or strong reflection.

9. The multimodal ship image fusion detection method for complex marine environments according to claim 1, characterized in that, Step S6 specifically includes: Multiscale characteristics after environmental adjustment Inputting the improved YOLO series detection head, ship target detection is performed at different scales; The detection head includes: a bounding box regression branch and a category prediction branch; The ship target detection loss function is calculated based on the improved YOLO series detection method, and the parameters of the entire ship target detection network are optimized by backpropagation.

10. A multimodal ship image fusion detection method for complex marine environments according to claim 9, characterized in that, Step S6 further includes: The ship target detection network combines multi-scale feature output results and employs various constraint loss functions, including modality consistency constraint loss function, scale consistency supervision loss function, and ship edge preservation loss function, to jointly optimize the modality conditional local kernel modulation module and the scale-aware residual interactive attention module. The training process of the ship target detection network involves repeating the steps of dual-stream feature extraction, modality conditional local kernel modulation module processing, scale-aware residual interactive attention module processing, environment-aware modal weight adaptive adjustment, and detection head and loss design, until the training is completed.

Images