Meteorological environment perception-based sea surface target multi-modal image adaptive fusion method

Abstract

The invention belongs to the technical field of computer image processing, and particularly relates to a sea surface target multi-modal image adaptive fusion method based on meteorological environment perception, which comprises the following steps: step 1, synchronously acquiring visible light, infrared and radar images and real-time meteorological data; 2, constructing an influence evaluation model of meteorological factors on the quality of each modal image; 3, dynamically distributing fusion weights based on an evaluation result and carrying out image preprocessing; step 4, realizing adaptive fusion guided by environmental correction factors at a feature level; and step 5, outputting a fused image and supporting closed-loop feedback optimization. According to the method, meteorological data is introduced into an image fusion process as an active regulation factor, the limitation of traditional static fusion is broken through, and the method has remarkable innovativeness.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology, specifically relating to an adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception. More specifically, this invention proposes a method and system for adaptive fusion of multimodal sea surface images, including visible light, infrared, and millimeter-wave radar images, by introducing real-time meteorological environment data as prior knowledge into the image fusion process. This method aims to improve the detection, identification, and tracking capabilities of sea surface targets such as ships and floating objects under adverse weather conditions. Background Technology

[0002] Multi-source image fusion is the process of combining image information from different sensors on the same scene into a more informative and robust image using a specific algorithm. By integrating complementary information from multiple images, the clarity, identifiability, and usability of the fused image can be significantly improved. Currently, sea surface target image fusion mainly relies on multi-sensor collaborative observation, and common technical solutions include: 1. Image fusion based on wavelet transform or multi-scale decomposition 1) Technical principle: Multi-scale decomposition of multimodal images is performed using discrete wavelet transform (DWT), non-subsampled contour wave transform (NSCT) or Laplacian pyramid. Coefficients are fused in different frequency bands using weighted averaging or maximum value selection strategies, and then the fused image is reconstructed through inverse transform.

[0003] 2) Representative application: Fusion of infrared and visible light images for nighttime target enhancement.

[0004] 3) Limitations: 2. End-to-end image fusion network based on deep learning 1) Technical principle: Using convolutional neural network (CNN), generative adversarial network (GAN) or Transformer architecture, the model is trained to directly output fused image from multimodal input.

[0005] 2) Representative models: FusionGAN, IFCNN, U²Fusion, etc.

[0006] 3) Limitations: The model has poor generalization ability and its performance drops sharply in extreme weather scenarios (such as heavy rainfall and high waves) not covered in the training; it lacks interpretability and is difficult to optimize for specific environments.

[0007] 3. Traditional methods based on feature-level fusion 1) Technical principle: Extract the edge, texture, saliency and other features of each modality image, perform registration and then fuse them to finally generate a fused image.

[0008] 2) Commonly used algorithms: SIFT / ORB feature matching + polynomial fitting + weighted fusion.

[0009] 3) Limitations: Feature extraction is susceptible to weather interference (such as fog causing blurred edges), and the registration failure rate is high; the systematic impact of environmental factors on sensor performance is not considered.

[0010] 4. Radar and optical image fusion 1) Technical Principle: Utilizing the all-weather, all-time, and cloud-penetrating capabilities of synthetic aperture radar (SAR) or millimeter-wave radar, and fusing it with visible light / infrared images, all-weather observation capabilities are enhanced. Optical and SAR images are processed independently (e.g., target detection, classification) to obtain their respective decision results (e.g., target location, category probability), and then a final decision is made through certain strategies (e.g., voting, weighted, Bayesian, DS evidence theory).

[0011] 2) Commonly used algorithms: weighted fusion, Bayesian inference, and DS evidence theory.

[0012] 3) Limitations: Optical imaging is passive reflection imaging, while SAR is microwave imaging. The two have significant differences in data distribution, contrast, and texture features, making feature alignment and fusion difficult. Furthermore, radar images have speckle noise and geometric distortion, and their resolution differs greatly from that of optical images. Traditional fusion methods struggle to balance detail preservation and noise suppression.

[0013] 5. Summary of common defects in existing technologies Summary of the Invention

[0014] (a) Technical problems to be solved The technical problem to be solved by this invention is: Existing marine target image fusion technologies generally suffer from the problem of "emphasizing images while neglecting the environment," failing to fully utilize external meteorological information to guide fusion decisions. This results in unstable fusion effects and high target miss rates under complex marine meteorological conditions.

[0015] This invention aims to solve the following key technical problems: 1. How to quantify the impact of meteorological environment on the imaging quality of different modal sensors? Existing methods lack modeling of physical laws such as "fog affecting visible light", "rain attenuating radar", and "temperature difference enhancing infrared".

[0016] 2. How to dynamically adjust the fusion strategy based on real-time meteorological data? An adaptive mechanism is needed to enable the fusion system to have "environmental awareness and response" capabilities.

[0017] 3. How to maintain effective detection of sea surface targets under adverse weather conditions? Especially when visibility is low and sea clutter is strong, it is necessary to enhance dominant modes (such as infrared and radar) and suppress inferior modes (such as blurred visible light).

[0018] 4. How to improve the interpretability and engineering practicality of integrated systems? Integrated decision-making should be transparent, traceable, and facilitate system debugging and deployment.

[0019] (II) Technical Solution To address the aforementioned technical problems, this invention provides an adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception, the method comprising: Step 1: Simultaneously acquire visible light, infrared, and radar images, as well as real-time meteorological data; Step 2: Construct an assessment model for the impact of meteorological factors on the image quality of each modality; Step 3: Dynamically assign fusion weights based on the evaluation results and perform image preprocessing; Step 4: Implement adaptive fusion guided by environmental correction factors at the feature level; Step 5: Output the fused image and support closed-loop feedback optimization.

[0020] In step 1, the meteorological data includes visibility, humidity, rainfall intensity, wind speed, and sea state level.

[0021] Step 1 includes: Step 1.1: Acquire visible light images of the same spatiotemporal region. Infrared images Millimeter-wave radar images ; Step 1.2: Synchronously collect meteorological data: visibility V relative humidity RH Rainfall intensity R Wind speed W Sea state rating S .

[0022] In step 2, the impact assessment model uses empirical formulas or machine learning regression models to output quality scores for each modality of the image.

[0023] Step 2 includes: Step 2.1, Visible light image quality scoring: ,like Where, e is the base of the natural logarithm, approximately equal to 2.71828; k1 is an empirical coefficient for adjusting the effect of humidity on image quality, used to adjust the degree of influence of relative humidity (RH) on image quality. The larger the value of k1, the stronger the attenuation effect of ambient humidity on image quality. The higher the relative humidity (RH), the lower the contrast and the blurrier the image; k2 is an empirical coefficient for adjusting the effect of rainfall on image quality; rainfall will cause background blur and reduce image usability; (1 R) indicates that the stronger the rainfall (the larger R is), the lower the percentage of remaining mass. Step 2.2, Infrared Image Quality Scoring: ,in The target is the temperature difference between the sea surface and the target. Wind speed; Step 2.3, Radar image signal-to-noise ratio: Heavy rainfall and high waves reduce the signal-to-noise ratio.

[0024] In step 3, the fusion weight is calculated based on the quality score using a power-law function normalization.

[0025] In step 3, according to , , Calculate the initial fusion weights for the i-th mode, i.e., visible light, infrared, and radar. This is used to represent the relative importance of the mode in multi-source fusion. , where i∈vis,ir,radar, is the sensitivity parameter in vis,ir,radar.

[0026] In step 3, targeted enhancement is performed on low-quality images. When the defogging algorithm is enabled, This is a visible light image quality threshold used to distinguish between "usable" and "needs enhancement" image quality boundaries; when When this is the case, enable speckle noise filtering. This is the signal-to-noise ratio threshold for radar images, used to identify low signal-to-noise ratio scenes that require preprocessing.

[0027] Step 4 includes: Step 4.1: Extract the saliency feature map from the i-th modality image. , used to reflect the importance of the mode in a specific region, where i∈vis,ir,radar, is the sensitivity parameter in vis,ir,radar; Step 4.2: Introduce environmental correction factors Dynamically adjust feature fusion weights: , in, To dynamically adjust the feature fusion weights, the fusion weights of each modality are dynamically adjusted based on its performance relative to the average level, highlighting the information contribution of high-quality modalities. Let be the initial fusion weights for the i-th mode; Average quality score calculated for all modes (vis,ir,radar) This is an adjustment coefficient used to control the intensity of environmental factors in adjusting the weights; The higher the value, the more significant the environmental impact; Step 4.3: Combine the adjusted weights of each modality with its saliency feature map, perform a weighted summation, and generate the final fused feature map: ; Step 4.4: The final fused image obtained by reconstructing based on the fused feature map. This preserves the advantages of each modality.

[0028] In step 5, the fusion result output and feedback optimization are performed. Output fused image And perform target detection; The detection accuracy and false alarm rate are fed back into the environmental impact model to continuously optimize the visible light image quality scoring function. Infrared image quality scoring function and radar image signal-to-noise ratio function The function parameters (such as coefficients and thresholds) are used to more accurately reflect the impact of the real environment on image quality, achieve closed-loop optimization, and improve the effect of future fusion tasks.

[0029] (III) Beneficial Effects Compared with the prior art, the advantages of this invention are: 1. Pioneering the concept of "meteorological environment synergy": For the first time, meteorological data is introduced as an active control factor into the image fusion process, breaking through the limitations of traditional "static fusion" and demonstrating significant innovation.

[0030] 2. Highly adaptive fusion strategy: It can dynamically adjust the weights and processing methods of each mode according to the real-time environment, and maintain stable performance even under complex sea conditions such as fog, rain, wind and waves.

[0031] 3. Improve target detection robustness: Automatically enhance infrared / radar modes under low visibility conditions and retain visible light details under clear weather conditions, significantly reducing the target false detection rate.

[0032] 4. High interpretability: The fusion decision is based on clear physical laws (such as fog attenuating visible light), the process is transparent, and it is easy to deploy and optimize in engineering.

[0033] 5. System closed-loop evolution: Supports feedback mechanism, which can continuously calibrate the environmental impact model based on actual application effects, and improve long-term accuracy.

[0034] 6. High application value: It is particularly suitable for scenarios with high requirements for all-weather observation capabilities, such as maritime security, search and rescue, and waterway monitoring. Attached Figure Description

[0035] The accompanying drawings of this invention illustrate the connection relationships and data flow between the meteorological sensing module, multimodal acquisition, impact assessment, adaptive fusion engine, and output module.

[0036] Figure 1 Flowchart of an adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception.

[0037] Figure 2 Schematic diagram of the impact model on meteorological sensor performance.

[0038] Figure 3 Adaptive fusion weight dynamic adjustment flowchart: The system assesses the image quality of each modality based on real-time meteorological data, automatically enhances low-quality data, dynamically calculates fusion weights based on quality, and incorporates environmental correction factors to achieve more robust feature-level fusion for target detection. Feedback is used to continuously optimize system performance. The flowchart illustrates the entire process: "Meteorological input → Quality assessment → Weight calculation → Image enhancement → Feature fusion → Output".

[0039] Figure 4 : Comparison diagram of fusion results: The images are presented side-by-side with those of the present invention in scenarios such as dense fog and rain, highlighting the advantages of the present invention in terms of target sharpness and background noise suppression.

[0040] Figure 5 Schematic diagram of the "environmental perception-fusion-feedback" closed-loop system: Data is collected from meteorological sensors and multi-source image sensors, fused and processed to output results, and then fed back to the environmental impact model to realize a closed-loop mechanism for system self-optimization. Detailed Implementation

[0041] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0042] To address the aforementioned technical problems, this invention provides an adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception, the method comprising: Step 1: Simultaneously acquire visible light, infrared, and radar images, as well as real-time meteorological data; Step 2: Construct an assessment model for the impact of meteorological factors on the image quality of each modality; Step 3: Dynamically assign fusion weights based on the evaluation results and perform image preprocessing; Step 4: Implement adaptive fusion guided by environmental correction factors at the feature level; Step 5: Output the fused image and support closed-loop feedback optimization.

[0043] In step 1, the meteorological data includes visibility, humidity, rainfall intensity, wind speed, and sea state level.

[0044] Step 1 includes: Step 1.1: Acquire visible light images of the same spatiotemporal region. Infrared images Millimeter-wave radar images ; Step 1.2: Synchronously collect meteorological data: visibility V relative humidity RH Rainfall intensity R Wind speed W Sea state rating S .

[0045] In step 2, the impact assessment model uses empirical formulas or machine learning regression models to output quality scores for each modality of the image.

[0046] Step 2 includes: Step 2.1, Visible light image quality scoring: ,like Where, e is the base of the natural logarithm, approximately equal to 2.71828; k1 is an empirical coefficient for adjusting the effect of humidity on image quality, used to adjust the degree of influence of relative humidity (RH) on image quality. The larger the value of k1, the stronger the attenuation effect of ambient humidity on image quality. The higher the relative humidity (RH), the lower the contrast and the blurrier the image; k2 is an empirical coefficient for adjusting the effect of rainfall on image quality; rainfall will cause background blur and reduce image usability; (1 R) indicates that the stronger the rainfall (the larger R is), the lower the percentage of remaining mass. Step 2.2, Infrared Image Quality Scoring: ,in The target is the temperature difference between the sea surface and the target. Wind speed; Step 2.3, Radar image signal-to-noise ratio: Heavy rainfall and high waves reduce the signal-to-noise ratio.

[0047] In step 3, the fusion weight is calculated based on the quality score using a power-law function normalization.

[0048] In step 3, according to , , Calculate the initial fusion weights for the i-th mode, i.e., visible light, infrared, and radar. This is used to represent the relative importance of the mode in multi-source fusion. , where i∈vis,ir,radar, is the sensitivity parameter in vis,ir,radar.

[0049] In step 3, targeted enhancement is performed on low-quality images. When the defogging algorithm is enabled, This is a visible light image quality threshold used to distinguish between "usable" and "needs enhancement" image quality boundaries; when When this is the case, enable speckle noise filtering. This is the signal-to-noise ratio threshold for radar images, used to identify low signal-to-noise ratio scenes that require preprocessing.

[0050] Step 4 includes: Step 4.1: Extract the saliency feature map from the i-th modality image. , used to reflect the importance of the mode in a specific region, where i∈vis,ir,radar, is the sensitivity parameter in vis,ir,radar; Step 4.2: Introduce environmental correction factors Dynamically adjust feature fusion weights: , in, To dynamically adjust the feature fusion weights, the fusion weights of each modality are dynamically adjusted based on its performance relative to the average level, highlighting the information contribution of high-quality modalities. Let be the initial fusion weights for the i-th mode; Average quality score calculated for all modes (vis,ir,radar) This is an adjustment coefficient used to control the intensity of environmental factors in adjusting the weights; The higher the value, the more significant the environmental impact; Step 4.3: Combine the adjusted weights of each modality with its saliency feature map, perform a weighted summation, and generate the final fused feature map: ; Step 4.4: The final fused image obtained by reconstructing based on the fused feature map. This preserves the advantages of each modality.

[0051] In step 5, the fusion result output and feedback optimization are performed. Output fused image And perform target detection; The detection accuracy and false alarm rate are fed back into the environmental impact model to continuously optimize the visible light image quality scoring function. Infrared image quality scoring function and radar image signal-to-noise ratio function The function parameters (such as coefficients and thresholds) are used to more accurately reflect the impact of the real environment on image quality, achieve closed-loop optimization, and improve the effect of future fusion tasks.

[0052] Example 1 This embodiment provides an adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception. First, it analyzes the impact of different meteorological conditions on sensor imaging, as follows:

[0053] The characteristics of multimodal image data are as follows:

[0054] Based on the above analysis, this invention proposes a closed-loop image fusion architecture that is "environmentally perceptive driven, goal-oriented, and adaptively fused". Its core idea is to use the meteorological environment as an active control variable, construct a mapping relationship between the environment, sensor, and fusion strategy, and realize the dynamic optimal fusion of multimodal images.

[0055] 1. System Overall Architecture The system consists of five main modules, coupled through a unified data bus: (1) Meteorological environment perception module (2) Multimodal image acquisition module (3) Environmental Impact Assessment and Modeling Module (4) Adaptive Fusion Engine (5) Fusion Image Output and Target Analysis Module 2. Methodology and Flow Step 1: Synchronous acquisition of multi-source images and meteorological data; including: Step 1.1: Acquire visible light images of the same spatiotemporal region. Infrared images Millimeter-wave radar images ; Step 1.2: Synchronously collect meteorological data: visibility V relative humidity RH Rainfall intensity R Wind speed W Sea state rating S .

[0056] Step 2: Construct a model to influence the performance of meteorological sensors; Establish experience- or data-driven models to quantify the impact of meteorological factors on the image quality of various modalities; including: Step 2.1, Visible light image quality scoring: ,like .

[0057] Step 2.2, Infrared Image Quality Scoring: ,in The target is the temperature difference between the sea surface and the target. Wind speed (affects heat dissipation).

[0058] Step 2.3, Radar image signal-to-noise ratio: Heavy rainfall and high waves reduce the signal-to-noise ratio.

[0059] Step 3: Dynamic weight allocation and enhanced preprocessing; according to , , Calculate the initial fusion weights: , in i ∈ vis,ir,radar This is the sensitivity parameter (adjustable).

[0060] Targeted enhancement of low-quality images, when When, enable dehazing algorithms (such as dark channel prior); when When this is the case, enable speckle noise filtering (such as Lee filtering).

[0061] Step 4: Feature-level adaptive fusion; including: Step 4.1: Extract saliency feature maps from each modality image. ; Step 4.2: Introduce environmental correction factors Dynamically adjust feature fusion weights: ,in For average mass, This is the adjustment coefficient.

[0062] Step 4.3: Fuse Feature Maps .

[0063] Step 4.4: Reconstruct and fuse the image This preserves the advantages of each modality.

[0064] Step 5: Optimization of Fusion Result Output and Feedback Output fused image And perform object detection (such as object detection models like YOLOv8); The detection accuracy and false alarm rate are fed back into the environmental impact model for continuous optimization. , , Function parameters.

[0065] In addition, the present invention also provides a sea surface target image fusion device, including a processor and a memory, wherein the memory stores program instructions, and when the program instructions are executed by the processor, the aforementioned sea surface target multimodal image adaptive fusion method based on meteorological environment perception is realized.

[0066] The device also includes a meteorological sensor interface, a multimodal image receiving unit, a fused image output unit, and a target detection module.

[0067] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception, characterized in that, The method includes: Step 1: Simultaneously acquire visible light, infrared, and radar images, as well as real-time meteorological data; Step 2: Construct an assessment model for the impact of meteorological factors on the image quality of each modality; Step 3: Dynamically assign fusion weights based on the evaluation results and perform image preprocessing; Step 4: Implement adaptive fusion guided by environmental correction factors at the feature level; Step 5: Output the fused image and support closed-loop feedback optimization.

2. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 1, characterized in that, In step 1, the meteorological data includes visibility, humidity, rainfall intensity, wind speed, and sea state.

3. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 2, characterized in that, Step 1 includes: Step 1.1: Acquire visible light images of the same spatiotemporal region. Infrared images Millimeter-wave radar images ; Step 1.2: Synchronously collect meteorological data: visibility V relative humidity RH Rainfall intensity R Wind speed W Sea state rating S .

4. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 3, characterized in that, In step 2, the impact assessment model uses empirical formulas or machine learning regression models to output quality scores for each modality of the image.

5. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 4, characterized in that, Step 2 includes: Step 2.1, Visible light image quality scoring: ,like Where, e is the base of the natural logarithm, approximately equal to 2.71828; k1 is an empirical coefficient for adjusting the effect of humidity on image quality, used to adjust the degree of influence of relative humidity (RH) on image quality. The larger the value of k1, the stronger the attenuation effect of ambient humidity on image quality. The higher the relative humidity (RH), the lower the contrast and the blurrier the image; k2 is an empirical coefficient for adjusting the effect of rainfall on image quality; rainfall will cause background blur and reduce image usability; (1 R) indicates that the stronger the rainfall (the larger the R), the lower the percentage of remaining mass. Step 2.2, Infrared Image Quality Scoring: ,in The target is the temperature difference between the sea surface and the target. Wind speed; Step 2.3, Radar image signal-to-noise ratio: Heavy rainfall and high waves reduce the signal-to-noise ratio.

6. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 5, characterized in that, In step 3, the fusion weight is calculated based on the quality score using a power-law function normalization.

7. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 6, characterized in that, In step 3, according to , , Calculate the initial fusion weights for the i-th mode, i.e., visible light, infrared, and radar. This is used to represent the relative importance of the mode in multi-source fusion. , where i∈vis,ir,radar, is the sensitivity parameter in vis,ir,radar.

8. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 7, characterized in that, In step 3, targeted enhancement is performed on low-quality images. When the defogging algorithm is enabled, This is a visible light image quality threshold used to distinguish between "usable" and "needs enhancement" image quality boundaries; when When this is the case, enable speckle noise filtering. This is the signal-to-noise ratio threshold for radar images, used to identify low signal-to-noise ratio scenes that require preprocessing.

9. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 8, characterized in that, Step 4 includes: Step 4.1: Extract the saliency feature map from the i-th modality image. , used to reflect the importance of the mode in a specific region, where i∈vis,ir,radar, is the sensitivity parameter in vis,ir,radar; Step 4.2: Introduce environmental correction factors Dynamically adjust feature fusion weights: ， in, To dynamically adjust the feature fusion weights, the fusion weights of each modality are dynamically adjusted based on its performance relative to the average level, highlighting the information contribution of high-quality modalities. Let be the initial fusion weights for the i-th mode; Average quality score calculated for all modalities (vis,ir,radar) This is an adjustment coefficient used to control the intensity of environmental factors in adjusting the weights; The higher the value, the more significant the environmental impact; Step 4.3: Combine the adjusted weights of each modality with its saliency feature map, perform a weighted summation, and generate the final fused feature map: ; Step 4.4: The final fused image obtained by reconstructing based on the fused feature map. This preserves the advantages of each modality.

10. The adaptive fusion method for multimodal images of sea surface targets based on meteorological environment perception as described in claim 8, characterized in that, In step 5, the fusion result output and feedback optimization are performed. Output fused image And perform target detection; The detection accuracy and false alarm rate are fed back into the environmental impact model to continuously optimize the visible light image quality scoring function. Infrared image quality scoring function and radar image signal-to-noise ratio function The function parameters are optimized to more accurately reflect the impact of the real environment on image quality, achieving closed-loop optimization and improving the performance of future fusion tasks.

Images