An image fusion enhancement method, device and storage medium

Abstract

The application provides an image fusion enhancement method and device and a storage medium, and belongs to the technical field of image processing.The method comprises the following steps: obtaining an original riverway infrared image through an infrared camera, and obtaining an original riverway visible light image through a visible light camera; performing resolution conversion on the original riverway infrared image and the original riverway visible light image to obtain a converted riverway infrared image and a converted riverway visible light image; performing brightness analysis on the converted riverway visible light image, and obtaining a target light mode according to the analysis result.The application solves the problems that the existing general fusion algorithm cannot adapt to a dynamic riverway environment and a single-mode enhancement technology cannot break through the difficulty of physical information loss, meets the requirement of real-time inspection, effectively combines temperature information of an infrared image and structural information of a visible light image, and improves image definition and distinguishability under low-light conditions.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image fusion enhancement method, apparatus, and storage medium. Background Technology

[0002] In the field of drone-based river inspection, the failure of visual perception in nighttime and low-light environments is a core challenge restricting all-weather inspection. Existing technologies primarily address this issue through image enhancement algorithms or dual-modal fusion techniques. While these technologies have applications in general vision or specific industries (such as power and transportation), none are specifically designed for the unique physical characteristics of river surfaces (such as specular reflection and slight temperature differences) and the real-time constraints of airborne edge computing.

[0003] Existing image processing methods mainly rely on histogram equalization of visible light images, Retinex theory, or deep CNNs for single-modal enhancement. These methods cannot capture effective physical information in complete darkness or extremely low light environments. Simply creating information out of thin air by algorithms can lead to severe noise amplification or loss of detail, and they cannot perceive the weak thermal radiation characteristics unique to river discharge outlets, resulting in the failure of nighttime inspections.

[0004] Existing infrared and visible light fusion technologies typically employ fixed weights or generic feature stitching strategies. These are not optimized for the unique "specular reflection" interference of water surfaces and lack the ability to dynamically adjust the fusion strategy based on ambient light intensity. In reflective areas of the water surface, generic models easily lose heat source information, making it impossible to distinguish between ambient light reflection and genuine heat radiation sources (such as sewage outlets). Furthermore, the large number of model parameters makes real-time operation at airborne edge environments difficult.

[0005] Existing inspection technologies in fields such as power and photovoltaics are mostly based on traditional image processing (wavelet transform, Otsu thresholding) or computationally intensive general-purpose depth models. These solutions typically rely on high-performance computing equipment for post-processing, do not address optimization for real-time deployment on resource-constrained UAV-borne devices (<50ms), and lack mechanisms for using infrared heat sources to guide visible light feature reconstruction, thus failing to simultaneously retain the dual information of "temperature rise area + pipeline outline" under low light conditions. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an image fusion enhancement method, apparatus and storage medium that address the shortcomings of the prior art.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: An image fusion enhancement method, comprising the following steps: The original infrared image of the river channel was obtained using an infrared camera, and the original visible light image of the river channel was obtained using a visible light camera. The original infrared image and the original visible light image of the river channel are respectively converted to obtain the converted infrared image of the river channel corresponding to each of the original infrared image of the river channel and the converted visible light image of the river channel corresponding to each of the original visible light image of the river channel. Brightness analysis is performed on the converted visible light image of the river channel, and the target illumination pattern is obtained based on the analysis results; Weight analysis is performed on the converted visible light image of the river channel and the target illumination pattern to obtain the weights to be processed; Based on the weights to be processed, the converted infrared image and the converted visible light image of the river channel are fused and analyzed to obtain a river channel fusion feature map. The river channel fusion feature map is enhanced, and the enhancement result is used as the image fusion enhancement result.

[0008] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: An image fusion enhancement device, comprising: The image acquisition module is used to acquire raw infrared images of the river channel via an infrared camera and raw visible light images of the river channel via a visible light camera. The resolution conversion module is used to convert the resolution of the original river infrared image and the original river visible light image respectively, so as to obtain the converted river infrared image and the converted river visible light image corresponding to each of the original river infrared images. A brightness analysis module is used to perform brightness analysis on the converted visible light image of the river channel and obtain the target illumination mode based on the analysis results. The weight analysis module is used to perform weight analysis on the converted river channel visible light image and the target illumination pattern to obtain the weights to be processed. The fusion analysis module is used to perform fusion analysis on the converted infrared image and the converted visible light image of the river channel according to the weight to be processed, so as to obtain a fused feature map of the river channel. The fusion enhancement result acquisition module is used to enhance the river channel fusion feature map and use the enhancement result as the image fusion enhancement result.

[0009] Based on the above-mentioned image fusion enhancement method, the present invention also provides an image fusion enhancement system.

[0010] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: an image fusion enhancement system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image fusion enhancement method described above is implemented.

[0011] Based on the above-described image fusion enhancement method, the present invention also provides a computer-readable storage medium.

[0012] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the image fusion enhancement method as described above.

[0013] The beneficial effects of this invention are as follows: A converted infrared image of the river channel is obtained through an infrared camera, and a converted visible light image of the river channel is obtained through a visible light camera. Brightness analysis is performed on the converted visible light image of the river channel, and the target illumination mode is obtained based on the analysis results. Weight analysis of the converted visible light image of the river channel and the target illumination mode yields the weights to be processed. Based on the weights to be processed, a fusion analysis of the converted infrared image and the converted visible light image of the river channel is performed to obtain a river channel fusion feature map. The river channel fusion feature map is then enhanced, and the enhancement result is used as the image fusion enhancement result. This solves the problems that existing general fusion algorithms cannot adapt to dynamic river channel environments and that single-modal enhancement technology cannot overcome the lack of physical information. It achieves adaptive fusion of the scene, effectively suppresses the interference of water surface mirror reflection, meets the requirements of real-time inspection, and effectively combines the temperature information of the infrared image and the structural information of the visible light image, improving image clarity and recognizability under low light conditions. Attached Figure Description

[0014] Figure 1 This is a schematic flowchart of the image fusion enhancement method provided in an embodiment of the present invention; Figure 2 This is a structural diagram of the dynamic weight generation module of the image fusion enhancement method provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the cross-modal attention fusion mechanism of the image fusion enhancement method provided in the embodiments of the present invention; Figure 4 This is a system flowchart of the image fusion enhancement method provided in the embodiments of the present invention; Figure 5 This is an edge deployment architecture diagram of the image fusion enhancement method provided in an embodiment of the present invention; Figure 6 This is a block diagram of an image fusion enhancement device provided in an embodiment of the present invention. Detailed Implementation

[0015] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0016] Figure 1This is a schematic flowchart of an image fusion enhancement method provided in an embodiment of the present invention.

[0017] like Figure 1 As shown, an image fusion enhancement method includes the following steps: S1: Obtain the original infrared image of the river channel through an infrared camera, and obtain the original visible light image of the river channel through a visible light camera; S2: Perform resolution conversion on the original river infrared image and the original river visible light image respectively to obtain the converted river infrared image and the converted river visible light image corresponding to each of the original river infrared images. S3: Perform brightness analysis on the converted visible light image of the river channel, and obtain the target illumination mode based on the analysis results; S4: Perform weight analysis on the converted visible light image of the river channel and the target illumination pattern to obtain the weights to be processed; S5: Perform fusion analysis on the converted infrared image and the converted visible light image of the river channel according to the weights to be processed to obtain a fused feature map of the river channel. S6: Enhance the river channel fusion feature map and use the enhancement result as the image fusion enhancement result.

[0018] It should be understood that the drone is equipped with a pair of rigorously calibrated infrared cameras (such as FLIR Lepton 3.5) and visible light cameras (such as Sony IMX378) to ensure that the field of view (FOV) and focal length of the two cameras are consistent, and that the time synchronization accuracy is ≤10ms.

[0019] It should be understood that the images (i.e., the original infrared image and the original visible light image of the river channel) are uniformly set to 1920×1080@30fps to ensure subsequent feature alignment.

[0020] Specifically, acquiring infrared images of the same scene in the same time and space. ir (i.e., the converted infrared image of the river channel) and visible light image I vis (i.e., the converted visible light image of the river channel), used as the fusion input source. Where, I ir (i.e., the converted infrared image of the river channel) captures thermal radiation information (sensitive to the temperature rise at the sewage outlet), I vis (i.e., the converted visible light image of the river channel) provides high-resolution structural details (clear outline of the pipes).

[0021] In the above embodiments, a converted infrared image of the river channel is obtained through an infrared camera, and a converted visible light image of the river channel is obtained through a visible light camera. Brightness analysis is performed on the converted visible light image of the river channel, and the target illumination mode is obtained based on the analysis results. Weight analysis is performed on the weights of the converted visible light image and the target illumination mode to obtain the weights to be processed. Based on the weights to be processed, a fusion analysis of the converted infrared and visible light images of the river channel is performed to obtain a river channel fusion feature map. The river channel fusion feature map is then enhanced, and the enhancement result is used as the image fusion enhancement result. This solves the problems that existing general fusion algorithms cannot adapt to dynamic river channel environments and that single-modal enhancement techniques cannot overcome the lack of physical information. It achieves adaptive fusion of the scene, effectively suppresses the interference of water surface specular reflection, meets the requirements of real-time inspection, and effectively combines the temperature information of the infrared image and the structural information of the visible light image, improving image clarity and recognizability under low-light conditions.

[0022] Optionally, as an embodiment of the present invention, the process of performing brightness analysis on the converted visible light image of the river channel and obtaining the target illumination mode based on the analysis results includes: The converted visible light image of the river channel is format-converted to obtain multiple brightness components; The global average brightness value is obtained by calculating all the brightness components using the first equation, which is: , in, This is the global average brightness value. To convert the visible light image height of the river channel, To convert the width of the visible light image of the river channel, For pixels The brightness component; Determine whether the global average brightness value is less than a preset brightness threshold. If so, use the preset first lighting mode as the target lighting mode; otherwise, use the preset second lighting mode as the target lighting mode.

[0023] It should be understood that the preset first lighting mode can be a low lighting mode (night / dense fog), and the preset second lighting mode can be a normal lighting mode.

[0024] Specifically, using only visible light images I vis The Y channel (luminance component) calculates the global average luminance value L. The judgment threshold (i.e., the preset luminance threshold) is set to L=50 (8-bit image, range 0~255): if L<50, it is judged as low light (night / dense fog); otherwise, it is normal light.

[0025] It should be understood that, in order to provide an environmental context for subsequent dynamic weight generation, this design avoids the introduction of additional sensors and utilizes existing visible light data to achieve low-cost, robust illumination sensing.

[0026] In the above embodiments, brightness analysis is performed on the converted visible light image of the river channel, and the target illumination pattern is obtained based on the analysis results. This avoids the introduction of additional sensors and achieves low-cost and highly robust illumination sensing.

[0027] Optionally, as an embodiment of the present invention, such as Figure 2 As shown, the process of performing weight analysis on the converted visible light image of the river channel and the target illumination pattern to obtain the weights to be processed includes: The first convolutional layer extracts features from the transformed visible light image of the river channel and the target illumination pattern to obtain a first visible light feature map of the river channel. The first visible light feature map of the river channel is mapped by the first ReLU activation function layer to obtain the second visible light feature map of the river channel. The visible light feature map of the second channel is normalized by the first normalization layer to obtain the visible light feature map of the third channel. The visible light feature map of the third channel is extracted by the second convolutional layer to obtain the visible light feature map of the fourth channel; The visible light feature map of the fourth channel is mapped by the second ReLU activation function layer to obtain the visible light feature map of the fifth channel; The visible light feature map of the fifth channel is normalized by the second batch of normalization layers to obtain the visible light feature map of the sixth channel. The visible light feature map of the sixth channel is extracted by the third convolutional layer to obtain the visible light feature map of the seventh channel; The visible light feature map of the seventh channel is mapped by the third ReLU activation function layer to obtain the visible light feature map of the eighth channel; The visible light feature map of the eighth channel is normalized by the third batch of normalization layers to obtain the visible light feature map of the ninth channel. The visible light feature map of the ninth river channel is predicted by a Sigmoid function layer to obtain the weights to be processed.

[0028] It should be understood that a lightweight CNN module (3 convolutional layers + ReLU + BN) is designed (i.e., first convolutional layer, first ReLU activation function layer, first batch normalization layer, second convolutional layer, second ReLU activation function layer, second batch normalization layer, third convolutional layer, third ReLU activation function layer, and third batch normalization layer), with input I. vis(That is, the converted visible light image of the river channel), the output is a single-channel weighted image α∈[0,1) H×W (i.e., the weights to be processed). Supervised learning is used for network training: daytime sample labels α→0.9 (emphasizing visible light details), nighttime sample labels α→0.2 (emphasizing infrared heat sources). The loss function is L1 Loss.

[0029] Specifically, pixel-level adaptive fusion is achieved. For example, in bright riverbank areas, α≈0.9 preserves rich textures; in dark water areas, α≈0.2 highlights weak thermal signals. This mechanism overcomes the problems of "overexposure during the day" or "blurring at night" caused by fixed-weight fusion.

[0030] Specifically, such as Figure 2 As shown, the input is a visible light image. After passing through three convolutional layers (i.e., the first, second, and third convolutional layers) and ReLU activation functions (i.e., the first, second, and third ReLU activation function layers), the output is a fusion weight α (i.e., the weight to be processed) between 0 and 1, which reflects the optimal fusion ratio of the infrared image and the visible light image under the current lighting conditions, so that the best image can be obtained under different lighting conditions. The quality is achieved through a 3-layer convolutional network structure (i.e., the first convolutional layer, the first ReLU activation function layer, the first batch normalization layer, the second convolutional layer, the second ReLU activation function layer, the second batch normalization layer, the third convolutional layer, the third ReLU activation function layer, and the third batch normalization layer): Layer 1 (32 channels, 3×3 convolution, stride 2), Layer 2 (64 channels, 3×3 convolution, stride 2), Layer 3 (1 channel, 1×1 convolution), each layer is followed by ReLU and batch normalization, and finally the [0,1] weights (i.e., the weights to be processed) are output through Sigmoid.

[0031] In the above embodiments, weight analysis is performed on the converted visible light image of the river channel and the target illumination mode to obtain the weight to be processed, realizing pixel-level adaptive fusion, preserving rich texture, and overcoming the problems of daytime overexposure or nighttime blurring caused by fixed weight fusion.

[0032] Optionally, as an embodiment of the present invention, such as Figure 3 As shown, the process of fusing and analyzing the converted infrared image and the converted visible light image of the river channel according to the weights to be processed to obtain the fused feature map of the river channel includes: The ResNet-18 model is used to extract features from the converted river infrared image and the converted river visible light image, respectively, to obtain the first river infrared feature map corresponding to the converted river infrared image and the tenth river visible light feature map corresponding to the converted river visible light image. The first river infrared feature map is processed by a global average pooling layer to obtain the second river infrared feature map. The second river infrared feature map is convolved by a 1×1 convolutional layer to obtain the third river infrared feature map. The spatial attention mask is obtained by mapping the infrared feature map of the third river channel through a Sigmoid activation function layer. The visible light feature map of the eleventh river channel is obtained by calculating the visible light feature map of the tenth channel and the spatial attention mask using the second equation, whereby the second equation is: , in, This is a visible light characteristic map of the eleventh river channel. For spatial attention mask, For element-wise multiplication, Visible light characteristic map of the tenth river channel; The weights to be processed are upsampled to obtain the target weights; The river channel fusion feature map is obtained by calculating the visible light feature map of the eleventh river channel, the target weight, and the infrared feature map of the first river channel using the third formula. The third formula is: , in, This is a map showing the characteristics of river channel integration. For the target weight, This is a visible light characteristic map of the eleventh river channel. This is the infrared feature map of the first river channel.

[0033] Specifically, the lightweight ResNet-18 branch with shared weights (i.e., the ResNet-18 model) is used to extract I. ir (i.e., the converted infrared image of the river channel) and I vis High-level feature map F (i.e., the converted visible light image of the river channel) ir (i.e., the infrared feature map of the first river channel) and (i.e., the visible light feature map of the tenth channel), construct a heat source-guided attention module: for F ir (i.e., the infrared feature map of the first river channel) Apply global average pooling + 1×1 convolution + Sigmoid to generate a spatial attention mask. (i.e., spatial attention mask), weighted by visible light features, as follows: (⊙ indicates element-wise multiplication). The fusion feature (i.e., the river channel fusion feature map) is as follows: , in The output weight map (i.e. the weights to be processed) is upsampled to the size of the feature map.

[0034] It should be understood that a semantic association between infrared heat sources and visible light structures is established. For example, when an infrared image detects a local temperature rise (suspected sewage outlet), the mask M will highlight the area, guiding the visible light features to reconstruct the corresponding pipe outline, effectively suppressing the interference of water surface mirror reflection.

[0035] It should be understood that, as Figure 3 As shown, first, the infrared features are channel compressed; second, spatial activation is performed; then, the visible light features are multiplied element-wise; and finally, they are fused with the infrared features according to dynamic weights.

[0036] Specifically, high-response regions in infrared images are used to generate spatial attention masks M, which are applied to the feature maps of visible light images, thereby enhancing the reconstruction of visible light details guided by infrared heat sources. This mechanism effectively combines the temperature information of infrared images with the structural information of visible light images, improving image clarity and recognizability under low-light conditions.

[0037] In the above embodiments, the converted infrared image and the converted visible light image of the river channel are fused and analyzed according to the weight to be processed to obtain a fused feature map of the river channel. This enhances the visible light detail reconstruction effect guided by the infrared heat source, effectively combines the temperature information of the infrared image and the structural information of the visible light image, and improves the image clarity and recognizability under low light conditions.

[0038] Optionally, as an embodiment of the present invention, the process of enhancing the river channel fusion feature map and using the enhancement result as the image fusion enhancement result includes: The river channel fusion feature map is enhanced by multiple transposed convolutional decoders to obtain an enhanced river channel image, which is then used as the image fusion enhancement result.

[0039] It should be understood that a 4-layer transposed convolutional decoder is used to convert F... fused (i.e., the river channel fusion feature map) reconstructed into an RGB enhanced image I out (i.e., enhanced river channel image).

[0040] It should be understood that this ensures the algorithm can run in real time on the drone's onboard edge device, meeting the timeliness requirements of inspection tasks.

[0041] In the above embodiments, the river channel fusion feature map is enhanced, and the enhancement result is used as the image fusion enhancement result, ensuring that the algorithm can run in real time on the UAV's onboard edge device and meet the timeliness requirements of the inspection task.

[0042] Alternatively, as another embodiment of the present invention, in view of the deficiencies of the prior art, the present invention aims to provide an adaptive infrared-visible light image fusion enhancement method for low-light river inspection, and the specific technical problems to be solved and the objectives to be achieved are as follows: Breakthrough in perception bottleneck in completely dark environments: A dual-modal complementary mechanism is provided, which utilizes infrared thermal radiation information to provide structural supplementation in extremely dark environments where visible light is completely ineffective, thus solving the physical bottleneck of "no information from single-modal visible light enhancement in completely dark environments".

[0043] Achieving scene-adaptive fusion: A dynamic weight generation mechanism driven by illumination and a heat source-guided cross-modal attention module are provided. Based on the ambient brightness (such as the weak illumination of a river at night) and the water surface reflection, the fusion ratio of infrared and visible light is adjusted in real time to effectively suppress the interference of water surface mirror reflection and achieve pixel-level feature interaction and semantic alignment.

[0044] Meeting the real-time constraints of airborne edge computing: Providing a lightweight end-to-end solution specifically for river inspection, achieving end-to-end real-time processing latency of <50ms on UAV airborne AI chips (such as Ascend 310) through model compression and hardware acceleration, meeting the requirements of real-time inspection.

[0045] Output high-quality enhanced images: Generate high-quality RGB enhanced images that simultaneously contain "thermal radiation features" and "visible light structural details", supporting downstream AI models (such as object detection and semantic segmentation) to run stably and reliably under harsh conditions such as night or haze.

[0046] Optionally, as another embodiment of the present invention, the present invention is an adaptive infrared-visible light image fusion enhancement method for low-light river inspection. It belongs to the method invention category in the interdisciplinary field of computer vision and edge intelligence. Its core lies in generating high-quality enhanced images by using a dynamic fusion mechanism driven by light perception and a cross-modal attention structure guided by heat sources, under the premise of meeting the real-time constraint of <50ms on the UAV on-board end, so as to support the stable operation of downstream AI models (such as YOLOv4) under harsh conditions such as night / fog.

[0047] Optionally, as another embodiment of the present invention, the present invention employs a knowledge distillation compression model: using the original large model as the teacher, a student network with <1M parameters is trained, resulting in an accuracy loss of <2%. Deployed on the Huawei Ascend 310 AI chip, INT8 quantization and operator fusion are enabled, with an end-to-end latency of <50ms.

[0048] Optionally, as another embodiment of the present invention, in order to ensure that the algorithm can run in real time on the UAV onboard edge device to meet the timeliness requirements of the inspection task, while outputting standard RGB images, it is seamlessly compatible with existing AI analysis platforms (such as YOLOv4 and Mask R-CNN).

[0049] Optionally, as another embodiment of the present invention, during a river inspection mission at the Wulongjiang Wetland Park in Fuzhou, Fujian, the UAV flew at an altitude of 100 meters and a speed of 8 m / s. At night, a suspected sewage outlet area was captured: the visible light image showed only a blurry black shadow, with no discernible structure; the infrared image showed a localized temperature rise (ΔT≈1.8°C), but lacked geometric details; after fusion by the present invention, the output image clearly presented both "temperature rise area + pipe outline" information; the downstream YOLOv4 model successfully identified it as "illegal sewage discharge" with a confidence level of 92%.

[0050] Alternatively, as another embodiment of the present invention, such as Figure 4 As shown, this invention describes a complete data stream starting with simultaneous acquisition from dual cameras, sequentially proceeding through illumination estimation, dynamic weight generation, cross-modal feature fusion, and image reconstruction, ultimately outputting an enhanced image. Tensors are passed between modules via memory buffers, supporting pipelined parallel processing. First, the infrared and visible light cameras on the UAV simultaneously acquire image data from the same field of view. Then, the illumination intensity estimation module assesses the current ambient illumination conditions to determine whether enhancement processing is needed. Next, the dynamic weight generation module automatically generates fusion weights based on the illumination conditions. Finally, cross-modal feature extraction and attention mechanisms are used to fuse the two images, ultimately outputting a high-quality enhanced image for subsequent analysis. This process achieves fully automated operation from image acquisition to final output.

[0051] Alternatively, as another embodiment of the present invention, such as Figure 5 As shown, the dual cameras (infrared and visible light) on the drone of this invention acquire image data in real time and run a lightweight fusion model through the Ascend 310 AI chip to instantly generate enhanced images. These enhanced images are then transmitted to a server via a 5G CPE for further analysis or archiving. The entire process ensures high efficiency and real-time performance of image processing while guaranteeing low-latency data transmission. The hardware connection relationship is: dual cameras → Ascend 310 AI chip (running the fusion model) → 5G CPE → cloud server. It is emphasized that the model is burned to the chip's NPU offline, and the inference process requires no network interaction, ensuring an end-to-end latency of <50ms.

[0052] Alternatively, as another embodiment of the present invention, the present invention fully covers algorithm design, core mechanisms and engineering implementation. Those skilled in the art can reproduce and deploy the present invention without creative effort by combining general deep learning frameworks (such as MindSpore and PyTorch) and edge development kits (such as CANN) based on the above description.

[0053] Optionally, as another embodiment of the present invention, the key point of the present invention lies in solving the problems that existing general fusion algorithms cannot adapt to dynamic river environments and that single-mode enhancement technology cannot overcome the lack of physical information. Specific protection points are as follows: A dynamic fusion strategy based on environment awareness (different from the fixed weights of general algorithms): This invention protects a light-driven dynamic fusion weight generation mechanism. Unlike existing general algorithms (such as SMIgusion) that use a preset loss function and a fixed network flow, this invention dynamically generates a pixel-level fusion weight map by estimating the global brightness of the visible light image in real time. This mechanism can automatically adjust the contribution ratio of infrared (thermal radiation) and visible light (texture) according to the lighting changes in the river environment (such as complete darkness, low light, and reflective areas on the water surface), thus solving the problem of general models losing heat source information under the mirror reflection of the water surface.

[0054] A heat source-guided cross-modal semantic interaction architecture (distinct from simple feature concatenation): This invention protects a heat-guided cross-modal attention module (TG-CAM). Existing technologies mostly employ feature stitching or simple weighted averaging. This invention utilizes infrared feature maps to generate spatial attention masks, directly modulating the reconstruction process of visible light features. This "heat-guided vision" mechanism is specifically designed to preserve the weak temperature difference features of sewage outlets in low signal-to-noise ratio environments and semantically align them with the geometric contours of visible light. This is something that existing single-modal algorithms (such as Retinex and Zero-DCE) and general bimodal algorithms do not possess.

[0055] Dedicated lightweight design for edge computing (as opposed to large cloud models): This invention protects an end-to-end lightweight network architecture adapted for airborne edge computing. Addressing the limited computing power of drones, this invention employs a lightweight backbone network with <1M parameters, coupled with a knowledge distillation training strategy, and combines the INT8 quantization and operator fusion technology of the Ascend AI chip to ensure real-time inference capabilities with a latency of <50ms. This is fundamentally different from solutions in existing academic papers that rely on high-performance servers (such as ResNeSt101 and Transformer large models).

[0056] Physical interference suppression methods for specific scenarios: The present invention provides a suppression logic for interference from water surface mirror reflection. The technical solution implicitly includes the identification and processing logic for the high-gloss area of ​​the water surface. Through a dynamic weighting mechanism, the weight of the visible light channel is automatically reduced in the reflection area to prevent ambient light reflection from obscuring the real heat radiation source, thereby ensuring the perception robustness in complex aquatic environments.

[0057] Alternatively, as another embodiment of the present invention, the present invention already has a clear and feasible implementation path: Product Integration: It can be directly used as a front-end perception enhancement module and seamlessly integrated into the existing 5G drone river inspection platform without requiring large-scale modifications to the existing system architecture.

[0058] Downstream compatibility: The algorithm outputs standard RGB format images, which are fully compatible with existing mainstream AI analysis models (such as YOLOv4 and Mask R-CNN), without requiring any retraining or adaptation of downstream models.

[0059] Engineering verification: The next step is to complete end-to-end deployment verification on Huawei Ascend 310 AI chip. The actual measured processing latency is consistently below 50ms, which meets the real-time requirements of drone inspection.

[0060] Product planning: This invention can be incorporated into the "Smart Water AI Suite" version as a core functional module to support continuous and reliable 24 / 7 monitoring of river pollution sources.

[0061] Figure 6 This is a block diagram of an image fusion enhancement device provided in an embodiment of the present invention.

[0062] Alternatively, as another embodiment of the present invention, such as Figure 6 As shown, an image fusion enhancement device includes: The image acquisition module is used to acquire raw infrared images of the river channel via an infrared camera and raw visible light images of the river channel via a visible light camera. The resolution conversion module is used to convert the resolution of the original river infrared image and the original river visible light image respectively, so as to obtain the converted river infrared image and the converted river visible light image corresponding to each of the original river infrared images. A brightness analysis module is used to perform brightness analysis on the converted visible light image of the river channel and obtain the target illumination mode based on the analysis results. The weight analysis module is used to perform weight analysis on the converted river channel visible light image and the target illumination pattern to obtain the weights to be processed. The fusion analysis module is used to perform fusion analysis on the converted infrared image and the converted visible light image of the river channel according to the weight to be processed, so as to obtain a fusion feature map of the river channel. The fusion enhancement result acquisition module is used to enhance the river channel fusion feature map and use the enhancement result as the image fusion enhancement result.

[0063] Optionally, as another embodiment of the present invention, the brightness analysis module is specifically used for: The converted visible light image of the river channel is format-converted to obtain multiple brightness components; The global average brightness value is obtained by calculating all the brightness components using the first equation, which is: , in, This is the global average brightness value. To convert the visible light image height of the river channel, To convert the width of the visible light image of the river channel, For pixels The brightness component; Determine whether the global average brightness value is less than a preset brightness threshold. If so, use the preset first lighting mode as the target lighting mode; otherwise, use the preset second lighting mode as the target lighting mode.

[0064] Optionally, as another embodiment of the present invention, the weight analysis module is specifically used for: The first convolutional layer extracts features from the transformed visible light image of the river channel and the target illumination pattern to obtain a first visible light feature map of the river channel. The first visible light feature map of the river channel is mapped by the first ReLU activation function layer to obtain the second visible light feature map of the river channel. The visible light feature map of the second channel is normalized by the first normalization layer to obtain the visible light feature map of the third channel. The visible light feature map of the third channel is extracted by the second convolutional layer to obtain the visible light feature map of the fourth channel; The visible light feature map of the fourth channel is mapped by the second ReLU activation function layer to obtain the visible light feature map of the fifth channel; The visible light feature map of the fifth channel is normalized by the second batch of normalization layers to obtain the visible light feature map of the sixth channel. The visible light feature map of the sixth channel is extracted by the third convolutional layer to obtain the visible light feature map of the seventh channel; The visible light feature map of the seventh channel is mapped by the third ReLU activation function layer to obtain the visible light feature map of the eighth channel; The visible light feature map of the eighth channel is normalized by the third batch of normalization layers to obtain the visible light feature map of the ninth channel. The visible light feature map of the ninth river channel is predicted by a Sigmoid function layer to obtain the weights to be processed.

[0065] Optionally, another embodiment of the present invention provides an image fusion enhancement system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image fusion enhancement method as described above. This system can be a computer or similar system.

[0066] Optionally, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image fusion enhancement method as described above.

[0067] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0071] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An image fusion enhancement method, characterized in that, Includes the following steps: The original infrared image of the river channel was obtained using an infrared camera, and the original visible light image of the river channel was obtained using a visible light camera. The original infrared image and the original visible light image of the river channel are respectively converted to obtain the converted infrared image of the river channel corresponding to each of the original infrared image of the river channel and the converted visible light image of the river channel corresponding to each of the original visible light image of the river channel. Brightness analysis is performed on the converted visible light image of the river channel, and the target illumination pattern is obtained based on the analysis results; Weight analysis is performed on the converted visible light image of the river channel and the target illumination pattern to obtain the weights to be processed; Based on the weights to be processed, the converted infrared image and the converted visible light image of the river channel are fused and analyzed to obtain a river channel fusion feature map. The river channel fusion feature map is enhanced, and the enhancement result is used as the image fusion enhancement result.

2. The image fusion enhancement method according to claim 1, characterized in that, The process of performing brightness analysis on the converted visible light image of the river channel and obtaining the target illumination pattern based on the analysis results includes: The converted visible light image of the river channel is format-converted to obtain multiple brightness components; The global average brightness value is obtained by calculating all the brightness components using the first equation, which is: ， in, This is the global average brightness value. To convert the visible light image height of the river channel, To convert the width of the visible light image of the river channel, For pixels The brightness component; Determine whether the global average brightness value is less than a preset brightness threshold. If so, use the preset first lighting mode as the target lighting mode; otherwise, use the preset second lighting mode as the target lighting mode.

3. The image fusion enhancement method according to claim 1, characterized in that, The process of performing weight analysis on the converted river channel visible light image and the target illumination pattern to obtain the weights to be processed includes: The first convolutional layer extracts features from the transformed visible light image of the river channel and the target illumination pattern to obtain a first visible light feature map of the river channel. The first visible light feature map of the river channel is mapped by the first ReLU activation function layer to obtain the second visible light feature map of the river channel. The visible light feature map of the second channel is normalized by the first normalization layer to obtain the visible light feature map of the third channel. The visible light feature map of the third channel is extracted by the second convolutional layer to obtain the visible light feature map of the fourth channel; The visible light feature map of the fourth channel is mapped by the second ReLU activation function layer to obtain the visible light feature map of the fifth channel; The visible light feature map of the fifth channel is normalized by the second batch of normalization layers to obtain the visible light feature map of the sixth channel. The visible light feature map of the sixth channel is extracted by the third convolutional layer to obtain the visible light feature map of the seventh channel; The visible light feature map of the seventh channel is mapped by the third ReLU activation function layer to obtain the visible light feature map of the eighth channel; The visible light feature map of the eighth channel is normalized by the third batch of normalization layers to obtain the visible light feature map of the ninth channel. The visible light feature map of the ninth river channel is predicted by a Sigmoid function layer to obtain the weights to be processed.

4. The image fusion enhancement method according to claim 1, characterized in that, The process of fusing and analyzing the converted infrared image and the converted visible light image of the river channel according to the weights to be processed to obtain a fused feature map of the river channel includes: The ResNet-18 model is used to extract features from the converted river infrared image and the converted river visible light image, respectively, to obtain the first river infrared feature map corresponding to the converted river infrared image and the tenth river visible light feature map corresponding to the converted river visible light image. The first river infrared feature map is processed by a global average pooling layer to obtain the second river infrared feature map. The second river infrared feature map is convolved by a 1×1 convolutional layer to obtain the third river infrared feature map. The spatial attention mask is obtained by mapping the infrared feature map of the third river channel through a Sigmoid activation function layer. The visible light feature map of the eleventh river channel is obtained by calculating the visible light feature map of the tenth channel and the spatial attention mask using the second equation, whereby the second equation is: ， in, This is a visible light characteristic map of the eleventh river channel. For spatial attention mask, For element-wise multiplication, Visible light characteristic map of the tenth river channel; The weights to be processed are upsampled to obtain the target weights; The river channel fusion feature map is obtained by calculating the visible light feature map of the eleventh river channel, the target weight, and the infrared feature map of the first river channel using the third formula. The third formula is: ， in, This is a map showing the characteristics of river channel integration. For the target weight, This is a visible light characteristic map of the eleventh river channel. This is the infrared feature map of the first river channel.

5. The image fusion enhancement method according to claim 1, characterized in that, The process of enhancing the river channel fusion feature map and using the enhancement result as the image fusion enhancement result includes: The river channel fusion feature map is enhanced by multiple transposed convolutional decoders to obtain an enhanced river channel image, which is then used as the image fusion enhancement result.

6. An image fusion enhancement device, characterized in that, include: The image acquisition module is used to acquire raw infrared images of the river channel via an infrared camera and raw visible light images of the river channel via a visible light camera. The resolution conversion module is used to convert the resolution of the original river infrared image and the original river visible light image respectively, so as to obtain the converted river infrared image and the converted river visible light image corresponding to each of the original river infrared images. A brightness analysis module is used to perform brightness analysis on the converted visible light image of the river channel and obtain the target illumination mode based on the analysis results. The weight analysis module is used to perform weight analysis on the converted river channel visible light image and the target illumination pattern to obtain the weights to be processed. The fusion analysis module is used to perform fusion analysis on the converted infrared image and the converted visible light image of the river channel according to the weight to be processed, so as to obtain a fused feature map of the river channel. The fusion enhancement result acquisition module is used to enhance the river channel fusion feature map and use the enhancement result as the image fusion enhancement result.

7. The image fusion enhancement apparatus according to claim 6, characterized in that, The brightness analysis module is specifically used for: The converted visible light image of the river channel is format-converted to obtain multiple brightness components; The global average brightness value is obtained by calculating all the brightness components using the first equation, which is: ， in, This is the global average brightness value. To convert the visible light image height of the river channel, To convert the width of the visible light image of the river channel, For pixels The brightness component; Determine whether the global average brightness value is less than a preset brightness threshold. If so, use the preset first lighting mode as the target lighting mode; otherwise, use the preset second lighting mode as the target lighting mode.

8. The image fusion enhancement apparatus according to claim 6, characterized in that, The weight analysis module is specifically used for: The first convolutional layer extracts features from the transformed visible light image of the river channel and the target illumination pattern to obtain a first visible light feature map of the river channel. The first visible light feature map of the river channel is mapped by the first ReLU activation function layer to obtain the second visible light feature map of the river channel. The visible light feature map of the second channel is normalized by the first normalization layer to obtain the visible light feature map of the third channel. The visible light feature map of the third channel is extracted by the second convolutional layer to obtain the visible light feature map of the fourth channel; The visible light feature map of the fourth channel is mapped by the second ReLU activation function layer to obtain the visible light feature map of the fifth channel; The visible light feature map of the fifth channel is normalized by the second batch of normalization layers to obtain the visible light feature map of the sixth channel. The visible light feature map of the sixth channel is extracted by the third convolutional layer to obtain the visible light feature map of the seventh channel; The visible light feature map of the seventh channel is mapped by the third ReLU activation function layer to obtain the visible light feature map of the eighth channel; The visible light feature map of the eighth channel is normalized by the third batch of normalization layers to obtain the visible light feature map of the ninth channel. The visible light feature map of the ninth river channel is predicted by a Sigmoid function layer to obtain the weights to be processed.

9. An image fusion enhancement apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image fusion enhancement method as described in any one of claims 1 to 5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image fusion enhancement method as described in any one of claims 1 to 5.

Images