Image restoration thermal fusion method and system

Abstract

The invention discloses an image restoration thermal fusion method and system, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a visible light image, an infrared light image and a low-light image, and judging whether the visible light image has environmental interference or not; if the visible light image does not have the environmental interference, performing thermal fusion on the visible light image and the infrared light image and / or the low-light image to obtain a first thermal fusion image; if the visible light image has environmental interference, inputting the visible light image into a pre-constructed visible light image restoration model for image restoration, and outputting a visible light restoration image; performing thermal fusion on the visible light restored image and the infrared light image and / or the low-light image to obtain a second thermal fusion image; and if the visible light image has environmental interference and image restoration cannot be carried out, carrying out thermal fusion on the infrared light image and the low-light image to obtain a third thermal fusion image. The adaptability and reliability of the thermal fusion night vision device can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image restoration thermal fusion method and system. Background Art

[0002] Thermal fusion night vision devices, as advanced visual aids, combine the advantages of infrared thermal imaging and visible light imaging to provide clear target identification in low-light environments. These devices typically incorporate infrared and visible light sensors, capturing and fusing thermal radiation and visible light images to enhance detail and contrast, providing more intuitive and rich visual information. Thermal fusion night vision devices are highly sought after in military, security, firefighting, and other fields for their superior performance in complex environments.

[0003] While thermal fusion night vision devices perform well on clear nights, their performance degrades significantly in extreme weather conditions, such as rainy nights, dense fog, snow, or smoky environments. This is because the quality of visible light images is severely limited under these conditions, resulting in reduced clarity and detail retention in the fused image. For example, on rainy nights, water droplets interfere with visible light sensors, while in dense fog or smoky environments, particulate matter scatters visible light, blurring the image. Furthermore, while thermal imaging is not affected by visible light, it can present challenges in certain scenarios, such as high-temperature fires or environments filled with toxic gases. High background temperatures can weaken the target's thermal signature, while certain chemicals can absorb or emit infrared radiation, affecting the accuracy of thermal images.

[0004] Existing thermal fusion night vision devices focus on optimizing image fusion algorithms, but lack adequate consideration of environmental adaptability. Furthermore, existing thermal fusion night vision devices have a relatively simple environmental perception module, relying primarily on infrared and visible light sensors. This limits their comprehensive environmental assessment and response capabilities in complex and changing battlefield environments.

[0005] Therefore, the performance of existing thermal fusion night vision devices is limited in special environments, and it is urgent to explore image restoration technology to improve the adaptability and reliability of thermal fusion night vision devices. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an image restoration thermal fusion method that can explore image restoration technology to improve the adaptability and reliability of thermal fusion night vision devices.

[0007] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0008] In one aspect, the present invention provides an image restoration thermal fusion method, comprising:

[0009] Acquire visible light images, infrared images, and low-light images;

[0010] If there is environmental interference in the visible light image, the visible light image is input into a pre-built visible light image restoration model to perform image restoration and output a visible light restored image; the visible light restored image is thermally fused with the infrared image and / or the low-light image to obtain a first thermally fused image;

[0011] The visible light image restoration model includes a convolutional Mamba module and a multi-scale selection kernel module connected in sequence, and the outputs of the visible light image and the multi-scale selection kernel module are converted into a visible light restoration image through a residual module;

[0012] The convolutional Maba module is obtained by adding a convolutional layer before the SSM projection layer in the Mamba module, and the weight of the convolutional Mamba module is dynamically adjusted through a gating unit; and the weight distribution of the multi-scale selection kernel module is dynamically adjusted through an attention mechanism.

[0013] Optionally, also include:

[0014] If there is no environmental interference in the visible light image, thermally fuse the visible light image and the infrared light image and / or the low-light image to obtain a second thermally fused image;

[0015] If the visible light image has environmental interference and cannot be restored, the infrared light image and the low-light image are thermally fused to obtain a third thermally fused image.

[0016] Optionally, the processing steps of the visible light image restoration model include:

[0017] In the convolutional Mamba module, local features and global features of the visible light image are extracted respectively, and the local features and global features are fused with environmental interference suppression to obtain a suppression feature map;

[0018] In the multi-scale selection kernel module, multi-scale feature extraction and fusion are performed on the suppressed feature map to obtain a visible light restored image.

[0019] Optionally, extracting local features and global features of the visible light image separately, performing environmental interference suppression fusion on the local features and the global features to obtain a suppression feature map, including:

[0020] ;

[0021] ;

[0022] ;

[0023] in, Represents local features; Represents 3×3 convolution; Represents a visible light image; Represents global features; represents the SSM projection; represents the suppression feature map; Represents the weight of the convolutional Mamba module, obtained through the gate unit; represents element-wise multiplication; Represents the sigmoid activation function.

[0024] Optionally, performing multi-scale feature extraction and fusion on the suppressed feature map to obtain a visible light restored image includes:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] in, 、 、 Represent the convolution results of each convolution branch respectively; represents the suppression feature map; 、 、 Represent 3×3 convolution, 5×5 convolution, and 7×7 convolution respectively; represents the weight of the j-th branch, obtained through the attention mechanism; 、 、 They represent the weight of the 3×3 convolution branch, the weight of the 5×5 convolution branch, and the weight of the 7×7 convolution branch respectively; represents the Softmax activation function; represents the MLP layer; represents global average pooling; Represents the characteristics after the fusion of each branch; represents the visible light restored image; Represents a visible light image.

[0030] Optionally, the training of the visible light image restoration model includes:

[0031] The visible light image restoration model is trained using a joint loss function consisting of reconstruction loss and contrast loss to obtain a trained visible light image restoration model.

[0032] Optionally, the joint loss function is expressed as:

[0033] ;

[0034] ;

[0035] ;

[0036] in, represents the joint loss function; represents the reconstruction loss; represents contrast loss; 、 represents a hyperparameter; represents the visible light restored image; Represents a visible light clean image; represents the structural similarity index; represents the L1 norm; Representing the feature representation of the visible light restored image; Feature representation of visible light clean image; represents the feature representation of the i-th noisy / degraded image; represents the temperature coefficient; Indicates the number of noisy / degraded images; represents the exponential function; Represents the similarity function.

[0037] In a second aspect, the present invention further provides an image restoration thermal fusion system, comprising:

[0038] An interference judgment module is used to obtain a visible light image, an infrared light image, and a low-light-level image, and judge whether the visible light image has environmental interference;

[0039] A first thermal fusion module is configured to: if the visible light image has environmental interference, input the visible light image into a pre-built visible light image restoration model to perform image restoration and output a visible light restored image; and thermally fuse the visible light restored image with the infrared image and / or the low-light image to obtain a first thermally fused image;

[0040] A model construction module is used for: wherein the visible light image restoration model includes a convolutional Mamba module and a multi-scale selection kernel module connected in sequence, and the outputs of the visible light image and the multi-scale selection kernel module are obtained through a residual module to obtain a visible light restoration image;

[0041] The convolutional Maba module is obtained by adding a convolutional layer before the SSM projection layer in the Mamba module, and the weight of the convolutional Mamba module is dynamically adjusted through a gating unit; and the weight distribution of the multi-scale selection kernel module is dynamically adjusted through an attention mechanism.

[0042] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image restoration and thermal fusion method as described in the first aspect.

[0043] The fourth invention provides a computer device including a processor and a storage medium;

[0044] The storage medium is used to store instructions;

[0045] The processor is configured to operate according to the instructions to execute the method according to the first aspect.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] By restoring visible light images, the present invention can improve the quality of visible light images in rainy nights, dense fog, snowy days, and smoky environments. Even under these adverse conditions, it can ensure the high clarity and rich details of the images, enhance the accuracy and reliability of target recognition, and effectively remove noise and restore true colors to address the processing difficulties of dark yellow images and noisy images. It can improve the overall quality of the image, so that clear and recognizable images can still be obtained in low-light and interference environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 FIG2 is a schematic flow chart of an image restoration and thermal fusion method according to an embodiment of the present invention;

[0049] Figure 2 FIG2 is a schematic structural diagram of a visible light image restoration component according to an embodiment of the present invention;

[0050] Figure 3 FIG2 is a schematic structural diagram of a visible light image restoration model according to an embodiment of the present invention;

[0051] Figure 4 FIG2 is a schematic diagram of the structure of a convolutional Mamba module in one embodiment of the present invention;

[0052] Figure 5 Shown is a schematic structural diagram of an image restoration thermal fusion system in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0054] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0055] Example 1

[0056] like Figure 1 As shown, this embodiment introduces an image restoration thermal fusion method, such as Figure 5 As shown, the system includes a first environment perception component, a second environment perception component, a decision control module, a protection component, a visible light image recognition component, a visible light image restoration component and an image thermal fusion component.

[0057] The first environmental perception component includes a decibel monitoring module, an air monitoring module, a body sensing monitoring module and a visible light collection module. The body sensing monitoring module includes monitoring of temperature and wind direction. The first environmental perception component integrates a series of sensors and monitoring devices for sensing and analyzing multiple key parameters of the surrounding environment.

[0058] The second environmental perception component includes an infrared light image acquisition module and a low-light level image acquisition module.

[0059] The judgment control module receives data information from the first environmental perception component and sets thresholds or related parameters to automatically judge the decibel, air, and body sensation data obtained by the first environmental perception component. When the value is within a reasonable range acceptable to the user, the judgment result is normal; otherwise, the judgment result is abnormal.

[0060] When the judgment result is abnormal, the protection component is automatically controlled and corresponding effective protection measures are taken to protect the user from damage caused by the environmental situation.

[0061] The protection components include a noise reduction module, an alarm module, a temperature prompt module, and a wind direction prompt module.

[0062] In battlefield environments, the visible light image discrimination component of thermal fusion night vision devices faces multiple challenges, especially in conditions such as severe weather and tactical smoke. Before thermal fusion of visible light, infrared light, and low light, visible light images can be discriminated first.

[0063] ① During the day, there is ample sunlight, rich colors, and high contrast. Image restoration requirements are relatively low, unless there is strong direct sunlight causing overexposure or backlight causing underexposure. In this case, high dynamic range (HDR) technology or local contrast enhancement is required.

[0064] ② Low light levels at night may increase image noise and cause loss of detail. High image restoration requirements require the use of low-light enhancement techniques such as noise reduction, brightness, and contrast adjustment to restore image details.

[0065] ③ During daytime fog, visibility is reduced, images may be blurry, and color saturation may decrease. This requires image restoration, and fog-penetrating algorithms, such as Dark Channel Prior, are needed to enhance image clarity and color saturation.

[0066] ④ Smoke can obstruct vision, blurring images. In particularly thick smoke, images can become almost invisible. This requires a smoke-penetrating algorithm, combined with image segmentation and edge enhancement, to recover objects obscured by smoke.

[0067] ⑤ Raindrops can interfere with the lens, causing streaks and spots in the image. High image restoration requirements necessitate the use of raindrop removal algorithms, such as deep learning-based methods, to remove raindrop traces and restore image clarity.

[0068] ⑥ When snow falls, the ground and air are filled with white particles, reducing contrast and detail. This places high demands on image restoration, requiring snow particle removal and contrast enhancement algorithms to restore image clarity and detail.

[0069] ⑦ Lightning is accompanied by strong instantaneous sunlight, which may cause overexposure and place high demands on image restoration. It requires dynamic exposure control or highlight suppression technology in post-processing.

[0070] ⑧ At the moment of artillery firing, the extremely bright flash may temporarily blind the image sensor, necessitating high-level image restoration, requiring automatic exposure control and overexposure correction technology. The physical vibrations associated with the shock wave may affect image stability, potentially requiring image restoration, requiring electronic image stabilization (EIS) or optical image stabilization (OIS).

[0071] ⑨ Heat waves are caused by rising hot air, which causes image distortion. The image restoration requirements are generally general, and image stabilization algorithms and thermal image compensation technologies are required.

[0072] The visible light image restoration component consists of a visible light image restoration model and a loss function.

[0073] The image fusion component can switch or fuse two or three images among infrared, low-light and visible light according to specific circumstances under different environments and tactical requirements to achieve the best visual effects and information acquisition capabilities.

[0074] For example:

[0075] In a completely dark environment, the infrared image acquisition module becomes critical because it is unaffected by darkness and generates images by detecting the temperature difference between the target and the background. This capability is extremely effective in night reconnaissance, identification of hidden enemy targets, and location of heat sources.

[0076] In low-light conditions, such as moonlight or starlight, the low-light image acquisition module can capture the weak light in the environment and provide images close to natural colors. This is especially useful for identifying camouflage, terrain features, and distant objects. In this case, low-light technology supplements the color information and details that infrared images may lack.

[0077] In environments with available visible light, such as at dawn or dusk, the visible light acquisition module in the first environmental perception module can provide the clearest and most realistic images. Under these conditions, visible light images can be fused with low-light or infrared images to enhance detail recognition, especially in missions requiring high visual accuracy, such as precision strike target confirmation.

[0078] In smoke or light fog, infrared image acquisition modules can penetrate these obstacles, while low-light image acquisition modules may be affected. In this case, fusing infrared images with visible light images can provide a comprehensive view that includes both temperature contrast and shape and details.

[0079] The method comprises:

[0080] Step 1: Image and environmental information collection, specifically:

[0081] The first environmental perception component is activated to collect visible light images and the perceived environmental values of the images, including ambient temperature, ambient humidity, ambient volume, and ambient gas composition. At the same time, the second environmental perception component is activated to obtain infrared light images and low-light images.

[0082] Step 2: Environmental parameter judgment and module control, specifically:

[0083] The decision control module comprehensively analyzes the information provided by the first environment perception component, comprehensively analyzes the perceived environment value of the image, and determines whether the perceived environment value of the image exceeds a threshold;

[0084] If the perceived environment value of the image exceeds the preset threshold, the corresponding protection module in the protection component is activated to take protective measures for the perceived environment of the image;

[0085] If the perceived environment value of the image does not exceed the preset threshold, no protective measures are taken for the perceived environment of the image.

[0086] Protection measures include environmental noise reduction, environmental alarms, environmental temperature prompts, and environmental wind direction prompts, etc.

[0087] Step 3: Initial analysis and judgment of visible light images, specifically:

[0088] The visible light image recognition component receives the visible light image from the first environmental perception component and preliminarily analyzes the quality of the visible light image to determine whether it is interfered with by adverse weather conditions (such as rain, fog, snow, smoke and dust):

[0089] If the quality of the visible light image is poor and there is environmental interference, that is, noise, dark yellow, fog, rain, snow, smoke, etc., the visible light image is input into a pre-built visible light image restoration model for image restoration processing, and a visible light restored image is output; then, the visible light restored image is thermally fused with the low-light level image and / or the infrared image to obtain a first thermally fused image;

[0090] If the quality of the visible light image is good and there is no environmental interference, the visible light image, the low-light image and / or the infrared image may be directly thermally fused to obtain a second thermally fused image;

[0091] If the visible light image has environmental interference and cannot be restored, that is, the image is completely black or difficult to restore, the visible light image is discarded, and the low-light image and the infrared image are thermally fused to obtain a third thermally fused image.

[0092] Step 4: Implement protective measures, specifically:

[0093] According to the instructions of the judgment control module, the protection component calls different internal modules to protect the user from damage caused by harsh environmental factors (such as high temperature, chemicals, toxic gases, etc.) and ensure the continuous and stable operation of the equipment.

[0094] Step 5: Image restoration processing, specifically:

[0095] When the visible light image discrimination component confirms that the image quality is poor, the visible light image restoration model automatically intervenes and applies the image restoration algorithm to process problems such as dark yellow images and noisy images, restoring the image clarity and details.

[0096] Fog, smoke, raindrops, and snowflakes can interfere with visible light images, creating spots and lines that affect image clarity and target identification. In visible light image restoration models, image processing algorithms are used to remove noise caused by rain and snow, while thermal imaging is relied upon to determine the target's position and shape, as thermal imaging is unaffected by precipitation.

[0097] The intense light produced by a white phosphorus bomb explosion can temporarily blind visible light sensors, causing image distortion or temporary failure. The system automatically adjusts sensor exposure based on light intensity to prevent overexposure caused by strong light. A visible light image restoration model is used to restore "bright spots" in visible light images, while thermal imaging is used to maintain target tracking.

[0098] Heat waves generated by the ground or high-temperature areas can cause changes in the air's refractive index, leading to image distortion. Therefore, image stabilization technology is implemented using a visible light image restoration model, using thermal imaging as a reference to correct for heat-induced image distortion and maintain a clear target outline.

[0099] In low-light conditions, image sensors may generate noise, degrading image quality. Using a visible light image restoration model and advanced digital signal processing techniques, such as noise reduction algorithms, we can reduce or eliminate noise in images, restore noisy and dark images, and maintain image clarity.

[0100] In actual battlefield applications, the visible light image recognition components of thermal fusion night vision devices need to be able to automatically or manually adjust to various environmental conditions. For example, in desert areas, heat waves and sandstorms may be common phenomena, while in forests or urban environments, smoke and artificial lighting may pose challenges. Thermal fusion night vision devices combine the advantages of visible light, infrared thermal, and low-light imaging. They can fuse two or more types of imagery through algorithms to provide clear target identification even in adverse conditions, which is crucial for tactical reconnaissance, target positioning, and battlefield navigation.

[0101] Therefore, for visible light images, the visible light image restoration model of the thermal fusion night vision device needs to have advanced image processing capabilities and environmental adaptability, including adjusting gain, exposure time, contrast, and brightness, as well as implementing image enhancement and denoising algorithms. In addition, to maintain effectiveness in complex and changing environments, the device also needs to have a certain degree of intelligence and be able to automatically optimize settings based on environmental conditions to ensure that users can obtain the best visual assistance in various situations.

[0102] like Figure 3 As shown in , the visible light image restoration model includes a convolutional Mamba module and a multi-scale selection kernel module connected in sequence, and the output of the visible light image and the multi-scale selection kernel module is obtained through a residual module to obtain a visible light restoration image to retain the original information; Figure 4 As shown, by adding a convolution layer before the SSM projection layer in the Mamba module, the convolutional Maba module is obtained, and the weight of the convolutional Mamba module is dynamically adjusted through the gating unit; the weight distribution of the multi-scale selection kernel module is dynamically adjusted through the attention mechanism;

[0103] The convolutional Mamba module suppresses image degradation caused by inclement weather. The multi-scale selective kernel module separates feature maps containing weather degradation information such as rain, snow, fog, and smoke into different scales, further helping the network restore high-quality images. This approach first suppresses global interference (such as haze) using the convolutional Mamba module, and then extracts local details (such as raindrops and smoke) using the multi-scale selective kernel, rather than using a traditional parallel architecture. The visible light restoration model leverages the multi-scale selective kernel module to further achieve a multi-scale receptive field.

[0104] In the convolutional Mamba module, local features and global features of the visible light image are extracted respectively, and the local features and global features are fused by suppressing environmental interference to obtain a suppressed feature map, including:

[0105] Use 3×3 depthwise convolution to extract local noise and details and output local features ,Right now:

[0106] ;

[0107] Local features Input SSM projection layer, model global dependencies through state space equations, and output global features ,Right now:

[0108] ;

[0109] Dynamically fuse local features through the Gating Unit With global features , and obtain the suppression feature map :

[0110] ;

[0111] in, Represents local features; Represents 3×3 convolution; Represents a visible light image; Represents global features; represents the SSM projection; represents the suppression feature map; Represents the weight of the convolutional Mamba module, obtained through the gate unit; represents element-wise multiplication; Represents the sigmoid activation function.

[0112] In the multi-scale selection kernel module, multi-scale feature extraction and fusion are performed on the suppression feature map to obtain a visible light restoration image, including:

[0113] Suppression feature map Perform multi-branch convolution, in this embodiment, 3-branch convolution, namely:

[0114] ;

[0115] Calculate the weight of each convolution branch and obtain it through the attention mechanism, that is:

[0116] ;

[0117] Merge each branch, that is:

[0118] ;

[0119] The suppressed feature map output by the convolutional Mamba is used as the input of the multi-scale module. The original information is retained through the cross-module residual connection, and the visible light restoration image is output, that is:

[0120] ;

[0121] in, 、 、 Represent the convolution results of each convolution branch respectively; represents the suppression feature map; 、 、 Represent 3×3 convolution, 5×5 convolution, and 7×7 convolution respectively; represents the weight of the j-th branch, obtained through the attention mechanism; 、 、 They represent the weight of the 3×3 convolution branch, the weight of the 5×5 convolution branch, and the weight of the 7×7 convolution branch respectively; represents the Softmax activation function; represents the MLP layer; represents global average pooling; Represents the characteristics after the fusion of each branch; represents the visible light restored image; Represents a visible light image.

[0122] Among them, the training of the visible light image restoration model includes:

[0123] like Figure 2As shown, a joint loss function consisting of reconstruction loss (Reconstruction Loss) and contrast loss (Contrastlearning Loss) is used to train the visible light image restoration model to obtain a trained visible light image restoration model. Reconstruction Loss can assist in reconstructing a more realistic low-interference image, and Contrastlearning Loss can make full use of the feature information provided by images interfered with by rain, snow, fog, smoke, etc. and clean images to restore the visible light image.

[0124] The joint loss function is expressed as:

[0125] ;

[0126] ;

[0127] ;

[0128] in, represents the joint loss function; represents the reconstruction loss; represents contrast loss; 、 represents the hyperparameter, =1.0, =0.5; Represents the visible light restored image (model output); Represents the visible light clean image (Ground Truth); Structural Similarity Index, which measures the similarity between two images in terms of brightness, contrast, and structure (the range is [0, 1], where 1 indicates complete identity). represents the L1 norm, which calculates the pixel-level absolute error between the visible light restored image and the visible light clean image; Represents the feature representation of the visible light restored image (usually extracted by CNN or Transformer); Feature representation of visible light clean image; represents the feature representation of the i-th noisy / degraded image; represents the temperature coefficient; Indicates the number of noisy / degraded images; represents the exponential function used for probability normalization in contrastive loss; Represents a similarity function, usually cosine similarity, which calculates the similarity between two features.

[0129] Step 6: Image thermal fusion, specifically:

[0130] If there is no environmental interference in the visible light image, the image thermal fusion component directly fuses the visible light image with the infrared image and / or low-light image to generate an image containing fusion information;

[0131] If there is environmental interference in the visible light image, the image thermal fusion component will fuse the visible light restored image with the infrared image and / or low-light image to generate a composite image containing fusion information to improve the accuracy and reliability of target recognition;

[0132] If the visible light image is subject to environmental interference and image restoration is impossible, the image thermal fusion component will discard the visible light restored image and directly fuse the infrared image and the low-light image to generate an image containing fusion information.

[0133] Through the above steps, the method of this embodiment can effectively cope with various harsh environments, provide high-quality image output, and greatly enhance its adaptability and reliability under complex conditions. The final thermal fusion image is presented to the user through the display unit or transmitted to the remote monitoring center for use in application scenarios such as military reconnaissance, security monitoring, and fire rescue, providing users with more comprehensive and clear visual assistance.

[0134] Example 2

[0135] Based on Example 1, this example introduces an experimental example of an image restoration thermal fusion method:

[0136] In the context of jungle warfare:

[0137] Environmental Information Collection-Step 1:

[0138] At night, special forces conduct covert reconnaissance missions deep into enemy-controlled rainforests, capturing visible light images while also collecting temperature, humidity, sound, and gas composition data, and acquiring infrared and low-light images to gain information about the surrounding environment even in complete darkness.

[0139] Environmental parameter judgment and module control - Step 2:

[0140] If high humidity and abnormal gas composition are detected and these values are judged to exceed the safety threshold, the environmental alarm will be activated immediately to alert soldiers to potential chemical weapons threats.

[0141] Initial analysis and judgment of visible light images - step 3:

[0142] After analyzing the received visible light image, we found that the image quality was very poor due to dense foliage and low light conditions at night. The system decided to first feed the visible light image into the visible light image restoration model for image restoration.

[0143] Implementation of protective measures - Step 4:

[0144] Chemical protection has been further enhanced according to the Protection Directive, ensuring that users are protected from the effects of toxic gases even in the event of a chemical warfare agent release.

[0145] Image restoration process - step 5:

[0146] The visible light image restoration model extracts trees, terrain features, and possible outlines of man-made structures from blurred visible light images, restores noisy images, dark yellow images, and images with weather conditions such as rain, fog, snow, and smoke, improves the clarity of visible light images, and obtains visible light restored images.

[0147] Image Hot Fusion - Step 6:

[0148] The visible light restored image is fused with the infrared image and / or low-light image. The infrared image provides the heat distribution of the human body and machinery, while the low-light image captures the weak natural light under the starlight. The visible light restored image is fused with one or more of them to generate a clear fused image, thereby improving the ability to recognize the target.

[0149] Through this series of steps, the final thermal fusion image is transmitted to the soldier's helmet display or a large screen in the command center. This enables soldiers to accurately identify friend or foe, terrain, and other key information in the pitch-black darkness and complex jungle environments, greatly improving mission success and soldier safety. Therefore, the method of this embodiment plays a vital role in jungle warfare, ensuring the efficient operation and survivability of special forces in extreme conditions.

[0150] In the context of nature reserves:

[0151] Environmental Information Collection-Step 1:

[0152] Collect visible light images along with environmental parameters such as temperature, humidity, sound, and gas composition. At the same time, acquire infrared and low-light images, especially at night or in low-light conditions, in order to monitor wildlife activity patterns.

[0153] Environmental parameter judgment and module control - Step 2:

[0154] By analyzing the data, when abnormal sounds (such as engine sounds, human conversations) and unusual temperature increases (possibly the body temperature of an illegal intruder) are detected, it is determined that these parameters are beyond the normal range under natural conditions. At this time, environmental alarms, ambient temperature prompts, and ambient noise reduction are triggered to remind the staff of the nature reserve to prepare to take corresponding measures.

[0155] Initial analysis and judgment of visible light images - step 3:

[0156] The visible light images were checked, but the image quality was poor due to dense fog and low light at night. A visible light image restoration model was used to improve the image quality, while thermal imaging and low-light images were relied upon for preliminary monitoring.

[0157] Implementation of protective measures - Step 4:

[0158] In accordance with protection instructions, the frequency of monitoring of sensitive areas has been increased, and drones and ground patrols have been dispatched to early warning locations for verification to prevent illegal poaching or destruction.

[0159] Image restoration process - step 5:

[0160] When weather conditions are within the range that can be improved, the visible light image restoration model processes the previously collected low-quality images to remove the impact of fog and enhance the contrast, thereby obtaining clearer images, helping to identify the location and number of specific species and obtain visible light restored images.

[0161] Image Hot Fusion - Step 6:

[0162] The visible light restored image, as well as one or more of the infrared light image and the low-light image are fused to generate a fused image. This image can not only show the outline and position of the animal, but also reveal the differences in their body temperature, which is especially useful for distinguishing different types of wild animals.

[0163] The resulting thermal fusion images are transmitted in real time to the monitoring center and night vision devices, enabling nature reserve staff to quickly locate wildlife, track their migration paths, and promptly respond to any illegal activities. Furthermore, these images are used in scientific research projects to help scientists understand the behavioral habits of wildlife and assess the state of the ecological balance in nature reserves. This allows nature reserves to effectively protect rare species and prevent criminal activities, while providing valuable data for scientific research and promoting continuous improvement in ecological protection efforts.

[0164] Example 3

[0165] Based on Example 1 or 2, this example introduces an image restoration thermal fusion system, including:

[0166] Interference judgment module, used to obtain visible light images, infrared light images and low-light images;

[0167] A first thermal fusion module is configured to: if the visible light image has environmental interference, input the visible light image into a pre-built visible light image restoration model to perform image restoration and output a visible light restored image; and thermally fuse the visible light restored image with the infrared image and / or the low-light image to obtain a first thermally fused image;

[0168] A model construction module, configured to: wherein the visible light image restoration model comprises a convolutional Mamba module and a multi-scale selection kernel module connected in sequence, and a residual connection is formed between the input layer of the convolutional Mamba module and the input layer of the multi-scale selection kernel module;

[0169] The convolutional Maba module is obtained by adding a convolutional layer before the SSM projection layer in the Mamba module, and the weight of the convolutional Mamba module is dynamically adjusted through a gating unit; and the weight distribution of the multi-scale selection kernel module is dynamically adjusted through an attention mechanism.

[0170] The specific functional implementation of each of the above modules can be found in the relevant contents of the method in Example 1 or 2 and will not be elaborated here.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0175] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A method for image restoration and thermal fusion, characterized in that: include: Acquire visible light images, infrared images, and low-light images; If there is environmental interference in the visible light image, the visible light image is input into a pre-built visible light image restoration model to perform image restoration and output a visible light restored image; the visible light restored image is thermally fused with the infrared image and / or the low-light image to obtain a first thermally fused image; The visible light image restoration model includes a convolutional Mamba module and a multi-scale selection kernel module connected in sequence, and the outputs of the visible light image and the multi-scale selection kernel module are converted into a visible light restoration image through a residual module; The convolutional Maba module is obtained by adding a convolutional layer before the SSM projection layer in the Mamba module, and the weight of the convolutional Mamba module is dynamically adjusted through a gating unit; and the weight distribution of the multi-scale selection kernel module is dynamically adjusted through an attention mechanism.

2. The image restoration thermal fusion method according to claim 1, characterized in that: Also includes: If there is no environmental interference in the visible light image, thermally fuse the visible light image and the infrared light image and / or the low-light image to obtain a second thermally fused image; If the visible light image has environmental interference and cannot be restored, the infrared light image and the low-light image are thermally fused to obtain a third thermally fused image.

3. The image restoration thermal fusion method according to claim 1, characterized in that: The processing steps of the visible light image restoration model include: In the convolutional Mamba module, local features and global features of the visible light image are extracted respectively, and the local features and global features are fused with environmental interference suppression to obtain a suppression feature map; In the multi-scale selection kernel module, multi-scale feature extraction and fusion are performed on the suppressed feature map to obtain a visible light restored image.

4. The image restoration thermal fusion method according to claim 3, characterized in that: Extracting local features and global features of the visible light image respectively, performing environmental interference suppression fusion on the local features and the global features to obtain a suppression feature map, including: ； ； ； in, Represents local features; Represents 3×3 convolution; Represents a visible light image; Represents global features; represents the SSM projection; represents the suppression feature map; Represents the weight of the convolutional Mamba module, obtained through the gate unit; represents element-wise multiplication; Represents the sigmoid activation function.

5. The image restoration thermal fusion method according to claim 3, characterized in that: Performing multi-scale feature extraction and fusion on the suppressed feature map to obtain a visible light restored image, including: ； ； ； ； in, 、 、 Represent the convolution results of each convolution branch respectively; represents the suppression feature map; 、 、 Represent 3×3 convolution, 5×5 convolution, and 7×7 convolution respectively; represents the weight of the j-th branch, obtained through the attention mechanism; 、 、 They represent the weight of the 3×3 convolution branch, the weight of the 5×5 convolution branch, and the weight of the 7×7 convolution branch respectively; represents the Softmax activation function; represents the MLP layer; represents global average pooling; Represents the characteristics after the fusion of each branch; represents the visible light restored image; Represents a visible light image.

6. The image restoration thermal fusion method according to claim 1, characterized in that: The training of the visible light image restoration model includes: The visible light image restoration model is trained using a joint loss function consisting of reconstruction loss and contrast loss to obtain a trained visible light image restoration model.

7. The image restoration thermal fusion method according to claim 6, characterized in that: The joint loss function is expressed as: ； ； ； in, represents the joint loss function; represents the reconstruction loss; represents contrast loss; 、 represents a hyperparameter; represents the visible light restored image; Represents a visible light clean image; represents the structural similarity index; represents the L1 norm; Representing the feature representation of the visible light restored image; Feature representation of visible light clean image; represents the feature representation of the i-th noisy / degraded image; represents the temperature coefficient; Indicates the number of noisy / degraded images; represents the exponential function; Represents the similarity function.

8. An image restoration thermal fusion system, characterized in that: include: Interference judgment module, used to obtain visible light images, infrared light images and low-light images; A first thermal fusion module is configured to: if the visible light image has environmental interference, input the visible light image into a pre-built visible light image restoration model to perform image restoration and output a visible light restored image; and thermally fuse the visible light restored image with the infrared image and / or the low-light image to obtain a first thermally fused image; A model construction module is used for: wherein the visible light image restoration model includes a convolutional Mamba module and a multi-scale selection kernel module connected in sequence, and the outputs of the visible light image and the multi-scale selection kernel module are obtained through a residual module to obtain a visible light restoration image; The convolutional Maba module is obtained by adding a convolutional layer before the SSM projection layer in the Mamba module, and the weight of the convolutional Mamba module is dynamically adjusted through a gating unit; and the weight distribution of the multi-scale selection kernel module is dynamically adjusted through an attention mechanism.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image restoration and thermal fusion method according to any one of claims 1 to 7 are implemented.

10. A computer device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the method according to any one of claims 1 to 7.

Images