Dual-mode image fusion enhancement method, system, device and storage medium

Abstract

This invention relates to the fields of optical imaging and intelligent image processing technology, and discloses a dual-modal image fusion enhancement method, system, device, and storage medium. The method includes: acquiring initial dual-modal images obtained through coaxial synchronous transient acquisition of the same scene; fusing the initial dual-modal images based on a saliency matrix; performing differentiated adaptive noise reduction processing on the low-frequency and high-frequency components separated from the fused image; performing dynamic anti-reflection compensation processing on the reconstructed high- and low-frequency images; performing a two-dimensional fast Fourier transform and centering the spectrum; constructing a notch mask and a low-frequency protection region at the center of the spectrum to suppress the narrow-band energy peak when a narrow-band energy peak corresponding to sensor row and column noise is identified; and then performing an inverse Fourier transform to output the enhanced image. Through the above methods, this invention can improve image sharpness, local contrast, and stripe suppression, while reducing artifacts generated during the enhancement process.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of optical imaging and intelligent image processing technology, specifically to a dual-modal image fusion enhancement method, system, device and storage medium. Background Technology

[0002] With the rapid development of industrial inspection, security monitoring, autonomous driving, and drone inspection, infrared thermal imaging technology has been widely used due to its all-weather, anti-interference, and temperature measurement capabilities. However, infrared images that rely solely on temperature radiation information often suffer from low resolution, lack of texture details, and blurred edge contours, resulting in limited target recognition rates.

[0003] To overcome the limitations of single-modality imaging, existing technologies typically employ dual-modality or multi-modality fusion to improve image quality. For example, by adding a mechanically rotating polarizer in front of an infrared thermal imager, images with different polarization directions can be acquired and fused to enhance the edge and material details of the target; or a dual-path fusion scheme of infrared and visible light can be used.

[0004] While the aforementioned technologies can theoretically improve image quality, they still have the following significant drawbacks and shortcomings in practical engineering implementation: 1. Asynchronous acquisition timing can easily lead to motion artifacts: Since two modal images are acquired sequentially by mechanical switching or time-sharing exposure, when observing dynamic scenes or the movement of the carrier itself, there will inevitably be parallax and motion blur between the two frames, resulting in ghosting and misalignment in subsequent fusion, which seriously affects the fusion accuracy.

[0005] 2. Severe image degradation and poor environmental adaptability: Due to the physical characteristics of the infrared focal plane detector (IRFPA) and other factors, the acquired raw images are often accompanied by degradation phenomena such as noise, stripes and blurring.

[0006] Therefore, the image quality output by existing technologies through dual-modal fusion needs to be improved. Summary of the Invention

[0007] In view of the above problems, embodiments of the present invention provide a dual-modal image fusion enhancement method, system, device and storage medium to solve the problems existing in the prior art.

[0008] According to one aspect of the present invention, a dual-modal image fusion enhancement method is provided, the method comprising: Acquire dual-modal initial images I1 and I2, which are obtained by coaxial synchronous transient acquisition of the same scene; The initial bimodal images I1 and I2 are registered and fused to obtain a fused image; wherein, the saliency matrices of the registered initial bimodal images I1 and I2 are calculated respectively, and the fusion process is performed based on the saliency matrices; The fused image is separated into low-frequency components and high-frequency components, and differential adaptive noise reduction processing is performed on the low-frequency components and the high-frequency components respectively to obtain the noise-reduced low-frequency components and high-frequency components. The denoised low-frequency components and high-frequency components are recombined to obtain the reconstructed image; Perform dynamic anti-reflection compensation processing on the reconstructed image; The compensated image is subjected to a two-dimensional fast Fourier transform and the spectrum is centered. Based on the spectrum, the existence of narrowband energy peaks corresponding to sensor row and column noise is identified. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency protection region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is then performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image.

[0009] In one alternative approach, the dual-mode initial images I1 and I2 are acquired at the same exposure time and on the same focal plane by a large-aperture metalens array composed of subwavelength nanoantennas. The effective light-passing diameter of the metalens array is 55 mm and the F-number is 0.365. The same incident field is simultaneously modulated by different metasurface units or different polarization selection regions, so that the dual-mode data are recorded synchronously on the same focal plane detector or adjacent detection regions, and are calibrated and mapped to be aligned dual-mode initial images I1 and I2.

[0010] In one optional approach, the step of calculating the saliency matrices of the registered bimodal initial images I1 and I2 respectively, and performing fusion processing based on the saliency matrices, includes: For each pixel in the registered bimodal initial images I1 and I2, a corresponding neighborhood window is defined with each pixel as the center. The sum of the absolute gray level differences of each pixel relative to all neighboring pixels in the corresponding neighborhood window is calculated to obtain the spatial neighborhood scatter matrix D1 of the first modality image and the spatial neighborhood scatter matrix D2 of the second modality image. The spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are normalized respectively. Based on the normalized spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2, the fusion weight is calculated. The bimodal initial images I1 and I2 are fused according to the fusion weight to obtain the fused image.

[0011] In an alternative approach, the spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are calculated using the following formulas: ; Where row and column are the total number of rows and columns of the first modal image or the second modal image, i and j are the coordinates of the pixel in the first modal image or the second modal image, g(i,j) is the gray value of the pixel, m and n are the coordinates of the neighboring pixels in the corresponding neighborhood window of the pixel, and g(m,n) is the gray value of the neighboring pixels. Ifuse = W1*I1 + W2*I2, (Image fusion) Where W1 and W2 are the fusion weights, W1=D1 / (D1+D2+ε), W2=D2 / (D1+D2+ε), and ε is the minimum constant.

[0012] In one alternative approach, the notch mask is dynamically adjusted according to the direction of the noise. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency guard region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image, including: If a narrowband energy peak corresponding to the column noise exists, a notch mask and a low-frequency protection region are constructed in the horizontal band at the center of the spectrum to eliminate the narrowband energy peak. If a narrowband energy peak corresponding to the line noise exists, a notch mask and a low-frequency protection region are constructed in the vertical band at the center of the spectrum to eliminate the narrowband energy peak. The edges of the constructed notch mask are smoothed using a Gaussian function, and the processed spectrum is subjected to an inverse Fourier transform to output the enhanced image.

[0013] In one alternative approach, after reconstructing the denoised low-frequency and high-frequency components to obtain the reconstructed image, the method further includes: The reconstructed image is subjected to edge-preserving reconstruction processing based on dual constraints of pixel grayscale difference and pixel spatial distance.

[0014] In one alternative approach, performing dynamic anti-reflection compensation processing on the reconstructed image includes: The background distribution of the reconstructed image is obtained by using a large-scale Gaussian low-pass filter, and the high-pass detail component is obtained by the difference between the reconstructed image and the background distribution. The high-pass detail component is then added back to the reconstructed image with a preset compensation intensity.

[0015] According to another aspect of the present invention, a dual-modal image fusion enhancement system is provided, the system comprising: The minimalist optical acquisition module includes a large-aperture superlens array composed of subwavelength nano-antennas, configured to coaxially and synchronously acquire dual-mode initial images of the same scene in transient mode. An image processing module is configured to execute the algorithm flow of the method as described in any one of claims 1 to 7; The control and storage module is electrically connected to the minimalist optical acquisition module and the image processing module, respectively.

[0016] According to another aspect of the present invention, a computer device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform the method described above.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the method described above.

[0018] This invention acquires dual-modal initial images obtained by coaxial synchronous transient acquisition of the same scene, calculates the saliency matrix of each dual-modal initial image, and fuses the dual-modal initial images based on the saliency matrix; separates the fused image into low-frequency and high-frequency components, and performs differentiated adaptive noise reduction on the low-frequency and high-frequency components respectively; reconstructs the noise-reduced low-frequency and high-frequency components; performs dynamic anti-reflection compensation on the reconstructed image; performs a two-dimensional fast Fourier transform on the compensated image and centers the spectrum; when narrowband energy peaks corresponding to sensor row and column noise are identified, a notch mask and a low-frequency protection region are constructed at the center of the spectrum to suppress the narrowband energy peaks; performs an inverse Fourier transform on the processed spectrum and outputs an enhanced image. This invention, through coaxial synchronous acquisition of initial images of both modalities, eliminates motion ghosting caused by asynchronous shooting, thereby improving the accuracy of dual-modal image fusion. Furthermore, through frequency band decomposition, differential restoration, anti-reflection compensation, and frequency domain stripe suppression algorithms, it solves degradation problems such as noise, stripes, and blurring, ultimately outputting an enhanced image with improved clarity and local contrast and suppressed artifacts.

[0019] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A schematic flowchart of the dual-modal image fusion enhancement method provided in an embodiment of the present invention is shown; Figure 2 This diagram illustrates the acquisition of visible light and infrared images in the dual-modal image fusion enhancement method provided in this embodiment of the invention. Figure 3 A schematic diagram of the structure of the dual-modal image fusion enhancement system provided in an embodiment of the present invention is shown; Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0021] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0022] Figure 1 The flowchart of the dual-modal image fusion enhancement method according to an embodiment of the present invention is shown. This method is executed by a computer, such as a laptop, tablet, or smart terminal. Figure 1 As shown, the method includes the following steps: Step 10: Obtain the dual-modal initial images I1 and I2, which are obtained by coaxial synchronous transient acquisition of the same scene.

[0023] The dual-mode can be an infrared intensity mode and an infrared polarization mode, or an infrared intensity mode and a visible light mode, or infrared polarization modes with different polarization angles, or an infrared intensity mode and a phase / structure modulation mode, etc. Preferably, the dual-mode in this embodiment is an infrared intensity mode and an infrared polarization mode.

[0024] Traditional methods, such as mechanical switching or time-sharing exposure, acquire images of two modalities sequentially. When observing dynamic scenes or the movement of the carrier itself, parallax and motion blur inevitably exist between the two images, leading to ghosting and misalignment in subsequent fusion, thus affecting fusion accuracy. This embodiment uses coaxial synchronous transient acquisition of the same scene to obtain initial images of both modalities. This synchronized acquisition timing eliminates motion ghosting caused by asynchronous shooting, thereby improving the fusion accuracy of dual-modal images.

[0025] It is worth noting that dual-modal initial images can be acquired coaxially or near coaxially; no specific limitation is made here.

[0026] In an optional embodiment, the dual-mode initial images I1 and I2 are acquired at the same exposure time and on the same focal plane by a large-aperture metalens array composed of subwavelength nanoantennas. The effective light-passing diameter of the metalens array is 55 mm, and the F-number is 0.365. The metalens array achieves polarization multiplexing or band multiplexing by designing different geometric phases and transmission phases of the nanopillars. For the infrared intensity and infrared polarization dual-mode, some metasurface units in the metalens array selectively focus the incident infrared light without polarization to obtain the total infrared radiation intensity image, while the remaining metasurface units integrate specific orientation polarization selection functions to focus the corresponding linear polarization components to obtain the infrared polarization image. The two image signals are synchronously read out by a single exposure of the same focal plane detector, and after pixel-level calibration mapping and interpolation registration, spatially aligned dual-mode initial images I1 and I2 are obtained, realizing coaxial synchronous transient acquisition of the same scene.

[0027] Traditional dual-modal acquisition schemes (especially polarization imaging) heavily rely on physical polarizers and sophisticated mechanical rotation devices to achieve time-division acquisition. This architecture is not only bulky and power-intensive in terms of optical structure, but its mechanical structure is also prone to failure under vibration, failing to meet the urgent needs of modern portable devices (such as drones, robots, and individual soldier equipment) for lightweight and miniaturization. Unlike the traditional method of first capturing one mode and then mechanically switching to capture the other, this embodiment, using a metalens array, simultaneously modulates the same incident field through different metasurface units, different polarization selection regions, or different imaging channels. This allows two modes to be synchronously recorded by the same focal plane detector or adjacent detection areas at the same exposure time. Subsequently, calibration mapping registers the two images into a dual-modal initial image of the same scene.

[0028] In other embodiments, for both visible light and infrared modes, such as Figure 2 As shown, each of the two lenses corresponds to a detector. The aperture and focal length of the two lenses are slightly different. They can be physically adjusted so that the two detectors are located on the same plane and maintain the same field of view. Finally, the overlapping part of the two images is used as the initial image of the dual-mode for subsequent image fusion to achieve coaxial synchronous acquisition of infrared and visible light.

[0029] Step 20: Register and fuse the bimodal initial images I1 and I2 to obtain a fused image; wherein, in an optional embodiment, the saliency matrix of the registered bimodal initial images I1 and I2 can be calculated, and the fusion process can be performed based on the saliency matrix.

[0030] The process involves using image processing algorithms (e.g., local contrast, frequency domain analysis, or deep learning models) to detect regions in the bimodal initial images I1 and I2 that attract human visual attention, generating grayscale images. Higher values ​​in the grayscale images indicate greater importance of the location, such as high-temperature targets in infrared light or texture edges in visible light.

[0031] Bimodal image fusion using saliency matrices involves first calculating the visual saliency maps of the initial bimodal images I1 and I2, i.e., the attention value matrix for each pixel. These maps are then normalized into dynamic weights, and the multi-scale components of the image (e.g., low-frequency structure, high-frequency details) are weighted and fused according to these weights. During the fusion process, regions with high saliency are automatically assigned higher weights to their corresponding modalities. For example, more infrared thermal radiation information is retained in salient areas of infrared targets, while more detail is retained in salient areas of visible light textures, thus accurately highlighting key targets and adaptively preserving complementary details.

[0032] Of course, image fusion can also be performed in other ways, such as multi-scale transformation fusion or deep learning feature-based fusion, etc., which are not limited here.

[0033] In an optional embodiment, the saliency matrices of the registered bimodal initial images I1 and I2 are calculated respectively, and the bimodal initial images I1 and I2 are fused based on the saliency matrices to obtain a fused image, including: For each pixel in the bimodal initial images I1 and I2, a corresponding neighborhood window is defined with each pixel as the center. The sum of the absolute gray level differences of each pixel relative to all neighboring pixels in the corresponding neighborhood window is calculated to obtain the spatial neighborhood scatter matrix D1 of the first modality image and the spatial neighborhood scatter matrix D2 of the second modality image. The spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are normalized respectively. Based on the normalized spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2, the fusion weight is calculated. The bimodal initial images I1 and I2 are fused according to the fusion weight to obtain the fused image.

[0034] The spatial neighborhood scatter matrix refers to the cumulative measure of the grayscale difference between each pixel and its local neighborhood pixels. It is used to describe the local structural saliency of the pixel's location in the corresponding modality. In this embodiment, the spatial neighborhood scatter matrix is ​​used as the saliency matrix to fuse the image. Fusion through spatial neighborhood scatter utilizes the statistical differences within the local spatial neighborhood to measure the information richness around the pixel or the consistency between modalities, and constructs adaptive weights accordingly for image fusion.

[0035] Furthermore, the spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are calculated using the following formulas: ; Where row and column are the total number of rows and columns corresponding to the first modal image or the second modal image, i and j are the coordinates of the pixel in the first modal image or the second modal image, g(i,j) is the gray value of the pixel, m and n are the coordinates of the neighboring pixels in the corresponding neighborhood window of the pixel, and g(m,n) is the gray value of the neighboring pixels. Ifuse = W1*I1 + W2*I2, (Image fusion) Where W1 and W2 are the fusion weights, W1=D1 / (D1+D2+ε), W2=D2 / (D1+D2+ε), and ε is the minimum constant.

[0036] The neighborhood window refers to a local window centered on the current pixel, such as a 3×3 or 5×5 window, which captures contextual information using the image's inherent grid topology. When fusing images, individual pixels are not processed separately; instead, the fusion weights are dynamically adjusted based on the spatial neighborhood divergence value. If a modality has a high divergence in its local neighborhood, it indicates a large amount of information and clear edges, thus being assigned a higher weight, and vice versa, thereby significantly improving the visual quality of the fused image.

[0037] Step 30: Separate the fused image into low-frequency components and high-frequency components, and perform differentiated adaptive noise reduction processing on the low-frequency components and the high-frequency components respectively to obtain the noise-reduced low-frequency components and high-frequency components.

[0038] This embodiment utilizes Gaussian blur. The fused image is separated into low-frequency and high-frequency components. The low-frequency component contains basic thermal distribution and large-scale structure, while the high-frequency component contains edge details and noise. For example, a Gaussian low-pass filter is used to obtain the low-frequency component. ,in Taking 5.0, the high-frequency components are obtained from H=IL. The larger the value, the more low-frequency background and thermal distribution are included; The smaller the value, the more edges and details will enter the high-frequency components.

[0039] Differential adaptive denoising is applied to the separated low-frequency and high-frequency components using iterative restoration processes based on depth plug-and-play priors: conservative restoration is applied to the low-frequency components to stabilize the background and large-scale thermal distribution, while relatively aggressive deconvolution denoising and detail restoration are applied to the high-frequency components to suppress sensor noise, blurring, and fragmented artifacts while preserving structural contours.

[0040] Specifically, since low-frequency components mainly consist of thermal distribution and large-scale structures, conservative parameters are used for DPIR deconvolution denoising to avoid over-sharpening of the background or artifacts. In a set of feasible parameters, the noise level of the low-frequency components can be set to 25 / 255, the first model scale parameter for iterative scheduling can be set to 8, the second model scale parameter to 35, and the weighting coefficient to 0.85, with approximately three rounds of iterative restoration. High-frequency components mainly consist of edges, fine lines, local textures, and noise; therefore, relatively aggressive detail recovery parameters are used for high-frequency components. In a set of feasible parameters, the noise level of the high-frequency components can be set to 12 / 255, the first model scale parameter to 4, the second model scale parameter to 20, and the weighting coefficient to 0.80, with approximately four rounds of iterative restoration. These model scale parameters are used to control the noise intensity scheduling and prior constraint strength in the plug-and-play restoration process, making the low-frequency components more smooth and stable, and the high-frequency components more edge-oriented.

[0041] Therefore, this embodiment effectively removes noise and improves the signal-to-noise ratio by decoupling the frequency bands and then performing adaptive noise reduction separately.

[0042] Step 40: Reconstruct the denoised low-frequency components and high-frequency components to obtain the reconstructed image.

[0043] In this method, the reconstructed image R = clip(L + H, 0, 255) is generated by directly adding the processed low-frequency component L and the processed high-frequency component H, and then truncating the pixel values ​​to the effective range of 0-255 to complete the image reconstruction and prevent data overflow. In other optional embodiments, adaptive weights can be set for different regions, so that the proportion of high-frequency components is appropriately increased in edge regions and the proportion of low-frequency components is appropriately increased in flat regions. However, in the main processing flow of this embodiment, low-frequency components and high-frequency components are directly reconstructed.

[0044] In an optional embodiment, after reconstructing the denoised low-frequency components and high-frequency components to obtain a reconstructed image, the method further includes performing edge-preserving reconstruction processing on the reconstructed image based on dual constraints of pixel grayscale difference and pixel spatial distance.

[0045] Edge-preserving reconstruction applies smoothing constraints to pixels with similar gray levels and spatial distances in the reconstructed image. This further smooths uniform background areas while preventing weighted averaging across boundaries at points of significant gray-level transitions. Simultaneously smoothing noise in flat areas of the image, it uses pixel gray-level constraints within the value range to block weighted averaging across edges, preserving edge sharpness and outputting a smooth reconstructed image that retains its edges. Edge-preserving reconstruction can be implemented using bilateral filtering or an equivalent value-spatial joint weighted filtering method.

[0046] Step 50: Perform dynamic anti-reflection compensation processing on the reconstructed image.

[0047] First, a large-scale Gaussian low-pass filter is applied to the reconstructed image to estimate the slowly changing background and hazy substrate. Then, the low-pass substrate is subtracted from the reconstructed image to obtain a high-pass detail component that reflects local contrast, edge, and detail variations. Subsequently, the high-pass detail component is added back to the reconstructed image with a preset compensation intensity, which enhances the mid-to-high frequency structures that were originally obscured by the hazy background. At the same time, since the low-pass substrate is still preserved, the flat background and overall thermal distribution are not directly deleted.

[0048] Further, the dynamic anti-reflection compensation processing on the reconstructed image includes: The background distribution of the reconstructed image is obtained by using a large-scale Gaussian low-pass filter, and the high-pass detail component is obtained by the difference between the reconstructed image and the background distribution. The high-pass detail component is then added back to the reconstructed image with a preset compensation intensity.

[0049] Specifically, firstly, a large-scale Gaussian low-pass filter is used to estimate the background and slowly changing heat distribution in the reconstructed image, resulting in a low-pass background map. Then, this low-pass background map is subtracted from the reconstructed image to obtain a high-pass detail component containing both positive and negative responses. Finally, this high-pass detail component is added back to the reconstructed image at a preset intensity. For example, the high-pass compensation filter scale can be 30 pixels, corresponding to a Gaussian standard deviation of one-sixth of the filter scale, and the compensation intensity can be 0.7. This processing can enhance edges and textures at scales higher than the background variation while preserving the original brightness baseline, improving fogging, blurring, and low-contrast degradation.

[0050] Step 60: Perform a two-dimensional fast Fourier transform on the compensated image and center the spectrum. Based on the spectrum, identify whether there is a narrowband energy peak corresponding to sensor row and column noise.

[0051] In this embodiment, a two-dimensional fast Fourier transform is first performed on the compensated image to convert the image from the spatial domain to the frequency domain, and the spectrum is shifted to center the low frequency. When the presence of sensor row and column fixed pattern noise is pre-determined or detected, the corresponding narrowband energy region is located in the spectrum according to the noise direction.

[0052] In the actual manufacturing process of infrared focal plane detectors, horizontal and vertical stripes are common. Vertical stripes are usually caused by fixed-mode noise of the sensor column, which stems from inconsistent gain or bias of the readout circuits of each column. In the centered spectrum, they usually correspond to a narrow horizontal band of energy passing through the center of the spectrum. Horizontal stripes are usually caused by fixed-mode noise of the sensor row or power supply interference. In the centered spectrum, they usually correspond to a narrow vertical band of energy passing through the center of the spectrum.

[0053] Step 70: If a corresponding narrowband energy peak exists, a notch mask and a low-frequency protection region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image.

[0054] This embodiment suppresses stripes by constructing a directional notch mask at the center of the spectrum, covering and attenuating narrow-band energy regions within the spectrum. Furthermore, since the center of the spectrum represents low-frequency information of the image, such as the overall outline, large-area grayscale distribution, and average brightness, a low-frequency protection region is set at the center of the spectrum to preserve the main low-frequency structure. This region allows spectral components to pass through. Finally, an inverse Fourier transform is used to recover the image to the spatial domain, outputting an enhanced image where stripes are suppressed and the main structure is preserved.

[0055] In an optional embodiment, the notch mask is dynamically adjusted according to the direction of the noise. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency guard region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image, including: If a narrowband energy peak corresponding to column noise exists, a notch mask and a low-frequency protection region are constructed in the horizontal band at the center of the spectrum to eliminate the narrowband energy peak. If a narrowband energy peak corresponding to line noise exists, a notch mask and a low-frequency protection region are constructed in the vertical band at the center of the spectrum to eliminate the narrowband energy peak. The edges of the constructed notch mask are smoothed using a Gaussian function, and the processed spectrum is subjected to an inverse Fourier transform to output the enhanced image.

[0056] In this embodiment, column noise and row noise have different narrowband energy directions in the centered spectrum. If vertical stripes corresponding to column noise exist, a notch mask and a low-frequency protection region are constructed in the horizontal band at the center of the spectrum; if horizontal stripes corresponding to row noise exist, a notch mask and a low-frequency protection region are constructed in the vertical band at the center of the spectrum. The width of the low-frequency protection region can be 15 pixels, and the notch bandwidth can be set from 3 to 7 pixels depending on the stripe intensity, for example, 5 pixels. This suppresses both horizontal and vertical stripes and protects the main image data.

[0057] Furthermore, if the edges of the notch mask are abruptly abruptly changed between 0 and 1, i.e., hard cutoff, a ringing effect will occur when the image is restored to the spatial domain by inverse transform, i.e., wavy lines appear at the image edges. In order to make the image edges smooth and not abrupt, this embodiment applies Gaussian blur to the edges of the notch mask, so that there is a smooth transition between the two states of the notch mask, which completely retains 1 and completely filters out 0, thereby eliminating the ringing effect caused by hard cutoff.

[0058] Compared with existing technologies, the embodiments of the present invention acquire dual-modal initial images obtained by coaxial synchronous transient acquisition of the same scene, calculate the saliency matrix of the dual-modal initial images respectively, and perform fusion processing on the dual-modal initial images based on the saliency matrix; separate the fused image into low-frequency components and high-frequency components, and perform differentiated adaptive noise reduction processing on the low-frequency components and high-frequency components respectively; reconstruct the noise-reduced low-frequency components and high-frequency components; perform dynamic anti-reflection compensation processing on the reconstructed image; perform two-dimensional fast Fourier transform on the compensated image and center the spectrum, and identify whether there are narrowband energy peaks corresponding to sensor row and column noise based on the spectrum; if they exist, construct a notch mask and a low-frequency protection region at the center of the spectrum to eliminate the narrowband energy peaks, perform inverse Fourier transform on the spectrum after eliminating the narrowband energy peaks, and output an enhanced image. This invention, through coaxial synchronous acquisition of initial images of both modalities, eliminates motion ghosting caused by asynchronous shooting, thereby improving the accuracy of dual-modal image fusion. Furthermore, by employing algorithms for frequency domain separation and noise reduction, anti-reflection compensation, and frequency domain elimination of narrowband energy peaks, it solves degradation problems such as noise, stripes, and blurring, ultimately outputting high-definition, artifact-free, high-quality enhanced images.

[0059] like Figure 3 As shown, Figure 3 This diagram illustrates the structure of a dual-modal image fusion enhancement system provided in an embodiment of the present invention. The system includes: The minimalist optical acquisition module 310 includes a large-aperture superlens array composed of subwavelength nano-antennas, configured to coaxially and synchronously acquire dual-mode initial images of the same scene in transient mode. Image processing module 330 is configured to execute the algorithm flow of the above-described embodiment of the dual-modal image fusion enhancement method; The control and storage module 320 is electrically connected to the simplified optical acquisition module and the image processing module, respectively.

[0060] The embodiments of this invention employ a minimalist superlens array in hardware. While achieving coaxial synchronous transient acquisition of dual-modal initial images and eliminating motion ghosting caused by asynchronous shooting, it can reduce the volume of the optical structure. Furthermore, through the image processing module, frequency band decomposition, differential restoration, anti-reflection compensation, and frequency domain stripe suppression algorithms are used to solve degradation problems such as noise, stripes, and blurring. Finally, an enhanced image with improved clarity and local contrast and suppressed artifacts is output.

[0061] Figure 4 The diagram shows a structural schematic of an embodiment of the computer device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computer device.

[0062] like Figure 4As shown, the computer device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0063] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other computer devices, such as clients or other server network elements. The processor 402 executes program 410, specifically performing the relevant steps in the above method embodiments.

[0064] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0065] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0066] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0067] Specifically, program 410 can be called by processor 402 to cause the computer device to perform the following operations: Acquire dual-modal initial images I1 and I2, which are obtained by coaxial synchronous transient acquisition of the same scene; The initial bimodal images I1 and I2 are registered and fused to obtain a fused image; wherein, the saliency matrices of the registered initial bimodal images I1 and I2 are calculated respectively, and the fusion process is performed based on the saliency matrices; The fused image is separated into low-frequency components and high-frequency components, and differential adaptive noise reduction processing is performed on the low-frequency components and the high-frequency components respectively to obtain the noise-reduced low-frequency components and high-frequency components. The denoised low-frequency components and high-frequency components are recombined to obtain the reconstructed image; Perform dynamic anti-reflection compensation processing on the reconstructed image; The compensated image is subjected to a two-dimensional fast Fourier transform and the spectrum is centered. Based on the spectrum, the existence of narrowband energy peaks corresponding to sensor row and column noise is identified. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency protection region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is then performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image.

[0068] In one alternative approach, the dual-mode initial images I1 and I2 are acquired at the same exposure time and on the same focal plane by a large-aperture metalens array composed of subwavelength nanoantennas. The effective light-passing diameter of the metalens array is 55 mm and the F-number is 0.365. The same incident field is simultaneously modulated by different metasurface units or different polarization selection regions, so that the dual-mode data are recorded synchronously on the same focal plane detector or adjacent detection regions, and are calibrated and mapped to be aligned dual-mode initial images I1 and I2.

[0069] In one optional approach, the saliency matrices of the registered bimodal initial images I1 and I2 are calculated separately, and a fusion process is performed based on the saliency matrices, including: For each pixel in the bimodal initial images I1 and I2, a corresponding neighborhood window is defined with each pixel as the center. The sum of the absolute gray level differences of each pixel relative to all neighboring pixels in the corresponding neighborhood window is calculated to obtain the spatial neighborhood scatter matrix D1 of the first modality image and the spatial neighborhood scatter matrix D2 of the second modality image. The spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are normalized respectively. Based on the normalized spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2, the fusion weight is calculated. The bimodal initial images I1 and I2 are fused according to the fusion weight to obtain the fused image.

[0070] In an alternative approach, the spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are calculated using the following formulas: ; Where row and column are the total number of rows and columns of the first modal image or the second modal image, i and j are the coordinates of the pixel in the first modal image or the second modal image, g(i,j) is the gray value of the pixel, m and n are the coordinates of the neighboring pixels in the corresponding neighborhood window of the pixel, and g(m,n) is the gray value of the neighboring pixels. Ifuse = W1*I1 + W2*I2, (Image fusion) Where W1 and W2 are the fusion weights, W1=D1 / (D1+D2+ε), W2=D2 / (D1+D2+ε), and ε is the minimum constant.

[0071] In one alternative approach, the notch mask is dynamically adjusted according to the direction of the noise. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency guard region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image, including: If a narrowband energy peak corresponding to the column noise exists, a notch mask and a low-frequency protection region are constructed in the horizontal band at the center of the spectrum to eliminate the narrowband energy peak. If a narrowband energy peak corresponding to the line noise exists, a notch mask and a low-frequency protection region are constructed in the vertical band at the center of the spectrum to eliminate the narrowband energy peak. The edges of the constructed notch mask are smoothed using a Gaussian function, and the processed spectrum is subjected to an inverse Fourier transform to output the enhanced image.

[0072] In one alternative approach, after reconstructing the denoised low-frequency and high-frequency components to obtain the reconstructed image, the method further includes: The reconstructed image is subjected to edge-preserving reconstruction processing based on dual constraints of pixel grayscale difference and pixel spatial distance.

[0073] In one alternative approach, performing dynamic anti-reflection compensation processing on the reconstructed image includes: The background distribution of the reconstructed image is obtained by using a large-scale Gaussian low-pass filter, and the high-pass detail component is obtained by the difference between the reconstructed image and the background distribution. The high-pass detail component is then added back to the reconstructed image with a preset compensation intensity.

[0074] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a computer device, causes the computer device to perform any of the above-described method embodiments.

[0075] This invention provides a computer program that can be invoked by a processor to cause a computer device to execute any of the above-described method embodiments.

[0076] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed on a computer, cause the computer to perform any of the above-described method embodiments.

[0077] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0078] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0079] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim.

[0080] Those skilled in the art will understand that modules in the computer device of the embodiments can be adaptively modified and placed in one or more computer devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or computer device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0081] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A dual-modal image fusion enhancement method, characterized in that, The method includes: Acquire dual-modal initial images I1 and I2, which are obtained by coaxial synchronous transient acquisition of the same scene; The initial bimodal images I1 and I2 are registered and fused to obtain a fused image. The saliency matrices of the registered initial bimodal images I1 and I2 are calculated respectively, and the fusion process is performed based on the saliency matrices. The fused image is separated into low-frequency components and high-frequency components, and differential adaptive noise reduction processing is performed on the low-frequency components and the high-frequency components respectively to obtain the noise-reduced low-frequency components and high-frequency components. The denoised low-frequency components and high-frequency components are recombined to obtain the reconstructed image; Perform dynamic anti-reflection compensation processing on the reconstructed image; The compensated image is subjected to a two-dimensional fast Fourier transform and the spectrum is centered. Based on the spectrum, the existence of narrowband energy peaks corresponding to sensor row and column noise is identified. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency protection region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is then performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image.

2. The method according to claim 1, characterized in that, The dual-mode initial images I1 and I2 are acquired at the same exposure time and on the same focal plane by a large-aperture metalens array composed of subwavelength nanoantennas. The effective light transmission diameter of the metalens array is 55 mm and the F number is 0.

365. The same incident field is simultaneously modulated by different metasurface units or different polarization selection regions, so that the dual-mode data are recorded synchronously in the same focal plane detector or adjacent detection areas. After calibration and mapping registration, they are aligned as the dual-mode initial images I1 and I2.

3. The method according to claim 1, characterized in that, The step of calculating the saliency matrices of the registered bimodal initial images I1 and I2 respectively, and performing fusion processing based on the saliency matrices, includes: For each pixel in the registered bimodal initial images I1 and I2, a corresponding neighborhood window is defined with each pixel as the center. The sum of the absolute gray level differences of each pixel relative to all neighboring pixels in the corresponding neighborhood window is calculated to obtain the spatial neighborhood scatter matrix D1 of the first modality image and the spatial neighborhood scatter matrix D2 of the second modality image. The spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2 are normalized respectively. Based on the normalized spatial neighborhood scatter matrix D1 and the spatial neighborhood scatter matrix D2, the fusion weight is calculated. The bimodal initial images I1 and I2 are fused according to the fusion weight to obtain the fused image.

4. The method according to claim 3, characterized in that, The spatial neighborhood divergence matrix D1 and the spatial neighborhood divergence matrix D2 are calculated using the following formulas: ； Where row and column are the total number of rows and columns of the first modal image or the second modal image, i and j are the coordinates of the pixel in the first modal image or the second modal image, g(i,j) is the gray value of the pixel, m and n are the coordinates of the neighboring pixels in the corresponding neighborhood window of the pixel, and g(m,n) is the gray value of the neighboring pixels. Ifuse = W1*I1 + W2*I2, (Image fusion) Where W1 and W2 are the fusion weights, W1=D1 / (D1+D2+ε), W2=D2 / (D1+D2+ε), and ε is the minimum constant.

5. The method according to claim 1, characterized in that, The notch mask is dynamically adjusted according to the direction of the noise. If a corresponding narrowband energy peak exists, a notch mask and a low-frequency guard region are constructed at the center of the spectrum to eliminate the narrowband energy peak. An inverse Fourier transform is performed on the spectrum after eliminating the narrowband energy peak to output an enhanced image, including: If a narrowband energy peak corresponding to the column noise exists, a notch mask and a low-frequency protection region are constructed in the horizontal band at the center of the spectrum to eliminate the narrowband energy peak. If a narrowband energy peak corresponding to the line noise exists, a notch mask and a low-frequency protection region are constructed in the vertical band at the center of the spectrum to eliminate the narrowband energy peak. The edges of the constructed notch mask are smoothed using a Gaussian function, and the processed spectrum is subjected to an inverse Fourier transform to output the enhanced image.

6. The method according to claim 1, characterized in that, After reconstructing the denoised low-frequency and high-frequency components to obtain the reconstructed image, the process further includes: The reconstructed image is subjected to edge-preserving reconstruction processing based on dual constraints of pixel grayscale difference and pixel spatial distance.

7. The method according to claim 1, characterized in that, The dynamic anti-reflection compensation process performed on the reconstructed image includes: The background distribution of the reconstructed image is obtained by using a large-scale Gaussian low-pass filter, and the high-pass detail component is obtained by the difference between the reconstructed image and the background distribution. The high-pass detail component is then added back to the reconstructed image with a preset compensation intensity.

8. A dual-modal image fusion enhancement system, characterized in that, The system includes: The minimalist optical acquisition module includes a large-aperture superlens array composed of subwavelength nano-antennas, configured to coaxially and synchronously acquire dual-mode initial images of the same scene in transient mode. An image processing module is configured to execute the algorithm flow of the method as described in any one of claims 1 to 7; The control and storage module is electrically connected to the minimalist optical acquisition module and the image processing module, respectively.

9. A computer device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a computer device, causes the computer device to perform the method as described in any one of claims 1-7.

Images