SNN-based infrared visible light image fusion enhancement method

Abstract

The invention discloses an infrared visible light image fusion enhancement method based on SNN, and belongs to the technical field of image enhancement. Comprising the following steps: firstly, using related image sensor equipment to input an infrared image and a visible light high-resolution image with time stamps aligned as input, and sending the input to corresponding AI computing equipment as input of an infrared visible light image fusion enhancement neural network; fine particle extraction of effective features is performed on the two images through a network model obtained through training of a large number of training data sets in advance, and an effect picture after fusion enhancement is given; according to the SNN-based infrared visible light image fusion enhancement method of the invention, the input image features of the infrared image and the visible light image can be efficiently fused, so that a high-quality fused image can be generated.

Description

Technical Field

[0001] The invention relates to an infrared and visible light image fusion enhancement method based on SNN, belonging to the technical field of image enhancement. Background Art

[0002] With the continuous advancement of sensor technology, image sources are showing a trend of diversification. Images collected by different sensors such as visible light, near infrared, far infrared, radar, etc. have different imaging principles and provide different aspects of characteristic information of real targets. These data from different sensors have different time, space, spectral resolution and different polarization modes. Since the amount of data provided by a single sensor is limited and it is difficult to meet application requirements, multi-sensor data fusion and analysis technology came into being under this background. It has a wide range of applications in earth science, military reconnaissance, medical imaging, computer vision and other fields. Image fusion is an important branch of this field. Through image fusion, images obtained by multiple different sensors of the same scene can be fused into a new image. The fused new image has better understandability, less blur and higher credibility, and is more suitable for human vision, computer detection, classification, recognition, understanding and other processing.

[0003] In the process of multi-sensor image acquisition, it is often necessary to fuse two different types of images. For example, Chinese patent application number: CN201710050787.6 discloses a visible light image and far-infrared image fusion method based on multi-scale analysis. Considering the influence of far-infrared image stripe noise on the final fusion result, hybrid total variation multi-scale analysis is used to process visible light image and far-infrared image according to the characteristics of far-infrared image stripe noise, and far-infrared noise is suppressed while obtaining sub-band images; then, according to the saliency distribution characteristics of human vision, the fusion rule of sub-band images is adjusted to fuse the visible light image with the far-infrared detail image; finally, according to the requirements of practical applications for fusion results, the reconstruction parameters are adjusted to obtain the fusion result; the above method can effectively suppress the stripe noise in the far-infrared image during the multi-scale decomposition of the image, and according to the saliency characteristics of human vision, the respective characteristics of the visible light image and the far-infrared image are combined to provide a fusion result with richer effective information and better image quality; however, this type of image fusion method cannot realize adaptive learning, and the processing process is relatively complicated and inefficient; therefore, in order to solve the above problems, it is urgent to propose an infrared visible light image fusion enhancement method based on SNN. Summary of the invention

[0004] To solve the above problems, the present invention proposes an infrared-visible light image fusion enhancement method based on SNN, which can efficiently fuse the input image features of infrared images and visible light images to generate high-quality fused images.

[0005] The infrared and visible light image fusion enhancement method based on SNN of the present invention comprises the following steps:

[0006] First, using relevant image sensor equipment, the timestamp-aligned infrared image and visible light high-resolution image are taken as input and sent to the corresponding AI computing device as the input of the infrared and visible light image fusion enhancement neural network. The network model trained with a large number of training data sets in advance performs fine-grained extraction of effective features of the two images and gives the effect of fusion and enhancement.

[0007] Furthermore, the infrared-visible-light image fusion enhancement neural network includes an input preprocessing module, a layer-by-layer convolution and pulse neuron module, a pixel rearrangement module and a final fusion module.

[0008] Furthermore, the method specifically comprises the following steps:

[0009] In the first step, the input preprocessing module is used to perform input preprocessing and downsampling.

[0010] Given an input image and First, the initial input is formed by channel concatenation: The input is then downsampled to a resolution of 1 / 161 / 161 / 16 using bilinear interpolation:

[0011] I' input =Interpolate(I input ,scale_factor=0.0625,mode=bilinear),

[0012] In the second step, layer-by-layer convolution and pulse neuron modules are used for feature extraction.

[0013] The layer-by-layer convolution and spiking neuron module contains two depth-wise separable convolution layers and one point-wise convolution layer to efficiently extract spatial information and simulate temporal dynamics through leaky integral spiking neurons; the process of each module can be described as:

[0014] a. Depth-wise separable convolution, given input First, through the vertical convolution Then through the horizontal convolution

[0015] b. Spike neuron processing: the result of convolution is processed by leaky integral spike neuron: S t ,M t =LIF(X t ,M t-1 ), where M t represents the neuron membrane potential at time step t, S tIndicates pulse emission, X t represents the input received by the neuron;

[0016] c. Point convolution: Use 1×1 point convolution to increase the number of output channels from C in Mapping to C out :Y=S*K pointwise , the output results of the layer-by-layer convolution and pulse neuron modules are added to the input through residual connections to alleviate the gradient vanishing problem and enhance the feature expression ability: Y'=Y+X;

[0017] The third step is feature fusion and upsampling.

[0018] The network consists of two core paths: the global path and the local path;

[0019] a. Global path,

[0020] in I' input Apply a series of layer-by-layer convolutions with spiking neuron modules and convolution operations:

[0021] F1=EERM_SNN(EERM_SNN(EERM_SNN(I' input ))),

[0022] Then, the number of channels is increased through the convolutional layer, and upsampling is performed using pixel reordering:

[0023] F global =PixelShuffle(Conv(F1)),

[0024] b. Local path,

[0025] Output F in the global path global Two layer-by-layer convolution and spiking neuron modules are further applied:

[0026] F2=EERM_SNN(EERM_SNN(F global ))),

[0027] Then the local features are obtained by rearranging the convolution kernel pixels:

[0028] F local =PixelShuffle(Conv(F2));

[0029] c. Final fusion,

[0030] The global feature F global 、Local Features local Concatenate with the initial input and generate the fusion result through the final convolution: I fused =Conv(concat(F local ,Iinput )),

[0031] The third step is time step iteration and multi-moment fusion.

[0032] The network iterates over multiple time steps to generate a sequence output, sharing the neuron state at each step:

[0033] I fused,t =Fusion_SNN(I' input ),

[0034] The final output sequence is aggregated through the stack operation:

[0035] I output = {I fused,t |t∈[1,T]}, where T is the total length of the time step.

[0036] Compared with the prior art, the SNN-based infrared and visible light image fusion enhancement method of the present invention uses a pulse neural network module to learn feature parameters from training images, adaptively learns feature activation functions, and improves the generalization ability of the model during reasoning; combined with the lightweight SNN network structure design, the number of network parameters is greatly reduced. When the input size is 1280*1024, the inference frame rate on a 6T computing power device is 25FPS. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the infrared and visible light image fusion enhancement network structure of the present invention.

[0038] Figure 2 It is a schematic diagram of the infrared image to be fused according to the present invention.

[0039] Figure 3 It is a schematic diagram of the visible light image to be fused according to the present invention.

[0040] Figure 4 It is a schematic diagram of the final fusion effect of the present invention. DETAILED DESCRIPTION

[0041] Embodiment 1:

[0042] like Figure 1 The infrared-visible image fusion enhancement method based on SNN shown in the figure efficiently fuses the input image features of the infrared image (x_infrade) and the visible light image (x_visible) through a fusion network (Fusion_SNN) based on a spike neural network (SNN) to generate a high-quality fused image; it uses images collected by relevant infrared and visible light image sensor devices as input, and uses a neural network to implement the fusion and detail enhancement technology of the two images; the method includes the following steps:

[0043] First, use relevant image sensor devices, such as Figure 2 and Figure 3 As shown in the figure, the infrared image and the visible light high-resolution image with aligned timestamps are used as input and sent to the corresponding AI computing device as the input of the infrared and visible light image fusion enhancement neural network. The network model trained with a large number of training data sets in advance extracts fine particles of effective features of the two images and gives the effect of fusion enhancement, as shown in the figure. Figure 4 As shown; wherein, the infrared visible light image fusion enhanced neural network includes an input preprocessing module, a layer-by-layer convolution and pulse neuron module (EERM_SNN), a pixel rearrangement module and a final fusion module; specifically, the method specifically includes the following steps:

[0044] In the first step, the input preprocessing module is used to perform input preprocessing and downsampling.

[0045] Given an input image and First, the initial input is formed by channel concatenation: The input is then downsampled to a resolution of 1 / 161 / 161 / 16 using bilinear interpolation:

[0046] I' input =Interpolate(I input ,scale_factor=0.0625,mode=bilinear),

[0047] In the second step, layer-by-layer convolution and spiking neuron module (EERM_SNN) is used for feature extraction.

[0048] The layer-by-layer convolution and spiking neuron module (EERM_SNN) contains two depth-wise separable convolution layers and one point-wise convolution layer to efficiently extract spatial information and simulate temporal dynamics through leaky integrate-and-fire neurons (LIF); the process of each module can be described as:

[0049] a. Depth-wise separable convolution, given input First, through the vertical convolution Then through the horizontal convolution

[0050] b. Spike neuron processing: the result of convolution is processed by leaky integral spike neuron: S t ,M t =LIF(X t ,M t-1 ), where M trepresents the neuron membrane potential at time step t, S t Indicates pulse emission, X t represents the input received by the neuron;

[0051] c. Point convolution: Use 1×1 point convolution to increase the number of output channels from C in Mapping to C out :Y=S*K pointwise , the output of the layer-by-layer convolution and spiking neuron module (EERM_SNN) is added to the input through residual connection to alleviate the gradient vanishing problem and enhance the feature expression ability: Y'=Y+X;

[0052] The third step is feature fusion and upsampling.

[0053] The network consists of two core paths: the global path and the local path;

[0054] a. Global path,

[0055] in I' input Apply a series of layer-by-layer convolutions with spiking neuron modules and convolution operations:

[0056] F1=EERM_SNN(EERM_SNN(EERM_SNN(I' input ))),

[0057] Then, the number of channels is increased through the convolution layer, and upsampling is performed using pixel shuffle:

[0058] F global =PixelShuffle(Conv(F1)),

[0059] b. Local path,

[0060] Output F in the global path global Two EERM_SNN modules are further applied on it:

[0061] F2=EERM_SNN(EERM_SNN(F global ))),

[0062] Then the local features are obtained by rearranging the convolution kernel pixels:

[0063] F local =PixelShuffle(Conv(F2));

[0064] c. Final fusion,

[0065] The global feature F global 、Local Features localConcatenate with the initial input and generate the fusion result through the final convolution: I fused =Conv(concat(F local ,I input )),

[0066] The third step is time step iteration and multi-moment fusion.

[0067] The network iterates over multiple time steps to generate a sequence output, sharing the neuron state at each step:

[0068] I fused,t =Fusion_SNN(I' input ),

[0069] The final output sequence is aggregated through the stack operation:

[0070] I output = {I fused,t |t∈[1,T]}, where T is the total length of the time step.

[0071] The infrared-visible-light image fusion enhancement method based on SNN of the present invention efficiently fuses the input image features of the infrared image and the visible-light image through a fusion network based on a pulse neural network to generate a high-quality fused image; the present invention uses a pulse neural network module to learn feature parameters from training images, and adaptively learns feature activation functions to improve the generalization ability of the model during reasoning; and combined with the lightweight SNN network structure design, the number of network parameters is greatly reduced. When the input size is 1280*1024, the inference frame rate on a 6T computing power device is 25FPS.

[0072] The above embodiments are only preferred implementations of the present invention, so all equivalent changes or modifications made according to the structures, features and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A method for infrared and visible light image fusion enhancement based on SNN, characterized in that: The following steps are involved: First, an image sensor device is used to send the timestamp-aligned infrared image and visible light high-resolution image as input to the AI ​​computing device as the input of the infrared and visible light image fusion enhancement neural network. The network model trained with a large number of training data sets in advance performs fine-grained extraction of effective features of the two images and gives the effect of fusion and enhancement.

2. The infrared and visible light image fusion enhancement method based on SNN according to claim 1 is characterized in that: The infrared and visible light image fusion enhancement neural network includes an input preprocessing module, a layer-by-layer convolution and pulse neuron module, a pixel rearrangement module and a final fusion module.

3. The infrared and visible light image fusion enhancement method based on SNN according to claim 1 is characterized in that: The method specifically The following steps are involved: In the first step, the input preprocessing module is used to perform input preprocessing and downsampling. Given an input image and First, the initial input is formed by channel concatenation: Where H and W are the height and width of the image; then, bilinear interpolation is used to downsample the input to a resolution of 1 / 161 / 161 / 16: I' input =Interpolate(I input ,scale_factor=0.0625,mode=bilinear), In the second step, layer-by-layer convolution and pulse neuron modules are used for feature extraction. The layer-by-layer convolution and spiking neuron module contains two depth-wise separable convolution layers and one point-wise convolution layer to efficiently extract spatial information and simulate temporal dynamics through leaky integral spiking neurons; the process of each module can be described as: a. Depth-wise separable convolution, given input First, through the vertical convolution Then through the horizontal convolution b. Spike neuron processing: the result of convolution is processed by leaky integral spike neuron: S t ,M t =LIF(X t ,M t-1 ), where M t represents the neuron membrane potential at time step t, S t Indicates pulse emission, X t represents the input received by the neuron; c. Point convolution: Use 1×1 point convolution to transform the input channel C in Mapped to output channel C out :Y=S*K pointwise , the output results of the layer-by-layer convolution and pulse neuron modules are added to the input through residual connections to alleviate the gradient vanishing problem and enhance the feature expression ability: Y′=Y+X; The third step is feature fusion and upsampling. The network consists of two core paths: the global path and the local path; a. Global path, in I′ input Apply a series of layer-by-layer convolutions with spiking neuron modules and convolution operations: F1=EERM_SNN(EERM_SNN(EERM_SNN(I' input ))), Then, the number of channels is increased through the convolutional layer, and upsampling is performed using pixel reordering: F global =PixelShuffle(Conv(F1)), b. Local path, Output F in the global path global Two layer-by-layer convolution and spiking neuron modules are further applied: <h2 style=";text-align:left;direction:ltr">F2 = EERM_SNN(EERM_SNN(F<h2 style=";text-align:left;direction:ltr"> global <h2 style=";text-align:left;direction:ltr"> ))), Then the local features are obtained by rearranging the convolution kernel pixels: F local =PixelShuffle(Conv(F2)); c. Final fusion, The global feature F global 、Local Features local Concatenate with the initial input and generate the fusion result through the final convolution: I fused =Conv(concat(F local ,I input )), The third step is time step iteration and multi-moment fusion. The network iterates over multiple time steps to generate a sequence output, sharing the neuron state at each step: TO fused,t =Fusion_SNN(I' input ), The final output sequence is aggregated through the stack operation: I output = {I fused,t |t∈[1,T]}, where T is the total length of the time step.

Images