A fast multimodal image fusion method, system, device and medium based on simplified adaptive parameter pulse coupled neural network

Abstract

A fast multimodal image fusion method, system, device, and medium based on a simplified adaptive parameter pulse-coupled neural network. The method comprises: converting the RGB channels of a color image into YUV channels, normalizing the Y channels of a grayscale image and a color image, decomposing the channels using a Laplace pyramid to obtain high-pass subbands and low-pass subbands, fusing the high-pass subbands using a SPAPCNN rule to obtain a high-frequency image, fusing the low-pass subbands using an MN-DWRE rule to obtain a low-frequency image, inversely decomposing the high-frequency and low-frequency images using an inverse Laplace pyramid to obtain the Y channel of the fused image, combining the Y channel with the U and V channels of the original image, converting the channel into an RGB channel, and performing normalization. The system, device, and medium are used to implement a fast multimodal image fusion method based on a simplified adaptive parameter pulse-coupled neural network. The present invention has the characteristics of fast decomposition speed, short fusion time, and good fusion effect.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a fast multimodal image fusion method, system, device and medium based on a simplified adaptive parameter pulse coupled neural network. Background Art

[0002] For medical image fusion, it is necessary to ensure the image fusion quality while shortening the fusion time. Currently commonly used fusion methods often require a lot of time for fusion, which is unacceptable for scenarios that require real-time fusion. Existing multimodal image fusion methods include:

[0003] (1) Image fusion based on spatial domain

[0004] Typical algorithms for spatial-domain image fusion include principal component analysis, differential evolution, and the gradient energy method. These methods operate directly on image pixel values. While computationally fast, these algorithms are poor at preserving image edge and contour information. Additionally, techniques such as fuzzy sets and random fields have also been used for spatial-domain image fusion. Spatial-domain fusion methods are simple and intuitive, offer rapid fusion speeds, and are suitable for real-time processing. However, due to their fusion rules based on simple overlap calculations, the signal-to-noise ratio and contrast of the fused image are significantly reduced.

[0005] (2) Image fusion based on transform domain

[0006] The general steps of transform domain fusion are as follows: first, the image to be fused is decomposed into high-frequency channels and low-frequency channels through a certain decomposition algorithm; then different fusion strategies are adopted for the high-frequency channels and low-frequency channels respectively; finally, image fusion is achieved through decomposition and inverse transformation; the transform domain method has higher feature extraction accuracy and better fusion effect, which is superior to the spatial domain method; however, the transform domain method also has some limitations: it is time-consuming and has poor real-time performance.

[0007] (3) Model-based image fusion

[0008] To adapt to these characteristics, new mathematical models for adaptive feature extraction have been proposed for image fusion, such as sparse representation (SR), pulse-coupled neural networks (PCNN), and deep learning-based neural networks. SR models are simple and can better handle noise errors; however, SR methods are time-consuming and can produce blurred edges and textures in the fused image. PCNNs are characterized by strong local detail extraction and high object recognition rates. Because image clarity is independent of features such as texture, they have a certain degree of universality. However, PCNNs have many parameters, whose values are obtained through extensive training, and their fusion results and performance are significantly affected by these parameters. The paper "A Novel Method of Multimodal Medical Image Fusion Based on Hybrid Approach of NSCT and DTCWT" (now proposed with a hybrid approach of DTCWT and NSCT) uses NSCT as a decomposition strategy and DTCWT as a fusion strategy. However, this method suffers from drawbacks such as long fusion time, color imbalance in the fused image, and unclear details in the fused image. Summary of the Invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a fast multimodal image fusion method, system, device and medium based on a simplified adaptive parameter pulse coupled neural network. The image is decomposed by using a Laplace pyramid, the high-pass subbands of the image are fused using a simplified adaptive parameter pulse coupled neural network, and the low-pass subbands of the image are fused using a multi-neighborhood distance weighted regional energy method, thereby ensuring high-quality fusion of the low-pass subbands of the two images, and having the characteristics of fast decomposition speed, short fusion time and good fusion effect.

[0010] A fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network includes the following steps:

[0011] Step 1: Convert the RGB channels of the color image in the original image to YUV channels, and use the Y channel of the color image as the fused image in the following steps;

[0012] Step 2: Normalize the grayscale image in the original image and use the Laplace pyramid decomposition to obtain high-pass subbands and low-pass subbands;

[0013] Normalize the Y channel image obtained in step 1 to obtain the normalized Y channel image, and decompose it using the Laplace pyramid to obtain high-pass subbands and low-pass subbands;

[0014] Step 3: The high-pass subbands obtained in step 2 are fused using a simplified adaptive parameter pulse coupled neural network (SPAPCNN rule) to obtain a fused high-frequency image;

[0015] Step 4: The low-pass subbands obtained in step 3 are fused using the multi-neighborhood distance weighted regional energy method (MN-DWRE rule) to obtain a fused low-frequency image;

[0016] Step 5: Use the inverse Laplacian pyramid to perform inverse decomposition on the fused high-frequency image obtained in step 3 and the fused low-frequency image obtained in step 4 to obtain the Y channel of the fused image;

[0017] Step 6: Combine the U and V channels of the original image in step 1 with the Y channel obtained in step 5, convert them into RGB channels, and perform normalization (SDZ) to obtain a fast multimodal image fusion result based on a simplified adaptive parameter pulse coupled neural network.

[0018] The specific process of step 1 is:

[0019] Use the following calculation formula to get the YUV channel of the image:

[0020] Y=0.2990R+0.5870G+0.1140B

[0021] U=-0.1684R-0.3316G+0.5B+128

[0022] V=0.5R-0.4187G-0.0813B+128

[0023] Among them: R, G, B are the red, green, and black color channels in the original image respectively; Y represents the brightness channel, that is, the grayscale value, and U and V are chromaticity.

[0024] The specific process of step 2 is:

[0025] Step 2.1: Downsample the Y channel image obtained in step 1 using the following formula to obtain the Gaussian pyramid GP of the image:

[0026] GP i =Down(GP i-1 )

[0027] Among them, the Down operation is to remove the even rows and even columns of each layer image;

[0028] Step 2.2: Upsample the Gaussian pyramid GP obtained in step 2.1, calculate the residual of the Gaussian pyramid GP, and obtain the Laplacian pyramid LP:

[0029]

[0030] The UP operation is upsampling, which maps the pixel at position (x, y) in the original image to the position (2x+1, 2y+1) in the target image. represents convolution, κ 5×5 is a 5×5 kernel, and the kernel is

[0031]

[0032] The specific process of step 3 is as follows:

[0033] The Gaussian pyramid GP obtained in step 2.1 is fused using a simplified adaptive parameter pulse coupled neural network. The calculation formula is as follows:

[0034] F ij [n]=S ij

[0035] The input is divided into two parts: S ij is the Y channel of the image obtained in step 1; L ij is the (i, j)th neuron, n is the number of iterations; F ij [n] is the overall input of the network;

[0036]

[0037] Among them, L ij is the (i, j)th neuron, n is the number of iterations;

[0038]

[0039] Where W is the synaptic weight of the pixel and its 8 neighboring points;

[0040]

[0041] is the attenuation degree of the last iteration, β is the link strength, F ij [n](1+βL ij ) is L ij and F ij Nonlinear connection between ij is the internal activity level;

[0042]

[0043] Y ijIgnition state: When Y ij If it is 1, it is ignition, otherwise if it is 0, it is non-ignition state;

[0044]

[0045] Among them, α e For E ij The exponential decay coefficient, V E For E ij Amplitude;

[0046] The coefficient α is determined dynamically using the following formula f , β, V E and α e :

[0047]

[0048]

[0049]

[0050]

[0051] Among them, S′ and S max represent the normalized Otsu threshold and the maximum intensity of the input image, respectively.

[0052] The specific process in step 4 is as follows:

[0053] Step 4.1: The Laplacian pyramid LP obtained in step 2.2 is fused using the multi-neighborhood distance weighted regional energy method. The calculation formula is as follows:

[0054]

[0055] Among them, WLE S (i, j) is the weighted activity level, S∈{M1,M2};

[0056] W is the weight matrix, which is calculated using the following formula:

[0057]

[0058] Among them, i, j are the coordinates of each point in the matrix;

[0059] Step 4.2: Calculate the stereo-weighted active energy focusing WAEF, which is calculated as follows:

[0060]

[0061] Where S∈{M1, M2}, m, n are the coordinate positions in the matrix, and r is the focus radius;

[0062] Step 4.3: Calculate the stereo energy SE as follows:

[0063]

[0064] Among them, i, j are the coordinate positions in the matrix;

[0065] Step 4.4 uses the following formula to obtain the low-frequency fusion result:

[0066]

[0067] Among them, WLE S (i, j) is the weighted activity level, S∈{M1,M2}; are the values of the corresponding pixels in images M1 and M2 respectively; L F (i, j) is the value of the pixel corresponding to the low-pass sub-band of the final fused image.

[0068] The specific process of step 5 is as follows:

[0069] Step 5.1: Upsample the high-pass subband fused in step 3. The calculation formula is as follows:

[0070] t i =UP(H i )

[0071] Among them, t i is the upsampling of the high-pass subband after fusion in step 3;

[0072] Step 5.2: Combine the low-pass subband obtained in step 4 with the t i Add them together to get the final fused Y channel, which is calculated as follows:

[0073] Y i =t i +L Fi

[0074] Among them, L Fi is the low-pass subband after fusion in step 4, and i is the number of decomposition layers.

[0075] The specific process of step 6 is as follows:

[0076] Step 6.1: Convert the UV channel in step 1 and the fused Y channel obtained in step 5 to RGB channels using the following formula:

[0077] R=Y+1.4075(V-128)

[0078] G=Y-0.3455(U-128)-0.7169(V-128)

[0079] B=Y+1.779(U-128)

[0080] Among them, R, G, and B are the red, green, and black color channels in the original image respectively; Y represents the brightness channel, that is, the grayscale value, and U and V represent the chromaticity.

[0081] A fast multimodal image fusion system based on a simplified adaptive parameter pulse coupled neural network, comprising:

[0082] Channel conversion module: used to convert RGB channels to YUV channels, and convert YUV channels to RGB channels;

[0083] Image decomposition module: used to decompose the normalized image into high-pass sub-bands and low-pass sub-bands using the Laplacian pyramid;

[0084] High-pass sub-band fusion module: used to fuse the high-pass sub-bands of the image;

[0085] Low-pass sub-band fusion module: used to fuse the low-pass sub-bands of the image;

[0086] Channel fusion module: used to decompose the fused high-frequency image and the fused low-frequency image using the inverse Laplace pyramid to obtain the Y channel of the fused image.

[0087] A fast multimodal image fusion device based on a simplified adaptive parameter pulse coupled neural network, comprising:

[0088] Memory: used for storing a computer program for implementing the fast multimodal image fusion method based on simplified adaptive parameter pulse coupled neural network;

[0089] A processor is used to implement the fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network when executing the computer program.

[0090] A computer-readable storage medium comprising:

[0091] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network can be implemented.

[0092] Compared with the prior art, the present invention has the following beneficial effects:

[0093] 1. In step 2 of the present invention, the Laplacian pyramid is used as the decomposition framework, which greatly reduces the decomposition time while retaining most of the information in the image, ensuring the final fusion effect.

[0094] 2. In step 3 of the present invention, a simplified adaptive parameter pulse coupled neural network (SPAPCNN rule) is used for fusion, so that the fused image retains most of the details in the original image and has extremely low time complexity, thereby significantly improving the quality of the fused image.

[0095] 3. In step 4 of the present invention, the multi-neighborhood distance weighted region energy method (MN-DWRE rule) is adopted, so it has the advantages of complete image feature extraction and more detail preservation.

[0096] 4. In step 5 of the present invention, an inverse Laplacian pyramid is used for inverse decomposition, so it has the characteristics of fast decomposition speed and good image fusion effect.

[0097] In summary, the present invention decomposes an image by using a Laplacian pyramid, fuses the high-pass subbands of the image using a simplified adaptive parameter pulse coupled neural network, and fuses the low-pass subbands of the image using a multi-neighborhood distance weighted regional energy method, thereby ensuring high-quality fusion of the low-pass subbands of the two images. A color image and a grayscale image can be quickly fused without training. Therefore, the present invention has the characteristics of fast decomposition speed, short fusion time, and good fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 Flow chart of the method of the present invention.

[0099] Figure 2 This is a Laplace pyramid decomposition flow chart of the present invention.

[0100] Figure 3 This is a flow chart of the simplified adaptive parameter pulse coupled neural network of the present invention.

[0101] Figure 4 This is a flow chart of the multi-neighborhood distance weighted regional energy method of the present invention.

[0102] Figure 5 This is a comparison chart of the effects of the present invention and other mainstream methods.

[0103] Figure 6 This is a comparison chart of the running time of the present invention and other mainstream methods. DETAILED DESCRIPTION

[0104] The working principle of the present invention will be described in detail below with reference to the accompanying drawings.

[0105] See also Figure 1, a method for fast multimodal image fusion based on simplified adaptive parameter pulse coupled neural network, comprising the following steps:

[0106] Step 1: Convert the RGB channels of the color image in the original image to YUV channels, and use the Y channel of the color image as the fused image in the following steps;

[0107] Use the following calculation formula to get the Y, U, and V channels of the two images:

[0108] Y=0.2990R+0.5870G+0.1140B

[0109] U=-0.1684R-0.3316G+0.5B+128

[0110] V=0.5R-0.4187G-0.0813B+128

[0111] Among them: R, G, B are the red, green, and black color channels in the original image respectively; Y represents the brightness channel, that is, the grayscale value, and U and V are chromaticity.

[0112] See also Figure 2 , normalize the gray image in the original image, use Laplace pyramid decomposition to obtain high-pass subband and low-pass subband;

[0113] Normalize the Y channel image obtained in step 1 to obtain the normalized Y channel image, and decompose it using the Laplacian pyramid to obtain high-pass subbands and low-pass subbands. Using the Laplacian pyramid as the decomposition framework greatly reduces the decomposition time while retaining most of the information in the image, ensuring the final fusion effect. The specific process is as follows:

[0114] Step 2.1: Downsample the Y channel image obtained in step 1 using the following formula to obtain the Gaussian pyramid GP of the image:

[0115] GP i =Down(GP i-1 )

[0116] Among them, the Down operation is to remove the even rows and even columns of each layer image;

[0117] Step 2.2: Upsample the Gaussian pyramid GP obtained in step 2.1, calculate the residual of the Gaussian pyramid GP, and obtain the Laplacian pyramid LP:

[0118]

[0119] The UP operation is upsampling, which maps the pixel at position (x, y) in the original image to the position (2x+1, 2y+1) in the target image. represents convolution, κ 5×5 is a 5×5 kernel, and the kernel is

[0120]

[0121] See also Figure 3 , Step 3: The high-pass subbands obtained in step 2 are fused using a simplified adaptive parameter pulse coupled neural network (SPAPCNN rule) to obtain a fused high-frequency image; thus, the quality of the fused image is significantly improved; the specific process is:

[0122] Step 3.1: The Gaussian pyramid GP obtained in step 2.1 is fused using a simplified adaptive parameter pulse coupled neural network. The calculation formula is as follows:

[0123] F ij [n]=S ij

[0124] The input is divided into two parts: S ij is the Y channel of the two images obtained in step 1; L ij is the (i, j)th neuron, n is the number of iterations; F ij [n] is the overall input of the network;

[0125]

[0126] Among them, L ij is the (i, j)th neuron, n is the number of iterations;

[0127]

[0128] Where W is the synaptic weight of the pixel and its 8 neighboring points;

[0129]

[0130] is the attenuation degree of the last iteration, β is the link strength, F ij [n](1+βL ij ) is L ij and F ij Nonlinear connection between ij For internal activity levels.

[0131]

[0132] Y ij Ignition state: When Y ijIf it is 1, it is ignition, otherwise if it is 0, it is non-ignition state;

[0133]

[0134] Among them, α e For E ij The exponential decay coefficient, V E For E ij Amplitude;

[0135] The coefficient α is determined dynamically using the following formula f , β, V E and α e :

[0136]

[0137]

[0138]

[0139]

[0140] Among them, S′ and S max represent the normalized Otsu threshold and the maximum intensity of the input image, respectively.

[0141] See also Figure 4 , Step 4: The low-pass subbands obtained in step 3 are fused using the multi-neighborhood distance weighted regional energy method (MN-DWRE rule) to obtain a fused low-frequency image; therefore, it has the advantages of complete image feature extraction and more detail preservation; the specific process is:

[0142] Step 4.1: The Laplacian pyramid LP obtained in step 2.2 is fused using the multi-neighborhood distance weighted regional energy method. The calculation formula is as follows:

[0143]

[0144] Among them, WLE S (i, j) is the weighted activity level, S∈{M1,M2};

[0145] W is the weight matrix, which is calculated using the following formula:

[0146]

[0147] Among them, i, j are the coordinates of each point in the matrix;

[0148] Step 4.2: Calculate the stereo-weighted active energy focusing WAEF, which is calculated as follows:

[0149]

[0150] Where S∈{M1, M2}, m, n are the coordinate positions in the matrix, and r is the focus radius;

[0151] Step 4.3: Calculate the stereo energy SE as follows:

[0152]

[0153] Among them, i, j are the coordinate positions in the matrix;

[0154] Step 4.4: Use the following formula to get the low-frequency fusion result:

[0155]

[0156] Among them, WLE S (i, j) is the weighted activity level, S∈{M1,M2}; are the values of the corresponding pixels in images M1 and M2 respectively; LF(i, j) is the value of the corresponding pixel in the low-pass subband of the final fused image.

[0157] Step 5: Use the inverse Laplacian pyramid to perform inverse decomposition on the high-frequency image obtained after fusion in step 3 and the low-frequency image obtained after fusion in step 4 to obtain the Y channel of the fused image. This has the characteristics of fast decomposition speed and good image fusion effect. The specific process is as follows:

[0158] Step 5.1: Upsample the high-pass subband fused in step 3. The calculation formula is as follows:

[0159] t i =UP(H i )

[0160] Among them, t i is the upsampling of the high-pass subband after fusion in step 3;

[0161] Step 5.2: Combine the low-pass subband obtained in step 4 with the t i Add them together to get the final fused Y channel, which is calculated as follows:

[0162] Y i =t i +L Fi

[0163] Among them, L Fi is the low-pass subband after fusion in step 4, and i is the number of decomposition layers.

[0164] Step 6: Combine the U and V channels of the original image in step 1 with the Y channel obtained in step 5, convert them into RGB channels, and perform normalization (SDZ) to obtain a fast multimodal image fusion result based on a simplified adaptive parameter pulse coupled neural network. The specific process is as follows:

[0165] Step 6.1: Convert the UV channel in step 1 and the fused Y channel obtained in step 5 to RGB channels using the following formula:

[0166] R=Y+1.4075(V-128)

[0167] G=Y-0.3455(U-128)-0.7169(V-128)

[0168] B=Y+1.779(U-128)

[0169] Among them, R, G, and B are the red, green, and black color channels in the original image respectively; Y represents the brightness channel, that is, the grayscale value, and U and V represent the chromaticity.

[0170] In order to evaluate the fusion effect of the present invention, the experimental results of the present invention are compared with the current mainstream image fusion methods. The experimental results of MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method are compared on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA, as shown in Table 1:

[0171]

[0172] Table 1

[0173] As can be seen from Table 1: on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA, the average test accuracy of the proposed method is the highest among MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method in terms of the PSNR indicator; on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA, the average accuracy of the proposed method is the highest among MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method in terms of the Qabf indicator; on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA, the average accuracy of the proposed method is the highest among MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method in terms of the SSIM indicator.

[0174] In order to evaluate the fusion efficiency of the model, the experimental results of the present invention are compared with the current mainstream image fusion methods. The experimental results of MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method are shown in Table 2 on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA:

[0175]

[0176] Table 2

[0177] As can be seen from Table 2, among the methods of MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method, on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA, on the MRI-PET image pairs, the proposed method takes the shortest time; among the methods of MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the proposed method, on the dataset from the Harvard Medical Library in Cambridge, Massachusetts, USA, on the MRI-SPECT image pairs, the proposed method takes the shortest time.

[0178] In summary, the fusion efficiency and fusion time of the proposed method are superior to those of other methods, which proves the feasibility of the proposed method.

[0179] See also Figure 5 Figure (a) MRI-PET is a comparison diagram of the fusion effect of the present invention (proposed method) with MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, and DTCWT, wherein the first two columns of MRI and PET are images to be fused. It can be seen that compared with the other methods, the present invention (proposed method) retains more details, and at the same time, the image does not have blurring, excessive noise, etc., and the color and contrast of the image are relatively balanced, and the information of the original two images can basically be retained; Figure (b) is a comparison diagram of the fusion effect of MRI-SPECT of the present invention (proposed method) with MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, and DTCWT, wherein the first two columns of MRI and SPECT are images to be fused. It can be seen that compared with the other methods, the present invention (proposed method) retains more details, and at the same time, the image does not have blurring, excessive noise, etc., and the color and contrast of the image are relatively balanced, and the information of the original two images can basically be retained.

[0180] See also Figure 6 , which are comparison diagrams of MMIF, NSCT-PAPCNN, CFL, NSCT, LLF-IOI, DTCWT and the present invention (proposed method) on MRI-PET and MRI-SPECT datasets, Pair1, Pair2, Pair3, Pair4, Pair5, Pair6, Pair7, Pair8, 8 groups of images. It can be seen that on the same dataset, in the same group of images, the present invention has the shortest fusion time and the best fusion effect.

[0181] A fast multimodal image fusion system based on a simplified adaptive parameter pulse coupled neural network, comprising:

[0182] Channel conversion module: used to convert RGB channels into YUV channels and YUV channels into RGB channels; used in step 1 of a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network; used in step 6 of a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network;

[0183] Image decomposition module: used to decompose the normalized image into high-pass subbands and low-pass subbands using a Laplacian pyramid; used in step 2 of a fast multimodal image fusion method based on a simplified adaptive parameter pulse-coupled neural network;

[0184] High-pass sub-band fusion module: used to fuse the high-pass sub-bands of the image; used in step 3 of a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network;

[0185] Low-pass sub-band fusion module: used to fuse the low-pass sub-bands of the image; used in step 4 of a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network;

[0186] Channel fusion module: used to decompose the fused high-frequency image and the fused low-frequency image using an inverse Laplace pyramid to obtain the Y channel of the fused image; used in step 5 of a fast multimodal image fusion method based on a simplified adaptive parameter pulse-coupled neural network.

[0187] A fast multimodal image fusion device based on a simplified adaptive parameter pulse coupled neural network, comprising:

[0188] Memory: used for storing a computer program for implementing the fast multimodal image fusion method based on simplified adaptive parameter pulse coupled neural network;

[0189] A processor is used to implement the fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network when executing the computer program.

[0190] A computer-readable storage medium comprising:

[0191] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network can be implemented.

Claims

1. A fast multimodal image fusion method based on simplified adaptive parameter pulse coupled neural network, characterized in that: The following steps are involved: Step 1: Convert the RGB channels of the color image in the original image into YUV channels, and use the Y channel of the color image as the fused image in the following steps; Step 2: Normalize the grayscale image in the original image and use the Laplace pyramid decomposition to obtain high-pass subbands and low-pass subbands; Normalize the Y channel image obtained in step 1 to obtain the normalized Y channel image, and decompose it using the Laplace pyramid to obtain high-pass subbands and low-pass subbands; The specific process of step 2 is: Step 2.1: Downsample the Y channel image obtained in step 1 using the following formula to obtain the Gaussian pyramid GP of the image: GP i =Down(GP i-1 ) Among them, the Down operation is to remove the even rows and even columns of each layer image; Step 2.2: Upsample the Gaussian pyramid GP obtained in step 2.1, calculate the residual of the Gaussian pyramid GP, and obtain the Laplacian pyramid LP: Among them, the UP operation is upsampling, mapping the pixel at position (x, y) in the original image to the position (2x+1, 2y+1) in the target image; represents convolution, K 5×5 For a 5×5 kernel, the kernel is: Step 3: The high-pass subbands obtained in step 2 are fused using a simplified adaptive parameter pulse coupled neural network (SPAPCNN rule) to obtain a fused high-frequency image; The specific process of step 3 is as follows: The Gaussian pyramid GP obtained in step 2.1 is fused using a simplified adaptive parameter pulse coupled neural network. The calculation formula is as follows: F ij [n]=S ij The input is divided into two parts: S ij is the Y channel of the image obtained in step 1; L ij is the (i, j)th neuron, n is the number of iterations; F ij [n] is the overall input of the network; Among them, L ij is the (i, j)th neuron, n is the number of iterations; Where W is the synaptic weight of the pixel and its 8 neighboring points; is the attenuation degree of the last iteration, β is the link strength, F ij [n](1+βL ij ) is L ij and F ij Nonlinear connection between ij is the internal activity level; Y ij Ignition state: When Y ij If it is 1, it is ignition, otherwise if it is 0, it is non-ignition state; Among them, α e For E ij The exponential decay coefficient, V E For E ij Amplitude; The coefficient α is determined dynamically using the following formula f , β, V E and α e : Among them, S' and S max represent the normalized Otsu threshold and the maximum intensity of the input image respectively; Step 4: The low-pass subbands obtained in step 3 are fused using the multi-neighborhood distance weighted regional energy method (MN-DWRE rule) to obtain a fused low-frequency image; The specific process of step 4 is as follows: Step 4.1: The Laplacian pyramid LP obtained in step 2.2 is fused using the multi-neighborhood distance weighted regional energy method. The calculation formula is as follows: Among them, WLE s (i, j) is the weighted activity level, S∈{M1,M2}; W is the weight matrix, which is calculated using the following formula: Among them, i, j are the coordinates of each point in the matrix; Step 4.2: Calculate the stereo-weighted active energy focusing WAEF, which is calculated as follows: Among them, S∈{M1,M2},m,n are the coordinate positions in the matrix, and r is the focus radius; Step 4.3: Calculate the stereo energy SE, the formula is as follows: Among them, i, j are the coordinate positions in the matrix; Step 4.4 uses the following formula to obtain the low-frequency fusion result: Among them, WLE s (i, j) is the weighted activity level, S∈{M1,M2}; L M1 (i,j),L M2 (i, j) are the values of the corresponding pixels in images M1 and M2 respectively; L F( i, j) is the value of the pixel corresponding to the low-pass sub-band of the final fused image; Step 5: Use the inverse Laplacian pyramid to perform inverse decomposition on the fused high-frequency image obtained in step 3 and the fused low-frequency image obtained in step 4 to obtain the Y channel of the fused image; Step 6: Combine the U and V channels of the original image in step 1 with the Y channel obtained in step 5, convert them into RGB channels, and perform normalization processing (SDZ) to obtain a fast multimodal image fusion result based on a simplified adaptive parameter pulse coupled neural network.

2. The fast multimodal image fusion method based on simplified adaptive parameter pulse coupled neural network according to claim 1, characterized in that: The specific process of step 1 is: Use the following calculation formula to get the YUV channel of the image: Y=0.2990R+0.5870G+0.1140B U=-0.1684R-0.3316G+0.5B+128 V=0.5R-0.4187G-0.0813B+128 Among them: R, G, B are the red, green, and black color channels in the original image respectively; Y represents the brightness channel, that is, the grayscale value, and U and V are chromaticity.

3. The fast multimodal image fusion method based on simplified adaptive parameter pulse coupled neural network according to claim 1, characterized in that: The specific process of step 5 is as follows: Step 5.1: Upsample the high-pass subband fused in step 3. The calculation formula is as follows: t i =UP(H i ) Among them, t i is the upsampling of the high-pass subband after fusion in step 3; Step 5.2: Combine the low-pass subband fused in step 4 with the t i Add them together to get the final fused Y channel, which is calculated as follows: Y i =t i +L Fi Among them, L Fi is the low-pass subband after fusion in step 4, and i is the number of decomposition layers.

4. The fast multimodal image fusion method based on simplified adaptive parameter pulse coupled neural network according to claim 1, characterized in that: The specific process of step 6 is as follows: Step 6.1: Convert the UV channel in step 1 and the fused Y channel obtained in step 5 to RGB channels using the following formula: R=Y+1.4075(V-128) G=Y-0.3455(U-128)-0.7169(V-128) B=Y+1.779(U-128) Among them, R, G, and B are the red, green, and black color channels in the original image respectively; Y represents the brightness channel, that is, the grayscale value, and U and V represent the chromaticity.

5. A fast multimodal image fusion system based on a simplified adaptive parameter pulse coupled neural network according to the method of claim 1, characterized in that: include: Channel conversion module: used to convert RGB channels to YUV channels, and convert YUV channels to RGB channels; Image decomposition module: used to decompose the normalized image into high-pass sub-bands and low-pass sub-bands using the Laplacian pyramid; High-pass sub-band fusion module: used to fuse the high-pass sub-bands of the image; Low-pass sub-band fusion module: used to fuse the low-pass sub-bands of the image; Channel fusion module: used to decompose the fused high-frequency image and the fused low-frequency image using the inverse Laplace pyramid to obtain the Y channel of the fused image.

6. A fast multimodal image fusion device based on a simplified adaptive parameter pulse coupled neural network, characterized in that: include: Memory: used to store a computer program for implementing a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network according to claims 1 to 4; A processor is used to implement the fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network of claims 1-4 when executing the computer program.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement claims 1-4, a fast multimodal image fusion method based on a simplified adaptive parameter pulse coupled neural network.

Images