A multimodal image fusion method based on the pulse nervous system of Ascaris lumbricoides

Abstract

This invention relates to the field of image fusion technology, specifically to a multimodal image fusion method based on the Ascaris lumbricoides spiking neural system. Registered multimodal source images are input into a biomimetic Ascaris lumbricoides spiking neural system model. The number of pulse excitations at each pixel location over a period of time is calculated, thereby converting the image information into a pulse count map. By comparing the size of two pulse count maps pixel by pixel, a corresponding binarized weight map is generated. Based on this weight map, the two source images are fused using a weighted average to obtain a final fused image with complementary information and enhanced features. Compared to spiking neural networks, this invention has a simpler structure, is easier to apply, can be deployed quickly, and has good interpretability. Related parameters can be flexibly adjusted according to needs. Unlike the application of spiking neural network excitation, this invention achieves the expected fusion effect through weight calculation based on pulse excitation counts.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a multimodal image fusion method based on the pulsatile nervous system of Ascaris lumbricoides. Background Technology

[0002] Artificial intelligence (AI) technology is a crucial supporting force for technological development in the new era. As a significant application of AI, intelligent fusion leverages the advantages of integrating data from multiple single sensors and their processing efficiency to create richer, multimodal target feature representations. Based on different fusion targets, intelligent fusion can be categorized into several types, including infrared and visible light image fusion, medical image fusion, multi-exposure image fusion, and remote sensing image fusion. Furthermore, intelligent fusion can provide higher-quality image data for downstream tasks, improving the efficiency and accuracy of applications such as image recognition, image classification, image enhancement, and target identification.

[0003] Scholars both domestically and internationally have conducted in-depth research on various image fusion methods. Zhai Hailin et al. proposed a parallel dual-branch heterogeneous image fusion method based on CNN and a channel-cosine position-encoded visual converter, and verified its effectiveness in target detection evaluation. Lin Boran et al. constructed a feature fusion network based on ResNet50, effectively improving the accuracy of ship target recognition. Chen Kunya et al. implemented an infrared-visible light fusion algorithm based on the RepVGG network framework. Pathak et al. proposed a medical image fusion method based on ensemble deep learning and Transformer. Laidouni et al. established an image fusion method using residual dense networks and cross-ConvNeXt. Advances in intelligent fusion methods have played a crucial role, accelerating the popularization of intelligent fusion in application scenarios, and possessing significant theoretical and practical value.

[0004] However, while fusion architectures based on deep learning and other methods can deeply express the features of images from different modalities and achieve good fusion results, they often rely on complex equipment conditions and lack interpretability. To improve the interpretability, effectiveness, and adaptability of fusion methods, this invention constructs a multimodal image fusion method based on the pulse neural system of Ascaris lumbricoides. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal image fusion method based on the Ascaris lumbricoides pulse nervous system, which solves the problem that existing fusion architectures based on deep learning and other methods require complex equipment conditions and lack interpretability.

[0006] In this invention, "Ascaris elegans spiking neural system" refers to a spiking neural network model inspired by the simple nervous system structure of the Ascaris elegans (specifically, Caenorhabditis elegans). This model achieves pulse encoding and processing of image information by simulating the dynamic accumulation and firing process of neuronal membrane potentials, and is a common term in the field of neuro-bionic computing. As a model organism, the Ascaris elegans has a clear and easily studied nervous system structure, and is frequently used in neuroscience and the construction of bio-inspired computing models; therefore, it is a common term in this technical field.

[0007] To achieve the above objectives, this invention provides a multimodal image fusion method based on the pulse nervous system of Ascaris lumbricoides, comprising the following steps:

[0008] Input the registered multimodal image, including the first modality image and the second modality image;

[0009] For the first modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model;

[0010] For the second modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model;

[0011] Based on the pulse count, weight values ​​for the first modal image and the second modal image are generated;

[0012] Based on the weight values, the first modal image and the second modal image are weighted and fused to obtain a fused image.

[0013] Specifically, for the first modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model, including:

[0014] For the input first modality image The calculation formula is as follows:

[0015] ;in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, The initial membrane potential is given by k, where k is the weight.

[0016] Set threshold The input images are statistically analyzed as follows: Number of pulse excitations:

[0017] if ,So,

[0018] ,and ;

[0019] in, Let (x, y) be the membrane potential at time t. Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment.

[0020] Specifically, for the second modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model, including:

[0021] For the input of the second modality image The calculation formula is as follows:

[0022] ;

[0023] in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, The initial membrane potential is given by k, where k is the weight.

[0024] Set threshold The input images are statistically analyzed as follows: Number of pulse excitations:

[0025] if ,So,

[0026] ,and ;

[0027] in, Let (x, y) be the membrane potential at time t. Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment.

[0028] Specifically, generating weight values ​​for the first modal image and the second modal image based on the pulse count includes:

[0029] like , =1

[0030] like , =1

[0031] in, Indicates the input image Count the pulses at position (x, y). Indicates the input image Count the pulses at position (x, y). For the input image Binary weights at position (x, y), For the input image Binary weights at position (x, y).

[0032] Specifically, the first modal image and the second modal image are weighted and fused according to the weight values ​​to obtain a fused image, including:

[0033]

[0034] Where F represents the final fusion result, Indicates the input image Binary weighted graph, Indicates the input image The binary weighted graph.

[0035] This invention discloses a multimodal image fusion method based on the Ascaris lumbricoides spiking neural system. First, registered multimodal source images are input into a biomimetic Ascaris lumbricoides spiking neural system model. This model simulates the accumulation and firing process of neuronal membrane potentials, calculating the number of pulse firings at each pixel location over a period of time, thus converting the image information into a pulse counting map. Next, by comparing the sizes of two pulse counting maps pixel-by-pixel, a corresponding binarized weight map is generated. Finally, the two source images are fused using a weighted average based on this weight map, resulting in a final fused image with complementary information and enhanced features. Compared to spiking neural networks, this invention's method has a simpler structure, is easier to apply, can be deployed quickly, and has good interpretability, allowing for flexible adjustment of relevant parameters according to needs. Unlike applications using spiking neural network excitation, this invention achieves the desired fusion effect by calculating weights based on pulse firing counts to take into account the information from multiple source images. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0037] Figure 1 The infrared input image of the present invention A schematic diagram.

[0038] Figure 2 The visible light input image of this invention A schematic diagram.

[0039] Figure 3 This is a schematic diagram of the multimodal infrared-visible light fusion effect F of the present invention.

[0040] Figure 4 This is a flowchart of the steps of the multimodal image fusion method based on the Ascaris lumbricoides pulse nervous system of the present invention. Detailed Implementation

[0041] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0042] Please see Figures 1 to 4 ,in, Figure 1 The infrared input image of the present invention A schematic diagram. Figure 2 The visible light input image of this invention A schematic diagram. Figure 3 This is a schematic diagram of the multimodal infrared-visible light fusion effect F of the present invention. Figure 4This is a flowchart of the steps of the multimodal image fusion method based on the Ascaris lumbricoides pulse nervous system of the present invention.

[0043] This invention provides a multimodal image fusion method based on the Ascaris lumbricoides pulse nervous system, comprising the following steps:

[0044] S101: Input the registered multimodal image, including the first modal image and the second modal image;

[0045] S102: For the first modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model;

[0046] S103: For the second modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model;

[0047] S104: Based on the pulse count, generate weight values ​​for the first modal image and the second modal image;

[0048] S105: Based on the weight values, the first modal image and the second modal image are weighted and fused to obtain a fused image.

[0049] Specifically, an introduction to the Ascaris lumbricoides pulse nervous system: Assuming the input image is I, the Ascaris lumbricoides pulse nervous system model is as follows: ;

[0050] in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, denoted as the initial membrane potential, and k as the weight.

[0051] Set threshold The number of pulse excitations is counted as follows:

[0052] like ,

[0053] ,and

[0054] in, Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment.

[0055] Based on the above method, the fusion steps of the Ascaris lumbricoides pulse nervous system are introduced using the fusion of registered infrared and visible light modal images as an example:

[0056] Step 1: Let the input image for the infrared mode be... ,like Figure 1 As shown, the input image for the visible light mode is ,like Figure 2 As shown, set the values ​​of the relevant parameters;

[0057] Step 2: For the input image The number of pulses in the images was detected using a model of the Ascaris lumbricoides pulse nervous system. ;

[0058] in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, denoted as the initial membrane potential, and k as the weight.

[0059] Set threshold The input images are statistically analyzed as follows: Number of pulse excitations:

[0060] if ,So,

[0061] ,and ;

[0062] in, Let (x, y) be the membrane potential at time t. Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment.

[0063] Step 3: For the input image The number of pulses in the images was detected using a model of the Ascaris lumbricoides pulse nervous system. ;

[0064] in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, denoted as the initial membrane potential, and k as the weight.

[0065] Set threshold The input images are statistically analyzed as follows: Number of pulse excitations:

[0066] if ,So,

[0067] ,and ;

[0068] in, Let (x, y) be the membrane potential at time t. Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment.

[0069] Step 4: After obtaining the pulse counts for each modality image from the above steps, calculate the weight value of each pixel in the image as follows:

[0070] like , =1

[0071] like , =1

[0072] in, Indicates the input image Count the pulses at position (x, y). Indicates the input image Count the pulses at position (x, y). For the input image Binary weights at position (x, y), For the input image Binary weights at position (x, y).

[0073] Step 5: Based on the obtained weight information, calculate the fusion result:

[0074]

[0075] Where F represents the final fusion result, Indicates the input image Binary weighted graph, Indicates the input image The binary weight map. The final fusion effect is as follows. Figure 3 As shown.

[0076] Inspired by the nervous system of Ascaris lumbricoides, this invention constructs a fusion method for the Ascaris lumbricoides spiking nervous system. Compared with spiking neural networks, this invention's method has a simpler structure, is easier to apply, can be deployed quickly, and has good interpretability, allowing for flexible adjustment of relevant parameters according to needs. Unlike applications triggered by spiking neural networks, to take into account information from multiple image sources, this invention achieves the desired fusion effect through weight calculation based on spiking counts.

[0077] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A multimodal image fusion method based on the pulse nervous system of Ascaris lumbricoides, characterized in that, Includes the following steps: Input the registered multimodal image, including the first modality image and the second modality image; For the first modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model; For the second modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model; Based on the pulse count, weight values ​​for the first modal image and the second modal image are generated; Based on the weight values, the first modal image and the second modal image are weighted and fused to obtain a fused image; Specifically, for the first modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model, including: For the input first modality image The calculation formula is as follows: ;in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, The initial membrane potential is given by k, where k is the weight. Set threshold The input images are statistically analyzed in the following manner. Number of pulse excitations: if ,So, ,and ; and with As a new Repeat the steps of calculating the membrane potential and counting the number of pulse excitations until the preset time period is reached; in, Let (x, y) be the membrane potential at time t. Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment; Specifically, for the second modality image, the pulse count at each pixel location is calculated using the Ascaris lumbricoides pulse nervous system model, including: For the input of the second modality image The calculation formula is as follows: ； in, For pixel position, Image position coordinates pixel size, To control the rate of change of potential, For pixels Neighborhood, (m, n) is the neighborhood Inner pixel position coordinates, Let be the membrane potential at position (m, n) in the neighborhood at time t. Let (x, y) be the membrane potential at time t. for Membrane potential at time (x, y) For time intervals, The initial membrane potential is given by k, where k is the weight. Set threshold The input images are statistically analyzed in the following manner. Number of pulse excitations: if ,So, ,and ; and with As a new Repeat the steps of calculating the membrane potential and counting the number of pulse excitations until the preset time period is reached; in, Let (x, y) be the membrane potential at time t. Count the pulses at time t at the image location (x, y). At image (x, y) Pulse count at any given moment; Specifically, generating weight values ​​for the first modal image and the second modal image based on the pulse count includes: like , =1 like , =1 in, Indicates the input image Count the pulses at position (x, y). Indicates the input image Count the pulses at position (x, y). For the input image Binary weights at position (x, y), For the input image Binary weights at position (x, y).

2. The multimodal image fusion method based on the Ascaris lumbricoides pulse nervous system as described in claim 1, characterized in that, Based on the weight values, the first modal image and the second modal image are weighted and fused to obtain a fused image, specifically including: Where F represents the final fusion result, Indicates the input image Binary weighted graph, Indicates the input image The binary weighted graph.

Images