An image fusion method in different lighting environments based on rattlesnake mechanism

Abstract

The present invention discloses an image fusion method based on the rattlesnake mechanism in different lighting environments, comprising: judging the lighting environment of the image to be fused according to the pixel value ratio distribution of the grayscale image of the visible light image in the image to be fused; the lighting environment includes: night image and day image; the image to be fused includes: visible light image and infrared light image; according to the lighting environment judgment result, enhancing and fusion processing are performed on the image to be fused through the ON-center receptive field and the OFF-center receptive field respectively, and the result is mapped to the R channel, the B channel and the G channel; and outputting the pseudo-color fusion image of the image to be fused. The method can significantly improve the fusion accuracy of daytime and nighttime images.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image fusion, and in particular to an image fusion method based on a rattlesnake mechanism in different lighting environments. Background Art

[0002] Image fusion is the process of generating richer and higher quality images from images of the same scene using different sensors through computer technology. Image fusion processing can extract effective information and available complementary information contained in the image, filter out redundant information and invalid information in the source image, generate a robust or informative image, and improve image quality. Therefore, fusion technology has shown great effectiveness in the field of computer vision in modern applications. Various forms of images, including visible light, infrared, computed tomography (CT) and magnetic resonance imaging (MRI), are very good information source images. Infrared images can also distinguish targets and scenes through radiation differences, and can also be well identified under all-weather and day and night conditions; visible light images can provide high spatial resolution and clarity of texture details, which are in line with the human visual system. Therefore, it is desirable to fuse these two types of infrared images and visible light images, which can combine the advantages of thermal radiation information in infrared images and detailed texture information in visible light images to obtain high-quality images. Moreover, the acquisition equipment of infrared and visible light images is relatively simple, while the requirements of imaging technologies such as CT and MRI are relatively strict. Moreover, infrared images and visible light images have complementary properties, resulting in a robust and information-rich fused image.

[0003] With the development of image fusion technology, traditional infrared and visible light fusion can no longer meet the needs of researchers. In 1981, Newman et al. found that the bimodal cells of rattlesnakes can simultaneously receive and process infrared and visible light image information, and automatically align them in neurons. Based on the fusion mechanism of its own six bimodal cells, it completes the fusion of infrared and visible light image information to perceive and judge the surrounding environment.

[0004] Li et al. improved the network structure and proposed an image enhancement algorithm based on biological visual fusion mechanism; Wang et al. proposed an improved image fusion of rattlesnake bimodal cell model. Jing et al. applied the mathematical model of rattlesnake bimodal cells to the Waxman network structure and realized pseudo-color image fusion of multi-band images. Liu et al. proposed image fusion based on rattlesnake visual receptive field model and pseudo-color fusion of infrared and visible images based on rattlesnake visual imaging system.

[0005] However, the above-mentioned studies were conducted in relatively bright environments. When the visible light is a night image, image fusion cannot be achieved. Night images have low visibility and poor image texture, so the fusion method is more difficult than that of daytime images. Zhu et al. proposed a night vision image fusion based on grayscale transformation and two-scale decomposition. Qiu et al. proposed a night surveillance image enhancement optimization method based on infrared and visible light image fusion. Among these methods, all are traditional image fusion methods, and there are few studies that use the biological perspective of rattlesnake visual perception mechanism to study the fusion of infrared and visible light images in night environments.

[0006] Therefore, based on the existing image fusion technology, how to significantly improve the fusion accuracy of daytime and nighttime images has become an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0007] In view of the above problems, the present invention proposes an image fusion method for different lighting environments based on the rattlesnake mechanism, which at least solves some of the above technical problems. The method can significantly improve the fusion accuracy of daytime and nighttime images.

[0008] The embodiment of the present invention provides an image fusion method in different lighting environments based on the rattlesnake mechanism, comprising the following steps:

[0009] S1. Determine the lighting environment of the image to be fused according to the pixel value ratio distribution of the grayscale image of the visible light image in the image to be fused; the lighting environment includes: nighttime image and daytime image; the image to be fused includes: visible light image and infrared light image;

[0010] S2. According to the illumination environment determination result, the to-be-fused images are enhanced and fused through the ON-center receptive field and the OFF-center receptive field, and the results are mapped to the R channel, the B channel and the G channel; and a pseudo-color fused image of the to-be-fused images is output.

[0011] Furthermore, when the number of pixels with pixel values ​​higher than 110 in the grayscale image of the visible light image accounts for less than 15% of the total number of pixels in the grayscale image of the visible light image, the image to be fused is determined to be a night image, otherwise it is a day image.

[0012] Furthermore, the step S2 comprises:

[0013] When the image to be fused is a daytime image, the visible light image and the infrared light image are respectively enhanced by ON-center receptive field processing to obtain a first visible light enhanced signal and a first infrared light enhanced signal respectively;

[0014] The visible light image and the infrared light image are enhanced and inverted by OFF-center receptive field processing to obtain a second enhanced visible light signal and a second enhanced infrared light signal, respectively;

[0015] Feeding the first enhanced visible light signal and the second enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a first fusion signal (also called infrared enhanced visible light signal, denoted as +VIS+IR);

[0016] Feeding the first enhanced visible light signal and the first enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a second fusion signal (also called infrared suppressed visible light signal, denoted as +VIS-IR);

[0017] Feeding the infrared light first enhanced signal and the visible light first enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a third fusion signal (also called visible inhibition infrared light signal, denoted as +IR-VIS);

[0018] Feeding the infrared light first enhanced signal and the visible light second enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a fourth fusion signal (also called visible enhanced infrared light signal, denoted as +IR+VIS);

[0019] Taking the second fusion signal and the third fusion signal as input signals, outputting infrared AND visible light signals (also called AND cell signals);

[0020] Taking the first fusion signal and the fourth fusion signal as input signals, outputting infrared OR visible light signals (also called OR cell signals);

[0021] Feeding the first visible light enhanced signal and the first fused signal into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a fifth fused signal;

[0022] Mapping the fifth fusion signal to the B channel; mapping the infrared OR visible light signal to the G channel;

[0023] Output a pseudo-color fused image of the image to be fused.

[0024] Furthermore, the step S2 further includes:

[0025] After the infrared AND visible light signal is subtracted from the infrared light first enhanced signal, the signal is mapped to the R channel.

[0026] Furthermore, the step S2 comprises:

[0027] When the image to be fused is a night image, the visible light image and the infrared light image are respectively enhanced by ON-center receptive field processing to obtain a first visible light enhanced signal and a first infrared light enhanced signal respectively;

[0028] The visible light image and the infrared light image are enhanced and inverted by OFF-center receptive field processing to obtain a second enhanced visible light signal and a second enhanced infrared light signal, respectively;

[0029] Feeding the first enhanced visible light signal and the second enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a first fusion signal (also called infrared enhanced visible light signal, denoted as +VIS+IR);

[0030] Feeding the first enhanced visible light signal and the first enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a second fusion signal (also called infrared suppressed visible light signal, denoted as +VIS-IR);

[0031] Feeding the infrared light first enhanced signal and the visible light first enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a third fusion signal (also called visible inhibition infrared light signal, denoted as +IR-VIS);

[0032] Feeding the infrared light first enhanced signal and the visible light second enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively, to obtain a fourth fusion signal (also called visible enhanced infrared light signal, denoted as +IR+VIS);

[0033] Taking the second fusion signal and the third fusion signal as input signals, outputting infrared AND visible light signals (also called AND cell signals);

[0034] Taking the first fusion signal and the fourth fusion signal as input signals, outputting infrared OR visible light signals (also called OR cell signals);

[0035] Feeding the infrared light first enhanced signal and the fourth fused signal into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a sixth fused signal;

[0036] Mapping the sixth fusion signal to the B channel; mapping the infrared OR visible light signal to the G channel;

[0037] Output a pseudo-color fused image of the image to be fused.

[0038] Furthermore, the step S2 further includes:

[0039] After the infrared AND visible light signal is subtracted from the infrared light first enhanced signal, the signal is mapped to the R channel.

[0040] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0041] An image fusion method based on the rattlesnake mechanism in different lighting environments provided by the embodiment of the present invention includes: judging the lighting environment of the image to be fused according to the pixel value ratio distribution of the grayscale image of the visible light image in the image to be fused; the lighting environment includes: night image and day image; the image to be fused includes: visible light image and infrared light image; according to the lighting environment judgment result, enhancing and fusion processing are performed on the image to be fused through the ON-center receptive field and the OFF-center receptive field respectively, and the result is mapped to the R channel, the B channel and the G channel; and outputting the pseudo-color fusion image of the image to be fused. This method can significantly improve the fusion accuracy of daytime and nighttime images.

[0042] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0043] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0045] Figure 1 A flow chart of an image fusion method for different lighting environments based on the rattlesnake mechanism provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a daytime image fusion structure provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a nighttime image fusion structure provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0049] The embodiment of the present invention provides an image fusion method for different lighting environments based on the rattlesnake mechanism, referring to Figure 1 As shown, including:

[0050] According to the pixel value ratio distribution of the grayscale image of the visible light image in the image to be fused, the lighting environment of the image to be fused is determined; the lighting environment includes: night image and day image; the image to be fused includes: visible light image and infrared light image;

[0051] According to the results of the illumination environment judgment, the fused image is enhanced and fused through the ON-center receptive field and the OFF-center receptive field, and the results are mapped to the R channel, the B channel and the G channel;

[0052] Output the pseudo-color fused image of the image to be fused.

[0053] The image fusion method based on the rattlesnake mechanism in different lighting environments provided in this embodiment can significantly improve the fusion accuracy of daytime and nighttime images.

[0054] Among them, the passive membrane equation is a dynamic description of the center-surround antagonistic domain, which is used to simulate the ion exchange of the cell membrane. Its dynamic equation is as follows:

[0055] Definition ON resistance system cellular response X k (i,j), then:

[0056]

[0057] Definition OFF opposition system cell response Y k (i,j), then:

[0058]

[0059] In the formula, A represents the attenuation constant, (i, j) is used to represent the pixel coordinates in the image, and D, It is used to represent the basal activity of two antagonistic cells, k represents the color channel of the corresponding color space, and E and F represent polarization constants, also known as Nernst potential.

[0060] Among them C k(i,j) is the excited center area of ​​the receiving domain:

[0061]

[0062] Where S k (i,j) is the domain-suppressed surrounding area:

[0063]

[0064] In the formula, I k (i, j) is the input image, * is the convolution operator, Wc and Ws are the Gaussian distribution functions of the central area and the surrounding area respectively, the sizes of the Gaussian templates of the central area and the surrounding area are m×n and p×q respectively, σ c , σ s are the space constants of the center and the surrounding area respectively. The superscripts C and S represent the center (Center) and the surrounding area (Surround) respectively.

[0065] Both (1) and (2) describe the instantaneous response of cells when stimulated. When the system is in equilibrium, let The following formula can be obtained:

[0066] ON countermeasure system output:

[0067]

[0068] OFF against system output:

[0069]

[0070] The ON adversarial system can enhance the common information and complementary information in the two source images. The OFF adversarial system suppresses the common information of the two images, and at the same time, the complementary information in the source images is reversely enhanced, that is, the bright area becomes darker and the dark area becomes brighter. Both adversarial systems can enhance the contrast and highlight the edge information.

[0071] There are a large number of bimodal cells in the optic tectum of rattlesnakes. Newman and Hartline discovered six types of bimodal cells that process visible light and infrared information, which can produce different nonlinear responses to infrared light stimulation and visible light stimulation. V (i,j) and f IR (i, j) represents the input visible light image and infrared image respectively, and the coordinates and functions accurately represent the pixel position and pixel value size of each input.

[0072] Infrared enhances visible light cell mechanism:

[0073] When the cell is stimulated by infrared light only, the cell does not respond. When it is stimulated by visible light only, the cell responds. When infrared light is used to stimulate the cell under the premise of visible light stimulation, the cell responds strongly. It can be judged that the infrared signal plays an auxiliary enhancement role on the visible light signal, so an exponential function is used to simulate this enhancement effect. The mathematical model is expressed as formula (7).

[0074] f +V←IR (i,j)=fV(i,j)exp[f IR (i,j)] (7)

[0075] Among them, f +V←IR (i, j) is the output result after infrared enhanced visible light processing.

[0076] From the visible light mathematical model image, infrared light mathematical model image, and infrared enhanced visible light cell mechanism mathematical model image, it can be seen that the same brightness area can better reflect the brightness area of ​​visible light. When there are strong boundaries in both visible light and infrared light images, the brightness in the result image is the brightest. When only one of the visible light and infrared light is brighter, the visible light is brighter than the infrared light.

[0077] Visible enhanced infrared photocell mechanism:

[0078] When the cell is stimulated by visible light only, the cell does not respond. When it is stimulated by infrared light only, the cell responds. When the cell is stimulated by visible light in addition to infrared light, the cell responds strongly. It can be judged that the visible light signal plays an auxiliary enhancement role on the infrared signal, so an exponential function is used to simulate this enhancement effect. The mathematical model is expressed as formula (8).

[0079] f +IR←V (i,j)=f IR (i,j)exp[f V (i,j)] (8)

[0080] Among them, f +IR←V (i, j) is the output result after visible enhanced infrared light processing.

[0081] From the visible light mathematical model image, infrared light mathematical model image, and visible enhanced infrared light cell mechanism mathematical model image, it can be seen that the same brightness area can better reflect the brightness area of ​​infrared light. When there are strong boundaries in both visible light and infrared light images, the brightness in the result image is the brightest. When only one of the visible light and infrared light is brighter, the infrared light is brighter than the visible light.

[0082] Infrared inhibits visible light cell mechanism:

[0083] When the cell is stimulated by infrared light only, the cell does not respond. When it is stimulated by visible light only, the cell responds. When infrared light is used to stimulate the cell under the premise of visible light stimulation, the cell responds weakly. It can be judged that the infrared signal plays an auxiliary inhibitory role on the visible light signal, so a logarithmic function is used to simulate this inhibitory effect. The mathematical model is expressed as formula (9).

[0084] f -V←IR (i,j)=f V (i,j)log[f IR (i,j)+1] (9)

[0085] Among them, f -V←IR (i, j) is the output result after infrared suppression of visible light processing.

[0086] From the visible light mathematical model image, infrared light mathematical model image, and infrared suppression of visible light cell mechanism mathematical model image, it can be seen that the results only display visible light information. When there is only visible light brightness, the result image has high brightness. When visible light and infrared light image information exist at the same time and when only infrared light image information exists, the result image has low brightness. Similarly, this mechanism will enhance the image boundary.

[0087] Visible inhibition of infrared light cell mechanism:

[0088] When the cell is stimulated by visible light only, the cell does not respond. When it is stimulated by infrared light only, the cell responds. When the cell is stimulated by visible light in addition to infrared light, the cell responds weakly. It can be judged that the visible light signal plays an auxiliary inhibitory role on the infrared signal, so a logarithmic function is used to simulate this inhibitory effect. The mathematical model is expressed as formula (10).

[0089] f -IR←V (i,j)=f IR (i,j)log[f V (i,j)+1] (10)

[0090] Among them, f -IR←V (i, j) is the output result after visible light suppression infrared light processing.

[0091] From the visible light mathematical model image, infrared light mathematical model image, and visible light suppression infrared light cell mechanism mathematical model image, it can be seen that the results only display infrared light information. When there is only infrared brightness, the result image has high brightness. When the information in the visible light and infrared light images exists at the same time and when only the visible light image information exists, the brightness in the result image is small. Similarly, this mechanism will also enhance the image boundary.

[0092] Cellular Mechanisms:

[0093] When only visible light or infrared light is used to stimulate the cell, it will produce only a weak response, and when both are used to stimulate the cell, it will produce a strong response. Consider using "weighted and" for simulation expression, and the mathematical model is expressed as formula (11) and (12).

[0094] a. When f V (i,j)>f IR (i,j),

[0095] The output is f AND (i,j)=b*f V (i,j)+a*f IR (i,j) (11)

[0096] b. When f V (i,j) <f IR (i,j),

[0097] The output is f AND (i,j)=a*f V (i,j)+b*f IR (i,j) (12)

[0098] Among them, a>0.5, b<0.5, f AND (i, j) is the output result after weighted processing.

[0099] The visible light AND infrared light mechanism uses the infrared suppressed visible light image and the visible light suppressed infrared image as the input images of the AND mechanism, and finally obtains the visible light AND infrared light fusion result.

[0100] From the visible light mathematical model image, infrared light mathematical model image, and cellular mechanism mathematical model image respectively, it can be seen that only the areas where both visible light and infrared light are bright appear in the result image. The brighter targets in the image represent both infrared light targets and visible light targets, as well as the boundary information where the two images exist at the same time.

[0101] or cellular mechanisms:

[0102] Any signal stimulus from visible light or infrared light will produce a response, and a response will also be produced when both are stimulated together. Consider using "weighted OR" for simulation expression, and the mathematical model is expressed as formulas (13) and (14).

[0103] a. When f V (i,j)>f IR (i,j),

[0104] The output is f OR (i,j)=a*f V (i,j)+b*f IR(i,j) (13)

[0105] b. When f V (i,j) <f IR (i,j),

[0106] The output is f OR (i,j)=b*f V (i,j)+a*f IR (i,j) (14)

[0107] Among them, a>0.5, b<0.5, f OR (i, j) is the output result after weighted processing.

[0108] From the visible light mathematical model image, infrared light mathematical model image, or cell mechanism mathematical model image, it can be seen that the result image contains both visible light and infrared light information. When both the infrared light target and the visible light target are bright, the result is brighter than either one alone. This mechanism can well show the hierarchy of visible light and infrared light information. This mechanism can also enhance the boundaries. When the boundaries of both images are strong, the enhancement is more obvious.

[0109] The fusion of infrared and visible light images under biological mechanisms was originally the classic fusion method of Waxman et al. from the MIT Laboratory of the Massachusetts Institute of Technology. This method is divided into two stages: the first stage of the method is the enhancement stage, specifically, the visible light image and the infrared light image are enhanced by ON-center receptive field processing to obtain ON_IR image (first infrared light enhanced signal) and ON_VIS image (first visible light enhanced signal), and the infrared light image is enhanced and inverted by OFF-center receptive field processing to darken the bright area and brighten the dark area to obtain OFF_IR image (second infrared light enhanced signal). The second stage of the method is the fusion stage, specifically, the enhanced signal (ON_VIS) of visible light after ON-type processing is fed into the central excited area of ​​an ON-center receptive field, and the enhanced signal (OFF_IR) of infrared light after OFF-type processing is fed into the surrounding inhibition area of ​​the receptive field, simulating the mechanism of infrared enhancing visible light cells, and obtaining a +VIS+IR fusion signal (first fusion signal); the enhanced signal (ON_VIS) of visible light after ON-type processing is fed into the central excited area of ​​an ON-center receptive field, and the enhanced signal (ON_IR) of infrared light after ON-type processing is fed into the surrounding inhibition area of ​​the receptive field, simulating the mechanism of infrared inhibiting visible light cells, and obtaining a +VIS-IR fusion signal (second fusion signal); finally, the +VIS+IR fusion signal is mapped to the R channel, and the +VIS-IR fusion signal is mapped to the B channel. ON_VIS is mapped to the G channel to generate a false color fusion image based on the RGB color space.

[0110] Furthermore, the fusion algorithm based on color space (Li Hui color space fusion algorithm) is: the signal mapped to the R channel is represented by the infrared self-enhancement map minus the infrared light AND visible light map. The specific operation is: the second fusion signal and the third fusion signal are used as input signals, and finally the infrared AND visible light signal (also called the cell signal) is obtained; the infrared light first enhancement signal (infrared self-enhancement map) minus the infrared light AND visible light signal is mapped to the R channel, in order to highlight the infrared signal that is weak in the visible light map but strong in the infrared light map, and the infrared light signal hidden in the visible light map is represented in red, which is more conducive to discovering infrared targets with important information. The G channel is used to represent the information in the visible light, and the B channel is used to represent the visible light information suppressed by the infrared information, so that targets of different properties are displayed in the final color result map, where the G and B channels are the same as the Waxman classic fusion method.

[0111] Among them, the fusion algorithm based on color space, the R channel signal of the algorithm is represented by the infrared self-enhancement image (infrared light first enhancement signal) minus the infrared AND visible light image, which is more conducive to discovering infrared targets with important information. This is the advantage of the fusion of visible light image and infrared light image, especially for night images. The second is the interference of special environments in daytime images (such as fog, fire, rain, cloud occlusion of remote sensing images), and tree occlusion of daytime images. Therefore, this embodiment selects the color space fusion algorithm for improvement.

[0112] Among the many methods under biological mechanisms, all of them process images with visible light during the day, and now night images are also widely concerned, but in the environment where visible light is at night, there is no good fusion method given under biological mechanisms. Therefore, this embodiment will provide an image fusion method of infrared light and visible light based on the visual perception mechanism of rattlesnakes for night images.

[0113] Specifically, the fusion method first determines whether the environment is daytime or nighttime. Rattlesnakes can determine whether the environment is daytime or nighttime based on their eye organs. Nighttime images often have low brightness, low visibility, weak contrast, and not rich texture information, and the pixel values ​​of their grayscale images are low. Therefore, it is chosen to determine whether the scene is daytime or nighttime based on the pixel value ratio distribution of the visible light grayscale image (or based on the histogram). When the number of pixels with pixel values ​​higher than 110 in the visible light grayscale image accounts for less than 15% of the total number of pixels in its grayscale image, the visible light image is considered to be a nighttime image, otherwise it is a daytime image.

[0114] When the visible light image is judged as a daytime image, the following is selected: Figure 2The network fusion structure shown in FIG. This structure adds an OR cell mechanism (OR) (or cell signal) to the G channel to replace the enhanced visible light signal in the color space-based fusion algorithm, because the OR mechanism can well display the hierarchy of visible light and infrared light information. This mechanism simultaneously displays visible light and infrared light information, that is, when both the infrared light target and the visible light target are bright, the result is brighter than either one alone, and can display all the information of visible light and infrared light, rather than just visible light information. In this way, the infrared information of the daytime image can also be captured, which is the advantage of the fusion of visible light and infrared light images.

[0115] When the visible light image is judged as a daytime image, the visible light image and the infrared light image are enhanced by ON-center receptive field processing to obtain a first enhanced visible light signal and a first enhanced infrared light signal respectively; the visible light image and the infrared light image are enhanced and inverted by OFF-center receptive field processing to obtain a second enhanced visible light signal and a second enhanced infrared light signal respectively; the first enhanced visible light signal and the second enhanced infrared light signal are fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a first fusion signal (also called infrared enhanced visible light signal, denoted as +VIS+IR); the first enhanced visible light signal and the first enhanced infrared light signal are fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a second fusion signal (also called infrared suppressed visible light signal, denoted as +VIS-IR); the first enhanced infrared light signal and the first enhanced visible light signal are fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively The first enhanced infrared signal and the second enhanced visible signal are respectively fed into the central excitation area and the surround inhibition area of ​​the ON-central receptive field to obtain a fourth fusion signal (also called visible enhanced infrared signal, denoted as +IR+VIS); the second fusion signal and the third fusion signal are taken as input signals to output an infrared AND visible light signal (also called AND cell signal); the first fusion signal and the fourth fusion signal are taken as input signals to output an infrared OR visible light signal (also called OR cell signal); the first enhanced visible light signal and the first fusion signal are respectively fed into the central excitation area and the surround inhibition area of ​​the ON-central receptive field to obtain a fifth fusion signal; the fifth fusion signal is mapped to the B channel; the infrared OR visible light signal is mapped to the G channel; the infrared light first enhanced signal is subtracted from the infrared AND visible light signal and then mapped to the R channel.

[0116] Because the ambient temperature is high in daytime images, the infrared image information is relatively weak, and the visible light texture is clear, so the visible light occupies a dominant position. At the same time, the infrared enhanced visible light cell mechanism (+VIS+IR, the first fusion signal) can better reflect the brightness area of ​​the visible light. When only one of the visible light and infrared light is brighter, the visible light is brighter than the infrared light. This cell mechanism can better reflect the information of the visible light. Therefore, in the B channel, the ON_VIS signal (the first enhanced signal of the visible light) obtained by processing the visible light image with the ON-center receptive field and the infrared enhanced visible light cell are combined. The mechanism (+VIS+IR) is fed into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field to obtain the +ON_VIS+VIS+IR image (the fifth fusion signal), which replaces the cell mechanism of pure infrared inhibition of visible light (+VIS-IR, the second fusion signal) in the fusion algorithm based on color space. The visible light enhancement signal (ON_VIS) and the cell mechanism of infrared enhancement of visible light (+VIS+IR) in this embodiment can better reflect the texture information of visible light of daytime images than the cell mechanism of pure infrared inhibition of visible light (+VIS-IR). Because the R channel can reflect the infrared information of daytime images, the network structure of the R channel remains unchanged. Finally, the network structure obtained by R, G, and B is mapped to the corresponding channels in sequence to output the final pseudo-color fusion image.

[0117] Night image fusion method:

[0118] When the visible light image is judged as a night image, the following is selected: Figure 3 The network fusion structure shown in FIG. Based on the structure of the fusion algorithm based on the color space, the structure also adds an OR cell mechanism (OR) in the G channel to replace the enhanced visible light signal of the fusion algorithm based on the color space, which can capture the infrared information of the night image and make the fused image higher in quality.

[0119] When the visible light image is judged as a night image, the visible light image and the infrared light image are enhanced by ON-center receptive field processing, and the first enhanced visible light signal and the first enhanced infrared light signal are obtained respectively; the visible light image and the infrared light image are enhanced and inverted by OFF-center receptive field processing, and the second enhanced visible light signal and the second enhanced infrared light signal are obtained respectively; the first enhanced visible light signal and the second enhanced infrared light signal are respectively fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field to obtain the first fusion signal (also called infrared enhanced visible light signal, denoted as +VIS+IR); the first enhanced visible light signal and the first enhanced infrared light signal are respectively fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field to obtain the second fusion signal (also called infrared suppressed visible light signal, denoted as +VIS-IR); the first enhanced infrared light signal and the first enhanced visible light signal are respectively fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field The first infrared light enhancement signal and the second visible light enhancement signal are respectively fed into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field to obtain a fourth fusion signal (also called visible enhanced infrared light signal, recorded as +IR+VIS); the second fusion signal and the third fusion signal are taken as input signals to output an infrared AND visible light signal (also called AND cell signal); the first fusion signal and the fourth fusion signal are taken as input signals to output an infrared OR visible light signal (also called OR cell signal); the first infrared light enhancement signal and the fourth fusion signal are respectively fed into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field to obtain a sixth fusion signal; the sixth fusion signal is mapped to the B channel; the infrared OR visible light signal is mapped to the G channel; the infrared light first enhancement signal is subtracted from the infrared AND visible light signal and then mapped to the R channel.

[0120] Because for night images, visible light can hardly provide effective visual information, and the visible light information is extremely weak, while the temperature is low in the night environment, so infrared light is dominant. At the same time, the visible enhanced infrared light cell mechanism (+IR+VIS, the fourth fusion signal) can better reflect the brightness area of ​​infrared light in the same brightness area. When only one of the visible and infrared light is brighter, the infrared light is brighter than the visible light. This cell mechanism can better reflect the information of infrared light. Therefore, in the B channel, the infrared image is processed by the ON-center receptive field to obtain the ON_IR signal (the first infrared enhancement signal) and the visible enhanced infrared cell mechanism (+IR+VIS) respectively fed into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field to obtain the +ON_IR+IR+VIS image (the sixth fusion signal), replacing the simple infrared suppression of visible light cell mechanism (+VIS-IR) of the fusion algorithm based on the color space, wherein the infrared suppression of visible light cell mechanism (+VIS-IR) only has visible light information displayed in the figure, when the infrared light and visible light brightness exist at the same time and when only the infrared light image brightness exists, the result brightness will be small, and the infrared information of the night image cannot be reflected. In this embodiment, the infrared enhancement signal (ON_IR) and the visible enhanced infrared cell mechanism (+IR+VIS) can better reflect the infrared information that dominates the night image. The R channel network structure remains unchanged, and finally the network structure obtained by R, G, and B is mapped to the corresponding channels in sequence to output the final pseudo-color fusion image.

[0121] In order to demonstrate the superiority of the method provided in this embodiment, 13 groups of representative infrared and visible light images were selected, and according to the pixel value ratio distribution of the visible light grayscale image, 6 groups were daytime images and 7 groups were nighttime images. The daytime images of these 13 groups of images include representative landmarks such as houses, trees, and signboards, as well as infrared heat sources that are difficult to find behind special environmental backgrounds (such as fire and smoke), and infrared heat sources that are similar in color to the visible light grayscale image and difficult to capture; nighttime images include roadblocks that cannot be captured by infrared sources, traffic signs, zebra crossings, street lights, business plaques and other landmarks, humans that cannot be captured by visible light sources, and vehicles with heat sources without turning on the lights in extremely dark environments. The 13 groups of images selected in this embodiment are derived from the data sets CVC-14Dataset and TNO Image Fusion Dataset. Specifically, standard deviation, information entropy, average gradient, Q AB / F , spatial frequency and other five objective image quality evaluation indicators are used to evaluate the fusion method proposed in this embodiment.

[0122] Q AB / F :

[0123] Q AB / FIt measures the amount of edge information transferred from the source image to the fused image, reflecting how much edge information in the source image is retained in the fused image. AB / F The larger the value is, the more the image edge information of visible light and infrared light is transferred to the fused image, which means the image is clearer.

[0124] Define Q as an image of size M×N AB / F for:

[0125]

[0126] in and are the edge strength value and direction value at the pixel point (i, j) respectively. X The weight representing the importance of each source image to the fused image.

[0127] Spatial Frequency:

[0128] The spatial frequency reflects the overall activity of the image in the spatial domain, that is, the rate of change of the image grayscale. The larger the spatial frequency, the better the quality of the fused image. The spatial frequency of an M×N image is defined as:

[0129]

[0130]

[0131]

[0132] Where SF represents the spatial frequency of the fused image, RF and CF represent the row frequency and column frequency of the fused image, respectively, and m and n represent the pixel positions of the fused image, respectively.

[0133] The final evaluation results show that the fusion method provided by this embodiment is closer to the real color of the object. The image has no noise interference, the image color is more natural, and the environmental level is clearer. The final fused image has natural color, brighter brightness, and high clarity.

[0134] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image fusion method for different lighting environments based on the rattlesnake mechanism, characterized in that: The steps include: S1. determining the lighting environment of the image to be fused according to the pixel value ratio distribution of the grayscale image of the visible light image in the image to be fused; The lighting environment includes: nighttime images and daytime images; the images to be fused include: visible light images and infrared light images; S2. According to the illumination environment determination result, the image to be fused is enhanced and fused through the ON-center receptive field and the OFF-center receptive field, and the result is mapped to the R channel, the B channel and the G channel; and a pseudo-color fused image of the image to be fused is output; Step S2 comprises: When the image to be fused is a daytime image, the visible light image and the infrared light image are respectively enhanced by ON-center receptive field processing to obtain a first visible light enhanced signal and a first infrared light enhanced signal respectively; The visible light image and the infrared light image are enhanced and inverted by OFF-center receptive field processing to obtain a second enhanced visible light signal and a second enhanced infrared light signal, respectively; Feeding the first enhanced visible light signal and the second enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a first fusion signal; Feeding the first enhanced visible light signal and the first enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively to obtain a second fusion signal; Feeding the infrared light first enhanced signal and the visible light first enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively to obtain a third fusion signal; Feeding the infrared light first enhanced signal and the visible light second enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively to obtain a fourth fusion signal; Taking the second fused signal and the third fused signal as input signals, and outputting infrared AND visible light signals; Taking the first fused signal and the fourth fused signal as input signals, and outputting an infrared OR visible light signal; Feeding the first visible light enhanced signal and the first fused signal into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a fifth fused signal; Mapping the fifth fusion signal to the B channel; mapping the infrared OR visible light signal to the G channel; After subtracting the infrared AND visible light signal from the infrared light first enhanced signal, the signal is mapped to the R channel; When the image to be fused is a night image, the visible light image and the infrared light image are respectively enhanced by ON-center receptive field processing to obtain a first visible light enhanced signal and a first infrared light enhanced signal respectively; The visible light image and the infrared light image are enhanced and inverted by OFF-center receptive field processing to obtain a second enhanced visible light signal and a second enhanced infrared light signal, respectively; Feeding the first enhanced visible light signal and the second enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a first fusion signal; Feeding the first enhanced visible light signal and the first enhanced infrared light signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively to obtain a second fusion signal; Feeding the infrared light first enhanced signal and the visible light first enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively to obtain a third fusion signal; Feeding the infrared light first enhanced signal and the visible light second enhanced signal into the central excitation area and the surrounding inhibition area of ​​the ON-central receptive field respectively to obtain a fourth fusion signal; Taking the second fused signal and the third fused signal as input signals, and outputting infrared AND visible light signals; Taking the first fused signal and the fourth fused signal as input signals, and outputting an infrared OR visible light signal; Feeding the infrared light first enhanced signal and the fourth fused signal into the central excitation area and the surrounding inhibition area of ​​the ON-center receptive field respectively to obtain a sixth fused signal; Mapping the sixth fusion signal to the B channel; mapping the infrared OR visible light signal to the G channel; After the infrared AND visible light signal is subtracted from the infrared light first enhanced signal, the signal is mapped to the R channel.

2. The image fusion method based on the rattlesnake mechanism in different lighting environments as claimed in claim 1, characterized in that: When the number of pixels with pixel values ​​higher than 110 in the grayscale image of the visible light image accounts for less than 15% of the total number of pixels in the grayscale image of the visible light image, the image to be fused is determined to be a night image, otherwise it is a day image.

Images