A method for generating non-pseudo-color images based on ON adversarial cells

Abstract

The present invention relates to a method for generating non-pseudo-color images based on ON adversarial cells, comprising the following steps: an enhancement stage, an operation stage, and a fusion stage. The method for generating non-pseudo-color images based on ON adversarial cells of the present invention simulates the visual imaging mechanism of snakes and achieves good results, so that the fused image not only highlights the target information but also retains richer detail information. In a comparative experiment between the method of the present invention and seven other methods in the prior art, 56.25% of the average gradient optimal value is generated in the method of the present invention, 68.75% of the information entropy optimal value is generated in the method of the present invention, and 87.5% of the spatial frequency optimal value is generated in the method of the present invention. Through the analysis of the obtained evaluation index data, it is found that the overall performance of the method of the present invention on the three evaluation indicators is better than that of other comparison methods.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a method for generating non-pseudo-color images based on ON antagonistic cells. Background Art

[0002] Image fusion, as an enhancement technology, aims to generate images that can provide accurate scene information by comprehensively processing images obtained by different types of sensors in the same scene to make up for the shortcomings of insufficient information in a single image. The key to an excellent fusion method lies in the effective extraction of image information and the appropriate application of fusion principles. It can extract useful information from the source image and integrate it into the fused image with the help of the complementary characteristics of sensors in different modes, generating a new description of the target object in a specific environment while avoiding the introduction of any artifacts. Based on this, image fusion algorithms have developed rapidly in theory and application. In the past few decades, many methods for fusing infrared and visible light images have been studied. According to the principles of these algorithms, they can be roughly divided into four types: methods based on multi-scale transformation, methods based on saliency, methods based on sparse representation, and methods based on neural networks.

[0003] Further research in zoology has given new possibilities to image fusion methods. Scientists in the field of image fusion have conducted in-depth research on the visual perception and infrared vision mechanisms of snakes based on zoology. Since the human eye can recognize targets faster in color images than in grayscale images, all fusion methods that simulate snakes to date have generated pseudo-color images, but the pseudo-color images generated by these methods lack details and have unnatural colors, which sometimes seriously affect the visual effects of the images. For example: In 1997, Waxman et al. proposed an adversarial fusion method for night vision images and infrared images based on the physiological mechanism of rattlesnakes. However, the pseudo-color images generated by this method have the same problem, which limits human observation. Specifically, if Figure 5 As shown, the adversarial fusion method of the night vision image and the infrared image includes:

[0004] The first stage: First, the infrared image (IR) is enhanced by the OFF central cells and the ON central cells to obtain the infrared OFF enhanced image and the infrared ON enhanced image, which are recorded as OFF_IR and ON_IR respectively; the visible light image (VI) is enhanced by the ON central cells to obtain the visible light ON enhanced image, which is recorded as ON_VI.

[0005] The second stage: further process the signals obtained in the previous stage, and input OFF_IR, ON_IR and ON_VI into two ON confrontation systems at the same time. First, OFF_IR is input into the surrounding inhibition area, corresponding to the inhibition signal, and ON_VI is input into the central excitation area, corresponding to the excitation signal. The output of the first ON confrontation system is infrared enhanced visible signal + VIS + IR; second, ON_IR is input into the surrounding inhibition area, corresponding to the inhibition signal, and ON_VI is input into the central excitation area, corresponding to the excitation signal. The output of the second ON confrontation system is infrared suppressed visible signal + VIS-IR; then the signal VI-IR and the signal IR-VI are simultaneously obtained through the AND cell (AND) to obtain the common information of the two; IR_ON minus the common information of the two and the weighted fusion of IR-VI to obtain the final fused image.

[0006] Finally, +VIS+IR, ON_VI, and +VIS-IR are mapped to the R channel, the G channel, and the B channel, respectively, to generate the final pseudo-color fused image.

[0007] After introducing the classic Waxman fusion structure, the simulation experiment of the Waxman classic fusion method is carried out. This fusion method involves core technology, so the experiment is based on the principle. The experimental results are as follows: Figure 6 As shown. Here, a set of infrared images and visible light images are used as experimental demonstrations. Among them, Figure (a), Figure (b), and Figure (c) respectively represent the infrared source image, the visible light source image, and the fused image of the Waxman method. Observing the final fusion result (c), it is found that since the final stage of this method is to directly map the enhanced image to the pseudo-color image formed by the RGB three channels, the entire image presents an unnatural dark purple color, and is not as clear as the visible light source image; and the target information of the finally generated fused image is not prominent, so the fusion result is not very ideal. Summary of the invention

[0008] The present invention aims to solve the technical problems in the prior art and provides a method for generating non-pseudo-color images based on ON antagonistic cells.

[0009] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0010] A method for generating a non-pseudo-color image based on ON adversarial cells comprises the following steps:

[0011] Step 1: Enhancement phase;

[0012] First, the infrared and visible light source images are recorded as IR and VI respectively; IR is enhanced by the ON confrontation system to obtain the enhanced infrared image IR_ON; VI is enhanced by the ON confrontation system to obtain the enhanced visible light image VI_ON; IR and VI are simultaneously input into the two ON confrontation systems for enhancement processing;

[0013] Step 2: Operation phase;

[0014] Input the signal VI-IR and the signal IR-VI into the cell at the same time to obtain the common information of the two: IR AND VI; then IR_ON minus IR ANDVI to obtain the unique information ir of IR_ON; ir and IR are enhanced through the ON confrontation system, ir is input into the surrounding inhibition area, corresponding to the inhibition signal, IR is input into the central excitation area, corresponding to the excitation signal, and finally the output signal X1 is obtained;

[0015] Step 3: Fusion phase;

[0016] X2 is obtained by weighted averaging X1 and VI_ON, so that the obtained X2 achieves a balance in the fusion of visible light information and infrared information; then the image contrast is adjusted through gamma correction to enhance the overall detail performance of the image and obtain the final fused image.

[0017] In the above technical solution, in step 1, IR and VI are simultaneously input into two ON countermeasure systems for enhanced processing, specifically:

[0018] VI input surrounds the inhibition area, corresponding to the inhibition signal, and IR inputs the central excitation area, corresponding to the excitation signal. The output of the first ON countermeasure system is the visible inhibition infrared signal VI-IR.

[0019] The IR input surrounds the inhibition area, corresponding to the inhibition signal, and the VI input is in the central excitation area, corresponding to the excitation signal. The output of the second ON countermeasure system is the infrared inhibition visible signal IR-VI.

[0020] The present invention has the following beneficial effects:

[0021] The method for generating non-pseudo-color images based on ON antagonistic cells of the present invention simulates the visual imaging mechanism of snakes and achieves good results, so that the fused image not only highlights the target information but also retains richer detail information.

[0022] In the comparative experiment between the method of the present invention and seven other methods in the prior art, 56.25% of the average gradient best value was generated in the method of the present invention, 68.75% of the information entropy best value was generated in the method of the present invention, and 87.5% of the spatial frequency best value was generated in the method of the present invention. Through the analysis of the obtained evaluation index data, it was found that the overall performance of the method of the present invention on the three evaluation indicators was better than that of other comparative methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0024] Figure 1 It is a schematic diagram of the fusion result of the method of the present invention and the Waxman method.

[0025] Figure 2 Schematic diagram of the fusion results obtained by simulating 8 methods using 16 sets of source images.

[0026] Figure 3 This is a schematic diagram of the local magnification results of the second group of images.

[0027] Figure 4 This is a schematic diagram of the local magnification results of the fourth group of images.

[0028] Figure 5 It is a schematic diagram of the Waxman structure in the prior art.

[0029] Figure 6 It is a schematic diagram of the simulation results of the Waxman model in the prior art. DETAILED DESCRIPTION

[0030] The inventive concept of the present invention is: when constructing a model simulating the visual mechanism of a snake, the present invention proposes an improved method based on receptive field to address the problems that the target information in the fusion results obtained by the Waxman fusion method in the prior art is not prominent and the color is unnatural. On the basis of the classic fusion method, the infrared information is strengthened, and in order to retain the detail information, the proportion of visible light information is also increased, so that the generated fused image retains more detail information.

[0031] The method for generating non-pseudo-color images based on ON confrontation cells of the present invention can be divided into three stages. The first is the enhancement stage: the visible light image VI and the infrared image IR are enhanced by the ON confrontation system to obtain VI_ON and IR_ON respectively; the infrared image and the visible light image are simultaneously input into two ON confrontation systems, and are output as visible suppression infrared signal VI-IR and infrared suppression visible signal IR-VI respectively; the second is the operation stage: VI-IR and IR-VI are simultaneously input into the AND cell (AND) to obtain the common information IR AND VI; then IR_ON minus IR ANDVI is obtained to obtain ir; then ir and IR_ON are enhanced by the ON confrontation system to obtain X1; the last is the fusion stage: X1 and VI are weighted averaged and then gamma corrected to obtain the final fused image. This method strengthens the infrared information on the basis of the Waxman fusion method, and also increases the proportion of visible light information in order to retain detail information. Three different non-reference image quality assessment methods are used to evaluate the quality of the non-pseudo-color images obtained by the present invention. Then these images are compared with the images obtained by the other seven fusion methods. By analyzing the obtained evaluation index data, it is found that 56.25% of the average gradient optimal value is generated in the method of the present invention, 68.75% of the information entropy optimal value is generated in the method of the present invention, and 87.5% of the spatial frequency optimal value is generated in the method of the present invention. The overall performance of the method of the present invention on the three evaluation indicators is better than that of other methods. Finally, the partial fusion image obtained by the method of the present invention is colorized to convert the grayscale image into a color image.

[0032] The present invention is described in detail below with reference to the accompanying drawings.

[0033] Through a brief analysis of the Waxman fusion structure and the fusion image finally generated by the model, it is found that the pseudo-color fusion image generated by the model is not natural enough, the target is not clear enough, and the processed image sometimes affects subsequent observation. Therefore, the present invention proposes an improved method for generating non-pseudo-color images based on ON adversarial cells.

[0034] Aiming at the problem that the target information of the infrared source image is not prominent and the color is unnatural in the fusion result obtained by the Waxman fusion method, an improved method is proposed. On the basis of the Waxman fusion method, the infrared information is strengthened, and the proportion of visible light information is also increased in order to retain the detail information. The model structure is as follows Figure 5 shown.

[0035] The method of the present invention can be divided into three stages.

[0036] The first stage is the enhancement stage. First, the infrared and visible light source images are recorded as IR and VI respectively. IR is enhanced by the ON confrontation system to obtain the enhanced infrared image IR_ON; VI is enhanced by the ON confrontation system to obtain the enhanced visible light image VI_ON; IR and VI are simultaneously input into two ON confrontation systems for enhancement processing. First, VI inputs the surrounding inhibition area, corresponding to the inhibition signal, and IR inputs the central excitement area, corresponding to the excitement signal. The output of the first ON confrontation system is the visible inhibition infrared signal VI-IR; second, IR inputs the surrounding inhibition area, corresponding to the inhibition signal, and VI inputs the central excitement area, corresponding to the excitement signal. The output of the second ON confrontation system is the infrared inhibition visible signal IR-VI.

[0037] The second stage is the operation stage. The signals VI-IR and IR-VI are simultaneously input into the AND cell (AND) to obtain the common information of the two: IR ANDVI; then IR_ON minus IR ANDVI obtains the unique information ir of IR_ON, thereby highlighting the infrared signal that is weak in the visible light image but strong in the infrared light image; ir and IR are enhanced through the ON countermeasure system, ir is input into the surrounding inhibition area, corresponding to the inhibition signal, and IR is input into the central excitation area, corresponding to the excitation signal, and finally the output signal X1 is obtained. This operation further strengthens the infrared information.

[0038] The third stage is the fusion stage. X2 is obtained by weighted averaging X1 and VI_ON, so that X2 achieves a balance in the fusion of visible light information and infrared information. Then the image contrast is adjusted through gamma correction to enhance the overall detail performance of the image and obtain the final fused image.

[0039] Figure 1 The fusion results of the Waxman method and the method of the present invention are demonstrated by selecting a set of images. (a) and (b) are visible light and infrared images respectively; (c) and (d) are the fusion results of the Waxman method and the improved method respectively. By comparing the results, it can be found that the target information of the fused image obtained by the improved method is more prominent, brighter and easier to identify.

[0040] Experimental results analysis

[0041] The improved method proposed above is simulated experimentally and compared with the other seven fusion methods. The image effects of the fusion results obtained by these eight methods are compared from two aspects: subjective evaluation, that is, the fusion image effect observed by the human eye, and objective evaluation of indicators such as spatial frequency (SF).

[0042] In order to more specifically observe the image fusion effect of the improved method proposed in this section, the present invention prepares a comparative experiment with other image fusion methods based on the rattlesnake vision mechanism. To facilitate the subsequent understanding of the experimental data, the image fusion method that will be compared with the method of the present invention is briefly introduced here.

[0043] First: The Waxman method proposed by A. Waxman et al. is one of the comparative experiments. In order to facilitate the identification of the comparative method in subsequent evaluations, it is recorded as Waxman here.

[0044] Second, the improved fusion method proposed by Li is used as one of the comparative experiments. To facilitate identification of the comparative method in subsequent evaluations, it is denoted as Li here.

[0045] Third, the improved color fusion method proposed by Zhang Zhen is used as one of the comparative experiments. To facilitate the identification of the comparative method in subsequent evaluations, it is denoted as Zhang here.

[0046] Fourthly, the improved image fusion method based on rattlesnake proposed by Wang and Liu et al. is used as one of the comparative experiments. To facilitate the identification of the comparative method in the subsequent evaluation, it is denoted as Wang.

[0047] Fifth, the image fusion method based on multi-resolution singular value decomposition proposed by Naidu et al. was used as one of the non-bionic comparison tests. In order to facilitate the identification of the comparison method in subsequent evaluations, it is denoted as MSVD.

[0048] Sixth, the fusion method based on anisotropic diffusion fusion and Karhunen-Loeve transformation proposed by Bavirisetti et al. was used as one of the comparative experiments. In order to facilitate the identification of the comparative method in subsequent evaluations, it is recorded as ADF.

[0049] Seventh, the multi-sensor image fusion method based on fourth-order partial differential equations proposed by Bavirisetti et al. was used as one of the comparative experiments. In order to facilitate the identification of the comparative method in subsequent evaluations, it is denoted as FPDE.

[0050] Eighth, in order to facilitate identification of the comparison method in subsequent evaluations, the method proposed in the present invention is denoted as Ours.

[0051] 1. Subjective evaluation

[0052] 1.1 Comparative Observation

[0053] Figure 2 This is the fusion result obtained by simulating 16 source images using the method of the present invention (Ours) and the corresponding comparative experiment. In this section, we first conduct a comparative analysis from the perspective of subjective evaluation, that is, the visual effect of the fused image observed by the human eye. Figure 2 From top to bottom are the visible light images, infrared images and 8 fused images from Group 1 to Group 16.

[0054] By observing the above sixteen groups of experimental results, it is found that the pseudo-color result images generated by the four algorithms of Waxman, Li, Zhang, and Wang have clear surrounding environment, are brighter overall, retain more visible light information, and are generally convenient for subsequent observation by the human eye, but the target information in the infrared image is not prominent, and the color is not very natural; the result images generated by MSVD, ADF, and FDPE are generally darker and more blurred, with more noise, and very little surrounding environment information is retained in the visible light image, but the target information in the infrared image is prominent, and the overall image is closer to the infrared image; the result image obtained by the method of the present invention (Ours) not only highlights the target information but also retains the environmental information, has the best visual effect, and the picture is brighter, which is generally more convenient for subsequent observation by the human eye.

[0055] 1.2 Local Observation

[0056] In order to judge the effect of the method of the present invention (Ours) after image fusion, the present invention proportionally enlarges the same part of the image results obtained by the method of the present invention (Ours) and other 6 image fusion methods for comparison, observes their visual effects with human eyes and conducts comparative analysis. Figure 3 This is a partially enlarged image of the person and his background on the right side of the fused image of group 2. Figure 4 These are the images after locally enlarging the car and its background on the upper right side of the four fused images in the group, where (a) is the local enlargement result of the Waxman fused image, (b) is the local enlargement result of the Li fused image, (c) is the local enlargement result of the Zhang fused image, (d) is the local enlargement result of the Wang fused image, (e) is the local enlargement result of the MSVD fused image, (f) is the local enlargement result of the ADF fused image, (g) is the local enlargement result of the FPDE fused image, and (h) is the local enlargement result of the Ours fused image.

[0057] Depend on Figure 3It can be seen that in the pseudo-color images (a), (b) and (c), the surrounding environment is clear and more visible light information is retained, but the target information in the infrared image is not prominent, and it is difficult to see the target person clearly even after local magnification; the target information in (d) is prominent, but there is a black area in the grass in the lower left corner, and there is a small amount of block distortion; the grass level in (e) is relatively fuzzy; (f) and (g) highlight the target information and the environment is relatively clear, but both images contain a lot of noise; (h) contains both environmental information such as roads and trees in the visible light image and highlights the target information standing in the grass in the infrared image. The grass level in (h) is also relatively clear.

[0058] Depend on Figure 4 It can be seen that in the pseudo-color images, (a) and (b) are the darkest and the target information is not prominent; the environmental information and target information in the infrared images in (c) and (d) are relatively clear, but the colors are unnatural, and there is a block-shaped distortion area at the partially enlarged car in (d); images (e), (f), and (g) are all darker overall, and the detail loss in (g) is the most serious, and the partially enlarged car is the most blurred; the noise in (h) is relatively small, and there is more car detail information. Apart from this, there is not much difference in other things; image (h) highlights the target information and retains the environmental information, with the best visual effect.

[0059] 2. Evaluation indicators

[0060] In the previous section, 16 sets of source images were used for the improved method, and experimental simulations were carried out with eight fusion algorithms. The fusion results obtained by these eight methods were subjectively evaluated, that is, the visual effects of these results observed by human eyes were evaluated. In this section, the performance of these eight fusion methods will be evaluated through three objective evaluation indicators such as spatial frequency (SF).

[0061] Table 1 Average gradient

[0062]

[0063] Table 1 is a data table of average gradient index obtained by experimental comparison of four bionic fusion methods, three non-bionic fusion methods and the method of the present invention (Ours) using sixteen groups of images; in the comparative experiment between the method of the present invention (Ours) and seven other methods, 56.25% of the average gradient best values ​​were generated in the method of the present invention (Ours), and the average gradient best values ​​of three groups were generated in the MSVD method, and the average gradient best values ​​of two groups were generated in the ADF and FPDE methods. This shows that among the seven fusion methods, the method of the present invention (Ours) has a better effect on the average gradient index, followed by the MSVD method.

[0064] Table 2 Standard deviation

[0065]

[0066] Table 2 is a data table of standard deviation evaluation indicators obtained by experimental comparison of seven comparative fusion methods and the improved algorithm proposed in this paper using sixteen sets of image pairs; analyzing the information in the data table, it is found that the optimal values ​​of the standard deviation are distributed in the method proposed by Wang et al. and the method of the present invention (Ours), of which 31.25% of the optimal information entropy values ​​are generated in the method proposed by Wang et al., and 68.75% of the optimal information entropy values ​​are generated in the method of the present invention (Ours). This shows that the best method in terms of information entropy indicators is the method of the present invention (Ours), and the method proposed by Wang et al. is second.

[0067] Table 3 is a data table of spatial frequency evaluation indicators obtained by experimentally comparing seven contrast fusion methods with the proposed improved algorithm using sixteen groups of images; analyzing the information in the data table, it is found that: 87.5% of the optimal spatial frequency values ​​are generated in the method of the present invention (Ours), which indicates that the method of the present invention (Ours) has the best effect on the spatial frequency indicator.

[0068] Table 3 Spatial frequency

[0069]

[0070] Through the analysis of the evaluation index data table obtained by experimentally comparing the eight fusion methods in Tables 1 to 3 using sixteen sets of image pairs, it is found that the method of the present invention (Ours) is overall superior to the other seven comparison methods in three evaluation indicators.

[0071] The method for generating non-pseudo-color images based on ON antagonistic cells of the present invention simulates the visual imaging mechanism of snakes and achieves good results, so that the fused image not only highlights the target information but also retains richer detail information.

[0072] In the comparative experiment between the method of the present invention and seven other methods in the prior art, 56.25% of the average gradient best values ​​were generated in the method of the present invention, 68.75% of the information entropy best values ​​were generated in the method of the present invention, and 87.5% of the spatial frequency best values ​​were generated in the method of the present invention. Through the analysis of the obtained evaluation indicators, it was found that the overall performance of the method of the present invention on the three evaluation indicators was better than that of the other comparison methods.

[0073] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for generating non-pseudo-color images based on ON adversarial cells, characterized in that: The following steps are involved: Step 1: Enhancement phase; First, the infrared and visible light source images are recorded as IR and VI respectively; IR is enhanced by the ON confrontation system to obtain the enhanced infrared image IR_ON; VI is enhanced by the ON confrontation system to obtain the enhanced visible light image VI_ON; IR and VI are simultaneously input into the two ON confrontation systems for enhancement processing; Step 2: Operation phase; Input the signal VI-IR and the signal IR-VI into the cell at the same time to obtain the common information of the two: IR AND VI; then IR_ON minus IR ANDVI to obtain the unique information ir of IR_ON; ir and IR are enhanced through the ON confrontation system, ir is input into the surrounding inhibition area, corresponding to the inhibition signal, IR is input into the central excitation area, corresponding to the excitation signal, and finally the output signal X1 is obtained; Step 3: Fusion phase; X2 is obtained by weighted averaging X1 and VI_ON, so that the obtained X2 achieves a balance in the fusion of visible light information and infrared information; then the image contrast is adjusted through gamma correction to enhance the overall detail performance of the image and obtain the final fused image; In step 1, IR and VI are simultaneously input into two ON countermeasure systems for enhancement processing, specifically: VI input surrounds the inhibition area, corresponding to the inhibition signal, and IR inputs the central excitation area, corresponding to the excitation signal. The output of the first ON countermeasure system is the visible inhibition infrared signal VI-IR. The IR input surrounds the inhibition area, corresponding to the inhibition signal, and the VI input is in the central excitation area, corresponding to the excitation signal. The output of the second ON countermeasure system is the infrared inhibition visible signal IR-VI.

Images