Natural color fusion method, device and medium for ultra-large field-of-view infrared low-light image

Abstract

The embodiments of this specification provide a method, device and medium for natural color fusion of ultra-large field infrared low-light images. The method includes super-pixel segmentation of ultra-large field low-light images and color reference images, respectively constructing super-pixel level feature sets, and establishing super-pixel initial matching sets based on the super-pixel feature similarity between each image and the image itself; adaptive color transfer of the low-light image, and initial color fusion of it with the ultra-large field infrared image to obtain an initial color fused image; extracting the infrared independent information colorized image; transferring its natural color, and adding the infrared independent information colorized image to the initial color fused image to obtain an ultra-large field infrared low-light natural color fused image. The present invention focuses on natural colorization of low-light images, and infrared images are mainly used to broaden the image band range, improve the image scene perception level, and expand the natural color space of fused images.

Description

Technical Field

[0001] The present embodiment relates to the field of image color fusion technology, and in particular to a method, device and system for natural color fusion of ultra-large field-of-view infrared low-light images. Background Art

[0002] The natural color fusion of the image increases the band information and color information of the image, significantly improving the personnel's ability to understand the scene and the perception of key targets. By adopting the "non-similar" imaging mechanism, the ultra-large field of view imaging technology expands the detection airspace. Among the ultra-large field of view imaging technologies, infrared and low-light night vision imaging technologies can effectively broaden the visual spectrum range of the human eye and improve visual perception. Infrared and low-light night vision technologies each have their own advantages and disadvantages, so infrared low-light fusion technology has been widely studied and applied to machine vision, security monitoring, situational awareness and other fields;

[0003] At present, natural colorization methods can be mainly divided into three categories: manual assisted colorization, sample reference based colorization and deep learning based colorization.

[0004] Among them, artificial assisted colorization mainly relies on manpower to annotate natural colors to the corresponding scenes in the grayscale image. This method relies on human operation, and the colorization effect is related to the operator's experience. The sample reference-based colorization method mainly includes two processes: image content matching and color transfer. Among them, image content matching mainly relies on image texture, brightness and other features to complete, and color transfer gives the grayscale image a color space based on the matching results. The colorization method based on deep learning constructs the relationship between grayscale information and color information by training a large number of color images. The grayscale image is input into a well-trained network to complete the colorization. This method has a high level of automation and rich color information, but it is highly dependent on network design, and for some specific objects, the color transfer effect is poor.

[0005] Compared with conventional small-field-of-view images, ultra-large-field-of-view images compress larger object scenes into images on the detector image plane, thereby achieving large-area perception and detection. However, ultra-large field-of-view introduces rich scene content, which requires the image content matching method to have more accurate and stable image expression capabilities; at the same time, image compression imaging leads to deformation of scene content, and matching methods based on features such as contours and shapes will be difficult to apply.

[0006] In view of this, this paper provides a comprehensive comparison of three types of color transfer methods and proposes a natural-looking color fusion method that is more suitable for colorization of ultra-large field-of-view grayscale images. Summary of the invention

[0007] One or more embodiments of this specification provide a method for natural color fusion of ultra-large field-of-view infrared low-light level images, including the following steps:

[0008] Taking the low-light image as the target image and the color reference image as the color source image, superpixel segmentation is performed on the ultra-large field of view low-light image and color reference image, and superpixel level feature sets are constructed respectively. The superpixel initial matching set is established based on the superpixel feature similarity between each image and the image itself.

[0009] Based on the established superpixel initial matching set, the low-light image is adaptively color-transferred, and the image is initially color-fused with the ultra-large field-of-view infrared image to obtain an initial color-fused image; the initial color-fused image and the low-light initial colorized image are differentiated to obtain an infrared independent information image;

[0010] The infrared independent information image is used as the target image for adaptive color transfer, and the colorized image obtained after adaptive color transfer is added to the initial color fusion image to obtain an ultra-large field of view infrared low-light natural color fusion image.

[0011] One or more embodiments of the present specification provide a computer electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the natural color fusion method for ultra-large field-of-view infrared low-light images as described above when executing the computer program.

[0012] One or more embodiments of the present specification provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for natural color fusion of ultra-large field-of-view infrared low-light images as described above is implemented.

[0013] The ultra-large field-of-view infrared low-light natural-color fusion method based on superpixel features provided by the present invention focuses on natural-colorization of low-light images, and infrared images are mainly used to widen the image band range, improve the image scene perception level, and expand the natural-color space of the fused image; the method chooses the method of first color transfer and then fusion, in order to solve the problem that although infrared images have the advantages of strong penetration ability and are not affected by factors such as weather, they do not conform to the visual habits of the human eye, that is, the characteristics of infrared and visible light images are obviously different, which can easily lead to the occurrence of mismatches. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0015] Figure 1 A schematic diagram of a process for natural color fusion of ultra-large field-of-view infrared low-light image provided in one or more embodiments of this specification;

[0016] Figure 2 A schematic diagram of the process framework of a method for natural color fusion of ultra-large field-of-view infrared low-light level images provided in one or more embodiments of this specification;

[0017] Figure 3 A superpixel feature constraint matching result diagram in the natural color fusion method for ultra-large field of view infrared low-light image provided in one or more embodiments of this specification;

[0018] Figure 4 A graph showing the result of the joint constraint matching of superpixel features and color clustering in the natural color fusion method for ultra-large field-of-view infrared low-light image provided in one or more embodiments of this specification;

[0019] Figure 5 A diagram of the color transfer result of a super-large-field-of-view visible light image in the super-large-field-of-view infrared low-light image natural-sense color fusion method provided in one or more embodiments of this specification;

[0020] Figure 6 A flowchart of fusing an infrared independent information colorized image and an initial colorized image in a method for natural color fusion of ultra-large field-of-view infrared low-light level images provided in one or more embodiments of this specification;

[0021] Figure 7 A fusion effect diagram of a natural color fusion method for ultra-large field-of-view infrared low-light images provided by one or more embodiments of this specification, wherein (a) the original infrared image; (b) the infrared weight map; (c) the initial color fusion image; (d) the infrared independent information image; (e) the infrared independent information colorization result; (f) the final color fusion result;

[0022] Figure 8 The first set of experimental image color transfer and color fusion result comparison diagrams are implemented by Welsh, Reinhard, Gupta and the present invention respectively in the experimental cases of this specification;

[0023] Fig. 9The second set of experimental image color transfer and color fusion result comparison diagrams are respectively implemented by Welsh, Reinhard, Gupta and the present invention method in the experimental cases of this specification;

[0024] Fig.10 The third group of experimental image color transfer and color fusion result comparison diagrams implemented by Welsh, Reinhard, Gupta and the present invention respectively in the experimental cases of this specification;

[0025] Fig.11 The fourth group of experimental image color transfer and color fusion result comparison diagrams implemented by Welsh, Reinhard, Gupta and the present invention respectively for the experimental cases of this specification;

[0026] Fig.12 The first group of fusion experimental results comparison diagrams using NSCT-PCNN, Zhang's method and the method of the present invention respectively for the experimental cases of this specification;

[0027] Fig.13 The second group of fusion experiment result comparison diagrams using NSCT-PCNN, Zhang's method and the method of the present invention respectively for the experimental cases of this specification;

[0028] Fig.14 The third group of fusion experiment results comparison diagrams using NSCT-PCNN, Zhang's method and the method of the present invention respectively for the experimental cases of this specification;

[0029] Fig.15 The fourth group of fusion experiment result comparison diagrams of the experimental cases in this specification using NSCT-PCNN, Zhang's method and the method of the present invention respectively;

[0030] Fig.16 Based on Figure 12-Figure 15 The ultra-large field of view infrared low-light image is realized by using grayscale fusion first and then color transfer.

[0031] Fig.17 A schematic diagram of the structure of a computer device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of this embodiment.

[0033] The present invention is described in detail below in conjunction with specific implementation methods and the accompanying drawings.

[0034] Method Embodiment

[0035] According to an embodiment of the present invention, a method for natural color fusion of ultra-large field-of-view infrared low-light image is provided. Figure 1-2 As shown, it is a schematic diagram and a process framework diagram of the ultra-large field of view infrared low-light image natural color fusion process provided by this embodiment. The ultra-large field of view infrared low-light image natural color fusion method according to the embodiment of the present invention includes the steps of:

[0036] Step 101: Taking the low-light image as the target image and the color reference image as the color source image, super-pixel segmentation is performed on the low-light image with a large field of view and the color reference image, and super-pixel level feature sets are respectively constructed. Based on the super-pixel feature similarity between each image and the image itself, a super-pixel initial matching set is established;

[0037] Step 102: Based on the established superpixel initial matching set, perform adaptive color transfer on the low-light image, and perform initial color fusion on it with the ultra-large field of view infrared image to obtain an initial color fusion image; perform a difference between the initial color fusion image and the low-light initial colorized image to obtain an infrared independent information image;

[0038] Step 103: Adaptively transfer the infrared independent information image as the target image, and add the colorized image obtained after the adaptive color transfer to the initial color fused image to obtain an ultra-large field of view infrared low-light natural color fused image.

[0039] The ultra-large field-of-view infrared low-light natural-color fusion method based on superpixel features provided in this embodiment is mainly used for natural-colorization of low-light images. Infrared images are mainly used to widen the image band range, improve the image scene perception level, and expand the natural-color space of fused images. In this embodiment, the method of first color transfer and then fusion is selected. This is to solve the problem that although infrared images have the advantages of strong penetration and are not affected by factors such as weather, they do not conform to the visual habits of the human eye, that is, the characteristics of infrared and visible light images are obviously different, which can easily lead to mismatching.

[0040] In this embodiment, in order to extract infrared image information independent of the low-light band, based on step 101, the initial color fusion image and the low-light initial colorized image are differentiated to obtain infrared independent information.

[0041] Compared with conventional small-field-of-view images, the "non-similar" imaging mechanism of the ultra-large-field-of-view in this embodiment leads to distortion of the image structure content, and the closer to the edge of the image, the greater the degree of distortion. Therefore, the commonly used matching methods based on image points, lines, regions and other features will be difficult to directly apply to the matching of ultra-large-field-of-view image content. Therefore, in order to further optimize the superpixel matching set and ensure the accuracy of image matching and the continuity of color transfer, this embodiment adopts an image content matching method based on grayscale space superpixel features;

[0042] Preferably, this embodiment transforms the color image into lαβ color space, where l represents the brightness space of the image and αβ represents the color space of the image, and adopts the SLIC method to perform superpixel segmentation on the l space of the ultra-wide field of view low-light image and the ultra-wide field of view color reference image, respectively, and establishes a superpixel initial matching set based on the superpixel feature similarity between the two images and the ultra-wide field of view low-light image itself.

[0043] In order to increase the robustness of superpixel matching, this embodiment extracts l-space superpixel features of the ultra-large field of view low-light image and the ultra-large field of view color reference image according to three levels: low, medium, and high.

[0044] The low-level features selected are image brightness distribution probability, mean and variance; the middle-level features selected are dense SIFT, LBP, LGP features; the high-level features selected are saliency features.

[0045] ①Low-level features.

[0046] Count the grayscale values ​​of pixels in each superpixel, divide them into 8 bins in the range of [0,255], and count the proportion of each bin, recorded as Calculate the grayscale mean and variance of all pixels, respectively recorded as The low-level superpixel feature is represented as

[0047] ②Middle-level characteristics.

[0048] In order to more effectively identify features such as texture, the dense SIFT features (128 dimensions), LBP features (59 dimensions) and LGP features (7 dimensions) of the entire image are calculated and normalized. In the superpixel, the mean of these features is calculated and recorded as The mid-level superpixel feature is represented as

[0049] ③High-level features.

[0050] The human visual system (HVS) always pays attention to the areas with significant changes. If the same object is significant in both images, the human eye can easily complete the recognition and matching. Based on this characteristic, a high-level feature based on saliency is used to achieve bionic matching. This embodiment adopts a saliency detection method, performs normalization operation processing on it, and the dimension is 1, denoted as f h .

[0051] In this embodiment, the Euclidean distance of the low, medium and high level features of the superpixel is used as the similarity. The smaller the Euclidean distance, the more similar the superpixel features are. For each superpixel in the low-light image, all superpixels in the color reference image l space are traversed and the similarity of the features of the two is calculated. If there are superpixels that meet the feature similarity requirements, these superpixels are regarded as the initial matching set of the superpixels of the current low-light image. At the same time, the feature similarity between all superpixels in the low-light image is calculated, and the superpixels that meet the similarity requirements are classified into one category, and the corresponding superpixel matching sets of the color image are merged.

[0052] In this embodiment, different weights are assigned to features at different levels to obtain better matching results. In this embodiment, the weights assigned to low-level, medium-level, and high-level features are set to 0.2, 0.5, and 0.3, respectively. Figure 3 As shown, Figure 3 (a) and 3(b) are respectively a super-wide field color image and a visible light grayscale image. By testing and selecting the superpixel with the highest feature similarity as the matching object, the superpixel matching results of the two images are shown in Figure 3 (c) and 3(d).

[0053] In the sample-based color transfer method of this embodiment, according to the superpixel matching results, the superpixels in the ultra-wide field of view low-light image can directly obtain the color from the superpixel with the highest feature similarity in the color reference image. However, the matching results based on superpixel features are easily affected by grayscale and texture changes, such as in the structural edges of the image, and in the ultra-wide field of view image, the non-uniformity of the image is also likely to become an interference factor for superpixel matching. Therefore, in order to improve the accuracy and consistency of superpixel matching, this embodiment proposes to establish a color classification model for superpixels in the color image by clustering the color space αβ of the color reference image, and guide and optimize the initial matching set of superpixels, as follows:

[0054] According to the color space constraint relationship of the color reference image, an improved Fuzzy ART (Fuzzy Adaptive Resonance Theory) network is proposed for superpixel color clustering of the color reference image.

[0055] Taking the color space αβ of color reference images as the research object, a color clustering method of color reference images based on improved Fuzzy ART neural network is proposed.

[0056] In order to optimize the Fuzzy ART structure, improve training accuracy and reduce complexity, this embodiment makes improvements in two aspects: on the one hand, the competition layer and the matching layer are swapped and adjusted to a positive correlation; on the other hand, the network learning rate is adaptively adjusted according to the input color similarity, that is, the higher the input color similarity, the faster the learning rate.

[0057] Specifically, the input matrix of the neural network is represented as [αβ1-α1-β], and the matching layer is represented as:

[0058]

[0059]

[0060] Where I is the input color feature, w j is the existing color feature type, and M is the number of color types.

[0061] The competition layer is expressed as:

[0062]

[0063] Where α = 10 -6 is to prevent calculation overflow, w J is the color type that is currently matched successfully, and n is the number of color types that are matched successfully;

[0064] The learning rate is adaptively adjusted to:

[0065]

[0066]

[0067] Where β0 is the initial learning rate, is the updated color type weight.

[0068] In this embodiment, the training output of the improved Fuzzy ART neural network is the classification result of superpixel color. Figure 3 The color clustering results of the ultra-large field of view color reference image are as follows: Figure 4Another advantage of using neural network training is that if a color reference image needs to be added, it can be directly input into the network for training to obtain the result, but the traditional clustering method requires secondary operation on all superpixel colors, which undoubtedly increases the complexity.

[0069] According to the superpixel color clustering results of the ultra-wide field color reference image and the clustering results using the SLIC method, in the same image, image content with similar colors is likely to have similar superpixel features. Therefore, the results of clustering segmentation can be used to constrain the matching relationship between superpixels of the same type, that is, in the matching set of superpixels in the ultra-wide field low-light image, the colors of the superpixels should belong to the same category. If there are superpixels with obvious color differences from other superpixels, they should be removed from the matching set. In this way, through the joint constraints of grayscale space superpixel features and color space clustering, the disadvantage of insufficient expression ability of superpixel features in certain areas is compensated, and the matching accuracy of superpixels is improved.

[0070] According to the above superpixel matching set optimization results, the superpixels with representative colors in the matching set are selected as matching objects. The superpixel matching results of the ultra-large field of view low-light image are as follows: Figure 4 (b) shows that Figure 3 (b) By comparison, it can be seen that the optimized superpixel categories are more continuous.

[0071] In this embodiment, in step 102 and step 103, the KNN algorithm is used to implement the adaptive color transfer method, which is as follows:

[0072] The KNN algorithm of this embodiment calculates the similarity or distance between the test data and K adjacent samples, and selects the category with the highest probability as the corresponding category of the current data. The specific algorithm is as follows:

[0073]

[0074] In the formula, sim represents the test sample x and the adjacent sample a i The similarity of i belongs to training sample c, then I(a i , c) is 1, otherwise it is 0.

[0075] Since the ultra-large field of view contains a larger scene airspace and relatively more color types, simple direct color transfer can easily lead to unnatural and discontinuous color transitions. In order to effectively and reasonably represent the rich scene information in the ultra-large field of view and improve the naturalness of color transfer, this embodiment proposes an adaptive color transfer method, and analyzes the following two situations for superpixels in ultra-large field of view low-light images:

[0076] (1) If its matching set is not empty and its color image superpixel matching set is not empty, then according to the principle that the smaller the Euclidean distance, the higher the similarity, the superpixel is given a greater weight in color transfer. The calculation formula is as follows:

[0077]

[0078] Where N represents the number of superpixels in the color reference image in the matching set, and C i is the αβ color space of the i-th superpixel, Euc i is the characteristic Euclidean distance between it and the current superpixel of the grayscale image, and C is the αβ color space obtained by the current superpixel of the grayscale image.

[0079] The above-mentioned superpixel feature matching set is the result of comparing the grayscale features of the color reference image and the low-light image. The color reference image itself has color, so there is a corresponding color image superpixel matching set.

[0080] I choose a larger weight because I eventually want to color each superpixel in the low-light image, and each superpixel in the low-light image may correspond to multiple superpixels in the color reference image after matching. I assign a larger weight to superpixel features with high similarity, but I do not choose the one with the highest similarity because it is easy to be distorted.

[0081] (2) If the matching set is empty, according to the KNN color transfer principle, in order to improve the continuity of color transfer, this embodiment optimizes the KNN algorithm to obtain the color by integrating the neighborhood of the current superpixel. Similar to formula (7), the Euclidean distance between the current superpixel and the neighborhood of k superpixels that have successfully transferred the color is calculated to adaptively obtain the color and transfer it. The specific calculation formula is as follows:

[0082]

[0083] Where k is the number of neighborhood superpixels with successful color transfer in the grayscale image, C j is the color space of the jth neighborhood superpixel that has successfully transferred color, Euc j ′ is the characteristic Euclidean distance between it and the current superpixel, and C′ is the color space of the current superpixel obtained based on the neighborhood superpixels.

[0084] The natural color transfer results of ultra-large field of view visible light images are as follows Figure 5 As shown, it can be seen that the initial colorized image better reflects the color of the scene in the image, such as trees, roofs and buildings in and on the edge of the snow.

[0085] In this embodiment, in step S103, the colorized image of the infrared independent information is added to the initial color fused image to perform a natural color fusion operation, which is specifically as follows.

[0086] Infrared images help to improve the brightness of low-light colorized images and increase infrared band information. After obtaining the low-light initial color fusion image, it is converted to the lαβ color space, and the image in the l space is fused with the infrared image. To this end, this embodiment proposes a grayscale image fusion rule based on robust principal component analysis (RPCA) and guided filtering to fuse the ultra-large field of view infrared image and the ultra-large field of view low-light initial colorized image. The process is as follows: Figure 6 As shown, the following steps are included:

[0087] Step 1101, using the RPCA method to decompose the colorized image of infrared independent information and the image of the l space of the initial color fusion image into a low-rank image and a sparse image part;

[0088] Step 1102: fusing the two low-rank images by weighted averaging to obtain a low-rank fused image;

[0089] The two sparse images are fused by Laplace filtering to obtain a sparse fused image.

[0090] Step 1103: Use the inverse RPCA method (the inverse transformation and reconstruction of RPCA) to fuse the low-rank fusion image and the sparse fusion image obtained in step 1102 to reconstruct a grayscale fusion image, and transform it together with the α and β color spaces of the initial color fusion image to obtain an ultra-large field of view infrared low-light natural color fusion image.

[0091] In this embodiment, preferably, in step 1102, when fusing two sparse images, Laplace filtering is used on the infrared image to construct a fused infrared weighted image, and then anisotropic diffusion filtering is used to maintain the salient areas of the image, and the weighted image is optimized using the infrared original image as a guide image.

[0092] In this embodiment, the infrared weight map of the ultra-large field of view image fusion and the initial color fusion image are as follows: Figure 7 (b) and 7(c).

[0093] In addition, if Figure 7As shown in (c), in the initial color fused image, a part of the content from the infrared image is not given color, which is defined as infrared independent information in this embodiment. By making a spatial difference between the grayscale fused image and the initial colorized image l, an infrared independent information image can be obtained. Using it as the target image for natural color transfer, a colorized result of the infrared independent information image can be obtained. Finally, its color space is transferred to the initial color fused image, and the uncolored independent infrared information has a natural color. It should be noted that this embodiment does not adopt the method of grayscale fusion of infrared and low light first and then color transfer, because the image features after fusion are no longer similar to the visible light image features, and it is difficult to complete superpixel matching.

[0094] The effectiveness and fusion effect of the above method are illustrated by specific experimental cases.

[0095] In order to verify the effect of the method proposed in this embodiment, the ultra-large field of view visible light infrared example image and three groups of ultra-large field of view infrared low-light images were selected for natural color fusion experiments. Among them, the proportion of ground scenery in the four groups of images was approximately 50%, 70%, 85% and 30%, respectively. At the same time, in order to verify the superiority of the method of this embodiment, comparative experiments were carried out by changing the color transfer method alone, changing the fusion method alone, and grayscale fusion first and then color transfer. In terms of hardware, the diagonal field of view of the ultra-large field of view infrared low-light lens is approximately 138° and 135°, respectively. The detectors use AutoNavi medium-wave infrared detection and "Sharp Core" low-light sensitive detectors, respectively. The experimental pictures used in this embodiment have been well registered and can be directly used for image fusion.

[0096] Based on the method of this embodiment, in the comparative experiment of changing the color transfer method separately, the color transfer methods of Welsh, Reinhard and Gupta are used respectively. The color transfer results of ultra-large field visible light / low light and the color fusion image are shown in Figure 2. Figure 8-Figure 11 As shown, in each figure, Figures (a1)-(a3) are the original low-light image, the original infrared image and the color reference image respectively; Figures (b1)-(b4) are the low-light color transfer results using the methods of Welsh, Reinhard, Gupta and this embodiment respectively; Figures (c1)-(c4) are the color fusion results of Figures (b1)-(b4) in one-to-one correspondence;

[0097] The experimental results of the first group of images are as follows Figure 8As shown in the figure, in the color transfer results of the Welsh method and the Reinhard method, the colors of the trees and houses are obviously insufficient and difficult to reflect their true natural colors; the color transfer results of the Gupta method and the method of this embodiment can basically reflect the color of the real scene. In comparison, the Gupta method has a better color transfer effect on the trees in the center of the image, but the snow on the playground and the trees at the bottom of the image are not good. In the color fusion image, the colors of the distant buildings in the four images are obvious, which verifies the correctness of the method of obtaining infrared independent information and its separate coloring in this embodiment; comprehensive comparison shows that the color space transition of the method of this embodiment is smoother, the color space consistency with the real scene is better, and the visual observation effect is better.

[0098] The experimental results of the second group of images are as follows Fig. 9 As shown, Figure 8 The results are similar. In the results of Welsh and Reinhard's method, only the color of the infrared independent information is more obvious, and the color transfer effect of other scenes is poor. In the color fusion image of Gupta's method, the colors of the wall and trees are mistransmitted; in the color fusion image of the method of this embodiment, some trees also have color mismatches, which may be caused by the low discrimination of superpixel features. The experimental results of the third group of images are shown in Figure 1. Fig.10 As shown in Figure 1, the image of the Welsh and Reinhard method is colored overall, but the color transition in the scene is not obvious enough, and the color of some areas is not ideal. Except for the mismatch of the light of the street lamp in the lower right corner of the image, the Gupta method and the method of this embodiment achieve similar results. The experimental results of the fourth group of images are shown in Figure 1. Fig.11 As shown, affected by the large amount of sky background in the image, the overall fusion color of the Welsh method is blue, and some trees are mis-transmitted. The Reinhard method has achieved certain results, and the color transition is relatively natural. The effects of the Gupta method and the method of this embodiment are basically the same, both of which can basically reflect the color of the real scene, and the boundaries are clearer.

[0099] To further verify the effect of the above-mentioned embodiment method, on the basis of the method of this embodiment, only the fusion method of the color transfer image and the infrared image is changed separately and a comparative experiment is conducted. The comparative method selects the method of NSCT-PCNN and Zhang's method (Multimodel fusion method via sparse representation at pixel-level and feature-level proposed by Fuyu Huang et al. in the SPIE journal on June 17, 2019). First, the color image is transformed into the lαβ color space, and then the infrared image and the color l space are gray-fused. Finally, the αβ color space is added to the gray-scale fused image and transformed into an RGB image. The experimental results are as follows. Figure 12-Figure 15 As shown, in each figure, Figure (a), (b), and (c) are respectively the effect diagrams of the fusion method using NSCT-PCNN, Zhang's method, and the method of this embodiment.

[0100] Compared with the comparison method, the fusion method of this embodiment retains the detail information and color information of ultra-large field visible light / low light to the greatest extent, and naturally integrates infrared information into visible light / low light information. At the same time, the visual observation effect of the method of this embodiment is better and more in line with the color experience of the human eye. In order to objectively verify the fusion effect of the method of this embodiment, the QG, QP and QE indicators are used for fusion evaluation, as shown in Table 1: The indicators of the method of this embodiment are basically the highest among the three methods, and are at least 19.5% higher than the indicators of the other two comparison methods.

[0101] Table 1 Evaluation results of color fusion indicators

[0102]

[0103] In order to quantitatively analyze the color effects of the four methods in the color fusion image, 30 military academy students with basic knowledge of image-related were organized as subjects to conduct color evaluation. It mainly involves three aspects: color naturalness, details, and similarity with the color reference image. The score is set to 1-5 points. The higher the score, the better the corresponding effect. The color evaluation results of the ultra-large field of view infrared low-light color fusion image are shown in Table 2. It can be seen from the table that most of the indicators of the method of this embodiment have the highest scores, and the average values ​​of the indicators are improved by at least 7.3% compared with the comparison method, which shows the accuracy and robustness of the image color fusion method proposed in this embodiment.

[0104] Table 2 Color evaluation of ultra-large field of view infrared low-light color fusion image

[0105]

[0106] In order to verify the effectiveness of the independent coloring of infrared information in the method of this embodiment, a comparative experiment was conducted by first merging grayscale images and then transferring the color of the grayscale fused images. Fig.16 As shown, it can be seen that the contrast method can also achieve a certain natural colorization effect, but there are unnatural colors and discontinuous transitions caused by mismatches in each image. The main reasons are two aspects: first, the scene content in the ultra-large field of view is rich, and many contents are similar, and the joint constraints of superpixel features and color clustering cannot effectively distinguish them; second, the ultra-large field of view infrared and low-light images are first fused, which changes the texture and other features of the original image to a certain extent, reducing its similarity with the color reference image. Therefore, the method of coloring the infrared independent information separately proposed in this embodiment not only ensures the natural color of the visible light, but also effectively broadens the spectrum range of the image.

[0107] The present invention proposes a method for natural color fusion of ultra-large field infrared low-light image. The method constructs super-pixel matching relationship and completes adaptive color transfer through the joint constraint of super-pixel features and color clustering. At the same time, it effectively fuses infrared image and low-light color transfer image, retains the significant information of dual bands, and improves the natural colorization level of fused image.

[0108] The experimental results of natural color fusion of ultra-large field-of-view infrared low-light images show that compared with the comparative method, this method has the advantages of good color consistency, clear dual-band details, and high comfort for human eye observation.

[0109] like Fig.17 As shown, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the natural color fusion method of ultra-large field of view infrared low-light image in the above embodiment is implemented, or when the computer program is executed by a processor, the natural color fusion method of ultra-large field of view infrared low-light image in the above embodiment is implemented. When the computer program is executed by the processor, the following method steps are implemented:

[0110] Step 101: Taking the low-light image as the target image and the color reference image as the color source image, super-pixel segmentation is performed on the low-light image with a large field of view and the color reference image, and super-pixel level feature sets are respectively constructed. Based on the super-pixel feature similarity between each image and the image itself, a super-pixel initial matching set is established;

[0111] Step 102: Based on the established superpixel initial matching set, perform adaptive color transfer on the low-light image, and perform initial color fusion on it with the ultra-large field of view infrared image to obtain an initial color fusion image; perform a difference between the initial color fusion image and the low-light initial colorized image to obtain an infrared independent information image;

[0112] Step 103: Adaptively transfer the infrared independent information image as the target image, and add the colorized image obtained after the adaptive color transfer to the initial color fused image to obtain an ultra-large field of view infrared low-light natural color fused image.

[0113] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0114] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A natural color fusion method for ultra-large field of view infrared low-light image, characterized in that: The following steps are involved: Taking the low-light image as the target image and the color reference image as the color source image, superpixel segmentation is performed on the ultra-large field of view low-light image and color reference image, and superpixel level feature sets are constructed respectively. The superpixel initial matching set is established based on the superpixel feature similarity between each image and the image itself. Based on the established superpixel initial matching set, the low-light image is adaptively color-transferred, and the image is initially color-fused with the ultra-large field-of-view infrared image to obtain an initial color-fused image; the initial color-fused image and the low-light initial colorized image are differentiated to obtain an infrared independent information image; The infrared independent information image is used as the target image for adaptive color transfer, and the colorized image obtained after adaptive color transfer is added to the initial color fusion image to obtain an ultra-large field of view infrared low-light natural color fusion image; The super-pixel segmentation of the ultra-large field low-light image and the color reference image comprises: The color image is transformed into the lαβ color space, and the SLIC method is used to perform superpixel segmentation on the l space of the ultra-wide field of view low-light image and the ultra-wide field of view color reference image respectively; The l-space superpixel features of the ultra-wide field of view low-light level image and the ultra-wide field of view color reference image are extracted according to three levels: low, medium and high.

2. The natural color fusion method for ultra-large field of view infrared low-light level images according to claim 1, characterized in that: The color space αβ of the color reference image is clustered based on the improved Fuzzy ART neural network, wherein the improvements of the improved FuzzyART neural network include: Swap the competition layer and the matching layer; The network learning rate is adaptively adjusted according to the input color similarity, that is, the higher the input color similarity, the faster the learning rate.

3. The natural color fusion method for ultra-large field of view infrared low-light level images according to claim 1, characterized in that: The KNN algorithm is used to realize the infrared independent information colorized image as the target image for adaptive color transfer.

4. The natural color fusion method for ultra-large field of view infrared low-light level images according to claim 3, characterized in that: The adaptive color transfer includes the following two cases: (1) If its matching set is not empty and its color image superpixel matching set is not empty, then according to the principle that the smaller the Euclidean distance, the higher the similarity, the superpixel is given a greater weight in color transfer and transferred; (2) If the matching set is empty, the KNN algorithm is optimized to obtain the color by integrating the neighborhood of the current superpixel. The Euclidean distance between the current superpixel and the neighborhoods of multiple superpixels that have successfully transferred colors is calculated to adaptively obtain the color and transfer it.

5. The natural color fusion method for ultra-large field of view infrared low-light level images according to claim 1, characterized in that: The grayscale image fusion rule based on RPCA method and guided filtering is realized by adding the colorized image with independent infrared information to the initial color fused image for fusion.

6. The natural color fusion method for ultra-large field of view infrared low-light level images according to claim 5, characterized in that: The grayscale image fusion rule based on the RPCA method and guided filtering realizes adding the infrared independent information colorized image to the initial color fused image for fusion, and comprises the following steps: The RPCA method is used to decompose the colorized image of infrared independent information and the l-space image of the initial color fusion image into a low-rank image and a sparse image part respectively; The two low-rank images are fused by weighted average to obtain a low-rank fused image; the two sparse images are fused by Laplace filtering to obtain a sparse fused image; The inverse RPCA method is used to fuse the low-rank fusion image and the sparse fusion image to reconstruct the grayscale fusion image, and then the grayscale fusion image is transformed together with the α and β color spaces of the initial color fusion image to obtain an ultra-large field of view infrared low-light natural color fusion image.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the natural-sense color fusion method for ultra-large-field-of-view infrared low-light-level images is implemented as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the natural-sense color fusion method for ultra-large-field-of-view infrared low-light-level images as described in any one of claims 1 to 6 is implemented.

Images