Multi-channel image fusion method and system based on multi-sensor image enhancement optimization

Abstract

The application provides a multi-channel image fusion method and system based on multi-sensor image enhancement optimization, comprising: obtaining a low-light image data set containing an EBCMOS image, a visible light black and white image and a near-infrared image, and then performing data preprocessing; performing image optimization enhancement, including: histogram equalization, contrast enhancement, and multi-scale enhancement; performing photometric feature information extraction; training a multi-channel image fusion framework according to the pictures contained in the low-light image data set, including: a synthetic learning stage and a reverse optimization stage, and then forming a multi-channel image fusion framework; inputting the pictures after data preprocessing into the multi-channel image fusion framework, and outputting the fused images. The multi-channel image fusion method provided by the application can perform image enhancement optimization on low-light images, and can effectively retain the illumination intensity and foreground / background information in each image while reducing the loss of effective information and the redundancy of invalid information in the learning process of the network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion processing, and particularly relates to a multi-channel image fusion method and system based on multi-sensor image enhancement optimization. BACKGROUND

[0002] The low-light image fusion method is a technology for fusing images captured by low-light sensors (night vision devices, infrared cameras, etc.) and images captured by visible light sensors (ordinary cameras), which can provide clearer and more detailed images in insufficiently lit or completely dark environments, helping to enhance the performance of target detection, recognition, and surveillance applications; the low-light image fusion method has a wide range of applications in military, security, night driving, etc., and it can help people better perceive and understand the environment in low-light conditions, improving the efficiency and accuracy of task execution.

[0003] At present, existing low-light image fusion methods generally include:

[0004] 1. Most of them use traditional fusion methods, such as multi-scale transform fusion, subspace-based fusion, and saliency-based fusion; although these methods have achieved good fusion results, these methods rely on predefined transformation patterns and corresponding scale components for source image decomposition and reconstruction, and traditional fusion methods are limited by design rules and cannot fully capture the complex features of images;

[0005] 2. A few use deep learning-based fusion methods, which, compared with traditional fusion methods, aim to provide information-rich fusion images by combining complementary information from different sensors; existing learning-based fusion methods try to construct various loss functions to preserve the complementary features of different channel source images, but ignore the discovery of the mutual relationship between different channels, resulting in redundant or even invalid information in the fusion results, and most of them focus on strengthening the network as the number of network layers increases, ignoring the importance of feature transmission, leading to degradation of important information and compromised fusion results.

[0006] Therefore, there is an urgent need for a multi-channel image fusion method that can perform image enhancement optimization on low-light images while effectively preserving the brightness and foreground / background information in the images, and reducing the loss of valid information and the redundancy of invalid information in the network during the learning process. SUMMARY

[0007] In order to fully capture the complex features of the image, optimize the image enhancement of the low-light image, and further reduce the loss of effective information and the redundancy of invalid information in the learning process of the network while effectively preserving the illumination intensity and foreground / background information in the image, one aspect of the present application provides a multi-channel image fusion method based on multi-sensor image enhancement optimization, which comprises the following steps:

[0008] Step S1: Obtain a low-light image dataset containing an EBCMOS image, a visible light black and white image and a near-infrared image, and then perform data preprocessing according to the low-light image dataset;

[0009] Step S2: Perform image optimization enhancement according to the low-light image dataset after data preprocessing, including histogram equalization, contrast enhancement and multi-scale enhancement;

[0010] Step S3: Perform photometric feature information extraction according to the low-light image dataset after image optimization enhancement, including constructing a multi-level attention model for illumination perception information extraction, constructing a photometric probability of an output image of a photometric information extraction network, constructing an auxiliary module for extracting common important features of the same target, and then using a convolution dimension reduction method to retain the selected optimal features;

[0011] Step S4: Train a multi-channel image fusion framework according to the pictures contained in the low-light image dataset, including a synthetic learning stage and a reverse optimization stage, and then fuse the synthetic learning stage and the reverse optimization stage to form a multi-channel image fusion framework;

[0012] Step S5: Input the picture after data preprocessing into the multi-channel image fusion framework to output a fused image.

[0013] Preferably, the data preprocessing method is:

[0014] Step S10: Use a semi-automatic feature matching based method to perform registration between multi-channel images, and output a registration picture with the same size as the calibrated camera;

[0015] Step S11: Crop the common area of the registration picture to retain the preset target;

[0016] Step S12: Use an LPDM method to perform image enhancement on the image below the preset illumination value.

[0017] Preferably, the method of using a semi-automatic feature matching based method to perform registration between multi-channel images is:

[0018] Step S100: the images of different channels are combined with the calibration camera image respectively by using the method of manual control point selection, a group of preset number of corresponding points with obvious features and prominent targets are manually selected, and then the corresponding homography matrix is calculated by least square method;

[0019] Step S101: for any group of images, the encoded method is used to realize automatic image registration under the same experimental environment by using the homography matrix.

[0020] Preferably, the image optimization enhancement method of histogram equalization is:

[0021] Step S20: input the image I(x, y) after data preprocessing, and generate the histogram H(i) of the input image, wherein i∈[0, L-1], represents the number of pixels of gray level i;

[0022] Step S21: calculate the cumulative distribution function:

[0023]

[0024] For each pixel point, map its gray value to a new gray value:

[0025]

[0026] Wherein, the gray scale range of the input image I(x, y) is [0, L-1], (x, y) is a pixel point, L is the number of gray levels, M and N are the width and height of the image, CDF min is the cumulative distribution function value of the first non-zero element.

[0027] Preferably, the image optimization enhancement method of contrast enhancement is:

[0028] Step S22: for each pixel point, map its gray value to a new gray value:

[0029]

[0030] Wherein, I max is the maximum pixel value of the input image, I min is the minimum pixel value of the input image.

[0031] Preferably, the image optimization enhancement method of histogram equalization combined with contrast enhancement is:

[0032] Step S23: for each pixel point, the gray values after histogram equalization and contrast enhancement are combined by weighting to obtain the enhanced gray value:

[0033] I final (x, y) = a × I enhanced(x, y) + (1 - a) x I eq (x, y)

[0034] wherein a is a weight parameter, and a has a value range of 0-1.

[0035] Preferably, the multi-scale enhanced image optimization enhancement method is:

[0036] Step S24: using wavelet transform to decompose the input image to obtain at least one high-frequency subband and low-frequency subband:

[0037] I(x, y) = å j B j (x, y) + A j (x, y)

[0038] wherein B j (x, y) is a high-frequency subband, A j (x, y) is a low-frequency subband, and j is different scales.

[0039] Step S25: performing enhancement processing on each high-frequency subband:

[0040]

[0041] and then synthesizing the enhanced high-frequency subband and the low-frequency subband to generate an enhanced image:

[0042]

[0043] Preferably, the method for training the multi-channel image fusion framework is:

[0044] Step S40: given a preset number of registered aligned pictures, and importing the pictures into the multi-channel image fusion framework;

[0045] Step S41: fusing the pictures by the multi-channel image fusion framework to generate a fused image;

[0046] Step S42: generating a mask of a pre-background for the EBCMOS image, and respectively superimposing the infrared image and the visible light black and white image;

[0047] Step S43: generating a mask of a post-background for the infrared image and the visible light black and white image, respectively superimposing the EBCMOS image, and then setting positive and negative samples, evaluating the similarity of the features, and setting the score of the similarity of the positive and negative samples of each channel image as the loss function of the channel image:

[0048]

[0049] wherein, N represents the number of layers of the VGG-19 network, M represents the number of negative samples set in each kind of channel image, V, E, I represent the visible light black and white image, the EBCMOS image, the infrared image respectively, w i represents the weight under the i-th kind of channel image, δ j represents the foreground and background features of the fusion image, represents the positive sample of the source image, represents the m-th negative sample of the source image

[0050] Step S44: learning minimizes the structural similarity of the source image and the fusion image, and the structural similarity is constrained by SSIM, and the loss function is defined as:

[0051] L SSIM = w a S(I V , I F ) + w b S(I R , I F )

[0052] wherein, S represents the calculation process of SSIM, w a , w b , respectively, represent the fusion proportion coefficients of the source image;

[0053] Step S45: learning minimizes the intensity similarity of the source image and the fusion image, and the intensity similarity is constrained by MSE, and the loss function is defined as:

[0054] L N = w c ·||I V -I F ||+ w d ·||I R -I F ||

[0055] wherein, w c , w d respectively represent the fusion proportion coefficients of the source image.

[0056] Preferably, the method for fusing the synthetic learning stage and the reverse optimization stage to form the multi-channel image fusion framework is:

[0057] Step S400: through the joint learning of the synthetic learning stage and the reverse optimization stage, minimizing the joint loss in the form of weighted sum, optimizing the multi-channel image fusion performance of the model;

[0058] Step S401: fusing the synthetic learning stage and the reverse optimization stage to form the multi-channel image fusion framework.

[0059] The application also provides a multi-channel image fusion system based on multi-sensor image enhancement optimization, comprising:

[0060] A data processing module is configured to acquire a low-light image data set comprising an EBCMOS image, a black-and-white visible light image and a near-infrared image, and then perform data preprocessing according to the low-light image data set.

[0061] An optimization enhancement module is configured to perform image optimization enhancement according to the low-light image data set after data preprocessing, including histogram equalization, contrast enhancement and multi-scale enhancement.

[0062] A multi-level attention module is configured to extract luminosity feature information according to the low-light image data set, including constructing a multi-level attention model for extracting illumination perception information, constructing a luminosity probability of an output image of a luminosity information extraction network, constructing an auxiliary module for extracting common important features of the same target, and then using a convolution dimension reduction method to retain the optimal features selected.

[0063] A fusion framework module is configured to train a multi-channel image fusion framework according to the pictures contained in the low-light image data set, including a synthetic learning stage and a reverse optimization stage, and then fusing the synthetic learning stage and the reverse optimization stage to form the multi-channel image fusion framework.

[0064] An image fusion module is configured to input the pictures after data preprocessing into the multi-channel image fusion framework, and output the fused images.

[0065] The application also provides a computer medium, wherein a computer program is stored on the computer medium, and the computer program is executed by a processor to realize the multi-channel image fusion method based on multi-sensor image enhancement optimization.

[0066] The application also provides a computer comprising the computer medium.

[0067] The above technical solution of the application has the following advantages compared with the prior art.

[0068] 1. The combination of image enhancement optimization of multi-sensor images and image fusion methods can improve image details and contrast, enhance image recognition, and fully capture complex features of images to obtain better enhancement effects.

[0069] 2. The multi-level attention model for extracting illumination perception information is introduced, which effectively retains luminosity information and important features of each channel image in the training process, exhibits the superiority of light perception utilization of target images, and reduces the phenomenon of missing effective information and redundant invalid information of the network in the learning process.

[0070] 3. A multi-channel image fusion method based on coupled learning is proposed, which fully retains the high-quality information of each channel image, excludes redundant information, and improves the fusion performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, a brief introduction will be given below to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0072] Figure 1 A flow chart of a multi-channel image fusion method is shown.

[0073] Figure 2 A schematic diagram of module connection of a multi-channel image fusion system is shown.

[0074] Figure 3 A flow chart of a data preprocessing method is shown.

[0075] Figure 4 A flow chart of a semi-manual feature matching multi-channel image registration method is shown.

[0076] Figure 5 A flow chart of an image optimization enhancement method of histogram equalization and contrast enhancement is shown.

[0077] Figure 6 A flow chart of an image optimization enhancement method of multi-scale enhancement is shown.

[0078] Figure 7 A flow chart of a photometric feature information extraction method is shown.

[0079] Figure 8 A flow chart of a multi-channel image fusion framework training method is shown.

[0080] Figure 9 A reference schematic diagram of a synthetic learning stage flow is shown.

[0081] Figure 10 A flow chart of a synthetic learning stage and reverse optimization stage fusion method is shown.

[0082] Explanation of reference signs:

[0083] 1, data processing module, 2, data processing module, 3, multi-level attention module, 4, fusion framework module, 5, image fusion module. DETAILED DESCRIPTION

[0084] Embodiments of the present application are described below in the detailed description and illustrated in the accompanying drawings by which like or similar elements, symbols and / or features are referred to by like or similar labels. Embodiments described below are exemplary in nature and are intended to be illustrative of the present application rather than to be limiting of the present application.

[0085] Referring to Figure 1 , Figures 3-10 , for a camera with multiple channels: visible light black and white channel, EBCMOS channel, infrared channel, and N sample quantity of real-time low-light image data set D = {(V i ,E i ,I i ), i = 1, 2, …, N}, the specific implementation method is as follows:

[0086] Step S1: According to N sample quantity of real-time low-light image data set containing EBCMOS image, visible light black and white image, near-infrared image, data preprocessing is performed.

[0087] As shown in reference Figure 3 , the preprocessing process is as follows:

[0088] Since the picture sizes taken by different channel cameras are different and have not been registered and aligned, therefore:

[0089] Step S10: A semi-manual feature matching based method is used for multi-channel image registration, and a registered picture with the same size as the calibrated camera is outputted;

[0090] Step S11: The common area of the registered picture is cropped to only keep the target object.

[0091] Step S12: LPDM algorithm is used for image enhancement for the image with extremely low brightness.

[0092] As shown in reference Figure 4 , the process of semi-manual feature matching multi-channel image registration is as follows:

[0093] Step S100: We manually select 4 groups of corresponding points with obvious features and prominent targets between a group of different channel images through manual control point selection, and calculate the homography matrix through least square method in MATLAB to find the transformation relationship.

[0094] Wherein, the method of manual control point selection is used to select corresponding points with obvious features and prominent targets between a group of different channel images, represented as (x i ,y i ) and (x' i ,y' i ), wherein i represents the number of control points;

[0095] The homography matrix H is calculated using the least squares method, and the coordinates of the corresponding points are expressed as homogeneous coordinates, where x and x' represent the coordinates of the input image and the output image:

[0096]

[0097] The normalized homogeneous coordinates are:

[0098] Step S101: For any set of images in the scene, use the homography matrix obtained in step S100 to achieve automatic image registration using an encoding method.

[0099] Where the automatic image registration is achieved based on the encoding method: for any set of images, the images I and I' are represented as m x n and m' x n' matrices respectively; assume I is the input image and I' is the output image:

[0100]

[0101]

[0102] Where k represents the number of control points selected;

[0103] The image I' is subjected to an affine transformation, and pixel mapping is performed according to the homography matrix H and the homogeneous coordinates of the image, to obtain the registered image I' registered :

[0104]

[0105] Where h ij represents the elements of the homography matrix H;

[0106] For example: suppose we have two images I and I' of the same size, and their corresponding control points (x i ,y i ) and (x' i ,y' i ); we manually select two control points (10,10) and (100,100), and calculate the homography matrix H using the least squares method:

[0107]

[0108] The images I and I' are represented as matrices:

[0109] I = [I(10,10), I(100,100)]

[0110] I' = [I'(10,10), I'(100,100)]

[0111] affine transformation of the image I' using the homography H, resulting in the registered image I registered :

[0112]

[0113] Finally, compare it with the original image I to observe whether the image is properly registered.

[0114] Step S2: Perform image optimization enhancement on the data-preprocessed glint image dataset, including histogram equalization, contrast enhancement, and multi-scale enhancement.

[0115] Reference Figure 5 histogram equalization is a technique used to enhance the contrast of an image by redistributing the pixel values to stretch the gray scale range, which can make the pixel values uniformly distributed across the entire gray scale range, thus enhancing the details of the image; the process of histogram equalization image optimization enhancement is as follows:

[0116] Given an input data-preprocessed image I(x, y) with a gray scale range of [0, L-1], where (x, y) is a pixel point and L is the number of gray levels (usually 256);

[0117] First, calculate the histogram H(i) of the input image, where i ∈ [0, L-1], representing the number of pixels at gray level i;

[0118] Then, calculate the cumulative distribution function (CDF):

[0119]

[0120] For each pixel point (x, y), map its gray value to a new gray value I eq (x, y):

[0121]

[0122] where M and N are the width and height of the image, and CDF min is the cumulative distribution function value of the first non-zero element.

[0123] where contrast enhancement is another commonly used image enhancement technique that can enhance the contrast of an image by increasing the difference between pixel values; the process of histogram equalization image optimization enhancement is as follows:

[0124] Given an input image I(x, y), assume the minimum pixel value is I min , and the maximum pixel value is I max , for each pixel point, map its gray value to a new gray value:

[0125]

[0126] where L is the number of gray levels (usually 256).

[0127] When histogram equalization and contrast enhancement are combined, the contrast and details of the image can be enhanced while preserving the overall brightness distribution of the image. The process of combining histogram equalization and contrast enhancement is as follows:

[0128] For each pixel point (x, y), the gray value after histogram equalization and contrast enhancement is weighted and combined to obtain the final enhanced gray value I final (x, y):

[0129] I final (x, y) = a × I enhanced (x, y) + (1-a) × I eq (x, y)

[0130] where a is a weight parameter used to control the relative contribution of histogram equalization and contrast enhancement, and in general, a takes a value between 0 and 1.

[0131] where image multiscale enhancement is a method of enhancing images at different scales to improve image details and contrast. This method can capture more detailed information in the image, which is particularly useful for low-light image fusion. Referring to Figure 6 the image optimization enhancement process of multiscale enhancement is as follows:

[0132] Using wavelet transform or other multiscale decomposition techniques, the input image is decomposed to obtain a series of image subbands at different scales. Wavelet transform is a time-frequency analysis method that can decompose an image into low-frequency and high-frequency parts. Each high-frequency subband corresponds to image details at different scales. Specifically:

[0133] Wavelet transform decomposes the image I(x, y) into at least one high-frequency subband B j (x, y) and a low-frequency subband A j (x, y):

[0134] I(x, y) = ∑ j B j (x, y) + A j (x, y)

[0135] j is the different scale.

[0136] For each scale of image subband, an enhancement method is used to enhance the details and contrast of the image. The specific process is as follows:

[0137]

[0138] The enhanced image subbands are inversely transformed, and the image subbands of all scales are combined into an enhanced image; the specific process is as follows:

[0139]

[0140] The image multi-scale enhancement method can enhance different scale details layer by layer, which helps to improve the observation effect and detail resolution of the image, thereby positively affecting the result of the low-light image fusion; in actual application, a suitable wavelet transform type and enhancement method can be selected according to specific needs to obtain the best enhancement effect.

[0141] Step S3: According to the low-light image data set optimized and enhanced by the image, the luminosity feature information is extracted, including: constructing a multi-level attention model for light perception information extraction, and then constructing a luminosity probability of the output image of the luminosity information extraction network, building an auxiliary module for extracting the same target public important features, and then using the method of convolution dimension reduction to retain the selected optimal features;

[0142] In the existing fusion method, the relevant workers only consider the fusion effect of the target object, ignoring the significant effect of the light information in each image on the vision; in the deep learning process, with the increase of the network layers, the effective information in the image will be lost, especially the light information, therefore, a multi-level attention model for light perception information extraction is constructed, which can retain the effective luminosity features of each path image in the network globally; the essence of the luminosity feature is to select the basic image for fusion, in the daytime scene, the light source comes from the visible light image; in the night scene, the light source is saved in the infrared image. Compared with the existing method which uses the visible light image as the basic image for fusion, the loss of light information cannot be eliminated, while the light perception can adaptively retain the optimal luminosity information, providing strong support for the fusion process.

[0143] Reference Figure 7 As shown in the figure, the luminosity feature information extraction process is as follows:

[0144] Step S30: Constructing a multi-level attention model for light perception information extraction, and then training the multi-level attention model through the low-light image data set, so that the multi-level attention model can benefit from the knowledge transfer of image fusion;

[0145] Step S31: a photometric information extraction network is constructed by using a method of building a ResNet-50 network structure, and the photometric probability corresponding to the image is output, so as to determine the current fusion basis image of the to-be-fused image and the fusion process image by using the trained multi-level attention model;

[0146] Step S32: according to the photometric information extraction network, an auxiliary module for extracting common important features of the same target in each channel image is constructed, and then the method of convolution dimension reduction is used to retain the screened optimal features, thereby facilitating the subsequent fusion.

[0147] Step S4: according to the pictures contained in the low-light image dataset, a multi-channel image fusion framework is trained, including a synthesis learning stage and a reverse optimization stage, and then the synthesis learning stage and the reverse optimization stage are fused to form the multi-channel image fusion framework.

[0148] Wherein, before inputting the neural network for training, for each group of pictures, a target mask is generated to facilitate coupled learning, the multi-channel image fusion framework adopts a VGG-19 network, and is divided into two stages: a synthesis learning stage and a reverse optimization stage; in addition, in order to reduce the inconsistency between the input and the output, the picture is subjected to ReLU activation and up-sampling operation, and is taken as the final output of the network.

[0149] The first stage: as shown in Figures 8-10 The general process of the synthesis learning stage is as follows:

[0150] First, given two registered and aligned pictures, the pictures are sent into the network;

[0151] Then, the multi-level attention block proposed in the second step generates a fusion image.

[0152] The second stage: the reverse optimization stage is divided into two parts: a coupled learning part and a self-adaptive weight learning module;

[0153] The coupled learning part constrains the fusion image by learning the foreground and background features in the source image, and the self-adaptive weight learning part changes the fixed weight distribution mode of the existing fusion method, so that the network finds the most suitable fusion weight in the training process.

[0154] First, the coupled learning part is mainly used for learning specific foreground and background information of the source image. For the EBCMOS image, the target outline of the obvious foreground is the part that interests us, and the information of the remaining part is redundant; for the visible light black and white image and the infrared image, the vivid texture details of their background are the parts that we tend to retain, and the dark foreground part is not desirable.

[0155] Specifically, we generate the mask of the pre-context for the EBCMOS image, and superimpose it with the infrared image and the visible black and white image respectively; generate the mask of the post-context for the infrared image and the visible black and white image, and superimpose it with the EBCMOS image respectively, set positive and negative samples, evaluate the similarity of the features, and set the score of the similarity of the positive and negative samples of each channel image as the loss function of the channel image:

[0156]

[0157] wherein N represents the number of layers of the VGG-19 network, M represents the number of negative samples set in each channel image, V, E and I represent the visible black and white image, the EBCMOS image and the infrared image respectively, w i represents the weight under the i-th channel image, δ j represents the pre / post-context feature of the fusion image, represents the positive sample of the source image, represents the m-th negative sample of the source image.

[0158] Secondly, the target of the adaptive weight learning part is to combine the advantageous features of the source images to provide the optimal fusion weight, so as to obtain an image rich in information, sufficient in details and prominent in targets. For the fusion task among multi-channel images, the structural similarity and intensity similarity between the source images and the fusion image are minimized, the structural similarity is constrained by SSIM, and the intensity similarity is constrained by MSE, and the loss functions are defined as follows:

[0159] L SSIM =w a S(I V ,I F )+w b S(I R ,I F )

[0160] L N =w c ·||I V -I F ||+w d ·||I R -I F ||

[0161] wherein S represents the calculation process of SSIM, w a , w b , w c , w d respectively represent the fusion proportion coefficients of the source images, which are often set as fixed values in the existing fusion methods, but the fixed values cannot fully utilize the characteristics of the source images, which will cause the loss of details, so in the above L SSIM , LN An adaptive weight is set in the loss function of the source image to better balance the part participating in the fusion.

[0162] Finally, the above synthesis learning stage and reverse optimization stage are fused to propose a multi-channel image fusion framework of reverse optimization, the learning goal of the multi-channel image fusion framework is to minimize the joint loss by joint learning of the synthesis learning stage and the reverse optimization stage, and the multi-channel image fusion performance of the model is optimized, wherein the joint loss combines the losses of the learning modules in the form of weighted sum.

[0163] Step S5: inputting the picture after data preprocessing into the multi-channel image fusion framework to output the fused image.

[0164] In the fourth step, end-to-end multi-channel image fusion is performed, and the process is as follows: inputting the preprocessed picture into the trained network, and outputting the fused image.

[0165] One aspect of the present application also provides a multi-channel image fusion system based on multi-sensor image enhancement optimization, which uses the above multi-channel image fusion method based on multi-sensor image enhancement optimization, and refers to Figures 1-10 As shown, it comprises:

[0166] The data processing module is used to acquire a low-light image dataset containing an EBCMOS image, a visible light black and white image and a near-infrared image, and then perform data preprocessing according to the low-light image dataset.

[0167] The optimization enhancement module is used to perform image optimization enhancement according to the low-light image dataset after data preprocessing, including histogram equalization, contrast enhancement and multi-scale enhancement.

[0168] The multi-level attention module is used to extract luminosity feature information according to the low-light image dataset, including constructing a multi-level attention model for extracting illumination perception information, constructing a luminosity probability of an output image of a luminosity information extraction network, constructing an auxiliary module for extracting common important features of the same target, and then using a convolution dimension reduction method to retain the optimal features selected.

[0169] The fusion framework module is used to train a multi-channel image fusion framework according to the pictures contained in the low-light image dataset, including a synthesis learning stage and a reverse optimization stage, and then fusing the synthesis learning stage and the reverse optimization stage to form a multi-channel image fusion framework.

[0170] The image fusion module is used to input the picture after data preprocessing into the multi-channel image fusion framework to output the fused image.

[0171] In one aspect of the present application, a computer medium is provided, wherein a computer program is stored on the computer medium, and the computer program is executed by a processor to implement the multi-channel image fusion method based on multi-sensor image enhancement optimization.

[0172] The present application also provides a computer comprising the computer medium.

[0173] Although the embodiments of the present application have been shown and described above, it should be understood by those skilled in the art that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A multi-channel image fusion method based on multi-sensor image enhancement optimization, characterized in that, The method comprises the following steps: Step S1: Obtain a low-light image data set comprising an EBCMOS image, a visible light black and white image, and a near-infrared image, and then perform data preprocessing on the low-light image data set; Step S2: Perform image optimization enhancement on the low-light image data set after data preprocessing, including histogram equalization, contrast enhancement, and multi-scale enhancement; Step S3: Perform photometric feature information extraction on the low-light image data set after image optimization enhancement, including constructing a multi-level attention model for illumination perception information extraction, then constructing a photometric probability of an output image of the photometric information extraction network, building an auxiliary module for extracting common important features of the same target, and then using a convolution dimension reduction method to retain the selected optimal features; Step S4: Train a multi-channel image fusion framework based on the pictures contained in the low-light image data set, including a synthetic learning stage and a reverse optimization stage, and then fuse the synthetic learning stage and the reverse optimization stage to form the multi-channel image fusion framework; Step S5: Input the pictures after data preprocessing into the multi-channel image fusion framework to output the fused images; The method for training the multi-channel image fusion framework is: Step S40: Given a preset number of registered and aligned pictures, and import the pictures into the multi-channel image fusion framework; Step S41: Fuse the pictures through the multi-channel image fusion framework to generate a fused image; Step S42: Generate a mask for the front background of the EBCMOS image, and superimpose it with the infrared image and the visible light black and white image respectively; Step S43: Generate a mask for the rear background of the infrared image and the visible light black and white image, superimpose it with the EBCMOS image respectively, then set positive and negative samples, evaluate the similarity of the features, and set the similarity score of each channel image positive and negative sample as the loss function of the channel image: Wherein, N represents the number of layers of VGG-19 network, M represents the number of negative samples set in each kind of channel image, V, E, I represent visible light black and white image, EBCMOS image, infrared image respectively, represents the weight under the i-th channel image, represents the foreground and background features of the fusion image, represents the positive sample of the source image, represents the m-th negative sample of the source image Step S44: Learn to minimize the structural similarity between the source image and the fused image, and the structural similarity is constrained by SSIM, and the loss function is defined as: Wherein, S represents the calculation process of SSIM, , , respectively represent the fusion proportion coefficient of the source image; Step S45: Learn to minimize the intensity similarity between the source image and the fused image, and the intensity similarity is constrained by MSE, and the loss function is defined as: wherein, , respectively represent the fusion proportionality coefficients of the source images.

2. A multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 1, characterized in that, The method for data preprocessing is: Step S10: Use a semi-automatic feature matching based method to register the multi-channel images, and output registered pictures with the same size as the calibrated camera; Step S11: Crop the common area of the registered pictures to retain the preset target; Step S12: Use the LPDM method to enhance the images below the preset illumination value.

3. A multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 2, characterized in that, The method for registering the multi-channel images using a semi-automatic feature matching based method is: Step S100: Use a manual control point selection method to combine the images of different channels with the calibrated camera images, manually select a group of corresponding points with obvious features and prominent targets, and then calculate the corresponding homography matrix by least squares method; Step S101: For any group of images, use an encoding method to realize automatic image registration under the same experimental environment using the homography matrix.

4. The multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 1, characterized in that, The image optimization enhancement method of histogram equalization is: Step S20: inputting the image after data preprocessing , generating a histogram of the input image , wherein, , the number of pixels of a gray level ; Step S21: Calculate the cumulative distribution function: For each pixel point, map its gray value to a new gray value: wherein the input image has a gray scale range of , is a pixel point, is the number of gray levels, and are the width and height of the image, is the cumulative distribution function value of the first non-zero element.

5. The multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 1, characterized in that, The image optimization enhancement method of contrast enhancement is: Step S22: For each pixel point, map its gray value to a new gray value: wherein, is the maximum pixel value of the input image, is the minimum pixel value of the input image.

6. A multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 4 or 5, characterized in that, The image optimization enhancement method of histogram equalization combined with contrast enhancement is: Step S23: For each pixel point, the gray value after histogram equalization and contrast enhancement is weighted and combined to obtain the enhanced gray value: wherein, is a weight parameter, has a value in the range 0-1.

7. The multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 1, characterized in that, The image optimization enhancement method of multi-scale enhancement is: Step S24: Use wavelet transform to decompose the input image to obtain at least one high-frequency subband and a low-frequency subband: wherein is a high frequency sub-band, is a low frequency sub-band, is a different scale; Step S25: Enhance each high-frequency subband: Then synthesize the enhanced high-frequency subband and the low-frequency subband to generate an enhanced image: 。 8. The multi-sensor image enhancement based optimized multi-channel image fusion method according to claim 1, characterized in that, The method for fusing the synthesis learning stage and the reverse optimization stage to form a multi-channel image fusion framework is: Step S400: Through joint learning of the synthesis learning stage and the reverse optimization stage, the joint loss is minimized in the form of weighted sum to optimize the multi-channel image fusion performance of the model; Step S401: Fuse the synthesis learning stage and the reverse optimization stage to form a multi-channel image fusion framework.

9. A multi-sensor image enhancement based optimized multi-channel image fusion system using the method of any one of claims 1-8, characterized in that, Comprise: A data processing module for obtaining a low-light image dataset containing EBCMOS images, visible light black and white images, and near-infrared images, and then performing data preprocessing according to the low-light image dataset; An optimization enhancement module for performing image optimization enhancement according to the low-light image dataset after data preprocessing, including histogram equalization, contrast enhancement, and multi-scale enhancement; A multi-level attention module for extracting photometric feature information according to the low-light image dataset, including constructing a multi-level attention model for extracting illumination perception information, then constructing a photometric probability of a photometric information extraction network output image, constructing an auxiliary module for extracting the same target common important features, and then using a convolution dimension reduction method to retain the selected optimal features; A fusion framework module for training a multi-channel image fusion framework according to the pictures contained in the low-light image dataset, including a synthesis learning stage and a reverse optimization stage, and then fusing the synthesis learning stage and the reverse optimization stage to form a multi-channel image fusion framework; An image fusion module for inputting the pictures after data preprocessing into the multi-channel image fusion framework and outputting the fused image.

Images