Image fusion method and device based on feedback illumination self-adaption

Abstract

The invention discloses an image fusion method and device based on feedback illumination self-adaption. The method comprises the following steps: acquiring a visible light image data set and an infrared image data set; inputting the visible light image data set into the illumination adaptive network to obtain an enhanced image data set; inputting the infrared image data set and the enhanced image data set into a fusion network to obtain a fusion image data set; inputting the fusion image data set into a pre-trained feedback network to obtain a plurality of groups of probability arrays, calculating a feedback loss function based on the plurality of groups of probability data, and performing joint training on the illumination adaptive network and the fusion network based on the feedback loss function; and inputting the visible light image to be fused into the illumination adaptive network subjected to joint training to obtain an enhanced image, and inputting the infrared image to be fused and the enhanced image into the fusion network subjected to joint training to obtain a fused image. According to the invention, the output fusion image can reach a rational illumination level, and the quality of the fusion image is improved.

Description

Technical Field

[0001] The present application relates to the field of image enhancement technology, and in particular to an image fusion method, device, storage medium and electronic device based on feedback illumination adaptation. Background Art

[0002] Infrared and visible light image fusion can effectively combine the advantages of the two source images, retaining important target information and rich texture details. In recent years, significant progress has been made in the field of infrared and visible light image fusion, and the main methods are divided into traditional methods and deep learning-based methods.

[0003] Traditional methods rely on multi-scale transformation, sparse representation, subspace clustering, optimization algorithms and hybrid methods. Although existing image fusion methods have achieved certain results, there are still several problems. For example, due to the limitation of ambient light, visible light images often show low brightness, insufficient contrast, lack of texture details and color distortion. Most of them are only for scenes with good lighting. It is difficult to maintain high quality of image fusion at night or in low light environments. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and electronic device for image fusion based on feedback illumination adaptation, which can obtain a high-quality fused image.

[0005] The embodiment of the present application provides an image fusion method based on feedback illumination adaptation, including: Obtain a visible light image dataset and an infrared image dataset; Inputting the visible light image dataset into an illumination adaptive network to obtain an enhanced image dataset; Inputting the infrared image dataset and the enhanced image dataset into a fusion network to obtain a fused image dataset; Inputting the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculating a feedback loss function based on the multiple groups of probability data, and jointly training the illumination adaptive network and the fusion network based on the feedback loss function; Obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into the jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into the jointly trained fusion network to obtain a fused image.

[0006] Furthermore, in the above-mentioned feedback-based illumination adaptation image fusion method, the fusion network includes a multi-scale convolution module, a convolution layer, a global average pooling layer, a fully connected layer, and an output layer; The fused image data set is input into a pre-trained feedback network to obtain multiple probability arrays, including: Inputting the fused image data set into a multi-scale convolution module for feature extraction and splicing to obtain a first feature map; Perform feature extraction on the first feature map through multiple convolutional layers in sequence to obtain a second feature map; Inputting the second feature map into the global average pooling layer for dimensionality reduction; Linearly transforming the second feature map after dimension reduction through a plurality of the fully connected layers; The second feature map after linear transformation is input into the output layer, and the value of the output node is converted into the probability array through the activation function, and the probability array includes the probability that the fused image is in an overexposed state, the probability that it is in a normal exposure state, and the probability that it is in an underexposed state.

[0007] Furthermore, in the above-mentioned image fusion method based on feedback illumination adaptation, the step of calculating the feedback loss function based on the multiple groups of probability data includes:

[0008] in, The probability of being in an overexposed state, is the probability of normal exposure, is the probability of underexposure, r is the number of iterations during training, is a minimum value that prevents the base of the logarithmic function from being 0. H and W are the height and width of the fused image, respectively.

[0009] Furthermore, in the above-mentioned image fusion method based on feedback illumination adaptation, the step of jointly training the illumination adaptation network and the fusion network based on the feedback loss function comprises: A first loss function is calculated based on the feedback loss function, and the first loss function is used to train the illumination adaptation network. The first loss function is calculated as follows:

[0010] in, , and is a hyperparameter, is the first loss function, is the spatial consistency loss function, is the exposure control loss function, Color invariance loss function, Smooth loss function, is the feedback loss function.

[0011] Furthermore, in the above-mentioned image fusion method based on feedback illumination adaptation, the calculation method of the spatial consistency loss function is:

[0012] Where M is the number of local regions, and the size of the local region is set to , express and The adjacent areas of are the upper, lower, left and right areas. I and E represent the average intensity values ​​of the local areas in the infrared image and enhanced image, respectively. The exposure control loss function is calculated as follows:

[0013] Where N is the number of local regions, and the size of the local region is set to , E represents the average intensity value of the local area in the enhanced image, is the preset good exposure level, the value range is ; The color constancy loss function is calculated as:

[0014] in, , and Represent the mean of red, green and blue channels respectively; The smooth loss function is calculated as follows:

[0015] in, are the horizontal and vertical coordinates of the pixel respectively.

[0016] Furthermore, in the above-mentioned image fusion method based on feedback illumination adaptation, the step of jointly training the illumination adaptation network and the fusion network based on the feedback loss function comprises: A second loss function is calculated based on the feedback loss function, and the second loss function is used to train the fusion network. The second loss function is calculated as follows:

[0017] in, , and is a hyperparameter, is the second loss function, is the gradient loss function, is the structural similarity loss function, is the color invariance loss function, is the feedback loss function.

[0018] Furthermore, in the above-mentioned image fusion method based on feedback illumination adaptation, the gradient loss function is calculated as follows:

[0019] Among them, H and W are the height and width of the infrared image respectively, Represents the Sobel gradient operator, which is used to calculate the spatial derivative of image intensity. , and They represent the fused image, infrared image and enhanced image after the illumination adaptive network respectively; The calculation method of the structural similarity loss function is:

[0020] in, Represents the structural similarity index measure.

[0021] The embodiment of the present application also provides an image fusion device based on feedback illumination adaptation, including: A training module is used to obtain a visible light image data set and an infrared image data set; input the visible light image data set into a light adaptive network to obtain an enhanced image data set; input the infrared image data set and the enhanced image data set into a fusion network to obtain a fused image data set; and input the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculate a feedback loss function based on the multiple groups of probability data, and jointly train the light adaptive network and the fusion network based on the feedback loss function; The application module is used to obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into the jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into the jointly trained fusion network to obtain a fused image.

[0022] An embodiment of the present application further provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned image fusion methods based on feedback illumination adaptation.

[0023] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any of the above-mentioned image fusion methods based on feedback illumination adaptation.

[0024] The image fusion method, device, storage medium and electronic device based on feedback illumination adaptation provided by the present application set a feedback network, calculate the feedback loss function through the probability array predicted by the feedback network, and jointly train the illumination adaptation network and the fusion network based on the feedback loss function and in combination with other functions to obtain the final fused image. The present application uses the feedback network to provide feedback and train the illumination adaptation network and the fusion network, thereby improving the performance of the model, so that the output fused image can reach a rational illumination level, and improves the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0026] Figure 1 A flowchart of an image fusion method based on feedback illumination adaptation provided in an embodiment of the present application.

[0027] Figure 2 Another flowchart of the image fusion method based on feedback illumination adaptation provided in an embodiment of the present application.

[0028] Figure 3 A schematic diagram of a process for generating an enhanced image provided in an embodiment of the present application.

[0029] Figure 4 A flowchart for generating a fused image provided in an embodiment of the present application.

[0030] Figure 5 A schematic diagram of the structure of the residual block provided in an embodiment of the present application.

[0031] Figure 6 A schematic diagram of the structure of a multi-scale spatial / channel attention fusion module provided in an embodiment of the present application.

[0032] Figure 7 A flowchart of generating a probability array provided in an embodiment of the present application.

[0033] Figure 8 A schematic diagram comparing the results of image fusion testing using the 12 methods provided in the embodiments of the present application.

[0034] Fig. 9 Another schematic diagram comparing the results of image fusion testing using the 12 methods provided in the embodiments of the present application.

[0035] Fig.10 A schematic diagram of the structure of an image fusion device based on feedback illumination adaptation provided in an embodiment of the present application.

[0036] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0037] Fig.12 Another schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0039] At present, traditional methods rely on multi-scale transformation, sparse representation, subspace clustering, optimization algorithms and hybrid methods. Although existing image fusion methods have achieved certain results, there are still several problems: (1) Ambient lighting limitations: Visible light images often show low brightness, insufficient contrast, and missing texture details. Most of them are only for scenes with good lighting, and it is difficult to maintain high quality of image fusion at night or in low-light environments. (2) Pixel intensity constraints: Existing night image fusion methods generally use pixel intensity constraints as the core mechanism, resulting in insufficient texture clarity and color performance of the fused image. Pixel intensity constraints emphasize brightness consistency, resulting in smoothing of texture areas, that is, texture information with high-frequency changes is low-pass filtered during the fusion process, reducing the local contrast and edge clarity of the image. (3) Existing enhancement-based image fusion methods only focus on the fine design of enhancement networks and fusion networks. Although these methods can enhance the visibility of images under certain conditions, when dealing with low-light or high-contrast scenes, they may cause skewed brightness distribution in the fused image, resulting in underexposure or overexposure, which is not conducive to obtaining an ideal light level in the fused image. In order to solve these problems, the present invention designs a feedback illumination adaptive infrared and visible light image fusion method.

[0040] To solve the above problems, the embodiments of the present application provide an image fusion method, device, storage medium and electronic device based on feedback illumination adaptation. The image fusion device based on feedback illumination adaptation provided in the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0041] See also Figure 1 and Figure 2 , Figure 1 This is a flowchart of the image fusion method based on feedback illumination adaptation provided in an embodiment of the present application. Figure 1 Another flow chart of the image fusion method based on feedback illumination adaptation provided in an embodiment of the present application, which is applied in an electronic device, the image fusion method based on feedback illumination adaptation includes the following steps: S1, obtain a visible light image dataset and an infrared image dataset.

[0042] Among them, the visible light image dataset includes several visible light images, and the several visible light images are annotated with corresponding real labels. The infrared image dataset includes several infrared images, and the several infrared images are annotated with corresponding real labels. The visible light image dataset and the infrared image dataset are used to train the illumination adaptive network, fusion network and feedback network. The LLVIP dataset and the MSRS dataset can be used.

[0043] S2, input the visible light image dataset into the illumination adaptation network to obtain an enhanced image dataset.

[0044] Figure 3 A schematic diagram of a process for generating an enhanced image provided in an embodiment of the present application is shown in FIG. Figure 3 As shown in Figure 1, the illumination adaptation network includes multiple convolutional layers (Conv) and multiple linear activation layers (ReLU). First, the illumination adaptation network first receives the visible light image. , and then the visible light image is transformed through a convolutional layer and ReLU activation function The processed result is then passed through three identical convolutional layers and activation functions. The illumination adaptive network uses residual connections, that is, after the 2nd, 3rd, and 4th convolutional layers, the feature map of the previous layer is directly passed to the subsequent layers. These residual connections help alleviate the problem of vanishing gradients in deep networks, enhance information flow, and retain more original image information. Finally, the enhanced image is obtained through a 3×3 convolutional layer and a Tanh activation function, whose value range is . The stride and padding size of each convolution kernel are set to 1.

[0045] The embodiment of the present application adopts the illumination enhancement curve as a basic method to adaptively adjust the brightness of the visible light image, thereby effectively restoring the brightness, texture and color of the visible light image.

[0046] S3, input the infrared image dataset and the enhanced image dataset into the fusion network to obtain a fused image dataset.

[0047] Figure 4 A flowchart for generating a fused image provided in an embodiment of the present application, Figure 5 A schematic diagram of the structure of the residual block provided in an embodiment of the present application, Figure 6 A schematic diagram of the structure of the multi-scale spatial / channel attention fusion module provided in the embodiment of the present application is shown in FIG. Figure 4-6 As shown in Figure 1, the fusion network consists of three parts: encoder, multi-scale spatial / channel attention fusion module and decoder. The encoder consists of multiple Convolutional layers, batch normalization, and Leaky ReLU activation functions. The encoder also includes two densely connected Sobel / Laplacian residual blocks. The infrared image and enhanced image are processed separately by the encoder to obtain their corresponding feature maps. The multi-scale spatial / channel attention fusion module performs multi-scale convolution on 128 channels, extracts features of different scales in the feature map, and integrates these features using spatial and channel attention mechanisms to obtain fused features. The decoder converts the fused features back to image format through multiple convolutional layers to generate the final fused image. . Among them, the step size of each convolution kernel is set to 1.

[0048] S4, inputting the fused image data set into the pre-trained feedback network to obtain multiple groups of probability arrays, calculating the feedback loss function based on the multiple groups of probability data, and jointly training the illumination adaptive network and the fusion network based on the feedback loss function.

[0049] Figure 7 A flowchart of generating a probability array provided in an embodiment of the present application, such as Figure 7 As shown, the fusion network includes a multi-scale convolution module, a convolution layer, a global average pooling layer, a fully connected layer, and an output layer. Step S4 includes the following steps: S41, pre-training the feedback network.

[0050] Specifically, the cross entropy loss function is used Training the feedback network: In order to obtain accurate prediction results, it is very important that the fusion network correctly outputs three probabilities (the probability of overexposure, the probability of normal exposure, and the probability of underexposure). This application introduces the cross entropy loss , defined as follows:

[0051] Where N is the number of categories, which is set to 3 here. is the one-hot encoding of the true label, is the probability that the feedback network predicts that the sample belongs to category i.

[0052] S42, input the fused image data set into a multi-scale convolution module for feature extraction and splicing to obtain a first feature map.

[0053] S43, extracting features from the first feature map through multiple convolutional layers in sequence to obtain a second feature map.

[0054] S44, input the second feature map into the global average pooling layer for dimensionality reduction.

[0055] S45, linearly transform the second feature map after dimensionality reduction through multiple fully connected layers.

[0056] S46, inputting the second feature map after linear transformation into the output layer, converting the value of the output node into a probability array through an activation function, the probability array including the probability that the fused image is in an overexposed state, the probability that it is in a normal exposure state, and the probability that it is in an underexposed state.

[0057] S47, calculate the feedback loss function based on multiple sets of probability data .

[0058] Specifically, it is calculated by the following formula:

[0059] in, The probability of being in an overexposed state, is the probability of normal exposure, is the probability of underexposure, r is the number of iterations during training, is a minimum value, which prevents the cardinality of the logarithmic function from being 0, and is set to 1×10 -6 , H and W are the height and width of the fused image respectively.

[0060] S48, calculating a first loss function based on the feedback loss function, the first loss function is used to train the illumination adaptive network, and the calculation method of the first loss function is:

[0061] in, , and is a hyperparameter, where , , , used to balance different losses, is the first loss function, is the spatial consistency loss function, is the exposure control loss function, Color invariance loss function, Smooth loss function, is the feedback loss function.

[0062] Specifically, in order to quantify and minimize the inconsistency of the predicted values ​​between adjacent pixels in the enhanced image and ensure that the semantic and structural features of the enhanced image in the local area are highly consistent with the original image, this application introduces a spatial consistency loss function. The calculation method of the spatial consistency loss function is:

[0063] Where M is the number of local regions, and the size of the local region is set to , express and The adjacent areas of are the upper, lower, left and right areas. I and E represent the average intensity values ​​of the local areas in the infrared image and enhanced image, respectively.

[0064] In order to control the exposure level of the image to suppress underexposed or overexposed areas and obtain a more balanced image brightness, this application introduces an exposure control loss function. The exposure control loss function is calculated as follows:

[0065] Where N is the number of local regions, and the size of the local region is set to , E represents the average intensity value of the local area in the enhanced image, is the preset good exposure level, the value range is , here it is set to 0.6.

[0066] In order to maintain the naturalness and consistency of color during the enhancement process, especially under different lighting conditions, the present invention introduces a color constant loss function. The calculation method of the color constant loss function is:

[0067] in, , and Represent the mean of red, green and blue channels respectively; In order to make the changes of adjacent pixels of the enhanced image smoother and reduce the noise and discontinuity of the image, the present invention introduces a smooth loss function. The calculation method of the smooth loss function is:

[0068] in, are the horizontal and vertical coordinates of the pixel respectively.

[0069] S49, calculating a second loss function based on the feedback loss function, the second loss function is used to train the fusion network, and the calculation method of the second loss function is:

[0070] in, , and is a hyperparameter, where , , , used to balance different losses, is the second loss function, is the gradient loss function, is the structural similarity loss function, is the color invariance loss function, is the feedback loss function.

[0071] Specifically, in order to retain the key edge information and texture details in the source image (infrared image and enhanced image), that is, to focus on the high-frequency part of the image, this application introduces a gradient loss function. The calculation method of the gradient loss function is:

[0072] Among them, H and W are the height and width of the infrared image respectively, Represents the Sobel gradient operator, which is used to calculate the spatial derivative of image intensity. , and They represent the fused image, infrared image and enhanced image after the illumination adaptation network respectively.

[0073] In order to ensure that the fused image is structurally consistent with the reference image, this application introduces a structural similarity loss function. The calculation method of the structural similarity loss function is:

[0074] in, Represents the structural similarity index measure.

[0075] The training data of the above illumination adaptive network, fusion network and feedback network are shown in Table 1 below: Table 1 Training data of illumination adaptive network, fusion network and feedback network

[0076] S5, obtaining a visible light image to be fused and an infrared image to be fused, inputting the visible light image to be fused into a jointly trained illumination adaptive network to obtain an enhanced image, and inputting the infrared image to be fused and the enhanced image into a jointly trained fusion network to obtain a fused image.

[0077] In order to verify the superior performance of the present invention in the field of image fusion, this application selected 50 pairs and 361 pairs of images on the LLVIP and MSRS datasets for fusion testing respectively. This application selected 9 advanced comparison methods, of which 2 methods are based on autoencoders, including RFN-Nest and SDCFusion. Among them, 6 methods are based on convolutional neural networks, including SDNet, SwinFusion, UMF-CMGR, MURF, LRRNet and PTET. The last method is based on a generative adversarial network, namely TarDAL. In addition, the present invention uses 5 indicators to objectively evaluate the fusion performance, namely information entropy, spatial frequency, standard deviation, visual information fidelity and average gradient.

[0078] The overall framework of the image fusion method based on feedback illumination adaptation provided in the embodiment of the present application includes three networks: an illumination adaptation network, a fusion network and a feedback network. First, the present invention adopts the illumination enhancement curve as the basic method to adaptively adjust the brightness of the visible light image, and effectively restores the brightness, texture and color of the visible light image. Secondly, the fusion network uses the Sobel / Laplacian residual block as the feature extraction block of the infrared and visible light images, and combines the multi-scale spatial / channel attention mechanism to ensure that the fused image contains rich semantic and spatial information. Then, the feedback network evaluates the illumination level of the fused image by calculating the probabilities of underexposure, normal exposure and overexposure. Finally, these probabilities are fed back to the illumination adaptation network and the fusion network in the form of a loss function, and guide them to perform joint training to ensure that the illumination level of the fused image reaches the optimal state. The present application has the following advantages: (1) The present application calculates the feedback loss function through the prediction result of the fusion network, and jointly trains the illumination adaptation network and the fusion network based on the feedback loss function, which effectively realizes the mutual promotion between the illumination adaptation network and the fusion network, and ensures that the fused image can reach the ideal illumination level. (2) In the joint training, different loss functions are set for training based on the requirements of the illumination adaptation network and the fusion network. For example, based on the requirement that the semantic and structural features need to be consistent with the original image, a structural consistency loss function is set, and based on the requirement to obtain a more balanced image brightness, an exposure control loss function is set. These loss functions improve the performance of the model, can adapt to various scenarios, and can improve the quality of the fused image. (2) The present application designs two novel modules in the fusion network. The Sobel / Laplacian residual block effectively fuses the deep features and strong and weak texture details of the source image, and the multi-scale spatial / channel attention module enables the fused image to contain rich semantic and spatial information. (3) The present application conducts pedestrian detection experiments on the fused image, and the superior detection performance verifies that the present invention can effectively promote advanced computer vision tasks.

[0079] Figure 8 The following is a schematic diagram showing the comparison of the results of image fusion testing using the 12 methods provided in the embodiments of the present application. Figure 8As shown, 6 groups of typical infrared and visible light image pairs were selected, including scenes such as pedestrians, inscriptions, buildings, and license plates. The thermal targets in the red box of RFN-Nest, SDNet, UMF-CMGR, MURF, and LRRNet are relatively blurred, and the details are not clear enough. SwinFusion has a low overall brightness, and the edge details are not obvious enough. TarDAL has a low overall contrast and insufficient sense of detail. Although PTET effectively highlights the thermal infrared targets, the color information is not realistic enough and the details are not well represented. There is a certain loss of texture details in the green box of SDCFusion, and the details are generally represented. In contrast, the image fusion method provided in this application can effectively highlight infrared thermal targets while retaining and enhancing rich texture details, which is more in line with human visual perception.

[0080] The quantitative results of the present invention on the LLVIP and MSRS datasets are shown in Tables 2 and 3. The present invention maintains the first place in all indicators. It is worth noting that the present invention is much higher than the other 9 advanced comparison methods in terms of spatial frequency, visual information fidelity and average gradient indicators.

[0081] Table 2 Quantitative results on the LLVIP dataset

[0082] Table 3 Quantitative results on the MSRS dataset

[0083] In order to verify that the present invention can promote the application of advanced visual tasks, the present invention adopts the YOLOv7 algorithm and selects 50 pairs of images on the LLVIP dataset for pedestrian detection experiments. Fig. 9 Another comparison diagram of the results of the fusion test of the image using the 12 methods provided in the embodiment of the present application is shown in FIG. Fig. 9 As shown in the figure, two typical scenes are selected. Fig. 9 It can be seen that the image fusion method provided by the present application has the highest detection accuracy and detection confidence. As shown in Table 4, AP@0.60 represents the average precision calculated when the intersection-over-union ratio threshold is 0.60, and the step size in mAP@[0.50:0.95] is 0.05. Obviously, the image fusion method provided by the present invention has the highest average precision.

[0084] Table 4 Quantitative results on the MSRS dataset

[0085] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an image fusion device based on feedback illumination adaptation. The image fusion device based on feedback illumination adaptation can be implemented as an independent entity or integrated in an electronic device. The electronic device can be a terminal, a server or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0086] See also Fig.10 , Fig.10 The image fusion device based on feedback illumination adaptation provided by the embodiment of the present application is specifically described, and is applied to an electronic device. The image fusion device based on feedback illumination adaptation may include: A training module is used to obtain a visible light image data set and an infrared image data set; input the visible light image data set into the illumination adaptive network to obtain an enhanced image data set; input the infrared image data set and the enhanced image data set into the fusion network to obtain a fused image data set; and input the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculate a feedback loss function based on the multiple groups of probability data, and jointly train the illumination adaptive network and the fusion network based on the feedback loss function; The application module is used to obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into a jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into a jointly trained fusion network to obtain a fused image.

[0087] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0088] In addition, the embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Fig.11 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0089] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0090] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions: Obtain a visible light image dataset and an infrared image dataset; Inputting the visible light image dataset into an illumination adaptive network to obtain an enhanced image dataset; Inputting the infrared image dataset and the enhanced image dataset into a fusion network to obtain a fused image dataset; Inputting the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculating a feedback loss function based on the multiple groups of probability data, and jointly training the illumination adaptive network and the fusion network based on the feedback loss function; Obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into the jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into the jointly trained fusion network to obtain a fused image.

[0091] The electronic device can implement the steps in any embodiment of the image fusion method based on feedback illumination adaptation provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any image fusion method based on feedback illumination adaptation provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0092] Fig.12 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown, and the electronic device can be used to implement the image fusion method based on feedback illumination adaptation provided in the above embodiment. The electronic device 500 can be a terminal, a server or other equipment, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC, Personal Computer), a micro processing box, or other equipment.

[0093] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0094] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0095] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0096] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0097] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0098] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0099] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0100] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: Obtain a visible light image dataset and an infrared image dataset; Inputting the visible light image dataset into an illumination adaptive network to obtain an enhanced image dataset; Inputting the infrared image dataset and the enhanced image dataset into a fusion network to obtain a fused image dataset; Inputting the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculating a feedback loss function based on the multiple groups of probability data, and jointly training the illumination adaptive network and the fusion network based on the feedback loss function; Obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into the jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into the jointly trained fusion network to obtain a fused image.

[0101] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0102] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the image fusion method based on feedback illumination adaptation provided by an embodiment of the present invention.

[0103] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0104] Since the instructions stored in the storage medium can execute the steps in any embodiment of the image fusion method based on feedback illumination adaptation provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image fusion method based on feedback illumination adaptation provided in the embodiments of the present invention can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0105] The above is a detailed introduction to the image fusion method, device, storage medium and electronic device based on feedback illumination adaptation provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image fusion method based on feedback illumination adaptation, characterized in that: The method comprises: Obtain a visible light image dataset and an infrared image dataset; Inputting the visible light image dataset into an illumination adaptive network to obtain an enhanced image dataset; Inputting the infrared image dataset and the enhanced image dataset into a fusion network to obtain a fused image dataset; Inputting the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculating a feedback loss function based on the multiple groups of probability data, and jointly training the illumination adaptive network and the fusion network based on the feedback loss function; Obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into the jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into the jointly trained fusion network to obtain a fused image.

2. The image fusion method based on feedback illumination adaptation according to claim 1, characterized in that: The fusion network includes a multi-scale convolution module, a convolution layer, a global average pooling layer, a fully connected layer, and an output layer; The fused image data set is input into a pre-trained feedback network to obtain multiple probability arrays, including: Inputting the fused image data set into a multi-scale convolution module for feature extraction and splicing to obtain a first feature map; Perform feature extraction on the first feature map through multiple convolutional layers in sequence to obtain a second feature map; Inputting the second feature map into the global average pooling layer for dimensionality reduction; Linearly transforming the second feature map after dimension reduction through a plurality of the fully connected layers; The second feature map after linear transformation is input into the output layer, and the value of the output node is converted into the probability array through the activation function, and the probability array includes the probability that the fused image is in an overexposed state, the probability that it is in a normal exposure state, and the probability that it is in an underexposed state.

3. The image fusion method based on feedback illumination adaptation according to claim 2, characterized in that: The calculating of the feedback loss function based on the multiple groups of probability data includes: in, The probability of being in an overexposed state, is the probability of normal exposure, is the probability of underexposure, r is the number of iterations during training, is a minimum value that prevents the base of the logarithmic function from being 0. H and W are the height and width of the fused image, respectively.

4. The image fusion method based on feedback illumination adaptation according to claim 3 is characterized in that: The jointly training the illumination adaptive network and the fusion network based on the feedback loss function includes: A first loss function is calculated based on the feedback loss function, and the first loss function is used to train the illumination adaptation network. The first loss function is calculated as follows: in, , and is a hyperparameter, is the first loss function, is the spatial consistency loss function, is the exposure control loss function, Color invariance loss function, Smooth loss function, is the feedback loss function.

5. The image fusion method based on feedback illumination adaptation according to claim 4, characterized in that: The calculation method of the spatial consistency loss function is: Where M is the number of local regions, and the size of the local region is set to , express and The adjacent areas of are the upper, lower, left and right areas. I and E represent the average intensity values ​​of the local areas in the infrared image and enhanced image, respectively. The exposure control loss function is calculated as follows: Where N is the number of local regions, and the size of the local region is set to , E represents the average intensity value of the local area in the enhanced image, is the preset good exposure level, the value range is ; The color constancy loss function is calculated as: in, , and Represent the mean of red, green and blue channels respectively; The smooth loss function is calculated as follows: in, are the horizontal and vertical coordinates of the pixel respectively.

6. The image fusion method based on feedback illumination adaptation according to claim 3, characterized in that: The jointly training the illumination adaptive network and the fusion network based on the feedback loss function includes: A second loss function is calculated based on the feedback loss function, and the second loss function is used to train the fusion network. The second loss function is calculated as follows: in, , and is a hyperparameter, is the second loss function, is the gradient loss function, is the structural similarity loss function, is the color invariance loss function, is the feedback loss function.

7. The image fusion method based on feedback illumination adaptation according to claim 6, characterized in that: The gradient loss function is calculated as follows: Among them, H and W are the height and width of the infrared image respectively, Represents the Sobel gradient operator, which is used to calculate the spatial derivative of image intensity. , and They represent the fused image, infrared image and enhanced image after the illumination adaptive network respectively; The calculation method of the structural similarity loss function is: in, Represents the structural similarity index measure.

8. An image fusion device based on feedback illumination adaptation, characterized in that: include: A training module, used to obtain a visible light image dataset and an infrared image dataset; Inputting the visible light image dataset into an illumination adaptive network to obtain an enhanced image dataset; Inputting the infrared image data set and the enhanced image data set into a fusion network to obtain a fused image data set; and inputting the fused image data set into a pre-trained feedback network to obtain multiple groups of probability arrays, calculating a feedback loss function based on the multiple groups of probability data, and jointly training the illumination adaptive network and the fusion network based on the feedback loss function; The application module is used to obtain a visible light image to be fused and an infrared image to be fused, input the visible light image to be fused into the jointly trained illumination adaptive network to obtain an enhanced image, and input the infrared image to be fused and the enhanced image into the jointly trained fusion network to obtain a fused image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the image fusion method based on feedback illumination adaptation according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It comprises a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the image fusion method based on feedback illumination adaptation as described in any one of claims 1 to 7.

Images