Image fusion method and device based on interest index, and storage medium

Abstract

This invention belongs to the field of image fusion technology, and particularly relates to an image fusion method based on interest indices, a computer device, and a storage medium. The image fusion method includes: selecting image evaluation indices as interest indices according to the characteristics of the images to be fused and the requirements of the application scenario; constructing a multi-objective optimization problem based on the interest indices and unknown fusion coefficients; solving the multi-objective optimization problem through a BP neural network to obtain optimized fusion coefficients; inputting the optimized fusion coefficients as fusion rules into the fusion method to complete the optimization of the interest indices and achieve image fusion; the entire fusion process only requires manual selection of evaluation indices, avoiding the selection of regions and the calculation of weighting coefficients, greatly reducing labor costs; moreover, it can be dynamically adjusted according to different situations, applicable to the fusion of various multi-band multi-dimensional optical information, and can also be applied to several arbitrary evaluation indices, and optimize them as a whole.

Description

Technical Field

[0001] This invention belongs to the field of image fusion technology, and particularly relates to an image fusion method based on multi-band multi-dimensional optical information of interest index, a computer device for performing the image fusion method, and a non-transient computer-readable storage medium. Background Technology

[0002] With the development of optical technology, traditional two-dimensional detection can no longer meet the increasingly complex detection needs. Multi-band, multi-dimensional optical information detection technology, which can detect more optical information and improve target recognition rate, has become the development trend of optical imaging systems. Faced with the richer optical information detected by various multi-band, multi-dimensional optical information detection systems, the question of how to efficiently process and utilize this optical information has attracted widespread attention from scholars. Image fusion technology for multi-band, multi-dimensional optical information utilizes the complementary advantages of different bands and dimensions of optical information to enhance information extraction capabilities, improve detection accuracy and reliability, and is widely applicable to the fusion of multi-band, multi-dimensional optical information such as polarization images and light field images of various bands.

[0003] Currently, image fusion technologies mainly include neural network-based fusion, region-based fusion, and pixel-level fusion. Neural network-based fusion requires large datasets for initial training, which is extremely costly for multi-band, multi-dimensional optical information detection involving multiple bands and broad information dimensions. Region-based and pixel-level fusion technologies first preprocess the images and then select appropriate fusion rules based on the characteristics of the original images to be fused. Traditional fusion rule selection methods include weighted average, extreme value, region averaging, and local energy methods. In multi-band, multi-dimensional optical information fusion, traditional fusion rule selection methods suffer from difficulties in determining weighting coefficients, high manual costs in finding feature regions, and often only highlight one or two indicators, failing to consider multiple indicators and resulting in low utilization of the detected optical information.

[0004] Currently, methods for selecting fusion rules for pixel-level and region-based image fusion mainly include weighted coefficient methods, extremum methods, and specific region weighting methods. Among these, the weighted coefficient method struggles to determine suitable weighting coefficients to balance multiple indicators when dealing with multi-band, multi-dimensional optical information fusion, especially for images with different bands and dimensions. The extremum method, by selecting the maximum gray value in a pixel or the maximum coefficient in a sub-band, focuses only on edge and detail information, leading to a decrease in the overall visual quality of the fused image, loss of some effective information, and severe susceptibility to noise. The specific region weighting method requires manual selection of regions and searching for appropriate weighting coefficients for each set of images to be fused, which is time-consuming and labor-intensive.

[0005] In 2023, Changchun University of Science and Technology proposed a dual-weighted polarization image fusion method based on quality assessment and attention mechanism. It combines information entropy, no-reference image quality assessment and local energy into a quality assessment unit, which achieves high-contrast fusion of polarization intensity map and polarization degree map. However, this method cannot select several arbitrary evaluation indicators to combine for different application requirements and the characteristics of the images to be fused, so as to achieve overall optimization of all selected evaluation indicators. It is limited to the optimization of a single indicator and cannot be universally applied to the fusion of multiple multi-band and multi-dimensional optical information. Summary of the Invention

[0006] In view of this, the present invention aims to provide an image fusion method based on interest indicators that can select several arbitrary evaluation indicators to combine for different application needs and characteristics of the images to be fused, thereby achieving overall optimization of all selected evaluation indicators, not limited to the optimization of a single indicator, and is universally applicable to the fusion of multiple multi-band and multi-dimensional optical information.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows:

[0008] This invention provides an image fusion method based on interest indices, specifically an image fusion method based on multi-band multi-dimensional optical information of interest indices, wherein the image fusion method based on interest indices is used to fuse several images to be fused.

[0009] The image fusion method based on interest indicators includes the following steps:

[0010] S1. Based on the characteristics of the images to be fused and the requirements of the application scenario, select several image evaluation indicators as interest indicators;

[0011] S2. Construct a multi-objective optimization problem based on the basic fusion method, the aforementioned interest index, and the unknown fusion coefficients;

[0012] S3. Solve the multi-objective optimization problem using a BP neural network to obtain the optimization fusion coefficients;

[0013] S4. Input the optimized fusion coefficient as a fusion rule into the basic fusion method to complete the optimization of the interest index and realize image fusion.

[0014] Furthermore, the plurality of images to be fused includes two or more images to be fused, and the plurality of image evaluation indicators includes 2 to 5 image evaluation indicators.

[0015] Furthermore, the expression for the constructed multi-objective optimization problem is:

[0016] ;

[0017] in, The expected value of the interest metric representing the images to be fused. These represent multiple unknown fusion coefficients for an image to be fused. M represents the image quality index of each image to be fused, where M is a positive integer; N represents the number of unknown fusion coefficients of an image to be fused, where N is a positive integer.

[0018] Furthermore, the training steps of the BP neural network include:

[0019] S31. Calculate the interest index value of the image to be fused;

[0020] S32. Obtain several sets of random numbers as the basic fusion coefficients of the image to be fused, and use the basic fusion coefficients as the output layer of the training set;

[0021] S33. Calculate the interest index value of the fused image based on the basic fusion coefficients;

[0022] S34. The interest index values ​​of the image to be fused obtained in S31 and the interest index values ​​of the fused image obtained in S33 are used as the input layer of the training set.

[0023] S35. Select 30% of the data in the training set as the test set; through continuous training, verify the performance and obtain the trained BP neural network.

[0024] Furthermore, the basic fusion method is selected from at least one of weighted fusion, wavelet transform algorithm, NSST transform algorithm, NSCT transform algorithm, Laplace pyramid algorithm, principal component analysis algorithm, total variational fusion, or HIS fusion.

[0025] Furthermore, if the basic fusion method is a weighted fusion, then the unknown fusion coefficients satisfy:

[0026] ; .

[0027] Furthermore, the basic fusion method is a wavelet transform algorithm, which includes decomposing the image to be fused into high-frequency components and low-frequency components. Then:

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] in, The unknown fusion coefficient characterizing the high-frequency components. The unknown fusion coefficient characterizing the low-frequency component.

[0033] Furthermore, the images to be fused include a first visible polarization image to be fused and an infrared image to be fused; the first interest index includes three evaluation indices: information entropy (EN), average gradient (AG), and standard deviation (STD); the second interest index includes three evaluation indices: structural similarity (SSIM), information entropy (EN), and peak signal-to-noise ratio (PSNR).

[0034] The image fusion method based on interest indices includes fusing the I-image, Aop-image, and Dolp-image from the first visible polarization image to be fused into a fused polarizable image, and fusing the fused polarizable image with the infrared image to be fused; including the following steps:

[0035] S11. The NSST transform algorithm is used to decompose the I-image, Aop-image, and Dolp-image in the detected visible polarization image to be fused into first high-frequency information and first low-frequency information in the frequency domain;

[0036] S12. The first fusion coefficient is predicted by the BP neural network on the decomposed first high-frequency information and first low-frequency information; the first fusion coefficient is used to achieve overall optimization of the first interest index;

[0037] S13. Multiply the first high-frequency information and the first low-frequency information by the first fusion coefficient and then add them to obtain the fused first high-frequency information and the fused first low-frequency information, respectively;

[0038] S14. Perform inverse NSST transform on the fused first high-frequency information and the fused first low-frequency information to obtain the fused visible polarization image;

[0039] S15. Preprocess the detected infrared spectrum image to obtain the original infrared image;

[0040] S16. The original infrared image is smoothed using a bilateral filter, and the smoothed infrared image is superimposed with the original infrared image to obtain the infrared image to be fused after edge enhancement;

[0041] S17. Register the fused visible polarization image with the infrared image to be fused, and crop the fused visible polarization image to obtain a second visible polarization image to be fused;

[0042] S18. The NSST transform algorithm is used to decompose the second visible polarization image to be fused and the infrared image to be fused into second high-frequency information and second low-frequency information in the frequency domain;

[0043] S19. Predict the second fusion coefficient of the second high-frequency information and the second low-frequency information using a BP neural network; the second fusion coefficient is used to achieve overall optimization of the second interest index;

[0044] S20. Multiply the second high-frequency information and the second low-frequency information by the second fusion coefficient and then add them together to obtain the fused second high-frequency information and the fused second low-frequency information, respectively;

[0045] S21. Perform inverse NSST transform on the fused second high-frequency information and the fused second low-frequency information to obtain a fused image.

[0046] The present invention also provides a computer device, comprising:

[0047] At least one processor; and

[0048] A memory communicatively connected to the at least one processor; wherein,

[0049] The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the image fusion method based on interest indicators described above.

[0050] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the above-described image fusion method based on interest indicators.

[0051] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0052] The image fusion method based on multi-band multi-dimensional optical information provided by this invention can select several evaluation indicators for different images to be fused and application requirements. These indicators can be dynamically adjusted according to different situations, making it suitable for the fusion of various multi-band multi-dimensional optical information. It can simultaneously take into account multiple evaluation indicators, so that the fused image can achieve good results in multiple aspects, improve the utilization rate of the detected multi-band multi-dimensional optical information, and avoid the loss of effective information. Moreover, by using a BP neural network to solve the multi-objective problem, the entire fusion process only requires manual selection of evaluation indicators, avoiding the selection of regions and the calculation of weighting coefficients, which greatly reduces labor costs. Attached Figure Description

[0053] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0054] Figure 1 A schematic diagram of the overall process of the image fusion method based on multi-band multi-dimensional optical information with interest index as described in the embodiments of the present invention;

[0055] Figure 2 A schematic diagram illustrating the specific process of the image fusion method based on multi-band multi-dimensional optical information with interest indices as described in the embodiments of the present invention;

[0056] Figure 3 A schematic diagram of the BP neural network in the image fusion method based on multi-band multi-dimensional optical information with interest index as described in the embodiments of the present invention;

[0057] Figure 4 A schematic diagram of a computer device for performing an image fusion method based on multi-band, multi-dimensional optical information with interest indices, as described in an embodiment of the present invention.

[0058] Explanation of reference numerals in the attached figures:

[0059] 12. Computer equipment; 14. External devices; 16. Processing unit; 18. Bus; 20. Network adapter; 22. I / O interface; 24. Display; 28. System memory; 30. RAM; 34. Storage system; 40. Program / utility; 42. Program module. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0061] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0062] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0063] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0064] In a specific embodiment of the present invention, an image fusion method based on interest indices is provided, specifically an image fusion method based on multi-band multi-dimensional optical information of interest indices. The image fusion method based on interest indices is used to fuse several images to be fused; here, several images to be fused refer to two or more images to be fused.

[0065] Specifically, such as Figure 1 The figure shows a schematic diagram of the overall process of the image fusion method based on multi-band multi-dimensional optical information with interest indices according to a specific embodiment of the present invention. As can be seen from the figure, the image fusion method based on interest indices includes the following steps:

[0066] S1. Based on the characteristics of the images to be fused and the application scenario requirements, select several image evaluation indicators as interest indicators; specifically, the several images to be fused here include two or more images to be fused, and the several image evaluation indicators are preferably 2 to 5 evaluation indicators; the application scenario requirements include various specific information that needs to be highlighted according to the application scenario, such as the need to highlight edge information and the need to highlight infrared characteristics.

[0067] In specific implementations, if the images to be fused include a visible polarization intensity map, a visible polarization angle map, and a visible polarization degree map, then the corresponding selected interest metrics include five image evaluation metrics: MSE (mean squared error), PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), EN (information entropy), AG (average gradient), and STD (standard deviation). If the images to be fused include a visible polarization intensity map, a visible polarization angle map, a visible polarization degree map, and an infrared image, then the corresponding interest metrics include four image evaluation metrics: PSNR, SSIM, EN, and MI (mutual information). If the images to be fused include multispectral images, such as visible images, infrared images, and RGB three-color images, then the interest metrics include PSNR, SSIM, EN, MI, and EP (edge...). Five image evaluation metrics (preservation, edge preservation); if the image to be fused includes a visible image, an infrared polarization intensity image, an infrared polarization angle image, and an infrared polarization degree image, then the interest metrics include four image evaluation metrics: PSNR, SSIM, EN, and MI; if the image to be fused includes an infrared polarization intensity image, an infrared polarization angle image, and an infrared polarization degree image, then the interest metrics include five image evaluation metrics: MSE, PSNR, SSIM, EN, AG, and STD; if the image to be fused includes a visible image and an infrared light field image, then the interest metrics include four image evaluation metrics: PSNR, SSIM, EN, and MI; if the image to be fused includes a light field image and a polarization image, then the interest metrics include PSNR, SSIM, EN, MI, and Depth. Consistency (depth consistency) is one of five image evaluation metrics. If the images to be fused include SAR and visible images, the interest metrics include PSNR, SSIM, EN, and MI. If the image fusion is for medical purposes and the images to be fused include CT, MRI, PET, and ultrasound, the interest metrics include MSE, MI, and PSNR. Among them, EN focuses on the complexity and uncertainty of information in the image, and is used to measure the amount of information contained in the image. The higher the value, the more complex the image information and the richer the details and textures. AG focuses on the intensity of gray value changes in the image, focusing on high-frequency information such as edge information, and is used to measure the image sharpness. The higher the value, the higher the image sharpness. STD focuses on the dispersion of gray values ​​in the image, reflecting the overall contrast and uniformity of the image. The higher the value, the higher the image contrast.SSIM focuses on the structural features of an image to evaluate the structural similarity between the source and fused images; a value closer to 1 indicates higher image similarity. EI focuses on precise edge information to evaluate the saliency of edges in an image; a higher value indicates higher fused image quality. MSE focuses on the grayscale changes of image pixels to evaluate the overall distortion of the image; a lower MSE value indicates lower image distortion and higher image quality. PSNR focuses on the highest possible pixel value in an image; it is a transformation of MSE, more closely approximating human vision; a higher PSNR value indicates lower image distortion and higher image quality. MI focuses on the similarity between the fused and unfused images. The MI (Mean Integrity) index measures the degree to which the fused image retains information from the source image; a higher MI value indicates a higher degree of information retention. EP (Edge Estimation) focuses on regions in the image where brightness or color changes abruptly, measuring the retention of edge information; a higher EP value indicates richer edge information. DepthConsisitency focuses on the depth information of the 3D image, measuring the consistency of depth information. This fully demonstrates that the image fusion method based on interest indices provided by this invention is applicable to various multi-band, multi-dimensional optical information. Specifically, appropriate image evaluation indices can be selected as interest indices according to the characteristics of the images to be fused and the application scenario requirements.

[0068] S2. Construct a multi-objective optimization problem based on the basic fusion method, the interest index, and the unknown fusion coefficients; specifically, the constructed multi-objective optimization problem is a system of equations, mainly used to define the relationship between the interest index and the unknown fusion coefficients, where the interest index is a constraint, and the unknown fusion coefficients are unknowns. The number of unknown fusion coefficients varies depending on the selected basic fusion method; the expression of the multi-objective optimization problem can be:

[0069] ;

[0070] in, The expected value of the interest index representing the images to be fused, which can take the maximum value in some cases and the minimum value in others; These represent multiple unknown fusion coefficients for an image to be fused. The image quality index represents each image to be fused. M is a positive integer, preferably 2 to 5. N is a positive integer, and N is the number of unknown fusion coefficients. The specific number of unknown fusion coefficients can be selected according to different basic fusion methods.

[0071] For any image, its image evaluation index is calculated from that image. When the fusion coefficient is used as the fusion rule, since the input image and the transformation method (IF) are fixed, the evaluation index of the fused image can be regarded as a function of the fusion coefficient. Only by changing the value of the fusion coefficient can the IQM be optimized. According to the characteristics of the image to be fused and the needs of the application background, a set of interest indices are selected, and it is expected that all selected interest indices can be optimized. At this time, the interest indices and the fusion coefficient constitute a multi-objective optimization problem. By solving this problem, the overall optimization of the interest indices can be achieved.

[0072] In specific implementations, the basic fusion method is selected from at least one or more combinations of fusion algorithms such as weighted fusion, wavelet transform algorithm, NSST transform (non-subsampled shear wave transform) algorithm, NSCT transform algorithm, Laplace pyramid algorithm, principal component analysis algorithm, total variational fusion, or HIS fusion (Hyper-SpectralImage, fusion of multispectral and panchromatic images).

[0073] In a specific implementation, if the selected basic fusion method is weighted fusion, then the unknown fusion coefficients satisfy:

[0074] ; .

[0075] In a specific implementation, the basic fusion method can be selected as multi-scale transform fusion, which includes decomposing the image to be fused into components of multiple scales, wherein the unknown fusion coefficients of each component satisfy:

[0076] ; ; ;

[0077] in, Unknown fusion coefficients characterizing each component.

[0078] Specifically, the multi-scale transform fusion can be a wavelet transform algorithm, which includes decomposing the image to be fused into high-frequency components and low-frequency components, then:

[0079] ; ; ; ; ;

[0080] in, The unknown fusion coefficient characterizing the high-frequency components. The unknown fusion coefficient characterizing the low-frequency component.

[0081] S3. Solve the multi-objective optimization problem using a backpropagation (BP) neural network to obtain fusion coefficients. Specifically, the dataset used to train the BP neural network is readily available and can be obtained as follows: Use a sufficient number of sets of random numbers as fusion coefficients for the images to be fused. These fusion coefficients will be used to train the neural network, serving as the output layer of the training set. Use the interest index values ​​of the images to be fused and the corresponding interest index values ​​of the fused images calculated based on these sets of random numbers as the input layer. When the training set is large enough, the neural network will help us predict a set of optimal fusion coefficients, achieving overall optimization of the interest index.

[0082] In a specific implementation, the training steps of the BP neural network include:

[0083] S31. Calculate the interest index value of the image to be fused;

[0084] S32. Obtain several sets of random numbers as the basic fusion coefficients of the image to be fused, and use the basic fusion coefficients as the output layer of the training set;

[0085] S33. Calculate the interest index value of the fused image based on the basic fusion coefficients;

[0086] S34. The interest index values ​​of the image to be fused obtained in S31 and the interest index values ​​of the fused image obtained in S33 are used as the input layer of the training set.

[0087] S35. Select 30% of the data in the training set as the test set; through continuous training, verify the performance and obtain the trained BP neural network.

[0088] Specifically, solving the multi-objective optimization problem using a BP neural network to obtain the fusion coefficients also includes the following steps:

[0089] S36. Using the expected maximum interest index value of the fused image as the input layer, the optimized fusion coefficient is predicted through the trained BP neural network.

[0090] S4. Input the optimized fusion coefficient as a fusion rule into the basic fusion method to complete the optimization of the interest index and realize image fusion.

[0091] The image fusion method based on multi-band multi-dimensional optical information with interest index provided in the specific embodiments of the present invention can realize the fusion of multiple images or even multiple sets of images. Specifically, multi-dimensional can refer to three-dimensional scene information, polarization information, light field information, and spectral information. The images containing this information may not be one or two. For example, polarization information includes intensity map, angle map, and degree of polarization map. Light field information is focused on images at different depths. Therefore, for image fusion under different conditions, the entire fusion method process may include more than one fusion process S1-S4.

[0092] In a specific implementation, taking the fusion of visible polarization image and infrared image as an example, the image to be fused includes a first visible polarization image to be fused and an infrared image to be fused; the first interest index includes three evaluation indicators: information entropy (EN), average gradient (AG), and standard deviation (STD); the second interest index includes three evaluation indicators: structural similarity index (SSIM), information entropy (EN), and peak signal-to-noise ratio (PSNR).

[0093] In a specific implementation method taking the fusion of visible polarization images and infrared images as an example, the image fusion method based on interest indices involves two fusion processes. First, the three images in the visible polarization image to be fused are fused, namely the I-image, Aop image, and Dolp image. Then, the fused visible polarization image is fused with the infrared spectral image. The specific steps include:

[0094] S11. The NSST transform algorithm is used to decompose the I-image, Aop-image, and Dolp-image in the detected visible polarization image to be fused into first high-frequency information and first low-frequency information in the frequency domain;

[0095] The I-image is a visible polarization intensity map, which shows the intensity information of the visible polarization images to be fused; the Aop-image is a visible polarization angle map, which shows the polarization angle information of the visible polarization images to be fused; and the Dolp-image is a visible polarization degree map, which shows the polarization degree information of the visible polarization images to be fused.

[0096] S12. The first fusion coefficient is predicted by the BP neural network on the decomposed first high-frequency information and first low-frequency information; the first fusion coefficient is used to achieve overall optimization of the first interest index;

[0097] S13. Multiply the first high-frequency information and the first low-frequency information by the first fusion coefficient and then add them to obtain the fused first high-frequency information and the fused first low-frequency information, respectively;

[0098] S14. Perform inverse NSST transform on the fused first high-frequency information and the fused first low-frequency information to obtain the fused visible polarization image;

[0099] S15. Preprocess the detected infrared spectrum image to obtain the original infrared image;

[0100] S16. The original infrared image is smoothed using a bilateral filter, and the smoothed infrared image is superimposed with the original infrared image to obtain the infrared image to be fused after edge enhancement;

[0101] S17. Register the fused visible polarization image with the infrared image to be fused, and crop the fused visible polarization image to obtain a second visible polarization image to be fused;

[0102] S18. The NSST exchange algorithm is used to decompose the second visible polarization image to be fused and the infrared image to be fused into second high-frequency information and second low-frequency information in the frequency domain;

[0103] S19. Predict the second fusion coefficient of the second high-frequency information and the second low-frequency information using a BP neural network; the second fusion coefficient is used to achieve overall optimization of the second interest index;

[0104] S20. Multiply the second high-frequency information and the second low-frequency information by the second fusion coefficient and then add them together to obtain the fused second high-frequency information and the fused second low-frequency information, respectively;

[0105] S21. Perform inverse NSST transform on the fused second high-frequency information and the fused second low-frequency information to obtain a fused image.

[0106] This invention also provides a computer device, comprising:

[0107] At least one processor; and

[0108] A memory communicatively connected to the at least one processor; wherein,

[0109] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image fusion method based on multi-band multi-dimensional optical information of interest indices as described above.

[0110] A specific embodiment of the present invention also provides a non-transient computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the image fusion method based on multi-band multi-dimensional optical information of interest index described above.

[0111] The image fusion method based on multi-band multi-dimensional optical information provided by this invention can select several evaluation indicators for different images to be fused and application requirements. These indicators can be dynamically adjusted according to different situations, making it suitable for the fusion of various multi-band multi-dimensional optical information. It can simultaneously take into account multiple evaluation indicators, so that the fused image can achieve good results in multiple aspects, improve the utilization rate of the detected multi-band multi-dimensional optical information, and avoid the loss of effective information. Moreover, by using a BP neural network to solve the multi-objective problem, the entire fusion process only requires manual selection of evaluation indicators, avoiding the selection of regions and the calculation of weighting coefficients, which greatly reduces labor costs.

[0112] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0113] like Figure 2 The figure shows a schematic diagram of the overall process of an image fusion method based on multi-band multi-dimensional optical information with interest indices according to a specific embodiment of the present invention. As can be seen from the figure, this specific embodiment takes the fusion of visible polarization image and infrared image as an example. The image fusion method includes the following steps:

[0114] Step 11: Use the NSST transform algorithm to decompose the I-map, Aop-map, and Dolp-map of the detected visible polarization portion into high-frequency and low-frequency information in the frequency domain;

[0115] Step 12: Use the trained BP neural network to predict the fusion coefficients for overall optimization of interest indicators by analyzing the high-frequency and low-frequency information of the decomposed I-graph, Aop-graph, and Dolp-graph.

[0116] Step 13: Multiply the high-frequency and low-frequency information of the decomposed I-graph, Aop-graph, and Dolp-graph with the corresponding fusion coefficients and then add them together to obtain the fused high-frequency and low-frequency information;

[0117] Step 14: Perform inverse NSST transform on the fused high-frequency and low-frequency information to obtain the fused visible polarization image;

[0118] In this specific embodiment, the visible polarization I-image contains intensity information of the polarized image, including the basic structure and brightness distribution of the scene; the Aop-image contains polarization angle information, mainly showing the surface shape and roughness of the target in the scene under test; and the Dolp-image contains polarization degree information, mainly showing the reflection characteristics of the target in the scene under test. Fusing the I-image, Aop-image, and Dolp-image can improve the overall quality of the visible polarization image, combining the main information of the scene under test, the main shape of the target, and the degree of reflectivity, reducing noise and errors, enhancing the ability to penetrate clouds and fog and perform low-light imaging, improving the target recognition capability, and providing a more comprehensive scene analysis.

[0119] Step 15: Preprocess the detected infrared spectral image to obtain an infrared image;

[0120] Step 16: Smooth the infrared image using a bilateral filter, and then overlay the smoothed image with the original image to obtain the edge-enhanced infrared image to be fused.

[0121] In this specific embodiment, a bilateral filter can smooth the image while preserving its edge information. The mid-wave infrared image is filtered using a bilateral filter to smooth it, and then the smoothed image is superimposed on the source image to enhance the edges of the mid-wave infrared image. In a preferred embodiment of the invention, during the fusion of the mid-wave infrared image and the visible polarization fusion image, it is necessary to preserve the edge information of the mid-wave infrared image to highlight the thermal radiation information of the target in the fused image and enhance the image contrast.

[0122] Step 17: Register the fused visible polarization image with the infrared image to be fused, and crop the visible polarization image to obtain the visible polarization image to be fused;

[0123] In this specific embodiment, the infrared image is a mid-wave infrared spectral image, which includes mid-wave infrared imaging information and mid-wave infrared spectral information. The mid-wave infrared imaging information mainly records the thermal radiation information of the target in the scene under test, and has all-weather imaging and a certain penetration capability. The mid-wave infrared spectral information records the spectral information of the target in the scene under test, which can assist in the identification of target materials and components. By fusing the fused visible polarization image and the mid-wave infrared spectral image, the capture of thermal radiation information can be enhanced on the basis of the original visible polarization fused image, highlighting targets with temperatures different from the ambient temperature, such as human bodies and moving vehicles, enhancing image details and contrast, and further enhancing the ability to image in low light and penetrate clouds and fog.

[0124] Step 18: Use the NSST algorithm to decompose the visible polarization image to be fused and the infrared image to be fused into high-frequency information and low-frequency information in the frequency domain;

[0125] Step 19: Predict the fusion coefficient of the high-frequency information and the low-frequency information using a BP neural network; the fusion coefficient is used to achieve overall optimization of the interest index;

[0126] In this specific embodiment, the steps for training the BP neural network and predicting the fusion coefficients are as follows:

[0127] Step 31: Calculate the interest index value of the images to be fused;

[0128] Step 32: Obtain a sufficient number of random numbers as fusion coefficients for the images to be fused by obtaining random numbers. These coefficients are used as the output layer of the training set.

[0129] Step 33: Calculate the interest index value of the fused image based on the obtained sets of random numbers;

[0130] Step 34: The interest index values ​​of the image to be fused obtained in step 34 and the interest index values ​​of the fused image obtained in step 33 are used as the input layer of the training set;

[0131] Step 35: Select 30% of the data in the training set as the test set to verify the performance of the neural network;

[0132] Step 36: Using the expected maximum interest index value of the fused image as the input layer, predict the optimization fusion coefficient to achieve overall optimization of the interest index.

[0133] S20. Multiply the high-frequency information and the low-frequency information by the fusion coefficient and then add them together to obtain the fused high-frequency information and the fused low-frequency information, respectively;

[0134] S21. Perform inverse NSST transformation on the fused high-frequency information and the fused low-frequency information to obtain a fused image.

[0135] In a preferred embodiment of the present invention, NSST performs non-downsampled pyramid decomposition on the input image, and then performs non-downsampled shear wave transform on the obtained high-frequency subbands. This multi-scale and multi-directional image decomposition effectively preserves high-frequency information such as edges, textures, and details. The use of non-downsampled transform avoids the loss of image details during reconstruction and provides strong noise resistance. Compared with non-downsampled contour wave transform (NSCT), it avoids multi-level processing and improves the transformation speed. In multi-band, multi-dimensional optical information fusion, it can effectively preserve details of source images from different bands and dimensions.

[0136] In this specific embodiment, such as Figure 2As shown, taking the fusion of visible polarization images and mid-infrared spectral images as an example, information entropy (EN), average gradient (AG), and standard deviation (STD) are used as interest indicators for the fusion of visible polarization intensity maps (I-map), visible polarization angle maps (Aop-map), and visible polarization degree maps (Dolp-map). Structural similarity (SSIM), information entropy (EN), and peak signal-to-noise ratio (PSNR) are used as interest indicators for the fusion of the visible polarization fused map and the mid-infrared spectral image. A trained BP neural network is used to solve for the fusion coefficients of the visible polarization image fusion and the fusion of the visible polarization fused map and the mid-infrared spectral image. A method combining non-subsampled shear wave transform (NSST) and a bilateral filter is used to fuse the solved fusion coefficients, resulting in the final fused visible polarization-infrared spectral image. Specifically, the fusion rule selects several image evaluation indicators of interest as interest indicators, and optimizes these interest indicators of the fused image as a whole. Multi-band and multi-dimensional optical information contains multiple information dimensions and spans a wide range of bands. Images of different bands and dimensions have different characteristics, and the focus of the fused images also varies depending on the application scenario. Based on the characteristics of images of different bands and dimensions and the needs of application scenarios, different image evaluation indicators can be flexibly selected to calculate the fusion coefficient, thereby improving the utilization rate of multi-band and multi-dimensional optical information.

[0137] Specifically, the principle of calculating the fusion coefficient using the fusion rules of this invention is as follows:

[0138] Taking visible polarization fusion as an example, there are fusion coefficients for I-plots, AOP plots, and Dolp plots. , , , , , The high-frequency and low-frequency information corresponding to the three images are respectively used to determine the selected interest indicators EN, AG, and STD. , , , , , , , , , , , These correspond to the I-image, Aop-image, Dolp-image, and the fused visible polarization image F-image, respectively. , , All , , , , , The functions are independent of each other, and the corresponding relationships are as follows:

[0139]

[0140]

[0141]

[0142] In a preferred embodiment of the present invention , , Both should take larger values, because , , They are unrelated, so we need to find a solution. , , , , , The value of constitutes a constrained multi-objective optimization problem, with the following constraints:

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149] The above-mentioned constrained multi-objective optimization problem is solved using a trained BP neural network, resulting in a set of... , , , , , The solution satisfies the condition that the EN, AG, and STD values ​​of the fused visible polarization image can all reach maximum values.

[0150] In this specific embodiment, the method for solving the fusion coefficients when fusing the mid-wave infrared image and the fused visible polarization image is the same as the method used for visible polarization fusion. A larger EN value indicates richer image information, a larger AG value indicates a clearer image, a larger STD value indicates better image contrast, a SSIM value closer to 1 indicates higher structural similarity between the fused image and the original image, and a larger PSNR value indicates lower image distortion. When fusing the I-image, Aop image, and Dolp image in the visible polarization fusion part, EN, AG, and STD are used to ensure the information content, contrast, and clarity of the fused image. When fusing the fused visible polarization image with the mid-wave infrared image, EN, SSIM, and PSNR are used to ensure the information content and structural similarity of the fused image and to avoid introducing excessive noise.

[0151] In this preferred embodiment, such as Figure 3 As shown, the input layer uses the interest index values ​​of the images to be fused and the expected maximum selected interest index value of the fused image, while the output layer uses the fusion coefficient that optimizes the overall interest index. A trained BP neural network is used to predict this coefficient. Specifically, the BP neural network used for predicting the fusion coefficient in the visible polarization fusion part... n =12, m =6, These correspond to the normalized values ​​of EN, AG, and STD of the detected I-graph, Aop-graph, and Dolp-graph. , , , , , , , , The normalized maximum expectation of the fused visible polarization EN, AG, and STD values. , , , Fusion coefficients corresponding to I-plot, AOP plot, and Dolp plot , , , , , A backpropagation neural network (BP neural network) with n=7 and m=4 was used for partial prediction of fusion coefficients in the fusion of visible polarization fusion map and mid-wave infrared image. The corresponding values ​​are the normalized EN, PSNR, and SSIM values ​​of the visible polarization fusion map and the detected mid-wave infrared image. , , , The normalized maximum expected value of EN and PSNR and the maximum theoretical value of SSIM of the fused final image. , , , The fusion coefficients of the visible polarization fusion map and the mid-wave infrared image , , , .

[0152] Accordingly, according to embodiments of the present invention, the present invention also provides a computer device, a readable storage medium, and a computer program product.

[0153] Figure 4 This is a schematic diagram of the structure of a computer device 12 provided in an embodiment of the present invention. Figure 4 A block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present invention is shown. Figure 4 The computer device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0154] like Figure 4 As shown, computer device 12 is represented in the form of a general-purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0155] The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0156] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0157] Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0158] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0159] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0160] Computer device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with computer device 12, and / or with any device that enables computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0161] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the image fusion method based on multi-band multi-dimensional optical information of interest index provided in the embodiments of the present invention.

[0162] This invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein the program, when executed by a processor, is the image fusion method based on multi-band multi-dimensional optical information with interest indices provided in all embodiments of this application.

[0163] The computer storage medium of this invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0164] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0165] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. The computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image fusion method based on the above-described multi-band multi-dimensional optical information according to an interest index.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0168] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image fusion method based on interest indices, characterized in that: The image fusion method based on interest indicators is used to fuse several images to be fused. The image fusion method based on interest indices includes the following steps: S1. Based on the characteristics of the images to be fused and the requirements of the application scenario, select several image evaluation indicators as interest indicators; S2. Construct a multi-objective optimization problem based on the basic fusion method, the interest index, and the unknown fusion coefficients; the expression of the constructed multi-objective optimization problem is: ； in, The expected value of the interest metric representing the images to be fused. These represent multiple unknown fusion coefficients for an image to be fused. The image quality index represents each image to be fused, where M is a positive integer, N is a positive integer, and N is the number of unknown fusion coefficients; S3. Solve the multi-objective optimization problem using a BP neural network to obtain the optimization fusion coefficients; S4. Input the optimized fusion coefficient as a fusion rule into the basic fusion method to complete the optimization of the interest index and realize image fusion.

2. The image fusion method based on interest indices according to claim 1, characterized in that: The plurality of images to be fused includes two or more images to be fused, and the plurality of image evaluation indicators includes 2 to 5 image evaluation indicators.

3. The image fusion method based on interest indicators according to claim 1, characterized in that: The training steps of the BP neural network include: S31. Calculate the interest index value of the image to be fused; S32. Obtain several sets of random numbers as the basic fusion coefficients of the image to be fused, and use the basic fusion coefficients as the output layer of the training set; S33. Calculate the interest index value of the fused image based on the basic fusion coefficients; S34. The interest index values ​​of the image to be fused obtained in S31 and the interest index values ​​of the fused image obtained in S33 are used as the input layer of the training set. S35. Select 30% of the data in the training set as the test set; through continuous training, verify the performance and obtain the trained BP neural network.

4. The image fusion method based on interest indicators according to claim 1, characterized in that: The basic fusion method is selected from at least one of weighted fusion, wavelet transform algorithm, NSST transform algorithm, NSCT transform algorithm, Laplace pyramid algorithm, principal component analysis algorithm, total variational fusion, or HIS fusion.

5. The image fusion method based on interest indicators according to claim 4, characterized in that: The basic fusion method is weighted fusion, then the unknown fusion coefficients satisfy: ； 。 6. The image fusion method based on interest indicators according to claim 4, characterized in that: The basic fusion method is a wavelet transform algorithm, which includes decomposing the image to be fused into high-frequency components and low-frequency components. ； ； ； ； ； in, The unknown fusion coefficient characterizing the high-frequency components, The unknown fusion coefficient characterizing the low-frequency component.

7. The image fusion method based on interest indicators according to claim 1, characterized in that: The images to be fused include a first visible polarization image to be fused and an infrared image to be fused; the first interest index includes three evaluation indices: information entropy (EN), average gradient (AG), and standard deviation (STD); the second interest index includes three evaluation indices: structural similarity index (SSIM), information entropy (EN), and peak signal-to-noise ratio (PSNR). The image fusion method based on interest indices includes fusing the I-image, Aop-image, and Dolp-image in the first visible polarization image to be fused into a fused polarizable image, and fusing the fused polarizable image with the infrared image to be fused. include step: S11. The NSST transform algorithm is used to decompose the I-image, Aop-image, and Dolp-image in the detected visible polarization image to be fused into first high-frequency information and first low-frequency information in the frequency domain; S12. Predict the first fusion coefficient of the decomposed first high-frequency information and first low-frequency information using the BP neural network; The first fusion coefficient is used to achieve overall optimization of the first interest index; S13. Multiply the first high-frequency information and the first low-frequency information by the first fusion coefficient and then add them to obtain the fused first high-frequency information and the fused first low-frequency information, respectively; S14. Perform inverse NSST transform on the fused first high-frequency information and the fused first low-frequency information to obtain the fused visible polarization image; S15. Preprocess the detected infrared spectrum image to obtain the original infrared image; S16. The original infrared image is smoothed using a bilateral filter, and the smoothed infrared image is superimposed with the original infrared image to obtain the infrared image to be fused after edge enhancement; S17. Register the fused visible polarization image with the infrared image to be fused, and crop the fused visible polarization image to obtain a second visible polarization image to be fused; S18. The NSST exchange algorithm is used to decompose the second visible polarization image to be fused and the infrared image to be fused into second high-frequency information and second low-frequency information in the frequency domain; S19. Predict the second fusion coefficient of the second high-frequency information and the second low-frequency information using a BP neural network; The second fusion coefficient is used to achieve overall optimization of the second interest index; S20. Multiply the second high-frequency information and the second low-frequency information by the second fusion coefficient and then add them together to obtain the fused second high-frequency information and the fused second low-frequency information, respectively; S21. Perform inverse NSST transform on the fused second high-frequency information and the fused second low-frequency information to obtain a fused image.

8. A computer device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the image fusion method based on interest indices as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the image fusion method based on interest indicators as described in any one of claims 1 to 7.

Images