Image fusion method and device based on interest index and storage medium

Abstract

The invention belongs to the technical field of image fusion, and particularly relates to an interest index-based image fusion method, computer equipment and a storage medium. The image fusion method comprises the steps of selecting an image evaluation index as an interest index according to characteristics of a to-be-fused image and application scene requirements; constructing a multi-objective optimization problem based on the interest index and the unknown fusion coefficient; solving a multi-objective optimization problem through a BP neural network to obtain an optimization fusion coefficient; and inputting the optimized fusion coefficient as a fusion rule into a fusion method, completing optimization of the interest index, and realizing image fusion. The whole fusion process only needs to manually select evaluation indexes, so that the selection of regions and the calculation of weighting coefficients are avoided, and the labor cost is greatly reduced; moreover, the method can be dynamically adjusted according to different conditions, is suitable for fusion of various multi-band and multi-dimensional optical information, can also be suitable for a plurality of arbitrary evaluation indexes, and can carry out the overall optimization of the evaluation indexes.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to an image fusion method based on multi-band and multi-dimensional optical information of interest indicators, a computer device for executing the image fusion method, and a non-transient computer-readable storage medium. Background Art

[0002] With the development of optical technology, traditional two-dimensional detection can no longer meet the increasingly complex detection needs. Multi-band and multi-dimensional optical information detection technology, which can detect more optical information and improve target recognition rate, has become the development trend of optical imaging systems. Faced with the richer optical information detected by various multi-band and multi-dimensional optical information detection systems, how to efficiently process and utilize this optical information has attracted widespread attention from scholars. Image fusion technology for multi-band and multi-dimensional optical information utilizes the complementary advantages of optical information in different bands and dimensions to enhance information extraction capabilities and improve detection accuracy and reliability. It is widely applicable to the fusion of multi-band and multi-dimensional optical information such as polarization images and light field images in various bands.

[0003] At present, image fusion technology mainly includes fusion based on neural network, fusion based on region and pixel-level fusion. Among them, the fusion technology based on neural network requires a large amount of data set for preliminary training. For multi-band and multi-dimensional optical information detection, which involves multiple bands and wide information dimensions, it is very costly to build a data set by yourself. The technology based on region and pixel-level fusion first pre-processes the image, and then selects appropriate fusion rules to fuse the image according to the characteristics of the original image to be fused. The current traditional fusion rule selection methods mainly include weighted average method, maximum value method, regional average method, local energy method, etc. In the fusion of multi-band and multi-dimensional optical information, the weight coefficient of the traditional fusion rule selection method is difficult to determine, the labor cost required to find the characteristic area is high, and it can usually only highlight one or two indicators, and cannot take into account multiple indicators, and the utilization rate of the detected optical information is low.

[0004] At present, the fusion rule selection methods for realizing pixel-level and regional image fusion mainly include weighted coefficient method, extreme value method, specific area weighted method, etc. Among them, the weighted coefficient method is difficult to determine the appropriate weighting coefficient to take into account multiple indicators for the images to be fused in different bands and dimensions when dealing with multi-band and multi-dimensional light information fusion. For the extreme value method, since this method selects the maximum gray value in the pixel or the maximum coefficient in the sub-band, it only focuses on the edge information and detail information of the image, resulting in a decrease in the overall visual effect of the fused image, the loss of some effective information, and serious noise impact. For the specific area weighted method, this method needs to manually select the area for each group of images to be fused and then find the appropriate weighting coefficient, which is time-consuming and labor-intensive.

[0005] In 2023, Changchun University of Science and Technology proposed a dual-weighted polarization image fusion method based on quality evaluation and attention mechanism, which combines information entropy, reference-free image quality evaluation and local energy to achieve high-contrast fusion of polarization intensity map and polarization degree map; however, this method cannot select several arbitrary evaluation indicators for combination according to different application requirements and characteristics of the image to be fused, and realize the overall optimization of all selected evaluation indicators. It is limited to the optimization of a single indicator and cannot be generally applied to the fusion of multiple multi-band and multi-dimensional optical information. Summary of the invention

[0006] In view of this, the present invention aims to provide an image fusion method based on interest indicators of multi-band and multi-dimensional optical information, which can select several arbitrary evaluation indicators for combination according to different application requirements and characteristics of the image to be fused, so as to achieve overall optimization of all selected evaluation indicators, is not limited to the optimization of a single indicator, and is generally applicable to the fusion of multiple multi-band and multi-dimensional optical information.

[0007] To achieve the above object, the technical solution created by the present invention is implemented as follows: The present invention provides an image fusion method based on interest index, specifically an image fusion method of multi-band multi-dimensional light information based on interest index, wherein the image fusion method based on interest index is used to fuse a plurality of images to be fused; The image fusion method based on interest index comprises the steps of: S1. Select several image evaluation indicators as interest indicators according to the characteristics of the image to be fused and the application scenario requirements; S2. Constructing a multi-objective optimization problem based on the basic fusion method, the interest index and the unknown fusion coefficient; S3. Solve the multi-objective optimization problem through a BP neural network to obtain an optimized fusion coefficient; S4. Input the optimized fusion coefficient as a fusion rule into the basic fusion method to complete the optimization of the interest index and realize image fusion.

[0008] Furthermore, the plurality of images to be fused include two or more images to be fused, and the plurality of image evaluation indicators include 2 to 5 image evaluation indicators.

[0009] Furthermore, the expression of the multi-objective optimization problem constructed is: ; in, Represents the expected value of the interest index of the image to be fused, Represents multiple unknown fusion coefficients of an image to be fused; Represents the image quality index of each image to be fused, M is a positive integer; N represents the number of unknown fusion coefficients of an image to be fused, N is a positive integer.

[0010] Furthermore, the training steps of the BP neural network include: S31. Calculate the interest index value of the image to be fused; S32. Obtaining several groups of random numbers as the basic fusion coefficients of the images to be fused, and the basic fusion coefficients are used as the output layer of the training set; S33. Calculating the interest index value of the fused image according to the basic fusion coefficient; S34. Using the interest index value of the image to be fused obtained in S31 and the interest index value of the fused image obtained in S33 as the input layer of the training set; S35. Select 30% of the data in the training set as the test set; through continuous training and verification of performance, obtain the trained BP neural network.

[0011] Furthermore, the basic fusion method is selected from at least one of weighted fusion, wavelet transform algorithm, NSST transform algorithm, NSCT transform algorithm, Laplace pyramid algorithm, principal component analysis algorithm, total variation fusion or HIS fusion.

[0012] Furthermore, the basic fusion method is weighted fusion, and the unknown fusion coefficient satisfies: ; .

[0013] Furthermore, the basic fusion method is a wavelet transform algorithm, and the wavelet transform algorithm includes decomposing the image to be fused into a high-frequency component and a low-frequency component, then: ; ; ; ; in, Characterizing the unknown fusion coefficient of the high frequency component, Characterize the unknown fusion coefficient of the low frequency component.

[0014] Furthermore, the image to be fused includes a first visible polarization image to be fused and an infrared image to be fused; the first interest index includes three evaluation indexes of information entropy EN, average gradient AG and standard deviation STD; the second interest index includes three evaluation indexes of structural similarity SSIM, information entropy EN and peak signal-to-noise ratio PSNR; The image fusion method based on the interest index includes fusing the I image, the Aop image, and the Dolp image in the first visible polarization image to be fused into a fused polarizable image, and fusing the fused polarizable image with the infrared image to be fused; including the steps of: S11. Decomposing the I image, Aop image, and Dolp image in the visible polarization image to be fused into first high-frequency information and first low-frequency information in the frequency domain by using the NSST transformation algorithm; S12. Predicting a first fusion coefficient for the decomposed first high-frequency information and the first low-frequency information through the BP neural network; the first fusion coefficient is used to achieve overall optimization of the first interest index; S13. multiplying the first high-frequency information and the first low-frequency information by the first fusion coefficient and then adding them together to obtain fused first high-frequency information and fused first low-frequency information respectively; S14. performing an inverse NSST transformation on the fused first high-frequency information and the fused first low-frequency information to obtain a fused visible polarization image; S15. Preprocessing the detected infrared spectrum image to obtain an original infrared image; S16. smoothing the original infrared image using a bilateral filter, and superimposing the smoothed infrared image with the original infrared image to obtain an infrared image to be fused after edge enhancement; S17. aligning the fused visible polarization image with the infrared image to be fused, and cropping the fused visible polarization image to obtain a second visible polarization image to be fused; S18. Decomposing the second visible polarization image to be fused and the infrared image to be fused into second high-frequency information and second low-frequency information in the frequency domain by using an NSST transformation algorithm; S19. Predicting a second fusion coefficient for the second high-frequency information and the second low-frequency information through a BP neural network; the second fusion coefficient is used to achieve overall optimization of the second interest index; S20. multiplying the second high-frequency information and the second low-frequency information by the second fusion coefficient and then adding them together to obtain fused second high-frequency information and fused second low-frequency information respectively; S21. Perform an inverse NSST transformation on the fused second high-frequency information and the fused second low-frequency information to obtain a fused image.

[0015] The present invention also provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image fusion method based on interest indicators of the present invention.

[0016] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned image fusion method based on interest indicators of the present invention.

[0017] Compared with the prior art, the invention can achieve the following beneficial effects: The image fusion method of multi-band multi-dimensional optical information based on interest index provided by the present invention can select several evaluation indexes according to different images to be fused and application requirements, can be dynamically adjusted according to different situations, and is suitable for the fusion of multiple multi-band multi-dimensional optical information; it can take into account multiple evaluation indexes at the same time, so that the fused image can achieve better effects in multiple aspects, improve the utilization rate of the detected multi-band multi-dimensional optical information, and avoid the loss of effective information; moreover, by adopting BP neural network to solve multi-objective problems, the entire fusion process only needs to manually select evaluation indexes, avoids the selection of regions and the calculation of weighting coefficients, and greatly reduces labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of the overall process of the image fusion method of multi-band multi-dimensional optical information based on interest indicators described in the implementation mode of the present invention; Figure 2 A schematic diagram of a specific process of the method for image fusion of multi-band and multi-dimensional optical information based on interest indicators according to an embodiment of the present invention; Figure 3 A schematic diagram of a BP neural network in an image fusion method of multi-band multi-dimensional optical information based on interest indicators described in an embodiment of the present invention; Figure 4 A schematic diagram of a computer device for executing an image fusion method of multi-band and multi-dimensional optical information based on interest indicators as described in an embodiment of the present invention.

[0019] Description of reference numerals: 12. Computer equipment; 14. External devices; 16. Processing unit; 18. Bus; 20. Network adapter; 22. I / O interface; 24. Display; 28. System memory; 30. RAM; 34. Storage system; 40. Programs / utilities; 42. Program modules. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0021] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0022] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0023] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0024] In a specific embodiment of the present invention, an image fusion method based on interest indicators is provided, specifically an image fusion method based on multi-band and multi-dimensional optical information of interest indicators, and the image fusion method based on interest indicators is used to fuse several images to be fused; here, several images to be fused refer to two or more images to be fused.

[0025] Specifically, Figure 1 As shown in FIG. 1 , it is a schematic diagram of the overall process of the image fusion method of multi-band multi-dimensional optical information based on interest indicators according to a specific embodiment of the present invention. It can be seen from the figure that the image fusion method based on interest indicators includes the steps of: S1. According to the characteristics of the images to be fused and the application scenario requirements, select several image evaluation indicators as interest indicators; specifically, the several images to be fused mentioned here include two or more images to be fused, and the several image evaluation indicators are preferably 2 to 5 evaluation indicators; the application scenario requirements include the need to highlight edge information, the need to highlight infrared characteristics and other specific information that needs to be highlighted according to the application scenario.

[0026] In a specific implementation manner, if the images to be fused include a visible polarization intensity map, a visible polarization angle map, and a visible polarization degree map, the corresponding selected interest indicators include five image evaluation indicators: MSE (mean squared error), PSNR (peak signal-to-noise ratio), SSIM (structural similarity index), EN (information entropy), AG (average gradient), and STD (standard deviation); if the images to be fused include a visible polarization intensity map, a visible polarization angle map, a visible polarization degree map, and an infrared image, the corresponding interest indicators include four image evaluation indicators: PSNR, SSIM, EN, and MI (mutual information); if the images to be fused include multispectral images, such as visible images, infrared images, RGB three-color images, etc., the interest indicators include PSNR, SSIM, EN, MI, and EP (Edge Preservation, edge preservation); if the images to be fused include visible images, infrared polarization intensity images, infrared polarization angle images, and infrared polarization degree images, the interest indicators include PSNR, SSIM, EN, and MI; if the images to be fused include infrared polarization intensity images, infrared polarization angle images, and infrared polarization degree images, the interest indicators include MSE, PSNR, SSIM, EN, AG, and STD; if the images to be fused include visible images and infrared light field images, the interest indicators include PSNR, SSIM, EN, and MI; if the images to be fused include light field images and polarization images, the interest indicators include PSNR, SSIM, EN, MI, Depth Consisitency (depth consistency); if the images to be fused include SAR images and visible images, the interest indicators include PSNR, SSIM, EN, and MI; if the image fusion is for the medical field, the images to be fused include CT, MRI, PET, and ultrasound, then the interest indicators include MSE, MI, and PSNR; among them, EN focuses on the complexity and uncertainty of the information in the image, and is used to measure the amount of information contained in the image. The higher the value, the more complex the image information, and the richer the details and textures; AG focuses on the intensity of the gray value change in the image, focusing on high-frequency information such as edge information, and is used to measure the clarity of the image. The higher the value, the higher the image clarity; STD focuses on the discreteness of the image gray value, reflecting the overall contrast and uniformity of the image. The higher the value, the higher the image contrast.SSIM focuses on the structural features of the image and is used to evaluate the structural similarity between the source image and the fused image. The closer the value is to 1, the higher the image similarity. EI focuses on precise edge information and is used to evaluate the significance of the edge in the image. The larger the value, the higher the quality of the fused image. MSE focuses on the grayscale value change of the image pixels and is used to evaluate the overall distortion of the image. The lower the MSE value, the lower the image distortion and the higher the image quality. PSNR focuses on the highest possible pixel value in the image and is a transformation of MSE, which is closer to human vision. The higher the PSNR value, the lower the image distortion and the higher the image quality. MI focuses on the similarity between the fused image and the image before and after fusion. Degree is used to measure the degree of information retention of the source image after fusion. The higher the MI value, the higher the information retention. EP focuses on the area in the image where the brightness or color changes sharply. It is used to measure the edge information retention of the image. The higher the value, the richer the edge information. DepthConsisitency focuses on the depth information of the three-dimensional image and is used to measure the consistency of the depth information of the image. It fully illustrates that the image fusion method based on interest index provided by the present invention is a method applicable to image fusion of various multi-band and multi-dimensional light information. Specifically, the corresponding image evaluation index can be selected as the interest index according to the characteristics of the image to be fused and the application scenario requirements. ;

[0027] S2. A multi-objective optimization problem is constructed based on the basic fusion method, the interest index and the unknown fusion coefficient; specifically, the constructed multi-objective optimization problem is a set of equations, which is mainly used to stipulate the relationship between the interest index and the unknown fusion coefficient, wherein the interest index is a constraint condition, and the unknown fusion coefficient is an unknown number. Depending on the selected basic fusion method, the number of corresponding unknown fusion coefficients is also different; the expression of the multi-objective optimization problem can be: ; in, Represents the expected value of the interest index of the image to be fused, that is, it can take the maximum value in some cases and the minimum value in some cases; Represents multiple unknown fusion coefficients of an image to be fused, Represents the image quality index of each image to be fused, M is a positive integer, preferably 2 to 5; N is a positive integer, N is the number of unknown fusion coefficients, and different numbers of unknown fusion coefficients can be selected according to different basic fusion methods.

[0028] For any image, its image evaluation index is calculated from the image. When the fusion coefficient is used as the fusion rule, since the input image is fixed and the transformation method (IF) is fixed, the evaluation index of the fused image can be regarded as a function of the fusion coefficient. The IQM can be optimized by simply changing the value of the fusion coefficient. A set of interest indicators is selected according to the characteristics of the image to be fused and the requirements of the application background, and it is expected that all the selected interest indicators can be optimized. At this time, the interest indicators and the fusion coefficient constitute a multi-objective optimization problem. By solving this problem, the overall optimization of the interest indicators can be achieved.

[0029] In a specific implementation, the basic fusion method is selected from at least one or more combinations of fusion algorithms such as weighted fusion, wavelet transform algorithm, NSST transform (non-subsampled shearlet transform) algorithm, NSCT transform algorithm, Laplace pyramid algorithm, principal component analysis algorithm, total variation fusion or HIS fusion (Hyper-SpectralImage, multispectral image and panchromatic image fusion).

[0030] In a specific implementation manner, the selected basic fusion method is weighted fusion, and the unknown fusion coefficient satisfies: ; .

[0031] In a specific implementation, the basic fusion method may be selected as multi-scale transform fusion, which includes decomposing the image to be fused into components of multiple scales, and the unknown fusion coefficient of each component satisfies: ; ; ; in, Characterize the unknown fusion coefficients of each component.

[0032] Specifically, the multi-scale transform fusion may be a wavelet transform algorithm, and the wavelet transform algorithm includes decomposing the image to be fused into a high-frequency component and a low-frequency component, then: ; ; ; ; ; in, Characterizing the unknown fusion coefficient of the high frequency component, Characterize the unknown fusion coefficient of the low frequency component.

[0033] S3. Solve the multi-objective optimization problem through BP neural network to obtain the fusion coefficient; specifically, the data set used to train the BP neural network is easy to obtain, and the acquisition method is as follows: use enough groups of random numbers as the fusion coefficients of the images to be fused, these fusion coefficients will be used for neural network training, serve as the output layer of the training set, and use the interest index values ​​of the images to be fused and the interest index values ​​of the corresponding fused images calculated according to these groups of random numbers as the input layer. When the number of training sets is large enough, the neural network will help us predict a set of optimal fusion coefficients to achieve overall optimization of the interest index.

[0034] In a specific implementation manner, the training steps of the BP neural network include: S31. Calculate the interest index value of the image to be fused; S32. Obtaining several groups of random numbers as the basic fusion coefficients of the images to be fused, and the basic fusion coefficients are used as the output layer of the training set; S33. Calculating the interest index value of the fused image according to the basic fusion coefficient; S34. Using the interest index value of the image to be fused obtained in S31 and the interest index value of the fused image obtained in S33 as the input layer of the training set; S35. Select 30% of the data in the training set as the test set; through continuous training and verification of performance, obtain the trained BP neural network.

[0035] Specifically, solving the multi-objective optimization problem by using a BP neural network to obtain a fusion coefficient also includes the following steps: S36. Taking the expected maximum interest index value of the fused image as the input layer, the optimized fusion coefficient is predicted through the trained BP neural network.

[0036] S4. Input the optimized fusion coefficient as a fusion rule into the basic fusion method to complete the optimization of the interest index and realize image fusion.

[0037] The image fusion method of multi-band and multi-dimensional optical information based on interest indicators provided in the specific embodiment of the present invention can realize the fusion of multiple images or even multiple groups of images. Specifically, multi-dimensionality can refer to three-dimensional scene information, polarization information, light field information, and spectral information. The images of this information may not be one or two. For example, polarization information includes intensity maps, angle maps, and polarization degree maps, and light field information is focused on images at different depths. Therefore, for image fusion in different situations, the entire fusion method process may include more than one S1-S4 fusion process.

[0038] In a specific implementation, taking the fusion of a visible polarization image and an infrared image as an example, the images to be fused include a first visible polarization image to be fused and an infrared image to be fused; the first interest index includes three evaluation indexes: information entropy EN, average gradient AG, and standard deviation STD; the second interest index includes three evaluation indexes: structural similarity index SSIM, information entropy EN, and peak signal-to-noise ratio PSNR.

[0039] In a specific implementation example of the fusion of a visible polarization image and an infrared image, the entire fusion process in the image fusion method based on interest indicators is performed twice, including first fusing the three images in the visible polarization image to be fused, that is, fusing the I image, the Aop image and the Dolp image, and then fusing the fused visible polarization image with the infrared spectrum image. The specific steps include: S11. Decomposing the I image, Aop image, and Dolp image in the visible polarization image to be fused into first high-frequency information and first low-frequency information in the frequency domain by using the NSST transformation algorithm; The I image is a visible polarization intensity image, which is the intensity information of the visible polarization image to be fused; the Aop image is a visible polarization angle image, which is the polarization angle information of the visible polarization image to be fused; the Dolp image is a visible polarization degree image, which is the polarization degree information of the visible polarization image to be fused.

[0040] S12. Predicting a first fusion coefficient of the decomposed first high-frequency information and the first low-frequency information through the BP neural network; the first fusion coefficient is used to achieve overall optimization of the first interest indicator; S13. multiplying the first high-frequency information and the first low-frequency information by the first fusion coefficient and then adding them together to obtain fused first high-frequency information and fused first low-frequency information respectively; S14. performing an inverse NSST transformation on the fused first high-frequency information and the fused first low-frequency information to obtain a fused visible polarization image; S15. Preprocessing the detected infrared spectrum image to obtain an original infrared image; S16. smoothing the original infrared image using a bilateral filter, and superimposing the smoothed infrared image with the original infrared image to obtain an infrared image to be fused after edge enhancement; S17. aligning the fused visible polarization image with the infrared image to be fused, and cropping the fused visible polarization image to obtain a second visible polarization image to be fused; S18. Decomposing the second visible polarization image to be fused and the infrared image to be fused into second high-frequency information and second low-frequency information in the frequency domain by using the NSST exchange algorithm; S19. Predicting a second fusion coefficient for the second high-frequency information and the second low-frequency information through a BP neural network; the second fusion coefficient is used to achieve overall optimization of the second interest index; S20. multiplying the second high-frequency information and the second low-frequency information by the second fusion coefficient and then adding them together to obtain fused second high-frequency information and fused second low-frequency information respectively; S21. Perform an inverse NSST transformation on the fused second high-frequency information and the fused second low-frequency information to obtain a fused image.

[0041] A specific embodiment of the present invention further provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image fusion method of multi-band and multi-dimensional optical information based on interest indicators of the present invention.

[0042] A specific embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned image fusion method of multi-band and multi-dimensional optical information based on interest indicators of the present invention.

[0043] The image fusion method of multi-band multi-dimensional optical information based on interest index provided by the present invention can select several evaluation indexes according to different images to be fused and application requirements, can be dynamically adjusted according to different situations, and is suitable for the fusion of multiple multi-band multi-dimensional optical information; it can take into account multiple evaluation indexes at the same time, so that the fused image can achieve better effects in multiple aspects, improve the utilization rate of the detected multi-band multi-dimensional optical information, and avoid the loss of effective information; moreover, by adopting BP neural network to solve multi-objective problems, the entire fusion process only needs to manually select evaluation indexes, avoids the selection of regions and the calculation of weighting coefficients, and greatly reduces labor costs.

[0044] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0045] like Figure 2 As shown, it is a schematic diagram of the overall process of an image fusion method of multi-band multi-dimensional light information based on interest indicators provided by a specific embodiment of the present invention. It can be seen from the figure that this specific embodiment takes the fusion of visible polarization image and infrared image as an example, and the image fusion method includes the following steps: Step 11: Use the NSST transformation algorithm to decompose the I map, Aop map, and Dolp map of the detected visible polarization part into high-frequency information and low-frequency information in the frequency domain; Step 12: Use the trained BP neural network to predict the fusion coefficient for achieving overall optimization of the interest index based on the high-frequency information and low-frequency information of the decomposed I graph, Aop graph, and Dolp graph; Step 13: multiply the high-frequency information and low-frequency information of the decomposed I-image, Aop-image, and Dolp-image by the corresponding fusion coefficients and then add them together to obtain the fused high-frequency information and low-frequency information; Step 14: Perform inverse NSST transformation on the fused high-frequency information and low-frequency information to obtain a fused visible polarization image; In this specific embodiment, the I image of visible polarization is the intensity information of the polarization image, which includes the basic structure and brightness distribution of the scene, the Aop image is the polarization angle information, which mainly shows the surface shape and roughness of the target in the scene to be measured, and the Dolp image is the polarization degree information, which mainly shows the reflection characteristics of the target in the scene to be measured. Fusion of the I image, the Aop image, and the Dolp image can improve the comprehensive quality of the visible polarization image, combine the main information in the scene to be measured, the main shape and the degree of reflection of the target, reduce noise and error, enhance the functions of cloud penetration and fog penetration and weak light imaging, improve the ability to identify the target, and provide a more comprehensive scene analysis.

[0046] Step 15: pre-process the detected infrared spectrum image to obtain an infrared image; Step 16: Use a bilateral filter to smooth the infrared image, and superimpose the smoothed image with the original image to obtain an infrared image to be fused after edge enhancement; In this specific embodiment, the bilateral filter can retain the edge information of the image while smoothing the image. The bilateral filter is used to filter the medium-wave infrared image, smooth the medium-wave infrared image, and then the smoothed image is superimposed on the source image to achieve edge enhancement of the medium-wave infrared image. In a preferred embodiment of the present invention, during the fusion of the medium-wave infrared image and the visible polarization fusion image, the edge information of the medium-wave infrared image needs to be retained to highlight the thermal radiation information of the target in the fused image and enhance the contrast of the image.

[0047] Step 17: aligning the fused visible polarization image with the infrared image to be fused, and cropping the visible polarization image to obtain the visible polarization image to be fused; In this specific embodiment, the infrared image is a medium-wave infrared spectral image, which includes medium-wave infrared imaging information and medium-wave infrared spectral information. The medium-wave infrared imaging information mainly records the thermal radiation information of the target in the scene to be measured, and has all-weather imaging and a certain penetration ability; the medium-wave infrared spectral information records the spectral information of the target in the scene to be measured, and can assist in the identification of target materials and components; the fused visible polarization image is fused with the medium-wave infrared spectral image, which can enhance the capture of thermal radiation information on the basis of the original visible polarization fusion image, highlight targets with different temperatures from the ambient temperature such as the human body and moving vehicles, enhance image details and contrast, and further enhance the ability of weak light imaging and penetrating clouds and fog.

[0048] Step 18: Using the NSST algorithm, the visible polarization image to be fused and the infrared image to be fused are decomposed into high-frequency information and low-frequency information in the frequency domain; Step 19: predicting the fusion coefficient of the high-frequency information and the low-frequency information through a BP neural network; the fusion coefficient is used to achieve overall optimization of the interest index; In this specific embodiment, the steps of BP neural network training and prediction of fusion coefficient are as follows: Step 31: Calculate the interest index value of the image to be fused; Step 32: obtaining enough groups of random numbers as fusion coefficients of the images to be fused by obtaining random numbers, and these coefficients are used as the output layer of the training set; Step 33: Calculate the interest index value of the fused image according to the obtained several groups of random numbers; Step 34: The interest index value of the image to be fused obtained in step 34 and the interest index value of the fused image obtained in step 33 are used as the input layer of the training set; Step 35: Select 30% of the data in the training set as the test set to verify the performance of the neural network; Step 36: The expected maximum interest index value of the fused image is used as the input layer to predict the optimized fusion coefficient for achieving overall optimization of the interest index.

[0049] S20. multiplying the high-frequency information and the low-frequency information by the fusion coefficient and then adding them together to obtain fused high-frequency information and fused low-frequency information respectively; S21. Perform inverse NSST transformation on the fused high-frequency information and the fused low-frequency information to obtain a fused image.

[0050] In a preferred embodiment of the present invention, NSST performs non-subsampled pyramid decomposition on the input image, and then performs non-subsampled shearlet transform on the obtained high-frequency subbands, decomposing the image in multiple scales and directions, which can better retain the high-frequency information such as the edge, texture, and details of the image. The use of non-subsampled transform avoids the loss of image details during reconstruction and has strong noise resistance. Compared with the non-subsampled contourlet transform NSCT, it avoids multi-level processing and improves the transformation speed. In the fusion of multi-band and multi-dimensional optical information, the details of the source images of different bands and dimensions can be well preserved.

[0051] In this specific embodiment, Figure 2 As shown in the figure, taking the fusion of visible polarization image and mid-infrared spectrum image as an example, information entropy (EN), average gradient (AG), standard deviation (STD) are used as interest indicators for the fusion of visible polarization intensity map (I map), visible polarization angle map (Aop map), and visible polarization degree map (Dolp map), and structural similarity (SSIM), information entropy (EN), peak signal-to-noise ratio (PSNR) are used as interest indicators for the fusion of visible polarization fusion map and mid-infrared spectrum image. The fusion coefficients of visible polarization image fusion and visible polarization fusion map and mid-infrared spectrum image are solved by the trained BP neural network. The non-subsampled shearlet transform (NSST) combined with the bilateral filter is used to fuse the solved fusion coefficients to obtain the final fused visible polarization-infrared spectrum image. Specifically, the fusion rule is to select several image evaluation indicators of interest as interest indicators, and optimize these interest indicators of the fused image as a whole. Multi-band and multi-dimensional optical information contains multiple information dimensions and spans a large number of bands. Images of different bands and dimensions have different characteristics, and the fusion of images in different application scenarios also focuses on different aspects. According to the characteristics of images of different bands and dimensions and the needs of application scenarios, different image evaluation indicators can be flexibly selected to calculate the fusion coefficient to improve the utilization rate of multi-band and multi-dimensional optical information.

[0052] Specifically, the principle of calculating the fusion coefficient by the fusion rule of the present invention is as follows: Taking visible polarization fusion as an example, there are fusion coefficients for I map, Aop map, and Dolp map , , , , , , respectively correspond to the high-frequency information and low-frequency information of the three pictures. For the selected interest indicators EN, AG, and STD, , , , , , , , , , , , They correspond to the I map, Aop map, Dolp map and the fused visible polarization image F map, respectively. , , Both , , , , , functions, and are independent of each other, and the corresponding relations are as follows:

[0053]

[0054]

[0055] In a preferred embodiment of the present invention, , , Both need to take larger values, because , , are unrelated, so the solution , , , , , The value of constitutes a multi-objective optimization problem with constraints. The constraints are as follows:

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] The trained BP neural network is used to solve the above multi-objective optimization problem with constraints, and a set of , , , , , The solution satisfies the condition that the EN, AG, and STD values ​​of the fused visible polarization image can all achieve maximum values.

[0062] In this specific embodiment, the method for solving the fusion coefficient when the medium-wave infrared image is fused with the fused visible polarization fusion image is the same as the method used for visible polarization fusion. The larger the EN value of the fused image, the richer the image information, the larger the AG value, the clearer the image, the larger the STD value, the better the image contrast, the closer the SSIM value is to 1, the higher the structural similarity between the fused image and the original image, and the larger the PSNR value, the lower the image distortion rate. When the visible polarization fusion part fuses the I image, Aop image, and Dolp image, EN, AG, and STD are used to ensure the information volume, contrast, and clarity of the fused image. When the fused visible polarization image is fused with the medium-wave infrared image, EN, SSIM, and PSNR are used to ensure the information volume and structural similarity of the fused image and avoid introducing too much noise.

[0063] In this specific preferred embodiment, Figure 3 As shown, the interest index value of the image to be fused and the expected maximum selected interest index value of the fused image are used as the input layer, and the fusion coefficient for realizing the overall optimization of the interest index is used as the output layer, and the coefficient is predicted by the trained BP neural network; specifically, the BP neural network used for predicting the fusion coefficient of the visible polarization fusion part is n =12, m =6, Corresponding to the normalized values ​​of EN, AG, and STD of the detected I map, Aop map, and Dolp map , , , , , , , , , the normalized maximum expected value of the visible polarization fusion image EN, AG, and STD after fusion , , , Fusion coefficients corresponding to I-map, Aop-map, and Dolp-map , , , , , . BP neural network n=7, m=4 for predicting the fusion coefficients of the visible polarization fusion image and the medium-wave infrared image fusion part. Corresponding to the normalized values ​​of EN, PSNR, and SSIM of the visible polarization fusion image and the detected medium-wave infrared image , , , , the normalized maximum expected value of EN, PSNR and the maximum theoretical value of SSIM of the final image after fusion , , , Fusion coefficients corresponding to visible polarization fusion image and medium-wave infrared image , , , .

[0064] Accordingly, according to an embodiment of the present invention, the present invention also provides a computer device, a readable storage medium and a computer program product.

[0065] Figure 4 FIG. 1 is a schematic diagram of the structure of a computer device 12 provided in an embodiment of the present invention. Figure 4 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0066] like Figure 4 As shown, computer device 12 is in the form of a general-purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0067] Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , a system memory 28 , and a bus 18 that connects various system components including system memory 28 and processing unit 16 .

[0068] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0069] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0070] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0071] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0072] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0073] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the image fusion method of multi-band multi-dimensional light information based on interest indicators provided in an embodiment of the present invention.

[0074] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the image fusion method of multi-band and multi-dimensional optical information based on interest indicators provided in all the inventive embodiments of the present application is implemented.

[0075] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, an apparatus, or a device.

[0076] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0077] The program code included in the computer-readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect to the Internet).

[0078] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned image fusion method of multi-band and multi-dimensional optical information based on interest indicators.

[0079] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0080] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image fusion method based on interest index, characterized by: The image fusion method based on interest index is used to fuse a plurality of images to be fused; The image fusion method based on interest index comprises the steps of: S1. Select several image evaluation indicators as interest indicators according to the characteristics of the image to be fused and the application scenario requirements; S2. Constructing a multi-objective optimization problem based on the basic fusion method, the interest index and the unknown fusion coefficient; S3. Solve the multi-objective optimization problem through a BP neural network to obtain an optimized fusion coefficient; S4. Input the optimized fusion coefficient as a fusion rule into the basic fusion method to complete the optimization of the interest index and realize image fusion.

2. The image fusion method based on interest index according to claim 1, characterized in that: The plurality of images to be fused include two or more images to be fused, and the plurality of image evaluation indicators include 2 to 5 image evaluation indicators.

3. The image fusion method based on interest index according to claim 1, characterized in that: The expression of the multi-objective optimization problem constructed is: ； in, Represents the expected value of the interest index of the image to be fused, Represents multiple unknown fusion coefficients of an image to be fused, Represents the image quality index of each image to be fused, M is a positive integer, N is a positive integer, and N is the number of unknown fusion coefficients.

4. The image fusion method based on interest index according to claim 1, characterized in that: The training steps of the BP neural network include: S31. Calculate the interest index value of the image to be fused; S32. Obtaining several groups of random numbers as the basic fusion coefficients of the images to be fused, and the basic fusion coefficients are used as the output layer of the training set; S33. Calculating the interest index value of the fused image according to the basic fusion coefficient; S34. Using the interest index value of the image to be fused obtained in S31 and the interest index value of the fused image obtained in S33 as the input layer of the training set; S35. Select 30% of the data in the training set as the test set; through continuous training and verification of performance, obtain the trained BP neural network.

5. The image fusion method based on interest index according to claim 1, characterized in that: The basic fusion method is selected from at least one of weighted fusion, wavelet transform algorithm, NSST transform algorithm, NSCT transform algorithm, Laplace pyramid algorithm, principal component analysis algorithm, total variation fusion or HIS fusion.

6. The image fusion method based on interest index according to claim 5 is characterized in that: The basic fusion method is weighted fusion, then the unknown fusion coefficient satisfies: ； 。 7. The image fusion method based on interest index according to claim 5, characterized in that: The basic fusion method is a wavelet transform algorithm, which includes decomposing the image to be fused into a high-frequency component and a low-frequency component. Then: ； ； ； ； ； in, Characterizing the unknown fusion coefficient of the high frequency component, Characterize the unknown fusion coefficient of the low frequency component.

8. The image fusion method based on interest index according to claim 1, characterized in that: The images to be fused include a first visible polarization image to be fused and an infrared image to be fused; the first interest index includes three evaluation indexes: information entropy EN, average gradient AG, and standard deviation STD; the second interest index includes three evaluation indexes: structural similarity index SSIM, information entropy EN, and peak signal-to-noise ratio PSNR; The image fusion method based on the interest index includes fusing the I image, the Aop image, and the Dolp image in the first visible polarization image to be fused into a fused polarizable image, and fusing the fused polarizable image with the infrared image to be fused; include step: S11. Decomposing the I image, Aop image, and Dolp image in the visible polarization image to be fused into first high-frequency information and first low-frequency information in the frequency domain by using the NSST transformation algorithm; S12. Predicting a first fusion coefficient of the decomposed first high-frequency information and the first low-frequency information through the BP neural network; The first fusion coefficient is used to achieve overall optimization of the first interest indicator; S13. multiplying the first high-frequency information and the first low-frequency information by the first fusion coefficient and then adding them together to obtain fused first high-frequency information and fused first low-frequency information respectively; S14. performing an inverse NSST transformation on the fused first high-frequency information and the fused first low-frequency information to obtain a fused visible polarization image; S15. Preprocessing the detected infrared spectrum image to obtain an original infrared image; S16. smoothing the original infrared image using a bilateral filter, and superimposing the smoothed infrared image with the original infrared image to obtain an infrared image to be fused after edge enhancement; S17. aligning the fused visible polarization image with the infrared image to be fused, and cropping the fused visible polarization image to obtain a second visible polarization image to be fused; S18. Decomposing the second visible polarization image to be fused and the infrared image to be fused into second high-frequency information and second low-frequency information in the frequency domain by using the NSST exchange algorithm; S19. Predicting a second fusion coefficient of the second high-frequency information and the second low-frequency information through a BP neural network; The second fusion coefficient is used to achieve overall optimization of the second interest indicator; S20. multiplying the second high-frequency information and the second low-frequency information by the second fusion coefficient and then adding them together to obtain fused second high-frequency information and fused second low-frequency information respectively; S21. Perform an inverse NSST transformation on the fused second high-frequency information and the fused second low-frequency information to obtain a fused image.

9. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image fusion method based on interest indicators according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the image fusion method based on interest indicators according to any one of claims 1 to 8.

Images