An image fusion method and system based on improved NSCT and PCNN

Abstract

The application discloses an image fusion method and system based on improved NSCT and PCNN, and the method comprises the following steps: acquiring an infrared image and a visible light image and importing the images into a hardware development board; based on the hardware development board, performing registration processing on the infrared image and the visible light image through an image registration algorithm to obtain a registered infrared image and a registered visible light image; based on the hardware development board, performing fusion processing on the registered infrared image and the registered visible light image through an improved NSCT algorithm combined with a PCNN model to obtain a fused image. Through the registration and then fusion mode combined with the hardware development board, the application can better balance the imaging speed and quality, thereby improving the fusion quality of the image. The application can be widely applied to the field of image fusion technology.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to an image fusion method and system based on an improved NSCT and PCNN. Background Technology

[0002] Existing methods for fusion of infrared and visible light images largely rely on contour wave transform. However, contour wave transform can cause spectral aliasing, leading to artifacts. When deploying fusion algorithms on the hardware side, targeted optimization and simplification are necessary based on the fusion objective. Improper handling of the characteristics of infrared and visible light images can result in loss of high-frequency details or noise amplification. Furthermore, especially when dealing with repetitive structures of power equipment in complex valve hall environments, achieving good fusion results requires implementing the algorithm on a PC. For portability and real-time performance, the algorithm needs to be deployed on hardware. However, infrared and visible light image fusion algorithms are highly complex, often using PCs as image processing platforms. But PCs suffer from high power consumption and large size, making them inconvenient to carry. In power equipment inspection applications, they cannot meet the requirements of real-time performance and portability. Moreover, existing technologies do not adequately balance the imaging speed of image fusion algorithms in hardware deployment, resulting in unsatisfactory final image fusion quality. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide an image fusion method and system based on an improved NSCT and PCNN. By employing a registration-then-fusion approach combined with a hardware development board, it can better balance imaging speed and quality, thereby improving the fusion quality of the images.

[0004] The first technical solution adopted in this invention is: an image fusion method based on improved NSCT and PCNN, comprising the following steps:

[0005] Acquire infrared and visible light images and import them into the hardware development board;

[0006] Based on the hardware development board, an image registration algorithm is used to register infrared and visible light images to obtain the registered infrared and visible light images.

[0007] Based on a hardware development board, the registered infrared image and the registered visible light image are fused using an improved NSCT algorithm combined with a PCNN model to obtain a fused image.

[0008] Furthermore, the step of registering the infrared image and the visible light image using an image registration algorithm based on a hardware development board to obtain the registered infrared image and the registered visible light image specifically includes:

[0009] Based on the PS end of the hardware development board, the infrared image and the visible light image are processed into grayscale to obtain the grayscale infrared image and the grayscale visible light image.

[0010] The feature triangles of the infrared image and the visible light image are obtained by performing feature corner detection processing on the grayscale infrared image and the grayscale visible light image respectively using the curvature scale space corner detection algorithm.

[0011] The feature triangles of the infrared image and the visible light image are processed by contour angle calculation to obtain the feature descriptors of the infrared image and the visible light image respectively;

[0012] Based on rotation angle estimation and linear consistency rules, coarse registration processing is performed on the feature descriptors of infrared images and visible light images to obtain preliminary registered infrared images and preliminary registered visible light images.

[0013] The preliminary registered infrared image and the preliminary registered visible light image are finely registered using the least squares method to obtain the registered infrared image and the registered visible light image.

[0014] Furthermore, the step of performing coarse registration processing on the feature descriptors of the infrared image and the visible light image based on rotation angle estimation and linear consistency rules to obtain a preliminary registered infrared image and a preliminary registered visible light image specifically includes:

[0015] The feature descriptors of infrared images and visible light images are derived using a bilateral registration algorithm to obtain the principal orientation angles of the feature descriptors of infrared images and visible light images. The principal orientation angles of the feature descriptors of infrared images include a first principal orientation angle and a second principal orientation angle, and the principal orientation angles of the feature descriptors of visible light images include a first principal orientation angle and a second principal orientation angle.

[0016] The feature descriptor rotation angle of the infrared image is obtained by subtracting the first principal orientation angle and the second principal orientation angle of the feature descriptor of the infrared image.

[0017] The rotation angle of the feature descriptor of the visible light image is obtained by subtracting the first principal orientation angle and the second principal orientation angle of the feature descriptor of the visible light image.

[0018] Based on the linear consistency rule, the rotation angles of the feature descriptors of the infrared image and the visible light image are rotated proportionally to obtain preliminary registered infrared and visible light images.

[0019] Furthermore, the step of performing fine registration processing on the initially registered infrared image and the initially registered visible light image based on the least squares method to obtain the registered infrared image and the registered visible light image specifically includes:

[0020] The optimal projection matrix of the infrared image and the optimal projection matrix of the visible light image are obtained by solving the preliminary registered infrared image and the preliminary registered visible light image using the least squares method.

[0021] Feature points of infrared images are obtained based on the optimal projection matrix of infrared images, and feature points of visible light images are obtained based on the optimal projection matrix of visible light images.

[0022] The fitting points of the infrared image are obtained based on the feature points of the infrared image, and the fitting points of the visible light image are obtained based on the feature points of the visible light image.

[0023] The residuals of infrared images are obtained by matching feature points with the fitting points of infrared images, and the residuals of visible light images are obtained by matching feature points with the fitting points of visible light images.

[0024] The residuals of the infrared image and the residuals of the visible light image are sorted in descending order, and the feature points corresponding to the top 25% are selected for fitting point replacement.

[0025] The process involves repeatedly performing fine registration steps on the initially registered infrared image and the initially registered visible light image until the sum of the residuals is zero, and then outputting the registered infrared image and the registered visible light image.

[0026] Furthermore, the step of fusing the registered infrared image and the registered visible light image using a modified NSCT algorithm combined with a PCNN model based on a hardware development board to obtain the fused image specifically includes:

[0027] Based on the PL end of the hardware development board, the registered infrared image and the registered visible light image are decomposed by the improved NSCT algorithm to obtain infrared multi-directional sub-band feature images and visible light multi-directional sub-band feature images.

[0028] The infrared multi-directional sub-band feature image is filtered by the CLAHE histogram equalization algorithm to obtain the low-frequency component and high-frequency component of the infrared image.

[0029] The low-frequency component and high-frequency component of the visible light image are obtained by filtering the visible light multi-directional sub-band feature image using the Sobel operator.

[0030] The PCNN model is simplified to construct a simplified PCNN model;

[0031] The low-frequency components of infrared images and visible light images are fused based on a simplified PCNN model to obtain the low-frequency fused components of the image.

[0032] The high-frequency components of the infrared image and the visible light image are fused based on the simplified PCNN model to obtain the high-frequency fused components of the image.

[0033] The low-frequency fusion component and the high-frequency fusion component of the image are reconstructed by the improved NSCT algorithm to obtain the fused image.

[0034] Furthermore, the improved NSCT algorithm includes a dual-channel two-dimensional filter and a non-downsampling directional filter bank. The dual-channel two-dimensional filter is used to decompose the registered infrared image and the registered visible light image, and the non-downsampling directional filter bank is used to reconstruct the low-frequency fusion component and the high-frequency fusion component of the image.

[0035] Furthermore, the step of filtering the infrared multi-directional sub-band feature image using the CLAHE histogram equalization algorithm to obtain the low-frequency and high-frequency components of the infrared image specifically includes:

[0036] The infrared multi-directional sub-band feature image is divided into blocks using the CLAHE histogram equalization algorithm to obtain the block histogram of the infrared image.

[0037] The gray-level distribution function of the block histogram of the infrared image is obtained and a contrast constraint is introduced for cropping to obtain the block histogram of the cropped infrared image.

[0038] The block histogram of the cropped infrared image is normalized and mapped to obtain the low-frequency component and high-frequency component of the infrared image.

[0039] Furthermore, the step of filtering the visible light multi-directional sub-band feature image using the Sobel operator to obtain the low-frequency component and high-frequency component of the visible light image specifically includes:

[0040] The gradient magnitude and pixel edge direction of the visible light multi-directional sub-band feature image are calculated by using the Sobel operator to obtain the gradient magnitude and pixel value of the visible light image.

[0041] Based on the gradient magnitude and pixel values ​​of the visible light image, the low-frequency and high-frequency components of the visible light image are determined.

[0042] Furthermore, the simplified expression of the PCNN model is as follows:

[0043] T ij [n] = T ij [n-1]+G ij [n]

[0044] In the above formula, G ij [n] represents one ignition pulse output, T ij [n] represents the total output of n ignitions, T ij [n-1] represents the total output of n-1 ignitions.

[0045] The second technical solution adopted in this invention is: an image fusion system based on improved NSCT and PCNN, comprising:

[0046] The acquisition module is used to acquire infrared and visible light images and import them into the hardware development board.

[0047] The registration module is used to register infrared images and visible light images based on the hardware development board using image registration algorithms, so as to obtain registered infrared images and registered visible light images.

[0048] The fusion module is used to fuse registered infrared and registered visible light images based on a hardware development board using an improved NSCT algorithm combined with a PCNN model to obtain a fused image.

[0049] The beneficial effects of the method and system of this invention are as follows: Based on a hardware development board, this invention performs registration processing on infrared and visible light images using an image registration algorithm. Based on the extracted feature points, the two images are appropriately rotated and cropped to improve image data quality before image fusion. Furthermore, a modified NSCT algorithm combined with a PCNN model is used to fuse the registered infrared and visible light images. Based on NSCT decomposition, the dynamic thresholding capability of PCNN is used to select and output image data. PCNN considers the local environment of each pixel during processing, dynamically adjusting the activation threshold of neurons to output high-quality fusion results. By registering before fusion, the feature information of the image is better emphasized, allowing for better balance between imaging speed and quality when deployed on a hardware development board. Attached Figure Description

[0050] Figure 1This is a flowchart of the steps of an image fusion method based on an improved NSCT and PCNN according to the present invention;

[0051] Figure 2 This is a structural block diagram of an image fusion system based on an improved NSCT and PCNN according to the present invention;

[0052] Figure 3 This is a schematic diagram of the internal data processing flow based on the ZYNQ development board provided in a specific embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram showing the comparison results of infrared images before and after grayscale processing according to a specific embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram showing the comparison results of visible light images before and after grayscale processing according to a specific embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram illustrating the process of image feature corner detection provided in a specific embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram illustrating the process of calculating the contour angle of an image according to a specific embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram of the steps for image registration provided in a specific embodiment of the present invention;

[0058] Figure 9 This is a schematic diagram of the image feature corner detection results provided in a specific embodiment of the present invention;

[0059] Figure 10 This is a schematic diagram of the image registration result provided in a specific embodiment of the present invention;

[0060] Figure 11 This is a schematic diagram of the result of coarse registration of an image provided in a specific embodiment of the present invention;

[0061] Figure 12 This is a schematic diagram illustrating the result of fine image registration provided in a specific embodiment of the present invention;

[0062] Figure 13 This is a schematic diagram of the structure of a dual-channel two-dimensional filter provided in a specific embodiment of the present invention;

[0063] Figure 14 This is a schematic diagram of the structure of the non-downsampling direction filter bank provided in a specific embodiment of the present invention;

[0064] Figure 15 This is a cumulative distribution diagram of CLAHE and bilateral filtering time provided in a specific embodiment of the present invention;

[0065] Figure 16 This is a cumulative distribution diagram of the Sobel and Laplace filtering time provided in a specific embodiment of the present invention;

[0066] Figure 17 This is a schematic diagram of the steps for image fusion provided in a specific embodiment of the present invention;

[0067] Figure 18 This is a schematic diagram of the image fusion experiment results under the first set of devices provided in a specific embodiment of the present invention;

[0068] Figure 19 This is a schematic diagram of the image fusion experiment results under the second set of devices provided in a specific embodiment of the present invention;

[0069] Figure 20 This is a schematic diagram of an objective evaluation simulation of the fusion experiment results of the first set of images provided in a specific embodiment of the present invention;

[0070] Figure 21 This is a schematic diagram illustrating the objective evaluation of the fusion experiment results of the second set of images provided in a specific embodiment of the present invention. Detailed Implementation

[0071] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0072] First, the technical terminology of the embodiments of the present invention will be explained. In the detection of overheating faults in power equipment, heterogeneous images often provide richer information about the equipment's characteristics. For example, infrared images can accurately reflect the temperature changes of the target equipment; visible light images can clearly show the appearance and location features of the equipment surface. After effectively fusing the dual-source images, the temperature information from the infrared image can be used to determine whether a fault has occurred; the detailed information from the visible light image can be used to quickly locate the faulty equipment's number and location, thereby improving detection efficiency and accuracy.

[0073] Furthermore, refer to Figure 1 This invention provides an image fusion method based on an improved NSCT and PCNN, which includes the following steps:

[0074] S1. Acquire infrared and visible light images and import them into the hardware development board;

[0075] Specifically, infrared and visible light images are acquired, and then the images are imported into a hardware development board. In this embodiment of the invention, the ZYNQ development board from MLK, using the XC7Z035 chip, is selected as the development platform for the image fusion and image transmission system. Its advantages lie in its fully programmable system architecture, as well as its ability to complete software and hardware co-design, enabling the software and hardware implementation of the system in a short period of time. The ZYNQ adopts a dual-processor architecture of ARM+FPGA, which meets the requirements of small size, low power consumption, and lightweight embedded devices, and is therefore widely used in the field of digital image processing. The ZYNQ platform can be mainly divided into two categories based on the hardware controller: the PS end and the PL end. The PS end is a processing unit with ARM as its core, while the PL end is an FPGA as its core.

[0076] In a specific embodiment of the present invention, such as Figure 3 As shown, the image is first imported from the SD card to the ARM (PS end), where image registration is performed. The PS end connects to the IP core VDMA on the PL end via the AXI interface, allowing for mutual conversion between AXI Stream and Memory Map data streams to facilitate data exchange between the PS and PL ends. The image fusion module is deployed on the FPGA (PL end). Ethernet and other parameters are configured on the PS end, which uses LwIP (Lightweight IP) to send data to the PC via UDP. Upon receiving the image data stream, the host computer performs packet reception, packet assembly, video display, and saves the data to its local disk.

[0077] S2. Based on the hardware development board, the infrared image and the visible light image are registered using an image registration algorithm to obtain the registered infrared image and the registered visible light image.

[0078] Specifically, firstly, the image is preprocessed, i.e., converted to grayscale. Then, feature corner points of the image are detected based on CSS to construct feature triangles and calculate the principal direction of the contour angles, generating SIFT feature descriptors. Secondly, based on bilateral matching, coarse registration is performed through linear consistency theory and rotation angle estimation to remove obviously erroneous feature point matching pairs. Finally, fine registration is performed through global iterative optimization, completing the registration process from coarse to fine.

[0079] S21. Based on the PS end of the hardware development board, perform grayscale processing on the infrared image and the visible light image to obtain the grayscale infrared image and the grayscale visible light image.

[0080] Specifically, although the pseudo-color image output by the thermal imager has more color information than the single-channel grayscale image, the rich color information occupies too much memory space, which is not conducive to subsequent image registration and fusion calculations.

[0081] In a specific embodiment of the present invention, grayscale processing of infrared images can improve processing speed while preserving infrared image feature information. This embodiment employs a weighted average method to simplify three-channel color information into single-channel component information through weighted value calculation. The formula for its calculation process is shown below:

[0082] O = R * 0.3 + G * 0.59 + B * 0.11

[0083] In the above formula, O represents the pixel value after grayscale processing, and R, G, and B represent the values ​​of the red, green, and blue channels at a certain point in the image, respectively.

[0084] like Figure 4 As shown, Figure 4 (a) represents the original infrared image. Figure 4 (b) represents the infrared image after grayscale transformation, which effectively preserves the target's temperature information while reducing memory usage.

[0085] Furthermore, while RGB images captured by visible light sensors can display rich detail and true color information, they consume a large amount of memory, which is detrimental to subsequent calculations. Therefore, visible light images also need to undergo grayscale transformation to reduce the computational load of the algorithm. The grayscale transformation of visible light images is consistent with the grayscale processing of infrared images. Figure 5 As shown, Figure 5 (a) represents the original visible light image. Figure 5 (b) represents the visible light image after grayscale transformation. Although the visible light image after grayscale transformation is no longer displayed in RGB form, the texture and contour feature information in the image is still preserved.

[0086] S22. The feature corner detection processing of the grayscale infrared image and the grayscale visible light image is performed by the curvature scale space corner detection algorithm to obtain the feature triangle of the infrared image and the feature triangle of the visible light image.

[0087] Specifically, corner detection is one of the effective methods for extracting image features, such as... Figure 6 As shown, Figure 6 (a) Figure 6 (b) Figure 6(c) Three scenarios encountered in corner detection are presented respectively. In the algorithm used in this embodiment, feature corner detection is based on curvature scale space. Curvature scale space (CSS) corner detection is based on a continuous change in scale from high to low, determining its precise location by retrieving curvature extrema on the edge contour. Because it considers multi-scale information, this detection technique has high accuracy and can effectively extract feature corners, thus providing strong support for registration work.

[0088] In a specific embodiment of the present invention, it is set that It is the Gaussian function at different scales The following expression is:

[0089]

[0090] First, search for T-shaped intersections on the contour; if they exist, record them as corner points. Second, calculate the curvature value of any point on the main contour curve at a high scale.

[0091]

[0092] in, This represents convolution calculation, where u represents the arc length parameter. and Gaussian function The first and second derivatives of u.

[0093] The maximum curvature point in the region is searched and used as a candidate special corner point. If the curvature value of a candidate corner point exceeds a set threshold, it is determined to be the correct corner point. Finally, the correct corner point is searched at a lower scale and compared with the T-shaped corner point.

[0094] Feature point detection is performed based on CSS, and the principal direction of the contour angle is calculated for the obtained CSS corner points, such as... Figure 9 The red arrow points to the main direction of this feature point. The number of feature points extracted from infrared images is significantly less than that from visible light images. Figure 9 (a) represents the feature point set of an infrared image. Figure 9 (b) represents the set of feature points in a visible light image.

[0095] S23. Perform contour angle calculation on the feature triangles of the infrared image and the visible light image respectively to obtain the feature descriptors of the infrared image and the visible light image.

[0096] Specifically, the principal direction allocation method is a feature point-based allocation method that can effectively overcome the problem of differences in image perspective. However, in complex valve hall scenes, the infrared and visible light images of electrical equipment with numerous repetitive structures have high similarity, making traditional registration methods unsuitable. Therefore, this embodiment of the invention uses the contour angle principal direction allocation method to register infrared and visible light images. Figure 7 As shown, the contour set M can be obtained through the edge detection algorithm in image preprocessing, which can generally be expressed as:

[0097]

[0098] In the above formula, N M Refers to the number of outlines, Ψ j N refers to the j-th contour in the contour set M, and N refers to the number of pixels in this contour.

[0099] In a specific embodiment of the present invention, it is assumed that the feature points of the contour are... Meanwhile, the left and right feature points of Q are defined as follows: Construct a feature triangle using these three points as vertices. Within the triangle... At the point, i.e., the feature point itself, the angle bisector vector is calculated. Its vector direction is the principal direction at this point. Regarding the direction of the profile angle... The calculation formula is as follows:

[0100]

[0101]

[0102]

[0103]

[0104] Based on the calculation formula for the comprehensive profile angle direction, it can be seen that... Furthermore, the principal direction of the contour angle corresponding to each feature point is uniquely determined.

[0105] Record the feature points and related information extracted in the previous step, save them as SIFT descriptors, and perform bilateral matching on them. Figure 10 The diagram shows the bilateral matching process. During this stage, there are a large number of mismatches of feature points, meaning that the lines connecting the feature points intersect and are not parallel.

[0106] S24. Based on rotation angle estimation and linear consistency rules, coarse registration processing is performed on the feature descriptors of the infrared image and the feature descriptors of the visible light image to obtain a preliminary registered infrared image and a preliminary registered visible light image.

[0107] Specifically, extracting feature points and feature point information in step S23 is only the first step in the registration process. Image registration requires constructing correct correspondences between the acquired feature points. Errors often exist in feature point matching between dual-source images, leading to invalid registration. This embodiment of the invention first performs coarse registration of feature points, using rotation angle estimation and linear consistency rules to remove obvious erroneous matching points. Secondly, a global iterative optimization method is employed to ensure accurate registration.

[0108] First, the rotation angle is estimated, and feature points are set. Two sets of principal orientation angles can be derived based on the bilateral registration algorithm. This allows us to determine the possible rotation angles between the images to be registered:

[0109]

[0110] If the pairing is correct, Δα is approximately equal to the actual chosen angle; if the pairing fails, the value of Δα will be very large.

[0111] Further, based on the continuity theory, let the ratio between the image to be registered and the reference image be λ, which is equal to the area ratio of the same region in the heterogeneous images at the same resolution. Then, a point O can be obtained by rotating it with the same ratio to obtain the corresponding point O'. The solution formula is shown below:

[0112]

[0113]

[0114] Where R refers to the orthogonality matrix when the infrared and visible light images are rotated to the same viewing angle, and α p Refers to the rotation angle, O' 2i Point O 2i The new point position generated after rotating by an angle α0.

[0115] Coarse registration is performed using linear consistency theory and rotation angle estimation to remove obvious erroneous feature points and incorrect matches. Figure 11 The image shows the coarse registration process. Although some incorrect feature point matching still exists, some of the incorrect matching has been removed. The feature point lines intersect in some places, but are mostly parallel.

[0116] S25. Based on the least squares method, perform fine registration processing on the preliminary registered infrared image and the preliminary registered visible light image to obtain the registered infrared image and the registered visible light image.

[0117] Specifically, since registered images often have different resolutions, sub-pixel level errors can lead to localization deviations. Using a global iterative optimization method to reduce feature point positional deviations is called fine registration. First, the optimal projection matrix β is solved using the least squares method, and its expression is as follows:

[0118] minf = ||O c1 β-O c2 ||2

[0119]

[0120] Find O again C2 Fitting point O 2fit :

[0121]

[0122] So O C2 and O 2fit The residual between two points is expressed as:

[0123] e(i) = ||O 2fit (i)-O C2 (i)||2

[0124] Where i = 1, 2, ..., N c N c This refers to the total number of correctly matched feature points. Sort e(i) in order of size, find the largest 1 / 4, and set O(i) to... C2 One-quarter of the elements are replaced by fitted points, and their expression is:

[0125] O C2 ={O C2 (1),....,O C2fit (i),....O C2fit (j),....O C2 (n)}

[0126] Where the fitting point O 2fit The result point with the higher residual value is then updated repeatedly until the sum of the residuals is zero, which means that the exact matching is over and the projection matrix β is obtained.

[0127] The least squares method is used for global iterative optimization to further remove incorrect matches and select the correctly matched feature points. Figure 12 The image shows the fine registration process, which preserves the correct feature point matching. All feature point lines are parallel and do not intersect.

[0128] In summary, the overall process of the image registration algorithm described in the embodiments of the present invention is as follows: Figure 8 As shown, the entire process from extracting feature descriptors and coarse registration to fine registration is realized.

[0129] S3. Based on the hardware development board, the registered infrared image and the registered visible light image are fused using the improved NSCT algorithm and the PCNN model to obtain the fused image.

[0130] Specifically, several image processing methods based on the transform domain all have some shortcomings. To address these issues, Cunha et al. proposed the NSCT algorithm, which not only retains the advantages of contourlet transform in multi-scale and multi-directional decomposition of the original image, but also possesses translation invariance by abandoning downsampling. Based on NSCT decomposition, the dynamic thresholding capability of PCNN is used to select and output image data. PCNN considers the local environment of each pixel during processing, dynamically adjusting the activation threshold of neurons to output high-quality fusion results.

[0131] However, this embodiment of the invention mainly focuses on fusing dual-spectral images of power equipment in a valve hall scene. Since infrared and visible light images have different characteristics, this embodiment of the invention will design different PCNN input methods based on their respective characteristics. By comparing the activation status of neurons at the same location in non-originating images, it is determined which image features need to be preserved. Furthermore, the image fusion algorithm needs to be deployed on the PL end of ZYNQ, so it is necessary to ensure good image fusion results and optimize the algorithm steps as much as possible to save computation time. Therefore, this embodiment of the invention will simplify the PCNN model and adjust and optimize the input model when improving the PCNN excitation source, specifically as follows... Figure 17 As shown.

[0132] S31. Based on the PL end of the hardware development board, the registered infrared image and the registered visible light image are decomposed by the improved NSCT algorithm to obtain infrared multi-directional sub-band feature images and visible light multi-directional sub-band feature images.

[0133] Specifically, the improved NSCT algorithm of this invention is first described. NSCT employs a dual-channel two-dimensional filter structure to achieve multi-scale, multi-directional decomposition and reconstruction of the non-subsampled pyramid (NSP), as shown below. Figure 13 As shown. The structure of NSP is not complex. By simply replacing the downsampling decomposition operations in the Laplace pyramid model with upsampling decomposition, the NSP model can be constructed. Figure 13 In the diagram, x and y correspond to the original and reconstructed signals, respectively, and {H0(z),H1(z)} and {G0(z),G1(z)} correspond to the analysis and synthesis filters, respectively. They satisfy the identity: H0(z)G0(z)+H1(z)G1(z)=1.

[0134] NSCT uses a nonsubsampled directional filter bank (NSDFB) to perform multi-directional decomposition of the high-frequency subbands of NSP's multi-scale decomposition, obtaining multi-directional subband feature images. NSDFB refers to a dual-channel nonsubsampled directional filter bank, which improves upon the directional filter (DFB) by replacing the upsampling and downsampling operations in the contour wave transform with upsampling operations of the corresponding filter, effectively avoiding frequency aliasing caused by the upsampling and downsampling characteristics of the original DFB. The structure of NSDFB is as follows... Figure 14 As shown, {U0(z),U1(z)} and {V0(z),V1(z)} correspond to the analysis and synthesis filters, respectively, and satisfy the conditions for complete reconstruction.

[0135] S32. The infrared multi-directional sub-band feature image is filtered by the CLAHE histogram equalization algorithm to obtain the low-frequency component and high-frequency component of the infrared image.

[0136] First, it's important to clarify that high-frequency components refer to high-scale features in an image, i.e., local detail features such as edge information and texture information. Low-frequency components, on the other hand, refer to low-scale features in an image, i.e., global approximate features, such as areas in the source image with small grayscale variations. As analyzed in the previous section, PCNN determines the fusion model of high and low frequency signals based on the number of firings, and the high and low frequency sub-band coefficients obtained from the NSCT decomposition of the source image are among the input parameters of the PCNN neuron algorithm.

[0137] This invention improves the step of using high-frequency and low-frequency components as input to PCNN neurons. Due to the differences in imaging between infrared and visible light images, this invention designs two filtering algorithms corresponding to the characteristics of each of the two different image sources, and selects the better-performing filter after testing and comparison.

[0138] Specifically, the CLAHE histogram equalization filter is used to smooth the infrared image, maintaining a roughly stable dynamic range by limiting the contrast threshold, while highlighting low-frequency details. The CLAHE algorithm is derived from histogram equalization, therefore, the infrared image first needs to be divided into blocks. Next, the grayscale distribution S(i) of each block is calculated, which can be expressed as:

[0139]

[0140] Where, r j Refers to the j-th gray level, P(r) j ) refers to the frequency of its corresponding gray level, n jThe number of times a pixel appears at gray level j is given, and the total number of pixels is given.

[0141] Next, the histogram contrast of each block is limited to prevent excessive enhancement from amplifying noise. The contrast limiting formula is shown below:

[0142] S'(i) = clip(S(i), limit)

[0143] Where S'(i) are the clipped histogram values, and the clip function ensures that each histogram does not exceed a preset threshold. Then, each clipped histogram is normalized to obtain the distribution function, which can be expressed as:

[0144]

[0145] By remapping the pixel values ​​in each small block according to the distribution function, the expression for pixel p'(x,y) in the infrared image after CLAHE filtering is obtained as follows:

[0146] p'(x,y)=round(C(p(x,y)×(MaxIntensity-1))

[0147] Where MaxIntensity refers to the maximum brightness value within the block, and round refers to the rounding function.

[0148] Furthermore, bilateral filtering considers both spatial proximity and pixel value similarity in the image, thus effectively preserving the richness of low-frequency components in infrared images. The formula for calculating bilateral filtering can be expressed as:

[0149]

[0150]

[0151]

[0152] Where I(k,l) refers to the pixel values ​​of the input image, and F space F is a spatial function of location distance. range It is the pixel value difference function, σ s and σ r These represent the parameters of the corresponding functions. K(x,y) refers to the output of the bilateral filter. After bilateral filtering, the expression for the infrared image pixel p'(x,y) is:

[0153] p'(x,y)=p(x,y)+K(x,y)

[0154] A simulation experiment was conducted on a PC using 42 actual infrared images of power equipment, and the algorithm's execution time was plotted. Figure 15 According to experimental results, the CLAHE algorithm extracts low-frequency signals within approximately 1.7 seconds, while the bilateral filtering process takes 3.4 seconds. Due to the high complexity of the bilateral filtering algorithm, its processing time far exceeds that of the CLAHE algorithm, which does not meet the real-time requirements of this invention. Therefore, this invention selects CLAHE as the low-frequency signal extraction filter for infrared images, and uses the output as the input to the neurons of the PCNN model.

[0155] S33. The visible light multi-directional sub-band feature image is filtered by the Sobel operator to obtain the low-frequency component and high-frequency component of the visible light image.

[0156] Specifically, the Laplacian filter outputs the second derivative value of each pixel in the image, which represents the rate of grayscale change in the image. Grayscale change is directly proportional to the richness of the visible light image; a larger grayscale change indicates more high-frequency signals. Therefore, the Laplacian filter can enhance edges and details in visible light images, making the high-frequency information of the image more prominent. This embodiment of the invention selects the Laplacian filter for sharpening visible light images. Let a binary function be f(x,y), then its Laplacian operator at (x,y) is defined as follows:

[0157]

[0158] For discrete digital images, the Laplacian response of each pixel can be calculated by performing a convolution operation on the original image. The convolution kernel of a Laplacian filter is typically a discrete approximation of the second derivative; however, considering the need for maximum simplicity in algorithm deployment, we will use a different kernel here.

[0159] After Laplace filtering, the expression for pixel p'(x,y) in the visible light image is as follows:

[0160]

[0161] The Sobel operator calculates gradients in the horizontal and vertical directions of a visible light image. A larger gradient indicates a faster change in image grayscale, corresponding to high-frequency signals in the visible light image. Therefore, the Sobel operator can effectively extract high-frequency signals such as contours and textures from visible light images. The Sobel operator first calculates the gradient magnitude G of the image and the edge direction α of each pixel, as shown in the following expression:

[0162]

[0163]

[0164]

[0165]

[0166] Where I(i,j) refers to the pixel value of the image, and G x G y These refer to the gradient values ​​in the horizontal and vertical directions, respectively, G. xij G yij These refer to the Sobel operator convolution kernels in the horizontal and vertical directions, respectively. To simplify calculations, this embodiment of the invention uses the horizontal convolution kernel. vertical direction

[0167] The expression for the visible light image pixel p'(x,y) after processing by the Sobel operator is as follows:

[0168] p'(x,y)=G(x,y)

[0169] Similarly, simulations were performed on visible light images of 42 sets of power equipment on a PC, and the time taken by the two algorithms was recorded, such as... Figure 16 As shown in the figure. Experimental results show that the Laplace filter completes the extraction of low-frequency signals in approximately 0.6 seconds, while the Sobel operator takes about 0.5 seconds to process.

[0170] Since both algorithms perform relatively simple convolution operations on images, their execution times are not significantly different. However, considering the different noise resistance capabilities of the two algorithms—the Sobel operator provides more accurate edge localization in low-noise conditions but may produce unnecessary responses in high-noise conditions; the Laplacian operator has strong noise resistance but is less sensitive to high-frequency signals than the Sobel operator—this embodiment of the invention performs image preprocessing and registration before deploying the fusion algorithm. In these two steps, most image noise can be suppressed, allowing the Sobel operator to achieve better performance after appropriate noise processing. Therefore, this embodiment of the invention selects the Sobel operator as the high-frequency component extraction filter for visible light images, using the output as the input to the PCNN model.

[0171] S34. Simplify the PCNN model and construct a simplified PCNN model;

[0172] Specifically, the PCNN model possesses adaptive characteristics and does not require tedious training, thus it is widely used in the field of image processing. PCNN is a single-layer neural network model composed of three main modules: a branching tree, a modulation coupler, and a pulse generator. The original mathematical model of a PCNN neuron involves numerous parameters. Considering the computing power and processing speed of embedded platforms, the algorithm in this embodiment simplifies the input values ​​and external activation values. The simplified PCNN model is shown below:

[0173] T ij [n] = T ij [n-1]+G ij [n]

[0174] In the neuron calculation formula, G ij [n] refers to the output of a single ignition pulse, T ij [n] refers to the sum of the outputs from n ignitions. Because the calculation formula for PCNN neurons is simplified, using the pixel value of each point in the image to be fused as input, the internal activity terms in the relevant neurons are greater than the dynamic threshold. In the application of the image fusion algorithm in this embodiment of the invention, after the initial ignition is completed, it is only necessary to count the number of ignitions corresponding to each pixel before fusing the source images.

[0175] S35. Based on the simplified PCNN model, the low-frequency components of the infrared image and the low-frequency components of the visible light image are fused to obtain the low-frequency fused components of the image.

[0176] S36. Based on the simplified PCNN model, the high-frequency components of the infrared image and the high-frequency components of the visible light image are fused to obtain the high-frequency fused components of the image.

[0177] S37. The low-frequency fusion component and the high-frequency fusion component of the image are reconstructed using the improved NSCT algorithm to obtain the fused image.

[0178] Finally, a simulation experiment was conducted according to an embodiment of the present invention. Figure 18 As can be seen, (a) is the visible light image of the power equipment numbered AL2, which clearly displays the equipment ID nameplate information "AL2"; (b) is the corresponding infrared image, which shows the heating situation of the power equipment, such as the temperature information of the Y-shaped tube and the complex structure below; (c) is the effect of simulating the fusion algorithm using MATLAB on the PC, which shows that the fused image is relatively clear and the outline features are well preserved; (d) is the result image received and saved in the host computer according to the embodiment of the present invention, which presents both the equipment ID information in the upper right corner of the visible light image and the temperature data in the infrared image. Although the overall visual effect of the result image of the embodiment of the present invention is better than that of Figure (c).

[0179] Depend on Figure 19 As can be seen, (a) is the visible light image of the AL4 power equipment, which clearly shows the equipment ID "AL4" and the supplier name "SIEMENS" in the lower right corner; (b) is the corresponding infrared image, showing the temperature information of the corresponding equipment; (c) is the effect of the fusion algorithm simulation using MATLAB on the PC, showing clear imaging with relatively complete preservation of details and edge features; (d) is the result image saved in the host computer. Observing the image, it can be found that the equipment ID "AL4" and "SIE" in the lower right corner from the visible light, as well as the temperature data of the Y-shaped component in the infrared image, are all presented. Compared with Figure (c), the overall imaging effect of the result image (d) of this invention is better.

[0180] In summary, the objective evaluation metrics for the simulation experiment in this embodiment of the invention are the mainstream evaluation metrics entropy (EN), standard deviation (STD), average gradient (AG), and spatial frequency (SF). These metrics are used to analyze the two sets of experimental images, and the experimental result of fusing the two sets of images is shown in the figure below. Figure 20 , Figure 21 As shown in Tables 1 and 2, the results of the objective evaluation indicators are presented.

[0181] Among them, Figure 20 middle, Figure 20 (a) represents a visible light image. Figure 20 (b) represents an infrared image. Figure 20 (c) represents the contour wave transform algorithm. Figure 20 (d) represents the NP algorithm. Figure 20 (e) represents the algorithm of an embodiment of the present invention.

[0182] exist Figure 21 middle, Figure 21 (a) represents a visible light image. Figure 21 (b) represents an infrared image. Figure 21 (c) represents the contour wave transform algorithm. Figure 21 (d) represents the NP algorithm. Figure 21 (e) represents the algorithm of an embodiment of the present invention.

[0183] Table 1 Objective quality evaluation indicators for the first group of experiments

[0184]

[0185] Table 2 Objective Quality Evaluation Indicators for the Second Group of Experiments

[0186]

[0187] Observing the two sets of experimental results, it is not difficult to find that although the three algorithms used in this embodiment of the invention are all image fusion algorithms based on multi-scale transformation, there are still slight differences in the fused images obtained by the three algorithms. However, in general, all three algorithms can ensure that the features of infrared and visible light images are preserved. Specifically, due to the frequency aliasing problem, the contour wave transform algorithm produces a wavy effect in the fusion result, and the imaging effect is not ideal; while the NP algorithm does not produce wavy effects, its extraction of high and low frequency information is not detailed enough; in order to solve the problem of high and low frequency information extraction, a preprocessing algorithm is added, and the algorithm in this embodiment of the invention shows more detailed image information in the experimental results.

[0188] Reference Figure 2 An image fusion system based on improved NSCT and PCNN includes:

[0189] The acquisition module is used to acquire infrared and visible light images and import them into the hardware development board.

[0190] The registration module is used to register infrared images and visible light images based on the hardware development board using image registration algorithms, so as to obtain registered infrared images and registered visible light images.

[0191] The fusion module is used to fuse registered infrared and registered visible light images based on a hardware development board using an improved NSCT algorithm combined with a PCNN model to obtain a fused image.

[0192] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0193] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image fusion method based on improved NSCT and PCNN, characterized in that, Includes the following steps: Acquire infrared and visible light images and import them into the hardware development board; Based on the hardware development board, an image registration algorithm is used to register infrared and visible light images to obtain the registered infrared and visible light images. Based on the PL end of the hardware development board, the registered infrared image and the registered visible light image are decomposed by the improved NSCT algorithm to obtain infrared multi-directional sub-band feature images and visible light multi-directional sub-band feature images. The infrared multi-directional sub-band feature image is filtered by the CLAHE histogram equalization algorithm to obtain the low-frequency component and high-frequency component of the infrared image. The low-frequency component and high-frequency component of the visible light image are obtained by filtering the visible light multi-directional sub-band feature image using the Sobel operator. The PCNN model is simplified to construct a simplified PCNN model; The low-frequency components of infrared images and visible light images are fused based on a simplified PCNN model to obtain the low-frequency fused components of the image. The high-frequency components of the infrared image and the visible light image are fused based on the simplified PCNN model to obtain the high-frequency fused components of the image. The low-frequency fusion component and the high-frequency fusion component of the image are reconstructed by the improved NSCT algorithm to obtain the fused image. The improved NSCT algorithm includes a dual-channel two-dimensional filter and a non-downsampling directional filter bank. The dual-channel two-dimensional filter is used to decompose the registered infrared image and the registered visible light image, and the non-downsampling directional filter bank is used to reconstruct the low-frequency fusion component and the high-frequency fusion component of the image.

2. The image fusion method based on improved NSCT and PCNN according to claim 1, characterized in that, The step of registering infrared and visible light images using an image registration algorithm based on a hardware development board to obtain registered infrared and visible light images specifically includes: Based on the PS end of the hardware development board, the infrared image and the visible light image are processed into grayscale to obtain the grayscale infrared image and the grayscale visible light image. The feature triangles of the infrared image and the visible light image are obtained by performing feature corner detection processing on the grayscale infrared image and the grayscale visible light image respectively using the curvature scale space corner detection algorithm. The feature triangles of the infrared image and the visible light image are processed by contour angle calculation to obtain the feature descriptors of the infrared image and the visible light image respectively; Based on rotation angle estimation and linear consistency rules, coarse registration processing is performed on the feature descriptors of infrared images and visible light images to obtain preliminary registered infrared images and preliminary registered visible light images. The preliminary registered infrared image and the preliminary registered visible light image are finely registered using the least squares method to obtain the registered infrared image and the registered visible light image.

3. The image fusion method based on improved NSCT and PCNN according to claim 2, characterized in that, The step of performing coarse registration processing on the feature descriptors of the infrared image and the visible light image based on rotation angle estimation and linear consistency rules to obtain a preliminary registered infrared image and a preliminary registered visible light image specifically includes: The feature descriptors of infrared and visible light images are derived using a bilateral registration algorithm to obtain the principal orientation angles of the feature descriptors of infrared and visible light images. The principal orientation angles of the feature descriptors of infrared and visible light images include a first principal orientation angle and a second principal orientation angle, respectively. The feature descriptor rotation angle of the infrared image is obtained by subtracting the first principal orientation angle and the second principal orientation angle of the feature descriptor of the infrared image. The rotation angle of the feature descriptor of the visible light image is obtained by subtracting the first principal orientation angle and the second principal orientation angle of the feature descriptor of the visible light image. Based on the linear consistency rule, the rotation angles of the feature descriptors of the infrared image and the visible light image are rotated proportionally to obtain preliminary registered infrared and visible light images.

4. The image fusion method based on improved NSCT and PCNN according to claim 3, characterized in that, The step of performing fine registration processing on the initially registered infrared image and the initially registered visible light image based on the least squares method to obtain the registered infrared image and the registered visible light image specifically includes: The optimal projection matrix of the infrared image and the optimal projection matrix of the visible light image are obtained by solving the preliminary registered infrared image and the preliminary registered visible light image using the least squares method. Feature points of infrared images are obtained based on the optimal projection matrix of infrared images, and feature points of visible light images are obtained based on the optimal projection matrix of visible light images. The fitting points of the infrared image are obtained based on the feature points of the infrared image, and the fitting points of the visible light image are obtained based on the feature points of the visible light image. The residuals of infrared images are obtained by matching feature points with the fitting points of infrared images, and the residuals of visible light images are obtained by matching feature points with the fitting points of visible light images. The residuals of the infrared image and the residuals of the visible light image are sorted in descending order, and the feature points corresponding to the top 25% are selected for fitting point replacement. The process involves repeatedly performing fine registration steps on the initially registered infrared image and the initially registered visible light image until the sum of the residuals is zero, and then outputting the registered infrared image and the registered visible light image.

5. The image fusion method based on improved NSCT and PCNN according to claim 4, characterized in that, The step of filtering the infrared multi-directional sub-band feature image using the CLAHE histogram equalization algorithm to obtain the low-frequency and high-frequency components of the infrared image specifically includes: The infrared multi-directional sub-band feature image is divided into blocks using the CLAHE histogram equalization algorithm to obtain the block histogram of the infrared image. The gray-level distribution function of the block histogram of the infrared image is obtained and a contrast constraint is introduced for cropping to obtain the block histogram of the cropped infrared image. The block histogram of the cropped infrared image is normalized and mapped to obtain the low-frequency component and high-frequency component of the infrared image.

6. The image fusion method based on improved NSCT and PCNN according to claim 5, characterized in that, The step of filtering the visible light multi-directional sub-band feature image using the Sobel operator to obtain the low-frequency component and high-frequency component of the visible light image specifically includes: The gradient magnitude and pixel edge direction of the visible light multi-directional sub-band feature image are calculated by using the Sobel operator to obtain the gradient magnitude and pixel value of the visible light image. Based on the gradient magnitude and pixel values ​​of the visible light image, the low-frequency and high-frequency components of the visible light image are determined.

7. The image fusion method based on improved NSCT and PCNN according to claim 6, characterized in that, The simplified expression for the PCNN model is shown below: ； In the above formula, This indicates the output of one ignition pulse. express The total output of each ignition cycle. express The total output of each ignition.

8. An image fusion system based on improved NSCT and PCNN, characterized in that, Includes the following modules: The acquisition module is used to acquire infrared and visible light images and import them into the hardware development board. The registration module is used to register infrared images and visible light images based on the hardware development board using image registration algorithms, so as to obtain registered infrared images and registered visible light images. The fusion module, used on the PL end of the hardware development board, decomposes the registered infrared image and the registered visible light image using the improved NSCT algorithm to obtain infrared multi-directional sub-band feature images and visible light multi-directional sub-band feature images. The infrared multi-directional sub-band feature image is filtered by the CLAHE histogram equalization algorithm to obtain the low-frequency component and high-frequency component of the infrared image. The low-frequency component and high-frequency component of the visible light image are obtained by filtering the visible light multi-directional sub-band feature image using the Sobel operator. The PCNN model is simplified to construct a simplified PCNN model; The low-frequency components of infrared images and visible light images are fused based on a simplified PCNN model to obtain the low-frequency fused components of the image. The high-frequency components of the infrared image and the visible light image are fused based on the simplified PCNN model to obtain the high-frequency fused components of the image. The low-frequency fusion component and the high-frequency fusion component of the image are reconstructed by the improved NSCT algorithm to obtain the fused image. The improved NSCT algorithm includes a dual-channel two-dimensional filter and a non-downsampling directional filter bank. The dual-channel two-dimensional filter is used to decompose the registered infrared image and the registered visible light image, and the non-downsampling directional filter bank is used to reconstruct the low-frequency fusion component and the high-frequency fusion component of the image.

Images