Dual light image matching fusion method and system

Abstract

The application provides a dual-optical image matching fusion method and system, which can effectively eliminate false matching point pairs, quickly and efficiently screen out the best matching point pairs, and improve the image matching fusion effect through the organic fusion of multi-dimensional information such as affine matrix prior information, clustering classification, distance, angle, and inscribed circle area. After the non-device area features are extracted through multi-scale image processing based on the Laplacian pyramid, infrared image binarization, corrosion, sharpening and other processing of the visible light image, the fusion can highlight the thermal radiation information in the infrared image and the appearance information in the visible light. The application can highlight the thermal radiation information in the infrared image and the appearance information of the device in the visible light image in the same video, and can display the fused video when the dual-optical camera performs the inspection task, so as to meet the application requirements of different scenes.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and artificial intelligence technology, and relates to a dual-light image matching and fusion method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Currently, substations mostly use inspection robots or fixed-point cameras to replace manual inspections and monitoring of substations, overcoming the problems of large workload, high risk, low efficiency and poor reliability faced by manual inspections, so as to ensure the safe operation of substations.

[0004] Currently, substations often rely on acquired infrared or visible light images for equipment inspection and diagnosis. Infrared images carry thermal radiation information of the electrical equipment, while visible light images carry appearance information. By registering and fusing these two images, multi-source information can be displayed on a single image. The fused image has higher spatial and spectral resolution than the original image, while highlighting multi-source information features and reducing noise. This facilitates the identification of equipment status and automatic detection and analysis by both the human eye and machines. Furthermore, by fusing dual-light images, it is possible to display and monitor more video feeds even under the limited hardware conditions at the substation site.

[0005] Therefore, some researchers have proposed technical solutions for fusing dual-light images, but all of them have certain problems. For example, Chinese invention patents CN202111074441.2 (a method for matching infrared and visible light images based on multi-feature similarity fusion), CN202010848163.0 (a feature matching method based on the consistency of connection features), and CN202111074441.2 (an infrared and visible light image matching method based on multi-feature similarity fusion) have all improved the matching accuracy to some extent. However, they also have problems such as time consumption, which does not meet the real-time application requirements; unsuitability for all substation scenarios, as robots and fixed-point cameras at substation sites need to perform regular inspection tasks, resulting in the infrared and visible light scenes not being constant each time; the need for known information such as imaging distance, which is inconvenient for on-site implementation and has poor applicability; most current fusion algorithms often rely on manual registration information, which is not automatic registration and requires human intervention or correction; and the algorithms have randomness, meaning that the registration results of two consecutive pairs of heterogeneous images may be different. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a dual-light image matching and fusion method and system. This invention can simultaneously highlight thermal radiation information in infrared images and equipment appearance information in visible light images within the same video feed, enabling dual-light cameras to display fused videos during inspection tasks and meeting the application needs of different scenarios on-site.

[0007] According to some embodiments, the present invention adopts the following technical solution:

[0008] A dual-light image matching and fusion method includes the following steps:

[0009] The acquired visible light and infrared images are preprocessed;

[0010] Extract the contour information of each preprocessed image;

[0011] Extract feature points and obtain a descriptor for each feature point;

[0012] Based on the descriptor information, preliminary matching of feature points is performed;

[0013] The preliminary matching results are classified, and a set of matching point pairs is selected from each class to calculate the affine matrix. Several points in one image are transformed into another image through affine transformation. The distance between the affine points and the origin is calculated. Matching point pairs with a distance lower than the threshold are selected until all pairs are traversed.

[0014] Infrared and visible light images are stitched together into a single image, and matching points are mapped. Based on the coordinates, distance, and angle information of the matching point pairs, a fusion comparison process is performed to eliminate mismatched point pairs.

[0015] Choose the best matching pair from the remaining matching pairs;

[0016] Based on the best matching point pairs, affine transformation and registration processing are performed on the two-light images;

[0017] Based on the registration results, post-processing is performed on the infrared and visible light images of the registered area, respectively;

[0018] Image fusion is achieved by fusing features of non-device regions in the post-processed infrared image into the post-processed visible light image.

[0019] As an alternative implementation, the specific process of preprocessing includes converting the visible light image and the infrared image to grayscale to obtain the corresponding grayscale images, and then normalizing the size of the infrared image and the visible light image based on the minimum height of the image.

[0020] As an alternative implementation, when extracting the contour information of each preprocessed image, an edge detection algorithm is used to binarize the preprocessed visible light image and infrared image respectively.

[0021] As an alternative implementation, when obtaining the descriptor of each feature point, it is necessary to calculate the centroid of the target region of the image to correct the main direction of the feature point.

[0022] As an alternative implementation method, during the initial matching process, the BBF algorithm is used to perform preliminary matching of feature points, and the matching results of the BBF algorithm are then corrected by matching again based on the features of scale invariance and consistent tilt angle.

[0023] As an alternative implementation method, the selection of a set of matching points from each class is achieved by random pairing.

[0024] As an alternative implementation, the specific process of removing mismatched point pairs from the matching point pairs includes, based on the known coordinate information of the matching point pairs in the stitched image, firstly calculating the distance between each matching point pair, obtaining the average distance, and removing erroneous matching point pairs whose difference is greater than a threshold by comparing with the average distance; then calculating the slope of the line connecting the remaining matching point pairs, removing erroneous matching point pairs whose slope difference is greater than a slope threshold; and finally obtaining the correctly matched point pairs.

[0025] As an alternative implementation, the specific process of selecting the best matching point pair from the remaining matching point pairs includes using a clustering method to classify the matching point pairs, selecting one point from each of the different classes to form a triangle, and selecting the pair with the largest inscribed circle area as the best matching point pair.

[0026] As an alternative implementation, the post-processing of the infrared image includes grayscale conversion, binarization, single-channel to three-channel conversion, and erosion processing of the infrared image of the registered area, followed by dilation, negation, erosion, and dilation processing.

[0027] As an alternative implementation, the post-processing of visible light images includes sampling reconstruction processing and sharpening processing.

[0028] A dual-light image matching and fusion system, comprising:

[0029] The preprocessing module is configured to preprocess the acquired visible light and infrared images;

[0030] The feature extraction module is configured to extract the contour information of each preprocessed image, extract feature points, and obtain a descriptor for each feature point;

[0031] The preliminary matching module is configured to perform preliminary matching of feature points based on the descriptor information;

[0032] The affine transformation module is configured to classify the preliminary matching results, select a set of matching point pairs from each class to calculate the affine matrix, transform several points in one image to another image through affine transformation, calculate the distance between the affine points and the origin, select matching point pairs with a distance lower than a threshold, and so on until all pairs are traversed.

[0033] The elimination module is configured to stitch infrared and visible light images into the same image, map each matching point, and perform fusion comparison processing based on the coordinates, distance and angle information of the matching point pairs to eliminate mismatched point pairs.

[0034] The optimal matching point selection module is configured to select the optimal matching point pair from the remaining matching point pairs, and perform affine transformation and registration processing on the two-light image based on the optimal matching point pair.

[0035] The post-processing module is configured to perform post-processing on the infrared and visible light images of the registered area based on the registration results.

[0036] The fusion module is configured to fuse features of non-device regions in the post-processed infrared image into the post-processed visible light image, thereby achieving image fusion.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention innovatively provides a dual-light image matching and fusion method utilizing multi-dimensional information features. It can break free from the constraints of manual registration, overcome the complex application environments caused by the differences in parameters of dual-light cameras from different brands, improve the registration rate of dual-light images with different resolutions, number of channels, and scenes, enhance algorithm efficiency, meet real-time requirements, and simultaneously highlight thermal radiation information in infrared images and equipment appearance information in visible light images in the same video stream. This allows dual-light cameras to display the fused video in real time when performing inspection tasks, meeting the application needs of different scenarios on site.

[0039] This invention innovatively proposes using clustering algorithms to classify matching point pairs. Affine matrix prior information can shorten the number of iterations in the algorithm, effectively reducing processing time and improving algorithm efficiency. By organically fusing multi-dimensional information such as affine matrix prior information, 3-means classification, distance, angle, and inscribed circle area, erroneous matching point pairs can be effectively eliminated, and the best matching point pairs can be quickly and efficiently selected. At the same time, the image matching and fusion effect is improved. By performing multi-scale image processing based on Laplacian pyramid, infrared image binarization, erosion, and sharpening on visible light images to extract non-device region features before fusion, the thermal radiation information in the infrared image and the appearance information in the visible light image can be highlighted. Attached Figure Description

[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0041] Figure 1 This is the overall flowchart of the two-light image matching and fusion method;

[0042] Figure 2 This is a flowchart of the dual-light image matching algorithm;

[0043] Figure 3 This is a flowchart of the dual-light image fusion algorithm. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] like Figure 1 As shown, the two-light image matching and fusion method mainly consists of two parts: two-light image matching and fusion. The fusion of two-light images is based on the two-light image matching process and is achieved by utilizing the best matching point pairs in the two-light image matching.

[0048] The following is a detailed introduction.

[0049] First, we will introduce image matching.

[0050] Specifically, it includes:

[0051] Step 1, Image Preprocessing: First, the visible light image and the infrared image are converted to grayscale to obtain grayscale images. Then, the dimensions of the infrared image and the visible light image are normalized based on the minimum height of the image.

[0052] The minimum height mentioned in this step refers to the smaller of the heights of the infrared image and the visible light image. Regarding distance, if the infrared image height is less than the visible light image height, the infrared image height is used to normalize the size of the visible light image.

[0053] Step 2: Use the Canny or Sobel edge detection algorithm to binarize the normalized infrared and visible light images to obtain the contour information in the images;

[0054] Step 3: Use the SIFT algorithm to extract feature points, and correct the principal direction of the feature points by calculating the centroid of the target region in the image; finally, obtain the descriptor of each feature point.

[0055] The SIFT feature extraction algorithm is used for scale space extremum detection, feature point extraction, calculation of principal direction, and finally obtaining the descriptor of each feature point.

[0056] Step 4: Based on the feature point descriptor information, the BBF algorithm is first used to perform preliminary matching of feature points. Then, the matching results of the BBF algorithm are matched and corrected again based on the features of scale invariance and consistent tilt angle.

[0057] That is, based on the BBF algorithm, the coarse matching results are used to remove erroneous matching point pairs according to the scale invariance and tilt angle consistency characteristics of the matching point pairs.

[0058] Step 5: The K-means algorithm is used to classify matching point pairs into three categories. Then, a random pairing method is used to select a pair of matching point pairs from each category to calculate the affine matrix. The pair of matching point pairs with smaller rotation coefficients are selected based on the rotation parameters in the affine matrix. Then, the three points in image A are transformed into image B through affine transformation. Finally, the distance between the affine points and the origin is calculated, and the pair of matching point pairs with a distance lower than a threshold are selected, until all pairs are traversed. By using k-means classification to classify matching point pairs and using prior information (rotation coefficients) to filter matching point pairs, the algorithm time can be greatly shortened and the algorithm efficiency improved.

[0059] If image A is an infrared image, then image B is a visible light image; if image A is a visible light image, then image B is an infrared image.

[0060] K-means clustering is an iterative clustering algorithm. Its steps are as follows: First, the data is divided into K groups. Then, K objects are randomly selected as initial cluster centers. Next, the distance between each object and each of its sub-cluster centers is calculated, and each object is assigned to the nearest cluster center. Each cluster center and the objects assigned to it represent a cluster. Each time a sample is assigned, the cluster centers are recalculated based on the existing objects in the cluster. This process is repeated until a termination condition is met. The termination condition could be that no (or a minimum number) objects have been reassigned to different clusters, no (or a minimum number) cluster centers have changed, or the sum of squared errors has reached a local minimum.

[0061] Of course, other clustering algorithms can be used in other embodiments.

[0062] Step 6: Stitch the infrared image and the visible light image into the same image, and at the same time map the matching point pairs of the two images accordingly;

[0063] After the preceding steps, which involve calculating the distance between each matching point pair in the stitched image based on the known coordinates of the matching point pairs, the average distance is obtained. By comparing this average distance with the average distance, incorrect matching point pairs with a difference greater than a threshold are eliminated. Then, the slope of the line connecting the remaining matching point pairs is calculated, and incorrect matching point pairs with a slope difference greater than a slope threshold are eliminated. Finally, the correctly matched point pairs are obtained.

[0064] This step can effectively eliminate incorrect matching pairs.

[0065] The image fusion process includes:

[0066] Step 7: Based on the 3-means clustering method, the matching point pairs are classified into 3 categories. By selecting one point from each of the 3 categories to form a triangle, the pair with the largest inscribed circle area is selected as the best matching point pair. The 3-means classification can improve the efficiency of the algorithm and shorten the time consumption. At the same time, using the inscribed circle area parameter can effectively select matching point pairs with large dispersion, which will help to improve the image fusion effect.

[0067] Step 8: Perform registration processing based on the best matching point pair. According to the size information of the infrared image and the visible light image, select the image with the smaller image size and perform an affine transformation on it, that is, perform an affine transformation on the best matching point in image A to image B.

[0068] Step 9: Based on the registration results, the infrared image of the registered area is processed sequentially by grayscale conversion, binarization, single-channel to 3-channel conversion, erosion, etc., which can better extract effective information such as thermal radiation characteristics of the infrared image, while preventing interference from useless information.

[0069] Upsampling and reconstructing the visible light image of the registration area can better extract information such as texture features of the visible light image, while preventing interference from useless information;

[0070] Step 10: Perform dilation, negation, erosion, and dilation on the processed infrared image to extract features of non-device regions; perform sharpening and sharpening on the processed visible light image; finally, fuse the features of non-device regions in the infrared image into the visible light image to complete image fusion; this step can preserve the thermal radiation information in the infrared image and the appearance information in the visible light image, and can simultaneously display the features of dual-light images in one image.

[0071] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0076] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A dual-light image matching and fusion method, characterized in that, Includes the following steps: The acquired visible light and infrared images are preprocessed; contour information of each preprocessed image is extracted; feature points are extracted, and a descriptor for each feature point is obtained; based on the descriptor information, preliminary matching of feature points is performed. The preliminary matching results are classified, and a set of matching point pairs is selected from each class to calculate the affine matrix. Several points in one image are transformed into another image through affine transformation. The distance between the affine points and the origin is calculated. Matching point pairs with a distance lower than the threshold are selected until all pairs are traversed. Infrared and visible light images are stitched together into the same image, and each matching point is mapped. Based on the coordinates, distance, and angle information of the matching point pairs, a fusion comparison process is performed to remove mismatched point pairs. First, the distance between each matching point pair is calculated to obtain the average distance. By comparing with the average distance, erroneous matching point pairs with a difference greater than a threshold are removed. Then calculate the slope of the line connecting the remaining matching point pairs, and remove incorrect matching point pairs whose slope difference is greater than the slope threshold; finally, obtain the correctly matched point pairs. Choose the best matching pair from the remaining matching pairs; Based on the best matching point pairs, affine transformation and registration processing are performed on the two-light images; Based on the registration results, post-processing is performed on the infrared and visible light images of the registered area, respectively; Image fusion is achieved by fusing features of non-device regions in the post-processed infrared image into the post-processed visible light image.

2. The dual-light image matching and fusion method as described in claim 1, characterized in that, The specific preprocessing process includes converting the visible light image and the infrared image to grayscale to obtain the corresponding grayscale images, and then normalizing the size of the infrared image and the visible light image based on the minimum height of the image.

3. The dual-light image matching and fusion method as described in claim 1, characterized in that, When extracting the contour information of each preprocessed image, the edge detection algorithm is used to binarize the preprocessed visible light image and infrared image respectively.

4. The dual-light image matching and fusion method as described in claim 1, characterized in that, When obtaining the descriptor for each feature point, it is necessary to calculate the centroid of the target region of the image to correct the principal orientation of the feature point.

5. The dual-light image matching and fusion method as described in claim 1, characterized in that, initially... During the matching process, the feature points are initially matched, and the initial matching results are then corrected based on features that are scale-invariant and have consistent tilt angles.

6. The dual-light image matching and fusion method as described in claim 1, characterized in that, The selection of a set of matching points from each category is achieved through random pairing.

7. The dual-light image matching and fusion method as described in claim 1, characterized in that, The specific process of selecting the best matching point pair from the remaining matching point pairs includes using clustering methods to classify the matching point pairs, selecting one point from each of the different classes to form a triangle, and selecting the pair with the largest inscribed circle area as the best matching point pair.

8. The dual-light image matching and fusion method as described in claim 1, characterized in that, The post-processing of infrared images includes grayscale conversion, binarization, single-channel to three-channel conversion, and erosion processing of the infrared images of the registered area, followed by dilation, negation, erosion, and dilation processing. The post-processing of visible light images includes sampling reconstruction and sharpening.

9. A dual-light image matching and fusion system, characterized in that, include: The preprocessing module is configured to preprocess the acquired visible light and infrared images; The feature extraction module is configured to extract the contour information of each preprocessed image, extract feature points, and obtain a descriptor for each feature point; The preliminary matching module is configured to perform preliminary matching of feature points based on the descriptor information; The affine transformation module is configured to classify the preliminary matching results, select a set of matching point pairs from each class to calculate the affine matrix, transform several points in one image to another image through affine transformation, calculate the distance between the affine points and the origin, select matching point pairs with a distance lower than a threshold, and so on until all pairs are traversed. The elimination module is configured to stitch infrared and visible light images into a single image and map each matching point. It performs a fusion comparison based on the coordinates, distance, and angle information of the matching point pairs, eliminating mismatched point pairs. First, it calculates the distance between each matching point pair to obtain the average distance, and then eliminates incorrect matching point pairs whose difference exceeds a threshold by comparing the average distance with the average distance. Next, it calculates the slope of the line connecting the remaining matching point pairs and eliminates incorrect matching point pairs whose slope difference exceeds a slope threshold. Finally, it obtains the correctly matched point pairs. The optimal matching point selection module is configured to select the optimal matching point pair from the remaining matching point pairs, and perform affine transformation and registration processing on the two-light image based on the optimal matching point pair. The post-processing module is configured to perform post-processing on the infrared and visible light images of the registered area based on the registration results. The fusion module is configured to fuse features of non-device regions in the post-processed infrared image into the post-processed visible light image, thereby achieving image fusion.

Images