A registration method and device between a visible light image and an infrared image of a circuit board

Abstract

The present invention discloses a registration method and device between a visible light image and an infrared image of a circuit board. The registration method comprises the following steps: using image region block statistical information entropy, and extracting its high entropy information region as the original image for subsequent feature point detection based on a threshold value, thereby improving the quality of feature points while also improving detection efficiency; selecting a method combining the FAST+SIFT algorithm to extract image feature points; adopting an improved SIFT feature point annular descriptor; and completing feature point matching and false matching elimination based on the FLANN matching method and the improved RANSAC algorithm, thereby achieving precise image matching. During image matching, the present invention selects points that better reflect image features for matching, thereby ensuring matching accuracy while improving the problems of the traditional SIFT algorithm, which has slow efficiency and low response strength when extracting feature points. Compared with traditional algorithms, the registration efficiency and accuracy are significantly improved.

Description

Technical Field

[0001] The present invention relates to an infrared and visible light image registration method based on a combination of image entropy and an improved SIFT algorithm in the field of image processing, and a registration device using the registration method, and in particular to a registration method and device between a visible light image and an infrared image of a circuit board. Background Art

[0002] A large number of circuit boards are used in various electronic products and electrical appliances. As their applications become increasingly widespread, their structures and functions become increasingly complex. When a circuit board fails, traditional contact-based diagnostic methods inevitably require significant time and effort. Infrared thermal imaging, a non-contact detection technology, has been successfully applied in many fields, with circuit board fault detection being one of its key applications. Each component on a circuit board emits different amounts of thermal radiation during operation. After capturing an infrared image and performing image processing, the processed infrared image of the faulty circuit board is compared with an infrared image of a healthy circuit board for feature analysis. Intelligent algorithms are then used to determine the location and component of the fault.

[0003] Due to the low resolution of infrared images, a lot of image information is lost during the imaging process, making it extremely difficult to determine the location of circuit board faults and components. Optical imaging, on the other hand, is clear and presents complete image information. Therefore, fusing infrared and optical images can achieve accurate detection of the distribution of fault defects on printed circuit boards. However, due to the different resolutions and shooting angles between infrared thermal imagers and optical cameras, direct fusion of infrared and optical images is not possible. Therefore, it is necessary to study the registration algorithm of infrared and visible light images.

[0004] Image registration is an important part of image processing. It refers to the correspondence between images of the same scene at two different time points. It is a fundamental problem in the field of computer vision research and also an application of computer vision. In recent years, image registration methods based on feature extraction have made rapid progress. Image registration based on feature points is the most widely used in the field of image registration.

[0005] Among many image registration algorithms, the SIFT method has good invariance under image rotation, scale transformation, and affine transformation conditions, making it the most stable algorithm at present. However, the SIFT method has limitations. It needs to process data volumes that are several times larger than the original image, and the feature descriptor generation process is very complex. The high dimensionality of the descriptor leads to a large amount of computation during feature matching. Summary of the Invention

[0006] In response to the technical problems of difficulty in registering infrared and visible light images and the high computational complexity of the SIFT algorithm itself, the present invention makes targeted improvements to the SIFT registration algorithm based on feature points, and proposes a method for registering infrared and optical images based on the combination of image entropy and an improved SIFT algorithm. The process is optimized to improve the algorithm's speed and accuracy. Specifically, the present invention provides a method and device for registering visible light images and infrared images of a circuit board.

[0007] The present invention is implemented by the following technical solution: a registration method between a visible light image and an infrared image of a circuit board, the registration method comprising the following steps:

[0008] Step 1: using a visible light image and an infrared image of the circuit board as input images;

[0009] Step 2: Use non-overlapping sliding windows to traverse both input images and split the windows, calculate the information entropy of the window area after segmentation, and define the local image area with a value above a given preset information entropy threshold as a high entropy area based on the histogram formed by the obtained information entropy, and define the local image area with a value above a given preset information entropy threshold as a high entropy area, and define the local image area with a value below a given information entropy threshold as a low entropy area. The high entropy area is used for subsequent algorithm feature extraction and participates in feature point detection, and the low entropy area does not participate in feature point detection;

[0010] Step 3: Use SIFT+FAST algorithm to detect feature points of the high entropy areas selected in the infrared image and the high entropy areas selected in the visible light image, and select representative points as their respective SIFT feature points;

[0011] Step 4: For the SIFT feature points detected in the two images, construct ring descriptors and perform PCA dimensionality reduction processing to obtain 64-dimensional feature vector descriptors for the visible light image and 64-dimensional feature vector descriptors for the infrared image respectively;

[0012] Step 5: Use Euclidean distance and cosine similarity as similarity measurement indicators of the two images, calculate the Euclidean distance and cosine similarity of the feature vectors of the feature points on the two images, use the nearest neighbor / next nearest neighbor FLANN algorithm to perform initial matching on the reference image and the image to be matched, and use the RANSAC algorithm to eliminate incorrect matches, and finally achieve precise matching between the visible light image and the infrared image.

[0013] As a further improvement of the above scheme, in step 2, the method of screening high entropy areas and low entropy areas using the information entropy threshold includes the following steps:

[0014] Firstly, non-overlapping sliding windows are used to segment visible light images and infrared images respectively. Multiple non-overlapping sliding windows are used to traverse each image, and each image is segmented according to the window size, and the information entropy of each window area is calculated.

[0015] Secondly, according to the histogram formed by obtaining the information entropy, the segmentation threshold, i.e., the information entropy threshold, is set to screen the window area where the information entropy is calculated, and the window area greater than the set information entropy threshold is retained for subsequent SIFT+FAST algorithm feature point extraction, while the window area less than the information entropy threshold is not subjected to subsequent feature point detection.

[0016] As a further improvement of the above scheme, for a discrete two-dimensional image, its information entropy P i,j The calculation formula is:

[0017] P i,j =f(i,j) / W·h

[0018]

[0019] Among them, W and h are the width and height of the image respectively, (i, j) is a two-tuple, i represents the grayscale value of the center of a sliding window, and j is the grayscale mean of the window except the center pixel; f(i, j) represents the number of times the two-tuple (i, j) appears in the entire image, and H is the two-dimensional grayscale entropy of the image.

[0020] As a further improvement to the above solution, in step 3, the detection method of using the SIFT+FAST algorithm to detect feature points in the high entropy area screened out in each image includes the following steps:

[0021] First, construct the Gaussian scale space;

[0022] The Gaussian scale space of an image is defined as the function L(x,y,σ):

[0023] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0024] Where I(x,y) is the input image, G(x,y,σ) is the scale-variable Gaussian function, (x,y) is the coordinate of the point on the image, and σ is the Gaussian blur coefficient. The Gaussian difference pyramid DOG is obtained by subtracting adjacent layers in each group. Subsequent feature point extraction is performed on the DOG pyramid. The formula of the DOG operator D(x,y,σ) is as follows:

[0025] D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ)

[0026] Where k is the proportionality coefficient;

[0027] Secondly, Gaussian scale spatial feature point detection and precise positioning;

[0028] Search all scales and image positions in the Gaussian scale space, locate the extreme point on each layer of the image at all scales, and draw a circle with a radius of 3 with the point as the center. When at least 12 of the 16 pixels on the edge meet the requirements of I x +T1 is larger or larger than I x -T1 hours, then this point is considered as a key point, and then the key point position and scale are accurately determined by fitting a three-dimensional quadratic function, where I x is the pixel value of the detection point, T1 is the pixel range threshold;

[0029] Next, remove low-contrast points and points at the edge of the image;

[0030] These two unstable points are removed by setting the contrast threshold and Hessian matrix;

[0031] Finally, calculate the direction of the feature points;

[0032] The gradient direction characteristics of the neighborhood pixels of the key points are used to achieve rotation invariance of the image; sampling is performed in multiple neighborhood windows centered on the feature point, and the gradient directions of the neighborhood pixels are counted using a histogram; the range of the gradient histogram is 0-360°, with every 45 degrees as a direction, and the histogram is divided into 8 directions, that is, each feature point has 8 gradient direction information; the peak of the histogram represents the main direction of the neighborhood gradient at the feature point, that is, the direction of the feature point; at the same time, the histogram is smoothed using a Gaussian function to reduce the impact of mutations. In the gradient direction histogram, when there is another peak equivalent to 80% of the energy of the main peak, this direction is considered to be the auxiliary direction of the feature point; a feature point may be assigned multiple directions, one main direction and more than one auxiliary direction, to enhance the robustness of matching.

[0033] Furthermore, when removing low-contrast points and points at the edge of the image, the extreme points are accurately calculated to the sub-pixel level by fitting a three-dimensional quadratic function, which is then substituted into the Taylor expansion, and only the first two terms are taken:

[0034]

[0035] in Where represents the offset relative to the interpolation center coordinates (x, y);

[0036] Preset a first contrast threshold, compare and analyze the contrast of the extreme points with the first contrast threshold, and select the extreme points with a contrast greater than the first contrast threshold as candidate feature points; simultaneously preset a second contrast threshold, the second contrast threshold being greater than the first contrast threshold, and continue to save the extreme points with a contrast greater than the second contrast threshold as candidate feature points;

[0037] Get the Hessian matrix H(x,y) of the feature point to be selected:

[0038]

[0039] Tr(H(x,y))=D xx (x,y)+D yy (x,y) represents the sum of the eigenvalues ​​of the matrix H(x,y), Det(H(x,y))=D xx (x,y)D yy (x,y)-(D xy (x,y)) 2 Denotes the determinant of the matrix H(x,y), where D xx (x,y),D xy (x,y),D yy The (x,y) value is obtained by the difference of the corresponding position of the candidate point neighborhood. The principal curvature of Det(H(x,y)) is proportional to the eigenvalue of H(x,y). Let represents the ratio of the maximum eigenvalue to the minimum eigenvalue of H(x,y), then In order to detect whether the principal curvature is below a certain threshold T2, it is only necessary to detect If the above formula is true, the feature point is removed, otherwise it is retained.

[0040] Furthermore, the method for calculating the direction of the feature point includes the following steps:

[0041] For the key points detected in the DOG pyramid, the gradient and direction distribution characteristics of the pixels in the 3σ neighborhood window of the Gaussian pyramid image are collected; the modulus and direction of the gradient are as follows:

[0042]

[0043] θ(x,y)=tan -1 ((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y)))

[0044] Where L(x,y) is the scale-space value of the key point at (x,y), L(x+1,y) is the scale-space value of the key point at (x+1,y), L(x-1,y) is the scale-space value of the key point at (x-1,y), L(x,y+1) is the scale-space value of the key point at (x,y+1), L(x,y-1) is the scale-space value of the key point at (x,y-1), m(x,y) is the gradient modulus, and θ(x,y) is the gradient direction.

[0045] As a further improvement to the above solution, in step 4, the method for obtaining 64-dimensional ring feature vector descriptors of the two images includes the following steps:

[0046] For any feature key point, a circle with a radius of 13 is drawn with the key point as the center in its scale space. Since the gradient distribution weight of the pixel point farther away from the center is smaller, the area is divided into 8 concentric annular circles with radii of 2, 3, 4, 5, 6, 8, 10, and 13 pixels to form 8 sub-areas. The key points in each sub-area have 8 gradient directions. Therefore, there are a total of 8×8=64 data, that is, 64-dimensional SIFT feature vectors.

[0047] As a further improvement to the above solution, in step 5, the matching method for performing initial matching using the FLANN algorithm combining Euclidean distance and cosine similarity includes the following steps:

[0048] After the SIFT feature vectors of the two images are generated, the Euclidean distance and cosine similarity between the feature vectors of the feature points in the two images are calculated. The distance and direction between the feature vectors are used as similarity judgment indicators. The feature points with the smallest distance and cosine similarity above a given threshold are selected as initial matching points. If the ratio of the Euclidean distance between the nearest neighbor and the next nearest neighbor is less than a ratio threshold T3 of 0.77, a pair of correct matching points is determined and incorrect matching points are removed. The matching points in the visible light image and the infrared image are then connected with a line. This achieves image registration.

[0049] The present invention also provides a registration device between a visible light image and an infrared image of a circuit board, the registration device comprising:

[0050] An acquisition module, which is used to acquire infrared images and visible light images of the circuit board;

[0051] An entropy region distinguishing module, which is used to remove low entropy regions according to the size of the image information entropy in the visible light image and the infrared image, and retain high entropy regions for subsequent feature point detection;

[0052] A construction module is used to construct a Gaussian scale space for high entropy areas and to establish image Gaussian pyramids and Gaussian difference pyramids;

[0053] A feature point screening module is used to obtain extreme points in different scale spaces in the Gaussian difference pyramid using a combined FAST+SIFT algorithm, and to accurately locate and screen feature points based on the extreme points;

[0054] The removal module is used to filter and remove unstable points using the threshold method and the Hessian matrix method, including points with low contrast and points at the edge of the image;

[0055] Feature point direction calculation module, which is used to calculate and determine the direction of feature points and construct a 64-dimensional ring descriptor of key points;

[0056] The key point matching module uses the Euclidean distance and cosine similarity between vectors as measurement indicators, applies the fast approximate nearest neighbor search FLANN to perform key point matching, and uses the RANSAC random sampling consensus algorithm to eliminate false matches.

[0057] As a further improvement of the above solution, the configuration device is further configured to adopt the above registration method between the visible light image and the infrared image of any circuit board.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. The image region is divided into blocks to calculate the statistical information entropy, and according to the threshold, the high entropy information area is extracted as the detection target image, which improves the accuracy of matching point pairs and has higher matching accuracy than traditional SIFT matching point pairs.

[0060] 2. The combination of SIFT and FAST improves the problems of slow efficiency and low response intensity when extracting feature points using traditional algorithms, improves the accuracy of matching point pairs, and has higher matching accuracy than traditional SIFT matching point pairs.

[0061] 3. The improved SIFT feature point ring descriptor is used to improve the overall running speed of the algorithm while ensuring the quality of registration.

[0062] When matching images, the present invention selects points that better reflect image features for matching, ensuring matching accuracy while improving the traditional SIFT algorithm's slow efficiency and low response strength when extracting feature points. This significantly improves both registration efficiency and accuracy compared to traditional algorithms. Therefore, the present invention uses visible light and infrared images as experimental data and compares them to the traditional SIFT algorithm. The results show significant improvements in registration efficiency and accuracy compared to traditional algorithms, and have broad application prospects in image fusion, remote sensing image processing, computer vision, visual fields, and power equipment diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of the registration method between the optical image and the infrared image of a circuit board provided by the present invention.

[0064] Figure 2 for Figure 1 Entropy histogram of visible light image.

[0065] Figure 3 for Figure 1 Mid-infrared image entropy histogram.

[0066] Figure 4 for Figure 1 Feature point detection diagram.

[0067] Figure 5 for Figure 1 Schematic diagram of the process of establishing a 64-dimensional ring descriptor.

[0068] Figure 6 This is the registration map of the traditional sift algorithm.

[0069] Figure 7 for Figure 1 Improved sift algorithm registration map.

[0070] Figure 8 The results comparison chart is shown in Figure 2. DETAILED DESCRIPTION

[0071] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0072] It should be noted that when a component is referred to as being "mounted on" another component, it may be on the other component or there may also be a central component. When a component is considered to be "set on" another component, it may be set on the other component or there may also be a central component. When a component is considered to be "fixed to" another component, it may be fixed to the other component or there may also be a central component.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0074] The registration method between the optical image and the infrared image of a circuit board of the present invention mainly includes five steps.

[0075] Step 1: Use the optical image and infrared image of the circuit board as input images to be registered.

[0076] Step 2: Use non-overlapping sliding windows to traverse both the optical image and the infrared image, and split the window, calculate the information entropy of the window area after segmentation, and define the local image area with high information entropy above a preset information entropy threshold as a high entropy area based on the histogram formed by the obtained information entropy, and define the local image area with low information entropy below the information entropy threshold as a low entropy area. The high entropy area is used for subsequent algorithm feature extraction and participates in feature point detection, and the low entropy area does not participate in feature point detection.

[0077] Step 3: Use the SIFT+FAST algorithm to perform feature point detection on the high entropy areas screened out in the optical image and the high entropy areas screened out in the infrared image, and select representative points as their respective SIFT feature points.

[0078] Step 4: Construct ring descriptors for the SIFT feature points of the optical image and the infrared image, and perform dimensionality reduction processing to obtain 64-dimensional feature vector descriptors of the optical image and 64-dimensional feature vector descriptors of the infrared image, respectively.

[0079] Step 5: Use Euclidean distance and cosine similarity as similarity measurement indicators between optical images and infrared images, calculate the Euclidean distance and cosine similarity of the feature vectors of the feature points on the two images, use the nearest neighbor / next nearest neighbor FLANN algorithm to perform initial matching on the optical image and infrared image, and use the RANSAC algorithm to eliminate incorrect matches, and finally achieve precise matching between the optical image and infrared image.

[0080] Of course, you can also refer to Figure 1 , which can be summarized into seven aspects:

[0081] 1) Collect infrared images and visible light images of PCB printed circuit boards;

[0082] 2) According to the size of the image information entropy, remove the low entropy area and retain the high entropy area for subsequent feature point detection;

[0083] 3) Constructing Gaussian scale space for the image and establishing image Gaussian pyramid and Gaussian difference pyramid;

[0084] 4) Using the FAST+SIFT combined algorithm, obtain the extreme points in different scale spaces in the Gaussian difference pyramid, and accurately locate and filter out feature points based on the extreme points;

[0085] 5) Use the threshold method and Hessian matrix method to filter and remove unstable points, including low-contrast points and points at the edge of the image;

[0086] 6) Calculate and determine the direction of the feature points and construct a 64-dimensional ring descriptor of the key points;

[0087] 7) Using the Euclidean distance and cosine similarity between vectors as measurement indicators, the fast approximate nearest neighbor search (FLANN) is used for key point matching, and the RANSAC random sampling consensus algorithm is used to eliminate false matches.

[0088] Next, each step is analyzed in detail.

[0089] For step 1, acquire infrared images and visible light images of the PCB printed circuit board;

[0090] For step 2, the specific method of using the image entropy threshold to screen the high entropy and low entropy window areas is as follows:

[0091] First, a non-overlapping sliding window is used to segment the reference image and the image to be registered. Multiple (e.g., 5*5) non-overlapping sliding windows are used to traverse the image, segmenting the image according to the window size and calculating the information entropy of each small window area. For a discrete two-dimensional image, the information entropy calculation formula is:

[0092] P i,j =f(i,j) / W·h

[0093]

[0094] Among them, W and h are the width and height of the image respectively, (i, j) is a two-tuple, i represents the grayscale value of the center of a sliding window, and j is the grayscale mean of the window except the center pixel; f(i, j) represents the number of times the two-tuple (i, j) appears in the entire image, and H is the two-dimensional grayscale entropy of the image.

[0095] Secondly, the histogram formed by the information entropy obtained (such as Figure 2 and Figure 3 As shown in the figure), set the segmentation threshold, filter the window area where the information entropy is calculated, and retain the window area larger than the set threshold for subsequent SIFT+FAST algorithm feature point extraction, and the window area smaller than the threshold will not be subjected to subsequent feature point detection.

[0096] For step 3, please combine Figure 4 ,The specific method of using SIFT+FAST algorithm to detect feature points in high entropy areas of images is as follows:

[0097] (1) Constructing Gaussian scale space: The Gaussian scale space of an image is defined as the function The Gaussian convolution kernel is the only linear kernel that can achieve scale transformation. (x, y) represents the coordinates of a point on the image, and σ represents the Gaussian blur coefficient. The magnitude of σ determines the smoothness of the image. Large scales correspond to the image's general features, while small scales correspond to its detailed features. The output image is I(x, y), i.e., L(x, y, σ) = G(x, y, σ) * I(x, y), where I(x, y) represents the input image and G(x, y, σ) represents the scale-variable Gaussian function. After creating the image Gaussian pyramid, to effectively detect stable keypoints in scale space, adjacent layers within each group are subtracted to form a Difference of Gaussian (DOG) pyramid. Subsequent feature point extraction is performed on the DOG pyramid.

[0098] (2) Gaussian scale space feature point detection and precise positioning: Search all scales and image positions in the Gaussian scale space, locate the extreme point on each layer of the image at all scales, and draw a circle with a radius of 3 with the point as the center. When at least 12 of the 16 pixels on the edge meet the requirements of I x +T1 is larger or larger than I x If -T1 is less than 0.05, the point is considered as a key point, and then the position and scale of the key point are accurately determined by fitting a three-dimensional quadratic function.

[0099] (3) Remove low-contrast points and points at the edge of the image: remove these two types of unstable points by setting the contrast threshold and Hessian matrix.

[0100] (4) Calculate the direction of the feature point: Utilize the gradient direction characteristics of the pixels in the neighborhood of the key point to achieve the rotation invariance of the image; sample multiple neighborhood windows such as 4×4 with the feature point as the center, and use the histogram to count the gradient direction of the neighborhood pixels; the range of the gradient histogram is 0-360°, and the histogram is divided into 8 directions, that is, each feature point has 8 gradient direction information. The peak of the histogram represents the main direction of the neighborhood gradient at the feature point, that is, the direction of the feature point. At the same time, the histogram is smoothed using a Gaussian function to reduce the impact of mutations. In the gradient direction histogram, when there is another peak with an energy equivalent to 80% of the main peak, this direction is considered to be the auxiliary direction of the feature point. A feature point may be assigned multiple directions, one main direction and one or more auxiliary directions, to enhance the robustness of the matching.

[0101] For step 4, please combine Figure 5 , the specific steps to obtain the 64-dimensional ring feature vector descriptors of the reference image and the image to be matched are as follows:

[0102] For any key point, a circle with a radius of 13 is drawn with the key point as the center in the scale space where it is located. Since the gradient distribution weight of the pixel point farther away from the center is smaller, this area is divided into 8 concentric annular circles with radii of 2, 3, 4, 5, 6, 8, 10, and 13, forming 8 sub-areas. The key points in each sub-area have 8 gradient directions. Therefore, there are a total of 8×8=64 data, that is, 64-dimensional SIFT feature vectors.

[0103] For step 5, the specific method of using the FLANN algorithm combining Euclidean distance and cosine similarity for initial matching is as follows:

[0104] After the SIFT feature vectors of the two images are generated, the Euclidean distance and cosine similarity of the feature vectors of the feature points on the two images are calculated. The distance and direction between the vectors are used as similarity judgment indicators. The feature points with the smallest distance and cosine similarity higher than a given threshold are used as initial matching points. If the ratio of the Euclidean distance between the nearest neighbor and the next nearest neighbor is less than a certain ratio threshold T of 0.77, they are determined to be a pair of matching points and the incorrect matching points are removed. The matching points in the reference image and the image to be registered are then connected with lines to achieve image registration. Please refer to Figure 6 、 Figure 7 、 Figure 8 ,in, Figure 6 Registration map for traditional sift algorithm; Figure 7 for Figure 1 Improved sift algorithm registration map; Figure 8 The results comparison chart is shown in Figure 2.

[0105] The registration method between the optical image and the infrared image of a circuit board of the present invention can be designed as embedded software or non-embedded software when applied. However, no matter how it is designed, it is nothing more than a registration device between the optical image and the infrared image of a circuit board that realizes the following functions.

[0106] The registration device includes an acquisition module, an entropy region differentiation module, a construction module, a feature point screening module, a removal module, a feature point direction calculation module, and a key point matching module.

[0107] The acquisition module is used to capture infrared and visible light images of the circuit board, using the visible light image (or optical image) as the reference image and the infrared image as the image to be matched. The entropy region discrimination module is used to remove low-entropy regions based on the information entropy of the reference and matching images, retaining high-entropy regions for subsequent feature point detection. The construction module is used to construct a Gaussian scale space for the high-entropy regions and create image Gaussian pyramids and Gaussian difference pyramids. The feature point screening module uses a combined FAST+SIFT algorithm to obtain extreme points at different scales within the Gaussian difference pyramid and accurately locate and filter feature points based on these extreme points. The removal module uses thresholding and the Hessian matrix method to filter and remove unstable points, including those with low contrast and those located at the edge of the image. The feature point direction calculation module is used to calculate and determine the directions of feature points and construct a 64-dimensional ring descriptor for the key points. The key point matching module is used to match key points using the Euclidean distance and cosine similarity between vectors as measurement indicators, and uses the fast approximate nearest neighbor search FLANN to perform key point matching, and the RANSAC random sampling consensus algorithm to eliminate false matches.

[0108] Image entropy is an estimate of the image's "busyness" expressed as the average number of bits in the image's grayscale levels. It also describes the average amount of information in the image source. Image entropy is a statistical characteristic that reflects the average amount of information in the image and indicates the clustering characteristics of the image's grayscale distribution. A higher image entropy indicates a greater number of high-contrast, high-quality feature points, and vice versa.

[0109] According to the image information entropy, high entropy areas are retained for subsequent feature point detection. Specifically, a non-overlapping sliding window is first used to traverse the visible light image and infrared image of the circuit board and segment the window, and the information entropy of the window area after segmentation is calculated; secondly, a threshold is set according to the information entropy of many local areas of the image, and then an appropriate threshold is selected based on the histogram formed by the obtained information entropy to retain the local image areas with high information entropy, while removing the low information entropy image areas. For the retained image areas, the improved SIFT algorithm is then used to extract feature points.

[0110] The specific process of constructing the Gaussian scale space and establishing the image Gaussian pyramid is as follows: after graying the visible light and infrared images, they are doubled and used as the first layer of the first group of the Gaussian pyramid. The first layer of the first group is located at the bottom of the Gaussian pyramid and is the image that is up-sampled successively. The image after Gaussian convolution of the first layer of the first group is used as the second layer of the first group of pyramids. The Gaussian convolution function is: Then multiply σ by a scaling factor k to get a new smoothing factor σ = k*σ, and use it to smooth the second layer image of the first group. The resulting image is used as the third layer of the first group. Repeat this operation to finally obtain the L layer image in the first group; for the second group of images, downsample the third-to-last layer image of the first group with a scaling factor of 2, and the resulting image is used as the first layer of the second group. Then, perform Gaussian smoothing on the first layer image of the second group with a smoothing factor of σ to obtain the second layer of the second group. Just like the above, the L layer image of the second group is obtained. The size of the images in the same group is the same, but the smoothing scale of the images is different. The corresponding smoothing coefficients are: 0, σ, kσ, 2kσ, k (L-2)σ .

[0111] Construct a Gaussian difference pyramid to detect and accurately locate feature points. Specifically, based on the image Gaussian pyramid constructed in the previous step, the adjacent layers in each group are subtracted to obtain a Gaussian difference pyramid (DOG). Subsequent SIFT feature point extraction is performed on the DOG pyramid. The first layer of the first group of the DOG pyramid is obtained by subtracting the first layer of the first group from the second layer of the first group of the Gaussian pyramid. This process is repeated to form a Gaussian difference pyramid. The Gaussian difference scale space is simplified, and the first layer scale space of the first group in the Gaussian difference scale space is removed. The extreme points are detected in the simplified Gaussian difference scale space. To determine whether a feature point K in a certain layer of the image is a feature point, a circle with a radius of 3 pixels and the point as the center can be drawn. There are 16 pixels on the circumference arc of the circle. By comparing the pixel values ​​of these 16 pixels with the center point to be tested, it is determined whether there are at least 12 consecutive pixels among the 16 pixels on the circumference that are all greater than I. k -t is larger, or both are larger than I k +t is small. If this requirement is met, K is considered a feature point. To reduce feature point detection time, for each point, pixels 1, 5, 9, and 13 are tested at four locations 90 degrees above, below, left, and right. If at least three of these four points meet the requirements, the point is tested across all 16 pixels in the area. Otherwise, the point is considered a non-feature point and is directly removed.

[0112] The specific process of removing low-contrast points and points at the edge of the image is: use a three-dimensional quadratic function to fit the extreme points to the sub-pixel level, substitute it into the Taylor expansion, and only take the first two terms: Preset a first contrast threshold, compare and analyze the contrast of the extreme points with the first contrast threshold, and select the extreme points with a contrast greater than the first contrast threshold as candidate feature points; at the same time, preset a second contrast threshold, which is greater than the first contrast threshold, and continue to save the extreme points with a contrast greater than the second contrast threshold as candidate feature points; because the DOG operator will produce a strong edge response, some less stable edge response points should be removed. Obtain the Hessian matrix of the candidate feature points: The D value can be obtained by taking the difference between adjacent pixels, and the eigenvalue of H is proportional to the principal curvature of D. Unstable edge response points are removed by setting the contrast threshold and the Hessian matrix.

[0113] In this embodiment, the Hessian matrix H(x,y) of the feature point to be selected is obtained:

[0114]

[0115] Tr(H(x,y))=D xx (x,y)+D yy (x,y) represents the sum of the eigenvalues ​​of the matrix H(x,y), Det(H(x,y))=D xx (x,y)D yy (x,y)-(D xy (x,y)) 2 Denotes the determinant of the matrix H(x,y), where D xx (x,y),D xy (x,y),D yy The (x,y) value is obtained by the difference of the corresponding position of the candidate point neighborhood. The principal curvature of Det(H(x,y)) is proportional to the eigenvalue of H(x,y). Let represents the ratio of the maximum eigenvalue to the minimum eigenvalue of H(x,y), then In order to detect whether the principal curvature is below a certain threshold T2, it is only necessary to detect If the above formula is true, the feature point is removed, otherwise it is retained.

[0116] Calculate the direction of the key point. Specifically, for the key point detected in the DOG pyramid, collect the gradient and direction distribution characteristics of the pixels in the 3σ neighborhood window of the Gaussian pyramid image. The modulus and direction of the gradient are as follows:

[0117]

[0118] θ(x,y)=tan -1 ((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y)))

[0119] Where L(x,y) is the scale-space value of the key point at (x,y), L(x+1,y) is the scale-space value of the key point at (x+1,y), L(x-1,y) is the scale-space value of the key point at (x-1,y), L(x,y+1) is the scale-space value of the key point at (x,y+1), L(x,y-1) is the scale-space value of the key point at (x,y-1), m(x,y) is the gradient modulus, and θ(x,y) is the gradient direction.

[0120] The operator is rotationally invariant by assigning a directional parameter to each keypoint using the gradient direction distribution of pixels in the keypoint's neighborhood. A gradient histogram is used to determine the keypoint's orientation. Image pixels within a 3σ neighborhood window of a Gaussian pyramid image, centered around the keypoint, are counted. The gradient and orientation of the pixels within the neighborhood are calculated using a histogram. The gradient histogram divides the directional range from 0 to 360° into 36 bins, each containing 10 degrees. The peak direction of the histogram represents the primary direction of the keypoint, with its contribution to the histogram decreasing with distance from the center. The histogram is smoothed using a Gaussian function to reduce the impact of sudden changes. In the gradient direction histogram, if there is another peak with an energy equivalent to 80% of the primary peak, this direction is considered the secondary direction of the feature point. A feature point may be assigned multiple directions: one primary direction and one or more secondary directions, to enhance matching robustness.

[0121] Construct keypoint descriptors and generate feature vectors: A circular descriptor is used. Because the circular descriptor is rotationally invariant, determining the main orientation of the feature points is unnecessary. With the keypoint as the center, a circular window with a radius of 13 is used as the feature point neighborhood. This neighborhood is divided into eight concentric circles with radii of 2, 3, 4, 5, 6, 8, 10, and 13 pixels, or eight subregions. The pixel gradients and directions of all pixels in each circular subregion are calculated in eight directions (one direction every 45°). Therefore, there are a total of 8 × 8 = 64 feature vectors. The feature vectors are sorted and weighted using a Gaussian window. To reduce the negative impact of illumination changes on the matching effect, the feature vectors are normalized.

[0122] The FLANN algorithm is used for key point matching: After the SIFT64-dimensional feature vectors of the two images are generated, the Euclidean distance and cosine similarity between the feature vectors of the feature points on the two images are calculated as similarity metrics. The first two feature points with the closest Euclidean distance to the reference image are found and called the nearest neighbor and the next nearest neighbor. If the distance of the nearest neighbor feature point divided by the distance of the next nearest neighbor feature point is less than a preset ratio threshold, and the cosine similarity is higher than a given threshold, the point group is considered to be matched successfully; otherwise, the feature point is considered to have failed to match, that is, there is no matching point. The matching points in the reference image and the image to be registered are then connected with lines to achieve image registration. After the initial matching of an image is completed, some false matches will occur. To eliminate these false matches, the RANSAC algorithm is used to eliminate false matching point pairs to achieve precise image matching.

[0123] Compared with the prior art, the present invention has the following beneficial effects:

[0124] 1. The image region is divided into blocks to calculate the statistical information entropy, and according to the threshold, the high entropy information area is extracted as the detection target image, which improves the accuracy of matching point pairs and has higher matching accuracy than traditional SIFT matching point pairs.

[0125] 2. The combination of SIFT and FAST improves the problems of slow efficiency and low response intensity when extracting feature points using traditional algorithms, improves the accuracy of matching point pairs, and has higher matching accuracy than traditional SIFT matching point pairs.

[0126] 3. The improved SIFT feature point ring descriptor is used to improve the overall running speed of the algorithm while ensuring the quality of registration.

[0127] This paper uses visible light and infrared images as experimental data and compares them with the traditional SIFT algorithm. The proposed method significantly improves registration efficiency and accuracy compared to the traditional algorithm. It has broad application prospects in image fusion, remote sensing image processing, computer vision, visual fields, and power equipment diagnosis.

[0128] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A registration method between a visible light image and an infrared image of a circuit board, characterized in that: The registration method comprises the following steps: Step 1: using a visible light image and an infrared image of the circuit board as input images; Step 2: Use non-overlapping sliding windows to traverse both input images and split the windows. Calculate the information entropy of the window area after segmentation. Based on the histogram formed by the obtained information entropy, define the local image area with a value above a given preset information entropy threshold as a high entropy area, and the local image area with a value below the given information entropy threshold as a low entropy area. The high entropy area is used for subsequent algorithm feature extraction and participates in feature point detection, while the low entropy area does not participate in feature point detection. Step 3: Use SIFT+FAST algorithm to detect feature points of high entropy areas selected from infrared images and high entropy areas selected from visible light images, and select representative points as their respective SIFT feature points. The feature point detection method includes the following steps: constructing Gaussian scale space; Gaussian scale space feature point detection and precise positioning; removing low contrast points and points at the edge of the image; calculating the direction of the feature point; wherein, the gradient direction characteristics of the pixels in the neighborhood of the key point are used to achieve the rotation invariance of the image; sampling in multiple neighborhood windows centered on the feature point, and using histogram statistics of the neighborhood pixels. The gradient direction of the gradient histogram is 0-360°, with every 45 degrees as a direction. The histogram is divided into 8 directions, that is, each feature point has 8 gradient direction information; the peak of the histogram represents the main direction of the neighborhood gradient at the feature point, that is, the direction of the feature point; at the same time, the histogram is smoothed using a Gaussian function to reduce the impact of mutations. In the gradient direction histogram, when there is another peak equivalent to 80% of the energy of the main peak, this direction is considered to be the auxiliary direction of the feature point; a feature point may be assigned multiple directions, one main direction and more than one auxiliary direction, to enhance the robustness of matching; Step 4: For the SIFT feature points detected in the two images, construct ring descriptors and perform PCA dimensionality reduction processing to obtain 64-dimensional feature vector descriptors for the visible light image and 64-dimensional feature vector descriptors for the infrared image respectively; Step 5: Use Euclidean distance and cosine similarity as similarity measurement indicators of the two images, calculate the Euclidean distance and cosine similarity of the feature vectors of the feature points on the two images, use the nearest neighbor / next nearest neighbor FLANN algorithm to perform initial matching on the reference image and the image to be matched, and use the RANSAC algorithm to eliminate incorrect matches, and finally achieve precise matching between the visible light image and the infrared image.

2. The registration method between a visible light image and an infrared image of a circuit board according to claim 1, characterized in that: In step 2, the method of using the information entropy threshold to screen high entropy areas and low entropy areas includes the following steps: Firstly, non-overlapping sliding windows are used to segment visible light images and infrared images respectively. Multiple non-overlapping sliding windows are used to traverse each image, and each image is segmented according to the window size, and the information entropy of each window area is calculated. Secondly, according to the histogram formed by obtaining the information entropy, the segmentation threshold, i.e., the information entropy threshold, is set to screen the window area where the information entropy is calculated, and the window area greater than the set information entropy threshold is retained for subsequent SIFT+FAST algorithm feature point extraction, while the window area less than the information entropy threshold is not subjected to subsequent feature point detection.

3. The registration method between a visible light image and an infrared image of a circuit board according to claim 2, characterized in that: For a two-dimensional image in discrete form, its information entropy P i,j The calculation formula is: P i,j =f(i,j) / W·h Among them, W and h are the width and height of the image respectively, (i, j) is a two-tuple, i represents the grayscale value of the center of a sliding window, and j is the grayscale mean of the window except the center pixel; f(i, j) represents the number of times the two-tuple (i, j) appears in the entire image, and H is the two-dimensional grayscale entropy of the image.

4. The registration method between a visible light image and an infrared image of a circuit board according to claim 1, characterized in that: In step three, the detection method includes: The Gaussian scale space of an image is defined as the function L(x,y,σ): L(x,y,σ)=G(x,y,σ)*I(x,y) Where I(x,y) is the input image, G(x,y,σ) is the scale-variable Gaussian function, (x,y) is the coordinate of the point on the image, and σ is the Gaussian blur coefficient. The Gaussian difference pyramid DOG is obtained by subtracting adjacent layers in each group. Subsequent feature point extraction is performed on the DOG pyramid. The formula of the DOG operator D(x,y,σ) is as follows: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ) Where k is the proportionality coefficient; Search all scales and image positions in the Gaussian scale space, locate the extreme point on each layer of the image at all scales, and draw a circle with a radius of 3 with the point as the center. When at least 12 of the 16 pixels on the edge meet the requirements of I x +T1 is larger or larger than I x -T1 hours, then this point is considered as a key point, and then the key point position and scale are accurately determined by fitting a three-dimensional quadratic function, where I x is the pixel value of the detection point, T1 is the pixel range threshold; These two unstable points are removed by setting the contrast threshold and Hessian matrix.

5. The registration method between a visible light image and an infrared image of a circuit board according to claim 4, characterized in that: When removing low-contrast points and points at the edge of the image, the extreme points are accurately calculated to the sub-pixel level by fitting a three-dimensional quadratic function, and then substituted into the Taylor expansion, taking only the first two terms: in Where represents the offset relative to the interpolation center coordinates (x, y); Preset a first contrast threshold, compare and analyze the contrast of the extreme points with the first contrast threshold, and select the extreme points with a contrast greater than the first contrast threshold as candidate feature points; simultaneously preset a second contrast threshold, the second contrast threshold being greater than the first contrast threshold, and continue to save the extreme points with a contrast greater than the second contrast threshold as candidate feature points; Get the Hessian matrix H(x,y) of the feature point to be selected: Tr(H(x,y))=D xx (x,y)+D yy (x,y) represents the sum of the eigenvalues ​​of the matrix H(x,y), Det(H(x,y))=D xx (x,y)D yy (x,y)-(D xy (x,y)) 2 Denotes the determinant of the matrix H(x,y), where D xx (x,y),D xy (x,y),D yy The (x,y) value is obtained by the difference of the corresponding position of the candidate point neighborhood. The principal curvature of Det(H(x,y)) is proportional to the eigenvalue of H(x,y). Let represents the ratio of the maximum eigenvalue to the minimum eigenvalue of H(x,y), then In order to detect whether the principal curvature is below a certain threshold T2, it is only necessary to detect If the above formula is true, the feature point is removed, otherwise it is retained.

6. The registration method between a visible light image and an infrared image of a circuit board according to claim 4, characterized in that: The method for calculating the direction of the feature point includes the following steps: For the key points detected in the DOG pyramid, the gradient and direction distribution characteristics of the pixels in the 3σ neighborhood window of the Gaussian pyramid image are collected; the modulus and direction of the gradient are as follows: θ(x,y)=tan -1 ((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y))) Where L(x,y) is the scale-space value of the key point at (x,y), L(x+1,y) is the scale-space value of the key point at (x+1,y), L(x-1,y) is the scale-space value of the key point at (x-1,y), L(x,y+1) is the scale-space value of the key point at (x,y+1), L(x,y-1) is the scale-space value of the key point at (x,y-1), m(x,y) is the gradient modulus, and θ(x,y) is the gradient direction.

7. The registration method between a visible light image and an infrared image of a circuit board according to claim 1, characterized in that: In step 4, the method for obtaining the 64-dimensional annular feature vector descriptors of the two images includes the following steps: For any feature key point, a circle with a radius of 13 is drawn with the key point as the center in its scale space. Since the gradient distribution weight of the pixel point farther away from the center is smaller, the area is divided into 8 concentric annular circles with radii of 2, 3, 4, 5, 6, 8, 10, and 13 pixels to form 8 sub-areas. The key points in each sub-area have 8 gradient directions. Therefore, there are a total of 8×8=64 data, that is, 64-dimensional SIFT feature vectors.

8. The registration method between a visible light image and an infrared image of a circuit board according to claim 1, characterized in that: In step 5, the matching method for performing initial matching using the FLANN algorithm combining Euclidean distance and cosine similarity includes the following steps: After the SIFT feature vectors of the two images are generated, the Euclidean distance and cosine similarity between the feature vectors of the feature points on the two images are calculated. The distance and direction between the feature vectors are used as similarity judgment indicators. The feature points with the smallest distance and cosine similarity higher than a given threshold are selected as initial matching points. The ratio of the Euclidean distance between the nearest neighbor and the next nearest neighbor is less than a certain ratio threshold T3 of 0.77, and they are determined to be a pair of correct matching points. The wrong matching points are then removed. The matching points in the visible light image and the infrared image are then connected with lines.

9. A registration device for a visible light image and an infrared image of a circuit board, characterized in that: The registration device comprises: An acquisition module, which is used to acquire infrared images and visible light images of the circuit board; An entropy region distinguishing module, which is used to remove low entropy regions according to the size of the image information entropy in the visible light image and the infrared image, and retain high entropy regions for subsequent feature point detection; A construction module is used to construct a Gaussian scale space for high entropy areas and to establish image Gaussian pyramids and Gaussian difference pyramids; A feature point screening module is used to obtain extreme points in different scale spaces in the Gaussian difference pyramid using a combined FAST+SIFT algorithm, and to accurately locate and screen feature points based on the extreme points; The removal module is used to filter and remove unstable points using the threshold method and the Hessian matrix method, including points with low contrast and points at the edge of the image; Feature point direction calculation module, which is used to calculate and determine the direction of feature points and construct a 64-dimensional ring descriptor of key points; The key point matching module uses the Euclidean distance and cosine similarity between vectors as measurement indicators, applies the fast approximate nearest neighbor search FLANN to perform key point matching, and uses the RANSAC random sampling consensus algorithm to eliminate false matches.

10. The registration device between a visible light image and an infrared image of a circuit board according to claim 9, characterized in that: The entropy region differentiation module includes: A non-overlapping sliding window is used to traverse both input images and split the windows. The information entropy of the window area after segmentation is calculated. According to the histogram formed by the obtained information entropy, the local image area with an entropy value above a given preset information entropy threshold is defined as a high entropy area, and the local image area with an entropy value below a given information entropy threshold is defined as a low entropy area. The high entropy area is used for subsequent algorithm feature extraction and participates in feature point detection, and the low entropy area does not participate in feature point detection.

11. The registration device between a visible light image and an infrared image of a circuit board according to claim 9, characterized in that: The building blocks include: The Gaussian scale space of an image is defined as the function L(x,y,σ): L(x,y,σ)=G(x,y,σ)*I(x,y) Where I(x,y) is the input image, G(x,y,σ) is the scale-variable Gaussian function, (x,y) is the coordinate of the point on the image, and σ is the Gaussian blur coefficient. The Gaussian difference pyramid DOG is obtained by subtracting adjacent layers in each group. Subsequent feature point extraction is performed on the DOG pyramid. The formula of the DOG operator D(x,y,σ) is as follows: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y)=L(x,y,kσ)-L(x,y,σ) Where k is the proportionality coefficient.

12. The registration device between a visible light image and an infrared image of a circuit board according to claim 9, characterized in that: The feature point screening module includes: Search all scales and image positions in the Gaussian scale space, locate the extreme point on each layer of the image at all scales, and draw a circle with a radius of 3 with the point as the center. When at least 12 of the 16 pixels on the edge meet the requirements of I x +T1 is larger or larger than I x -T1 hours, then this point is considered as a key point, and then the key point position and scale are accurately determined by fitting a three-dimensional quadratic function, where I x is the pixel value of the detection point, and T1 is the pixel range threshold.

13. The registration device between a visible light image and an infrared image of a circuit board according to claim 9, characterized in that: The feature point direction calculation module includes: The gradient direction characteristics of the neighborhood pixels of the key points are used to achieve rotation invariance of the image; sampling is performed in multiple neighborhood windows centered on the feature point, and the gradient directions of the neighborhood pixels are counted using a histogram; the range of the gradient histogram is 0-360°, with every 45 degrees as a direction, and the histogram is divided into 8 directions, that is, each feature point has 8 gradient direction information; the peak of the histogram represents the main direction of the neighborhood gradient at the feature point, that is, the direction of the feature point; at the same time, the histogram is smoothed using a Gaussian function to reduce the impact of mutations. In the gradient direction histogram, when there is another peak equivalent to 80% of the energy of the main peak, this direction is considered to be the auxiliary direction of the feature point; a feature point may be assigned multiple directions, one main direction and more than one auxiliary direction, to enhance the robustness of matching.

Images