Image feature point matching method and device, and nonvolatile storage medium

Abstract

The application discloses a kind of image feature point matching method, device and nonvolatile storage medium.Therein, the method includes: using corner detection algorithm to obtain the first feature point set of target infrared image, and the second feature point set of target visible light source image;According to the first feature point set and the second feature point set, using the Jaccard distance method, obtain the first matching pair set;Determine the second matching pair set based on the slope and length of the line between the first feature point and the corresponding first matching point;Determine the first main direction of the third feature point in the second matching pair set, and the second main direction of the corresponding second matching point;Determine the feature point matching result based on the direction difference between the first main direction and the second main direction.The application solves the technical problems of large image feature point mis-matching probability and low image matching accuracy caused by objective factors such as equipment assembly difference, shooting time, distance and viewing angle difference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image matching, in particular to a method and device for matching image feature points and a nonvolatile storage medium. BACKGROUND

[0002] Existing infrared and visible light image registration methods can be divided into three categories: calibration parameter-based, region-based and feature-based. The calibration parameter-based method is a non-automatic registration method, which can only register a group of images taken at the same time, and its registration accuracy depends on the calibration parameter accuracy. The region-based registration method depends on the linear correlation degree of image gray scale and the degree of field of view overlap, and has poor adaptability to complex scenes with perspective, spectral differences and distortion, and high computational complexity. On the contrary, the feature-based registration method has strong performance stability and is more robust in dealing with various complex image registration scenes, among which the point feature-based registration method is widely applied and researched. However, due to objective factors such as sensor assembly differences, shooting time, distance and perspective differences, there are inevitably scale and perspective differences between images when the inspection robot and the unmanned aerial vehicle take pictures, which leads to a high probability of image feature point misregistration and low image matching accuracy.

[0003] At present, there is no effective solution to the above problems. SUMMARY

[0004] The embodiments of the present application provide a method and device for matching image feature points and a nonvolatile storage medium to at least solve the technical problems of high probability of image feature point misregistration and low image matching accuracy caused by objective factors such as device assembly differences, shooting time, distance and perspective differences.

[0005] According to an aspect of the embodiments of the present application, a method for matching image feature points is provided. The method comprises: obtaining a first set of feature points in a target infrared image and a second set of feature points in a target visible light image by using a corner detection algorithm; determining a first matching point corresponding to a first feature point in the first set of feature points from the second set of feature points by using a Jaccard distance method, and obtaining a first matching pair set; determining a slope of a line connecting the first feature point and the first matching point in the first matching pair set, and a length of the line; determining a second matching pair set from the first matching pair set based on the slope and the length, wherein the second matching pair set comprises a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; determining a first principal direction corresponding to the third feature point and a second principal direction corresponding to the second matching point in the second matching pair set; determining a direction difference between the first principal direction and the second principal direction; and determining a feature point matching result between the target infrared image and the target visible light image based on the direction difference.

[0006] Optionally, the determining of the first matching point corresponding to the first feature point in the first set of feature points from the second set of feature points by using the Jaccard distance method comprises: calculating a Jaccard distance between the first feature point in the first set of feature points and a second feature point in the second set of feature points; and taking the second feature point with the minimum Jaccard distance in the second set of feature points as the first matching point corresponding to the first feature point; and obtaining the first matching pair set based on the first feature point in the first set of feature points and the corresponding second feature point.

[0007] Optionally, the determining of the second matching pair set from the first matching pair set based on the slope and the length comprises: determining a fourth feature point in the first matching pair set, wherein the slope of the fourth feature point is within a preset slope range and the length of the fourth feature point is within a preset length range; and obtaining the second matching pair set based on the fourth feature point in the first matching pair set and a third matching point corresponding to the fourth feature point.

[0008] Optionally, the determining the first principal direction corresponding to the third feature point and the second principal direction corresponding to the second matching point in the second matching pair set comprises: determining a first image contour corresponding to the third feature point in a preset first range and a second image contour corresponding to the second matching point in a preset second range; constructing a first triangle based on the third feature point and the first image contour line; and constructing a second triangle based on the second matching point and the second image contour line; determining a first median direction of the first triangle and a second median direction of the second triangle; obtaining the first principal direction based on the first median direction and obtaining the second principal direction based on the second median direction.

[0009] Optionally, the constructing the first triangle based on the third feature point and the first image contour line and the constructing the second triangle based on the second matching point and the second image contour line comprises: determining a first direction vector between the third feature point and a first end point of the first image contour line, a second direction vector between the third feature point and a second end point of the first image contour line, a third direction vector between the second matching point and a third end point of the second image contour line, and a fourth direction vector between the second matching point and a fourth end point of the second image contour line; constructing the first triangle based on the third feature point, the first direction vector, and the second direction vector; and constructing the second triangle based on the second matching point, the third direction vector, and the fourth direction vector.

[0010] Optionally, the determining the feature point matching result between the target infrared image and the target visible light source image based on the direction difference comprises: determining a first distribution interval of the direction difference corresponding to the third feature point included in the second matching pair set; determining a target distribution interval based on the first distribution interval; determining a fifth feature point in the second matching pair set, wherein the direction difference of the fifth feature point is located in the target distribution interval, and a fourth matching point corresponding to the fifth feature point; and obtaining the feature point matching result based on the fifth feature point and the fourth matching point corresponding to the fifth feature point.

[0011] Optionally, in a case where the first distribution interval is multiple, the determining the target distribution interval based on the first distribution interval comprises: determining a number of the direction difference included in each of the multiple first distribution intervals; taking a first distribution interval with the largest number of the direction difference in the multiple first distribution intervals as a second distribution interval; and obtaining the target distribution interval based on the second distribution interval.

[0012] Optionally, before the first feature point set in the target infrared image and the second feature point set in the target visible light source image are obtained by using the corner point detection algorithm, the method further comprises: obtaining an initial infrared image and an initial visible light source image corresponding to the initial infrared image; performing grayscale processing on the initial infrared image and the initial visible light source image respectively to obtain a first infrared image and a first visible light source image corresponding to the first infrared image; and performing image enhancement processing on the first infrared image and the first visible light source image respectively to obtain the target infrared image and the target visible light source image corresponding to the target infrared image, wherein the target infrared image and the target visible light source image have the same longitudinal resolution.

[0013] According to another aspect of the embodiments of the present application, an image feature point matching device is further provided, which comprises: a first obtaining module, configured to obtain a first feature point set in a target infrared image and a second feature point set in a target visible light source image by using a corner point detection algorithm; a second obtaining module, configured to determine a first matching point corresponding to a first feature point included in the first feature point set from the second feature point set by using a Jaccard distance method to obtain a first matching pair set; a first determining module, configured to determine a slope of a line connecting the first feature point and the corresponding first matching point and a length of the line in the first matching pair set; a second determining module, configured to determine a second matching pair set from the first matching pair set based on the slope and the length, wherein the second matching pair set includes a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; a third determining module, configured to determine a first principal direction corresponding to the third feature point and a second principal direction of the corresponding second matching point in the second matching pair set; a fourth determining module, configured to determine a direction difference between the first principal direction and the corresponding second principal direction; and a fifth determining module, configured to determine a feature point matching result between the target infrared image and the target visible light source image based on the direction difference.

[0014] According to another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, characterized in that the non-volatile storage medium stores a plurality of instructions, and the instructions are adapted to be loaded and executed by a processor to implement any one of the image feature point matching methods.

[0015] In the embodiment of the present application, the first feature point set in the target infrared image and the second feature point set in the target visible light source image are obtained by using the corner detection algorithm; the first matching point corresponding to the first feature point included in the first feature point set is determined from the second feature point set by using the Jaccard distance method, and the first matching pair set is obtained; the slope of the line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and the length of the line are determined; the second matching pair set is determined from the first matching pair set based on the slope and the length, wherein the second matching pair set includes the third feature point and the second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; the first principal direction corresponding to the third feature point and the second principal direction of the corresponding second matching point included in the second matching pair set are determined; the direction difference between the first principal direction and the corresponding second principal direction is determined; the feature point matching result between the target infrared image and the target visible light source image is determined based on the direction difference, which achieves the purpose of accurately obtaining the matching points of infrared images and visible light images, thereby realizing the technical effects of improving image matching accuracy and reducing misfit points, and further solving the technical problems of high image feature point mis-matching probability and low image matching accuracy caused by objective factors such as device assembly difference, shooting time, distance and viewing angle difference. BRIEF DESCRIPTION OF DRAWINGS

[0016] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of an optional image feature point matching method according to an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of an image feature point matching device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0020] It is to be understood that the terms "first", "second", and the like used in the description and the claims of the present application as well as the above-described drawings do not necessarily have to connote any ordinal, sequential or chronological order, but are merely used to distinguish a different set of objects. It is to be understood that the terms so used in the description and the claims are interchangeable under appropriate circumstances and embodiments of the application described herein are capable of operating in other sequences than the one explicitly given in the description and claims. Also, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises, has or includes a list of steps or elements does not necessarily comprise, have or include only those steps or elements but can include additional steps or elements not expressly listed or inherent to such process, method, article or apparatus.

[0021] First, for the convenience of understanding the embodiments of the present application, the following will explain some terms or nouns involved in the present application:

[0022] Jaccard distance (Jaccard similarity coefficient): an index used to measure the difference between two sets.

[0023] Binary Robust Independent Features (BRIEF): an algorithm for generating binary descriptors, which only needs simple Hamming distance matching and can be completed by XOR operation between bits.

[0024] Image registration is the process of matching and superimposing two or more images acquired at different times, different sensors (imaging devices) or different conditions (weather, illumination, camera position and angle, etc.), which has been widely applied in remote sensing data analysis, computer vision, image processing and other fields. In the power system, manual processing of massive image data will occupy a large amount of human resources, resulting in low operation efficiency and resource utilization of the equipment diagnosis system. Therefore, automatic registration of infrared and visible light images of power equipment is of great significance to improve the accuracy and efficiency of the autonomous diagnosis system.

[0025] The purpose of image registration is to obtain the spatial mapping relationship of different images and align the spatial positions of the same target in different images. Image registration is a necessary preprocessing step for image fusion and three-dimensional reconstruction technology, and combined with deep learning image recognition technology, it can fuse the multi-state information (such as type, temperature, mechanical structure, etc.) of the equipment into a single image to improve the degree of equipment information visualization and diagnosis efficiency.

[0026] The existing infrared and visible light image registration methods can be divided into three categories of calibration parameter based, region based and feature based. The calibration parameter based method is a non-automatic registration method, which can only register a group of images taken at the same time, and the registration accuracy depends on the calibration parameter accuracy. The region based registration method depends on the linear correlation degree of image gray and the field of view overlap degree, and has poor adaptability to complex scenes with perspective, spectral difference and distortion, and high computational complexity. On the contrary, the feature based registration method has strong performance stability and is more robust in dealing with various complex image registration scenes, among which the point feature based registration method is widely applied and researched. However, when the inspection robot and the unmanned aerial vehicle take pictures, due to the differences in sensor assembly and the objective factors such as shooting time, distance and perspective, there are inevitably scale and perspective differences between images. And the existing main direction calculation method depends on the image gradient correlation, and the registration accuracy is low in the scene with scale and perspective differences between infrared and visible light images, which further leads to low descriptor correlation in the main direction, and finally cannot accurately match the infrared and visible light images. Further, the image feature point mismatching probability is large, and the image matching accuracy is low.

[0027] Based on the above problems, the embodiment of the present application provides a method for matching image feature points. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0028] Figure 1 The flowchart of the image feature point matching method according to the embodiment of the present application is shown in FIG. 1, which comprises the following steps: Figure 1

[0029] In step S102, an angle point detection algorithm is used to obtain a first feature point set in the target infrared image and a second feature point set in the target visible light image.

[0030] Optionally, the target infrared image and the target visible light image are obtained by preprocessing the initial infrared image and the initial visible light image obtained by using the same image acquisition device. The size of the target infrared image and the target visible light image is the same, and they have the same best matching scale. The angle points in the target infrared image and the visible light image are identified by using the angle point detection algorithm, and the angle points are used as the first feature point set in the target infrared image and the second feature point set in the target visible light image. Thus, the purpose of accurately obtaining the feature point set is achieved.

[0031] ​In an optional embodiment, before the above-mentioned respectively acquiring the first feature point set in the target infrared image and the second feature point set in the target visible light source image by using the corner point detection algorithm, the above-mentioned method further comprises: acquiring an initial infrared image and an initial visible light source image corresponding to the initial infrared image; respectively performing grayscale processing on the initial infrared image and the initial visible light source image to obtain a first infrared image and a first visible light source image corresponding to the first infrared image; respectively performing image enhancement processing on the first infrared image and the first visible light source image to obtain the target infrared image and the target visible light source image corresponding to the target infrared image, wherein the target infrared image and the target visible light source image have the same longitudinal resolution. In the above manner, the input initial infrared image and initial visible light source image obtained by photographing the same power equipment are subjected to grayscale processing to obtain the first infrared image and the first visible light source image with the grayscale value range linearly stretched to [0, 255]. Further, the acquired first infrared image and first visible light source image are respectively subjected to image enhancement processing to obtain the target infrared image and the target visible light source image with the same longitudinal resolution, so as to achieve the purpose of parameter unification of the infrared image and the corresponding visible light source image, improve the image definition, and eliminate the interference caused by the image definition, resolution and other factors on the feature point matching.

[0032] In step S104, the first matching points corresponding to the first feature points included in the first feature point set are determined from the second feature point set by using the Jaccard distance method, and a first matching pair set is obtained.

[0033] In an optional embodiment, the first matching points corresponding to the first feature points included in the first feature point set are determined from the second feature point set by using the Jaccard distance method to obtain the first matching pair set, which comprises: respectively calculating the Jaccard distance between the first feature points included in the first feature point set and the second feature points included in the second feature point set; taking the second feature point with the smallest Jaccard distance in the second feature point set as the first matching point corresponding to the first feature point; and obtaining the first matching pair set based on the first feature points included in the first feature point set and the corresponding second feature points. In the above manner, the matching of the feature points of the visible light source image and the infrared image is performed based on the principle of the smallest Jaccard distance, and the second feature point with the smallest Jaccard distance is taken as the first matching point corresponding to the first feature point. The Jaccard distance is used to measure the distance between the matching point pairs, which improves the correct matching rate in the initial matching point set and improves the accuracy of the feature point mis-matching elimination.

[0034] Optionally, one first feature point is selected from the first feature point set, and the Jacard distance between the one first feature point and each second feature point in the second feature point set is calculated, and the second feature point corresponding to the smallest Jacard distance is taken as the first matching point corresponding to the one first feature point.

[0035] Optionally, one first feature point a is selected from the first feature point set, and one second feature point b is selected from the second feature point set; the feature descriptor algorithm is used to determine the first multi-dimensional two-dimensional vector corresponding to the first feature point a and the second multi-dimensional two-dimensional vector corresponding to the second feature point b; the first dimension number of the corresponding values of the first multi-dimensional two-dimensional vector and the second multi-dimensional two-dimensional vector being 0 is determined , the second dimension number of the corresponding values of the first multi-dimensional two-dimensional vector being 0 and the corresponding values of the second multi-dimensional two-dimensional vector being 1 , the third dimension number of the corresponding values of the first multi-dimensional two-dimensional vector being 1 and the corresponding values of the second multi-dimensional two-dimensional vector being 0 , the fourth dimension number of the corresponding values of the first multi-dimensional two-dimensional vector and the second multi-dimensional two-dimensional vector being 1 ; based on the first dimension number , the second dimension number , the third dimension number , and the fourth dimension number , the Jacard distance between the first feature point a and the second feature point b is obtained by the following method .

[0036] Step S106, the slope of the line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and the length of the line are determined.

[0037] Optionally, the target infrared image and the target visible light original image of the same size are placed at a predetermined plane position (such as adjacent positions on the same plane), for example, the target infrared image and the target visible light original image of the same size are spliced on the same plane, and the feature point matching of the target infrared image and the target visible light original image is performed by using the feature points between the two images which meet certain correlation (such as the correlation of the slope, the correlation of the line length, etc.), so as to improve the accuracy of the feature point matching result.

[0038] Step S108, based on the slope and the length, a second matching pair set is determined from the first matching pair set, wherein the second matching pair set includes a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set.

[0039] In an optional embodiment, the determining, from the first matching pair set, a second matching pair set based on the slope and the length comprises: determining fourth feature points in the first matching pair set that have the slope within a preset slope range and the length within a preset length range; and obtaining the second matching pair set based on the fourth feature points in the first matching pair set and third matching points corresponding to the fourth feature points.

[0040] It should be noted that the target infrared image and the target visible light original image of the same size are placed at a predetermined plane position, and the slope correlation and the length correlation between the feature points and the corresponding matching points are satisfied, that is, the slope of the line connecting the first feature point and the corresponding first matching point in the first matching pair set is the same, in parallel relationship, and the length of the line is equal. Therefore, the fourth feature points in the first matching pair set that have the slope within a preset slope range and the length within a preset length range are removed by using the characteristics of the same scale feature point direction angle and the invariant interval to obtain the second matching pair set, and the accuracy of the matching result of the infrared image and the visible light original image feature points is further improved.

[0041] In step S110, the first main direction corresponding to each of the third feature points included in the second matching pair set and the second main direction corresponding to the second matching points are determined.

[0042] In an optional embodiment, the determining, from the first matching pair set, a second matching pair set based on the slope and the length comprises: determining fourth feature points in the first matching pair set that have the slope within a preset slope range and the length within a preset length range; and obtaining the second matching pair set based on the fourth feature points in the first matching pair set and third matching points corresponding to the fourth feature points.

[0043] Optionally, a first original image contour around (e.g., within a preset first range) the third feature point in the target infrared image is determined, and a second original image contour around (e.g., within a preset second range) the second matching point in the target visible light image is determined. The first original image contour and the second original image contour are respectively subjected to a smoothing filter processing to obtain a first image contour corresponding to the third feature point and a second image contour corresponding to the second matching point. A first triangle is constructed with the tangent of the first image contour as a side and the third feature point as a vertex, and a second triangle is constructed with the tangent of the second image contour as a side and the second matching point as a vertex. The length of the side of the first triangle is determined by a directional vector between the end point of the first image contour and the third feature point, and the length of the side of the second triangle is determined by a directional vector between the end point of the second image contour and the second matching point.

[0044] In an optional embodiment, the constructing the first triangle based on the third feature point and the first image contour line, and the constructing the second triangle based on the second matching point and the second image contour line, comprises: determining a first directional vector between the third feature point and a first end point of the first image contour line, a second directional vector between the third feature point and a second end point of the first image contour line, a third directional vector between the second matching point and a third end point of the second image contour line, and a fourth directional vector between the second matching point and a fourth end point of the second image contour line; constructing the first triangle based on the third feature point, the first directional vector, and the second directional vector; and constructing the second triangle based on the second matching point, the third directional vector, and the fourth directional vector. In this way, the first triangle is constructed with the third feature point as a vertex, and the length of the side of the first triangle is determined by the directional vectors between the end points (i.e., the first end point and the second end point) of the first image contour line and the third feature point; and the second triangle is constructed with the second matching point as a vertex, and the length of the side of the second triangle is determined by the directional vectors between the end points (i.e., the third end point and the fourth end point) of the second image contour line and the second matching point.

[0045] Optionally, the first end point is a starting end point of the first image contour line, and the second end point is a terminal end point of the first image contour line; the third end point is a starting end point of the second image contour line, and the fourth end point is a terminal end point of the second image contour line.

[0046] In step S112, the direction difference between the first main direction and the corresponding second main direction is determined.

[0047] It should be noted that the above target infrared image and the above target visible light original image of the same size are placed at a predetermined plane position, and the direction difference between the first main direction of the third feature point and the second main direction of the corresponding second matching point satisfies a certain correlation, that is, for the feature points matched with each other in the target infrared image and the target visible light original image, the corresponding direction differences should be the same or similar. Based on this, the unmatched points in the second matching pair set are further removed based on the direction difference, so as to improve the accuracy of the feature point matching result.

[0048] In step S114, the feature point matching result between the target infrared image and the target visible light original image is determined based on the above direction difference.

[0049] In an optional embodiment, the determination of the feature point matching result between the target infrared image and the target visible light original image based on the above direction difference includes: determining a first distribution interval of the above direction difference corresponding to the third feature points included in the second matching pair set respectively; determining a target distribution interval based on the first distribution interval; determining a fifth feature point in the second matching pair set, and a fourth matching point corresponding to the fifth feature point, in which the direction difference of the fifth feature point is located in the target distribution interval; and obtaining the feature point matching result based on the fifth feature point and the fourth matching point corresponding to the fifth feature point.

[0050] In an optional embodiment, when the above first distribution interval is multiple, the determination of the target distribution interval based on the first distribution interval includes: determining the number of the above direction differences included in the multiple first distribution intervals respectively; taking the first distribution interval with the largest number of the above direction differences in the multiple first distribution intervals as a second distribution interval; and obtaining the target distribution interval based on the second distribution interval.

[0051] Optionally, the above second distribution interval can be directly taken as the target distribution interval, or a certain domain range is added to the second distribution interval, and the second distribution interval after the addition is taken as the target distribution interval. For example, when the above second distribution interval is [5°, 10°], [5°, 10°] is taken as the target distribution interval, or [5°-0.05°, 10°+0.05°] is taken as the target distribution interval.

[0052] It can be understood that, in the case that the third feature points corresponding to the direction difference in the above-mentioned second matching pair set are distributed in a plurality of different first distribution intervals, according to the matching feature point direction difference centralized distribution principle, the third feature points and the corresponding second matching points in the above-mentioned second matching pair set whose direction difference is located in the above-mentioned target distribution interval (i.e. the distribution interval with the most concentrated direction difference) are retained, and other third feature points and corresponding other second matching points whose direction difference is located outside the above-mentioned target distribution interval are eliminated, so as to obtain the final feature point matching result between the above-mentioned target infrared image and the above-mentioned target visible light source image.

[0053] Through the above steps S102 to S114, the purpose of accurately obtaining the matching points of the infrared image and the visible light image can be achieved, thereby realizing the technical effects of improving the image matching accuracy and reducing the misaligned points, and further solving the technical problems of high image feature point misalignment probability and low image matching accuracy caused by objective factors such as device assembly difference, shooting time, distance and viewing angle difference.

[0054] Based on the above embodiments and optional embodiments, the present application proposes an optional implementation method, which comprises the following steps:

[0055] Step S1: pre-processing the images to be aligned. The input initial infrared image and initial visible light source image obtained by shooting the same power equipment are subjected to gray scale processing to obtain a first infrared image and a first visible light source image with the gray value range linearly stretched to [0, 255]. The obtained first infrared image and first visible light source image are further subjected to image enhancement processing to obtain a target infrared image and a target visible light source image with the same longitudinal resolution. An angle point detection algorithm is used to obtain a first feature point set in the target infrared image and a second feature point set in the target visible light source image.

[0056] Step S2: calculating the Jaccard distance between the first feature points included in the first feature point set and the second feature points included in the second feature point set; matching the feature points of the visible light source image and the infrared image according to the principle of minimum Jaccard distance, taking the second feature point with the minimum Jaccard distance in the second feature point set as the first matching point corresponding to the first feature point; and obtaining a first matching pair set based on the first feature points included in the first feature point set and the corresponding second feature points.

[0057] Step S3, the same size of the target infrared image and the target visible light original image are spliced in the same plane, the slope of the line between the first feature point included in the first matching pair set and the corresponding first matching point is calculated, and the length of the line is calculated. The feature points between the two images have certain correlation (such as the correlation of the slope and the length of the line). The feature point matching of the target infrared image and the target visible light original image is carried out.

[0058] Step S4, the same size of the target infrared image and the target visible light original image are placed in a predetermined plane position, the fourth feature point in the first matching pair set whose slope is within a predetermined slope range and whose length is within a predetermined length range is removed by using the characteristics of the same scale feature point direction angle and the invariable interval to remove the mismatched points, and the second matching pair set is obtained.

[0059] Step S5, the first original image contour around the third feature point (such as within a predetermined first range) in the target infrared image is determined, and the second original image contour around the second matching point (such as within a predetermined second range) in the target visible light source image is determined. The first image contour corresponding to the third feature point and the second image contour corresponding to the second matching point are obtained by respectively performing smoothing filtering processing on the first original image contour and the second original image contour. The first triangle is constructed with the tangent of the first image contour as the side and the third feature point as the vertex, and the second triangle is constructed with the tangent of the first image contour as the side and the second matching point as the vertex. The length of the side of the first triangle is determined by the direction vector between the end point of the first image contour line and the third feature point, and the length of the side of the first triangle is determined by the direction vector between the end point of the second image contour line and the second matching point.

[0060] Step S6, the direction difference between the first main direction and the corresponding second main direction is determined, and the mismatched points in the second matching pair set are further removed based on the direction difference.

[0061] Step S7, in the case that the direction difference corresponding to the third feature point included in the second matching pair set is distributed in a plurality of different first distribution intervals, the third feature point in the second matching pair set whose direction difference is located in the target distribution interval (i.e. the distribution interval with the most concentrated direction difference) and the corresponding second matching point are retained according to the principle of centralized distribution of matching feature point direction difference, and other third feature points whose direction difference is located outside the target distribution interval and the corresponding other second matching points are removed, and the final feature point matching result between the target infrared image and the target visible light source image is obtained.

[0062] The embodiment of the present application can achieve the following technical effects: (1) a main direction distribution method based on a feature point contour triangle median line method is proposed, and the scale and perspective invariance of the infrared image and the visible light image of the power equipment is achieved. The method does not depend on the similarity between the infrared and visible light image gray scales, but focuses on the image contour line features with higher similarity. (2) An identical scale feature point direction angle and interval invariant feature matching method is proposed to achieve accurate matching between the infrared and visible light image feature points, and the method has the characteristics of few mismatched points and high accuracy.

[0063] In the embodiment, an image feature point matching device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" "device" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0064] According to the embodiment of the present application, a device embodiment for implementing the above-mentioned image feature point matching method is also provided, Figure 2 is a structural schematic diagram of an image feature point matching device according to the embodiment of the present application, as Figure 2 shown, the above-mentioned image feature point matching device comprises a first acquisition module 200, a second acquisition module 202, a first determination module 204, a second determination module 206, a third determination module 208, a fourth determination module 210, and a fifth determination module 212, wherein:

[0065] The first acquisition module 200 is used to acquire a first feature point set in a target infrared image and a second feature point set in a target visible light image by using a corner detection algorithm.

[0066] The second acquisition module 202 is connected to the first acquisition module 200 and is used to determine a first matching point corresponding to a first feature point included in the first feature point set from the second feature point set by using a Jaccard distance method, to obtain a first matching pair set.

[0067] The first determination module 204 is connected to the second acquisition module 202 and is used to determine the slope of a line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and the length of the line.

[0068] The second determining module 206 is connected to the first determining module 204, configured to determine a second matching pair set from the first matching pair set based on the slope and the length, wherein the second matching pair set comprises a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set;

[0069] The third determining module 208 is connected to the second determining module 206, configured to determine a first principal direction corresponding to the third feature point and a second principal direction corresponding to the second matching point in the second matching pair set respectively.

[0070] The fourth determining module 210 is connected to the third determining module 208, configured to determine a direction difference between the first principal direction and the corresponding second principal direction.

[0071] The fifth determining module 212 is connected to the fourth determining module 210, configured to determine a feature point matching result between the target infrared image and the target visible light source image based on the direction difference.

[0072] In the embodiment of the present application, by setting the first acquisition module 200, the first feature point set in the target infrared image and the second feature point set in the target visible light source image are acquired by using the corner detection algorithm; the second acquisition module 202 is connected to the first acquisition module 200, and the first matching point corresponding to the first feature point included in the first feature point set is determined from the second feature point set by using the Jaccard distance method, and the first matching pair set is obtained; the first determination module 204 is connected to the second acquisition module 202, and the slope of the line connecting the first feature point and the corresponding first matching point included in the first matching pair set and the length of the line are determined; the second determination module 206 is connected to the first determination module 204, and the second matching pair set is determined from the first matching pair set based on the slope and the length, wherein the second matching pair set includes the third feature point and the second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; the third determination module 208 is connected to the second determination module 206, and the first principal direction corresponding to the third feature point included in the second matching pair set and the second principal direction of the corresponding second matching point are determined; the fourth determination module 210 is connected to the third determination module 208, and the direction difference between the first principal direction and the corresponding second principal direction is determined; the fifth determination module 212 is connected to the fourth determination module 210, and the feature point matching result between the target infrared image and the target visible light source image is determined based on the direction difference, which achieves the purpose of accurately acquiring the matching points of the infrared image and the visible light image, thereby realizing the technical effects of improving the image matching accuracy and reducing the misfit points, and further solving the technical problems of high image feature point mis-matching probability and low image matching accuracy caused by the objective factors such as device assembly difference, shooting time, distance and viewing angle difference.

[0073] It should be noted that each of the above modules can be realized by software or hardware. For example, for the latter, each of the above modules can be located in the same processor, or any combination of the above modules can be located in different processors.

[0074] It should be noted that the first obtaining module 200, the second obtaining module 202, the first determining module 204, the second determining module 206, the third determining module 208, the fourth determining module 210, and the fifth determining module 212 correspond to steps S102 to S114 in the embodiments, and the above modules have the same instances and application scenarios as the corresponding steps, but are not limited to the above disclosed embodiments. It should be noted that the above modules can run in a computer terminal as part of the device.

[0075] It should be noted that the optional or preferred embodiments of the present embodiment can refer to the related description in the embodiments, which will not be repeated here.

[0076] The image feature point matching device described above can further include a processor and a memory, and the first obtaining module 200, the second obtaining module 202, the first determining module 204, the second determining module 206, the third determining module 208, the fourth determining module 210, and the fifth determining module 212 are stored in the memory as program modules, and the processor executes the above program modules stored in the memory to realize the corresponding functions.

[0077] The processor includes a core, and the core retrieves the corresponding program modules from the memory. The above core can be one or more. The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0078] According to the embodiments of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in the present embodiment, the non-volatile storage medium includes a stored program, wherein the program controls the device where the non-volatile storage medium is located to execute any of the above image feature point matching methods when the program is running.

[0079] Optionally, in the present embodiment, the non-volatile storage medium can be located in any one of a computer terminal group in a computer network or in any one of a mobile terminal group, and the non-volatile storage medium includes a stored program.

[0080] Optionally, the device where the non-volatile storage medium is located performs the following functions when the program is running: using a corner detection algorithm to obtain a first feature point set in the target infrared image and a second feature point set in the target visible light source image respectively; using a Jaccard distance method to determine a first matching point corresponding to a first feature point included in the first feature point set from the second feature point set, to obtain a first matching pair set; determining a slope of a line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and a length of the line; determining a second matching pair set from the first matching pair set based on the slope and the length, wherein the second matching pair set includes a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; determining a first principal direction corresponding to the third feature point and a second principal direction of the corresponding second matching point included in the second matching pair set respectively; determining a direction difference between the first principal direction and the corresponding second principal direction; and determining a feature point matching result between the target infrared image and the target visible light source image based on the direction difference.

[0081] According to the embodiments of the present application, an embodiment of a processor is also provided. Optionally, in the embodiment, the processor is used to run a program, wherein the program performs any of the image feature point matching methods when running.

[0082] According to the embodiments of the present application, an embodiment of a computer program product is also provided, which is adapted to execute the program initialized with any of the image feature point matching method steps when executed on a data processing device.

[0083] Optionally, the computer program product described above, when executed on a data processing device, is adapted to execute the program steps of: acquiring a first feature point set in a target infrared image and a second feature point set in a target visible light source image respectively by using a corner point detection algorithm; determining a first matching point corresponding to a first feature point included in the first feature point set from the second feature point set by using a Jaccard distance method, to obtain a first matching pair set; determining a slope of a line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and a length of the line; determining a second matching pair set from the first matching pair set based on the slope and the length, wherein the second matching pair set includes a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; determining a first principal direction corresponding to the third feature point and a second principal direction of the corresponding second matching point included in the second matching pair set; determining a direction difference between the first principal direction and the corresponding second principal direction; and determining a feature point matching result between the target infrared image and the target visible light source image based on the direction difference.

[0084] The electronic device provided by the embodiment of the present application includes a processor, a memory, and a program stored on the memory and executable on the processor. When the processor executes the program, the following steps are implemented: a first feature point set in a target infrared image and a second feature point set in a target visible light source image are acquired respectively by using a corner point detection algorithm; a first matching point corresponding to a first feature point included in the first feature point set is determined from the second feature point set by using a Jaccard distance method, to obtain a first matching pair set; a slope of a line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and a length of the line are determined; a second matching pair set is determined from the first matching pair set based on the slope and the length, wherein the second matching pair set includes a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; a first principal direction corresponding to the third feature point and a second principal direction of the corresponding second matching point included in the second matching pair set are determined; a direction difference between the first principal direction and the corresponding second principal direction is determined; and a feature point matching result between the target infrared image and the target visible light source image is determined based on the direction difference.

[0085] The serial numbers of the embodiments of the present application described above are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0086] In the above-mentioned embodiments of the present application, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0087] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the above-mentioned modules can be a logical function division, and actual implementation can have another division mode, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between modules or modules, which can be electrical or other forms.

[0088] The above-mentioned modules described as separate components can be or can not be physically separated, and the components displayed as modules can be or can not be physical modules, that is, they can be located in one place, or they can be distributed to a plurality of modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, the functional modules in each embodiment of the present application can be integrated in a processing module, or each module can exist physically, or two or more modules can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of software functional module.

[0090] The above-mentioned integrated module, if realized in the form of software functional module and sold or used as an independent product, can be stored in a computer readable non-volatile storage medium. Based on this understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of software product, which is stored in a non-volatile storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the embodiments of the present application. The above-mentioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and various program code storage media.

[0091] The above merely is the preferred embodiment of the present application, it should be pointed out that, for ordinary skilled in the art, without departing from the principles of the present application, can also make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for matching image feature points, characterized in that, include: A corner detection algorithm is used to obtain the first set of feature points in the infrared image of the target and the second set of feature points in the visible light source image of the target, respectively. Using the Jaccard distance method, a first matching point corresponding to the first feature point included in the first feature point set is determined from the second feature point set, thus obtaining a first matching pair set; Determine the slope of the line connecting the first feature point and the corresponding first matching point included in the first matching pair set, as well as the length of the line; Based on the slope and the length, a second set of matching pairs is determined from the first set of matching pairs, wherein the second set of matching pairs includes a third feature point and a second matching point corresponding to the third feature point, and the second set of matching pairs is less than or equal to the first set of matching pairs; Determine the first principal direction corresponding to the third feature point included in the second matching pair set, and the second principal direction corresponding to the second matching point; Determine the direction difference between the first principal direction and the corresponding second principal direction; Based on the directional difference, the feature point matching result between the target infrared image and the target visible light source image is determined; The step of determining the first principal direction corresponding to the third feature points included in the second matching pair set, and the second principal direction corresponding to the second matching points, includes: determining the first image contour corresponding to the third feature points within a preset first range, and the second image contour corresponding to the second matching points within a preset second range; constructing a first triangle based on the third feature points and the first image contour line; constructing a second triangle based on the second matching points and the second image contour line; determining the first median direction of the first triangle and the second median direction of the second triangle; obtaining the first principal direction based on the first median direction, and obtaining the second principal direction based on the second median direction.

2. The method according to claim 1, characterized in that, The method employs Jaccard distance to determine first matching points from the second set of feature points that correspond to the first feature points included in the first set of feature points, thereby obtaining a first set of matching pairs, including: Calculate the Jaccard distance between the first feature points included in the first feature point set and the second feature points included in the second feature point set; The second feature point with the smallest Jaccard distance in the second feature point set is taken as the first matching point corresponding to the first feature point; Based on the first feature points included in the first feature point set and the corresponding second feature points, the first matching pair set is obtained.

3. The method according to claim 1, characterized in that, Determining the second set of matching pairs from the first set of matching pairs based on the slope and the length includes: Identify the fourth feature point in the first set of matching pairs whose slope is within a preset slope range and whose length is within a preset length range; The second set of matching pairs is obtained based on the fourth feature point in the first set of matching pairs and the third matching point corresponding to the fourth feature point.

4. The method according to claim 1, characterized in that, The first triangle is constructed based on the third feature point and the first image contour line; And based on the second matching point and the second image contour line, constructing a second triangle, including: Determine a first direction vector between the third feature point and the first endpoint of the first image contour line, a second direction vector between the third feature point and the second endpoint of the first image contour line, a third direction vector between the second matching point and the third endpoint of the second image contour line, and a fourth direction vector between the second matching point and the fourth endpoint of the second image contour line. Based on the third feature point, the first direction vector, and the second direction vector, construct the first triangle; Based on the second matching point, the third direction vector, and the fourth direction vector, the second triangle is constructed.

5. The method according to claim 1, characterized in that, The step of determining the feature point matching result between the target infrared image and the target visible light source image based on the direction difference includes: Determine the first distribution interval of the directional difference corresponding to the third feature points included in the second matching pair set; Based on the first distribution interval, determine the target distribution interval; Determine the fifth feature point in the second set of matching pairs where the directional difference is located in the target distribution interval, and the fourth matching point corresponding to the fifth feature point; Based on the fifth feature point and the fourth matching point corresponding to the fifth feature point, the feature point matching result is obtained.

6. The method according to claim 5, characterized in that, When there are multiple first distribution intervals, determining the target distribution interval based on the first distribution intervals includes: Determine the number of directional differences included in each of the multiple first distribution intervals; The interval with the largest number of directional differences among the plurality of first distribution intervals shall be designated as the second distribution interval; Based on the second distribution interval, the target distribution interval is obtained.

7. The method according to any one of claims 1 to 6, characterized in that, Before acquiring the first set of feature points in the infrared image of the target and the second set of feature points in the visible light source image of the target using the corner detection algorithm, the method further includes: Acquire an initial infrared image and an initial visible light source image corresponding to the initial infrared image; The initial infrared image and the initial visible light source image are processed in grayscale to obtain a first infrared image and a first visible light source image corresponding to the first infrared image; Image enhancement processing is performed on the first infrared image and the first visible light source image respectively to obtain the target infrared image and the target visible light source image corresponding to the target infrared image, wherein the target infrared image and the target visible light source image have the same vertical resolution.

8. An image feature point matching device, characterized in that, include: The first acquisition module is used to acquire a first set of feature points in the infrared image of the target and a second set of feature points in the visible light source image of the target using a corner detection algorithm. The second acquisition module is used to use the Jaccard distance method to determine the first matching point corresponding to the first feature point included in the first feature point set from the second feature point set, so as to obtain the first matching pair set; The first determining module is used to determine the slope of the line connecting the first feature point and the corresponding first matching point included in the first matching pair set, and the length of the line; The second determining module is used to determine a second matching pair set from the first matching pair set based on the slope and the length, wherein the second matching pair set includes a third feature point and a second matching point corresponding to the third feature point, and the second matching pair set is less than or equal to the first matching pair set; The third determining module is used to determine the first principal direction corresponding to the third feature points included in the second matching pair set, and the second principal direction corresponding to the second matching point; The fourth determining module is used to determine the direction difference between the first main direction and the corresponding second main direction; The fifth determining module is used to determine the feature point matching result between the target infrared image and the target visible light source image based on the direction difference; The third determining module is further configured to determine a first image contour corresponding to the third feature point within a preset first range, and a second image contour corresponding to the second matching point within a preset second range; construct a first triangle based on the third feature point and the first image contour line; construct a second triangle based on the second matching point and the second image contour line; determine a first median direction of the first triangle and a second median direction of the second triangle; obtain a first principal direction based on the first median direction and a second principal direction based on the second median direction.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores multiple instructions adapted for loading by a processor and executing the image feature point matching method according to any one of claims 1 to 7.

Images