A real-time heterogeneous target matching method based on gradient features

Abstract

The present disclosure provides a real-time heterogeneous target matching method based on gradient features. Key point gradient features in a visible light reference image are extracted offline, converted into feature point pairs that can be subjected to real-time coordinate transformation, and saved locally. During online matching, the feature point pairs are read, subjected to coordinate transformation, and then compared with key point features in a real-time infrared image for heterogeneous feature similarity calculation, thereby effectively improving matching speed and accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a real-time heterogeneous target matching method based on gradient features. Background Technology

[0002] Image matching technology is a fundamental technique in image processing, used to match two or more images that are offset from each other. It has wide applications in medical image analysis, object identification and change detection, and target recognition. Heterogeneous image matching is an emerging and hot research topic in the field, with infrared and visible light image matching serving as the foundation for multi-source remote sensing data fusion, scene matching navigation, and other technologies, showing great promise for future development.

[0003] Currently, most commonly used target matching algorithms are designed for matching targets from the same source, and have achieved good matching results. However, due to the differences between infrared and visible light images in terms of sensor structure and imaging principles, the image information of targets in images from different sources varies significantly. Directly applying commonly used target matching algorithms to match targets from different sources in infrared and visible light images yields poor results. Furthermore, differences in shooting angle and distance necessitate a series of image transformations before matching, a computationally intensive process that struggles to meet the real-time requirements of matching algorithms. Summary of the Invention

[0004] To address the issues that existing same-source image matching algorithms cannot adapt to heterogeneous matching and that online feature extraction cannot meet real-time requirements, this disclosure proposes a real-time heterogeneous target matching method based on gradient features, which can effectively improve the matching speed and accuracy.

[0005] The real-time heterogeneous target matching method based on gradient features disclosed herein includes the following steps:

[0006] Step S1: Extract key points and their gradient features from the visible light reference image of the target to be matched; generate feature point pairs for each key point based on the gradient features; save the feature point pairs as a local file;

[0007] Step S2: Extract key points and their gradient features from the current frame infrared image;

[0008] Step S3: Based on the relevant parameters of the infrared detector, calculate the transformation matrix M1 from the infrared camera coordinate system to the pixel coordinate system; based on the relevant information of the real-time infrared camera, calculate the transformation matrix M2 from the world coordinate system to the infrared camera coordinate system; based on matrices M1 and M2, calculate the transformation matrix M3 from the world coordinate system to the pixel coordinate system.

[0009] Step S4: Read the feature point pairs in the local file and perform coordinate transformation from world coordinate system to pixel coordinate system based on matrix M3; calculate the corresponding key point gradient features after coordinate transformation based on the feature point pairs after coordinate transformation.

[0010] Step S5: Perform heterogeneous feature similarity calculation between the transformed keypoint gradient features and the keypoint gradient features in the current frame infrared image, and determine whether there is a matching target in the current frame. If there is, output the target's position coordinates; otherwise, take the next frame infrared image as the current frame and proceed to step S2.

[0011] Furthermore, in step S1, the method for extracting key points and their gradient features specifically includes the following steps:

[0012] Obtain a visible light reference image of the target to be matched and adjust its resolution to 1, that is, a 1m×1m area in world coordinates occupies 1 pixel in the image;

[0013] Using the Sobel operator, the gradient features in the X and Y directions of each point are extracted, and their magnitude and orientation field are calculated.

[0014] Based on the amplitude, several feature points are selected as key points, and the orientation field angle corresponding to each key point is used as the key point gradient feature.

[0015] Furthermore, in step S1, the method for calculating the amplitude and direction field includes:

[0016] The formula for calculating the amplitude at each location is as follows:

[0017]

[0018] Where dx represents the gradient in the X direction at that location, dy represents the gradient in the Y direction at that location, and MagG represents the magnitude at that location;

[0019] The formulas for calculating the directional field at each location are as follows:

[0020]

[0021] The formula for calculating angle is as follows:

[0022]

[0023] Furthermore, in step S1, the method for generating feature point pairs for each key point includes:

[0024] At the key point coordinate P1(x1,y1), corresponding to the direction field direction and with a step size of 1, select coordinate point P2(x2,y2). This coordinate point P2 and the key point P1 form a feature point pair (P1,P2) of point P1.

[0025] Furthermore, in step S3, the method for calculating the coordinate transformation matrix M1 includes:

[0026] Assuming the infrared detector's relevant parameters include: field-of-view center coordinates (u0, v0), pixel width dx, pixel height dy, and focal length f, then according to the camera projection principle, the coordinate transformation matrix M1 is calculated as follows:

[0027]

[0028] The formula for calculating matrix M2 is as follows:

[0029]

[0030] Where T represents the 3×1 translation matrix of the camera relative to the target, and R is the 3×3 rotation matrix based on the affine transformation of the three attitude angles, which refer to the azimuth angle α, pitch angle β, and roll angle δ of the camera coordinate system relative to the world coordinate system.

[0031] The formula for calculating matrix M3 is as follows:

[0032] M3 = M1M2.

[0033] Furthermore, in step S4, the method for transforming the feature point pairs based on matrix M3 includes:

[0034] When the visible light reference image is scaled to 1, each pixel in the image is directly translated to world coordinates. The conversion formula is as follows:

[0035]

[0036] Where (x,y,z) represents the coordinates of the pixel in the world coordinate system, (c,r) represents the coordinate position of the pixel in the reference image, W represents the total number of pixels in the horizontal direction of the reference image, and H represents the total number of pixels in the vertical direction of the reference image.

[0037] The formula for calculating the coordinate transformation of feature point pairs based on matrix M3 is as follows:

[0038]

[0039]

[0040] Where (u,v) represents the pixel coordinates of the feature point in the transformed image, Z c This indicates the directional position of the feature point on the camera's Z-axis.

[0041] Furthermore, in step S4, the method for calculating the gradient features of the corresponding key points after coordinate transformation includes:

[0042] Calculate the orientation field angles of the key points after coordinate transformation. The formula for calculating the orientation field is the same as that in step S1, but the definitions of dx and dy in the formula are:

[0043]

[0044] Where (x1′,y1′) and (x2′y′) are feature point pairs after coordinate transformation;

[0045] Gradient diffusion is performed on the transformed keypoint gradient features, that is, each pixel and its n×n neighborhood, where n is a natural number, are ORed according to the discretized direction. Each keypoint gradient feature contains more than one directional field angle.

[0046] Furthermore, n = 5.

[0047] Furthermore, step S5 specifically includes the following steps:

[0048] Calculate similarity using the following formula:

[0049]

[0050] Where Sa represents the orientation field angle of the key point in the real-time infrared image, Ta represents the orientation field angle of the gradient feature of the key point after the visible light reference image is transformed, and L(Sa,Ta) is used to calculate the feature similarity between two key points.

[0051] Using a sliding window with a step size of t, the real-time infrared images are traversed, and the similarity between the gradient features of key points in all regions and the visible light reference image after coordinate transformation is calculated. The similarity calculation formula is as follows:

[0052]

[0053] Where k represents the total number of key points extracted from the reference image, m represents the presence of m directional field angle features at the location of the key point, Sa represents the directional field angle of the key point in the real-time infrared image corresponding to the key point in the visible light reference image, and Ls represents the similarity between the region and the reference image, with a lower Ls indicating a higher similarity.

[0054] When the similarity of a certain region is less than the threshold K, it means that the target has been successfully matched, and the coordinates of the target's location are output.

[0055] Otherwise, the next infrared image frame is taken as the current frame, and the process proceeds to step S2.

[0056] The real-time heterogeneous target matching method based on gradient features disclosed herein extracts the gradient features of key points in a visible light reference image offline and converts them into feature point pairs that can be transformed in real time. During online matching, the similarity between these feature point pairs after coordinate transformation and the key point features in a real-time infrared image is calculated, thereby effectively improving the matching speed and accuracy.

[0057] Compared with the prior art, the beneficial effects of this disclosure are: ① reducing the computational load in real-time matching by generating feature point pairs that can be transformed online offline; ② improving matching speed; ③ improving the adaptability of gradient features to heterogeneous matching by designing a similarity calculation method for heterogeneous features; ④ improving matching accuracy; ⑤ improving the anti-interference ability of the matching algorithm by using gradient diffusion to diffuse key point features into the neighborhood. Attached Figure Description

[0058] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0059] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present disclosure. Detailed Implementation

[0060] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0061] Figure 1 A flowchart illustrating an exemplary embodiment of this disclosure is shown, including the following steps:

[0062] Step S101: Input the visible light reference image of the target to be matched, adjust the resolution of the reference image and extract its key point gradient features.

[0063] The reference image resolution indicates how many pixels a 1m×1m area occupies in the image in world coordinates. By scaling the reference image, its resolution is adjusted to 1, meaning that a 1m×1m area occupies 1 pixel in the image in world coordinates.

[0064] Extracting key point gradient features: The Sobel operator is used to extract gradient features in the X and Y directions respectively. The magnitude and orientation field are calculated respectively. Based on the magnitude, several feature points are filtered to obtain key points. The orientation field angle corresponding to each key point is used as the key point gradient feature.

[0065] The formula for calculating the amplitude at each location is as follows:

[0066]

[0067] In the formula, dx represents the gradient in the X direction at that position, dy represents the gradient in the Y direction at that position, and MagG represents the magnitude at that position;

[0068] The formulas for calculating the directional field at each location are as follows:

[0069]

[0070] The formula for calculating angle is as follows:

[0071]

[0072] Step S102: Based on the key point features, calculate and generate feature point pairs corresponding to each key point.

[0073] The method for generating feature point pairs includes: selecting coordinate point P2(x2,y2) at the position with a direction field step size of 1 corresponding to the key point coordinate P1(x1,y1). This coordinate point P2 and the key point P1 form a feature point pair, and this feature point pair (P1,P2) is used to describe the key point features.

[0074] Step S103: Store the feature point pairs to generate a local storage file. In this embodiment, a YAML file is used to store the feature data.

[0075] Step S104: Input the current frame image of the infrared detector and extract the key point features of the current frame image. The extraction of key point features is the same as in step S101.

[0076] Step S105: Obtain relevant parameters of the infrared detector and calculate the coordinate transformation matrix M1 from the infrared camera coordinate system to the pixel coordinate system (i.e., the image coordinate system).

[0077] The relevant parameters of the infrared detector include the field-of-view center coordinates (u0, v0), pixel width dx, pixel height dy, and focal length f. Based on the camera projection principle, the coordinate transformation matrix M1 is calculated using the following formula:

[0078]

[0079] Step S106: Obtain real-time camera attitude, distance from the target, and other relevant information, and calculate the coordinate transformation matrix M2 from the world coordinate system to the infrared camera coordinate system.

[0080] The attitude of an infrared camera refers to the azimuth angle α, pitch angle β, and roll angle δ of the camera coordinate system relative to the world coordinate system. According to the principle of affine coordinate transformation, the formula for calculating M2 is as follows:

[0081]

[0082] Where T represents the 3×1 translation matrix of the camera relative to the target, and R is the 3×3 rotation matrix based on the affine transformation of the three attitude angles.

[0083] Step S107: Combine the matrices M1 to M2 to calculate the coordinate transformation matrix M3 from the world coordinate system to the pixel coordinate system.

[0084] The coordinate transformation matrix M3 refers to the transformation matrix from world coordinates to pixel coordinates, and its calculation formula is as follows: M3 = M1M2

[0085] Step S108: Read the local storage file and perform coordinate transformation on the stored feature point pairs based on matrix M3 to obtain the transformed feature point pairs.

[0086] Calculate the new feature point pairs after transformation of all feature point pairs in the reference image. First, according to step S101, the ratio of pixel coordinates in the reference image to world coordinates is 1:1. Therefore, each pixel in the reference image can be directly translated into world coordinates. The transformation formula is as follows:

[0087]

[0088] Where (x,y,z) represents the coordinates of the pixel in the world coordinate system, (c,r) represents the coordinate position of the pixel in the reference image, W represents the total number of pixels in the horizontal direction of the reference image, and H represents the total number of pixels in the vertical direction of the reference image. The transformation calculation formula is as follows:

[0089]

[0090]

[0091] Where (u,v) represents the pixel coordinates of the feature point in the transformed image, z c This indicates the position of the feature point along the Z-axis of the camera coordinate system.

[0092] Step S109: Based on the transformed feature point pairs, calculate the transformed key point features corresponding to each feature point pair.

[0093] Based on the coordinates of the transformed feature points obtained in step S108, the transformed key point features, i.e., the orientation field angles corresponding to the transformed key points, can be calculated. The orientation field calculation formula is the same as that in step S101, but the meanings of dx and dy in the formula are different from those in step S101, as follows:

[0094]

[0095] Where (x1′,y1′) and (x2′y′) are feature point pairs after coordinate transformation;

[0096] Preferably, to ensure a certain degree of fault tolerance in the matching process and improve its anti-interference capability, gradient diffusion is performed on the transformed keypoint features. This involves performing an OR operation on each pixel and its n×n neighborhood according to the discretized directions. Each keypoint feature contains more than one directional field angle. In this embodiment, n = 5.

[0097] Step S110: Perform heterogeneous feature similarity calculation between the transformed key point features and the key point features of the current frame image.

[0098] Calculation formula:

[0099]

[0100] In the formula, Sa represents the orientation field angle of the key point in the real-time infrared image, and Ta represents the orientation field angle of the key point feature after the reference image transformation. L(Sa,Ta) is used to calculate the feature similarity between two key points.

[0101] Using a sliding window with a step size of t, the real-time infrared images are traversed, and the similarity between the key point features of all regions and the transformed reference image is calculated. The similarity calculation formula is as follows:

[0102]

[0103] In the formula, n represents the number of key points extracted from the reference image, m represents the presence of m directional field angle features at the location of the key point, Sa represents the directional field angle of the key point in the real-time infrared image at the location corresponding to the key point in the reference image, and Ls represents the similarity between the region and the reference image, with a lower Ls indicating a higher similarity.

[0104] Step S111: If there is a target in the current frame that meets the matching conditions, proceed to step S112; otherwise, set the next frame of the current frame as the current frame and proceed to step S104.

[0105] The matching condition refers to setting a similarity threshold K. When the similarity of a certain region is less than K, it indicates a successful match with the target. In this embodiment, K = 0.5.

[0106] Step S112: Output the target's position coordinates, and the method ends.

[0107] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A real-time heterogeneous target matching method based on gradient features, comprising the following steps: Step S1: Extract key points and their gradient features from the visible light reference image of the target to be matched; generate feature point pairs for each key point based on the gradient features; save the feature point pairs as a local file; Step S2: Extract key points and their gradient features from the current frame infrared image; Step S3: Calculate the transformation matrix from the infrared camera coordinate system to the pixel coordinate system based on the relevant parameters of the infrared detector. M 1. Based on the real-time infrared camera information, calculate the transformation matrix from the world coordinate system to the infrared camera coordinate system. M 2; According to the matrix M 1 and M 2. Calculate the transformation matrix from the world coordinate system to the pixel coordinate system. M 3; Step S4: Read the feature point pairs from the local file and, based on the matrix... M 3. Perform coordinate transformation from world coordinate system to pixel coordinate system; based on the feature point pairs after coordinate transformation, calculate the corresponding key point gradient features after coordinate transformation; Step S5: Perform heterogeneous feature similarity calculation between the transformed keypoint gradient features and the keypoint gradient features in the current frame infrared image, and determine whether there is a matching target in the current frame. If there is, output the target's position coordinates; otherwise, take the next frame infrared image as the current frame and proceed to step S2.

2. The method according to claim 1, characterized in that, In step S1, the method for extracting key points and their gradient features specifically includes the following steps: Obtain a visible light reference image of the target to be matched and adjust its resolution to 1, that is, a 1m×1m area in world coordinates occupies 1 pixel in the image; Using the Sobel operator, the gradient features in the X and Y directions of each point are extracted, and their magnitude and orientation field are calculated. Based on the amplitude, several feature points are selected as key points, and the orientation field angle corresponding to each key point is used as the key point gradient feature.

3. The method according to claim 2, characterized in that, In step S1, the calculation method for the amplitude and direction field includes: The formula for calculating the amplitude at each location is as follows: in, dx This represents the gradient in the X direction at that location. dy This represents the gradient in the Y direction at that location, and MagG represents the magnitude at that location. The formulas for calculating the directional field at each location are as follows: The formula for calculating angle is as follows: 。 4. The method according to claim 2 or 3, characterized in that, In step S1, the method for generating feature point pairs for each key point includes: At key point coordinates P1 Select coordinate point P2 at the position corresponding to the direction field direction and with a step size of 1. The coordinate point P2 and the key point P1 form a feature point pair (P1, P2) of point P1.

5. The method according to claim 2, characterized in that, In step S3, the coordinate transformation matrix M The calculation methods for 1 include: Assume the relevant parameters of the infrared detector include: coordinates of the center of the field of view. Pixel width dx Pixel High dy and focal length f According to the principle of camera projection, the coordinate transformation matrix... M The calculation formula is as follows: ； The matrix M The formula for calculating 2 is as follows: in, T This represents the 3×1 translation matrix of the camera relative to the target. R The rotation matrix is ​​a 3×3 rotation matrix based on the affine transformation of three attitude angles, which refer to the azimuth angle α, pitch angle β, and roll angle δ of the camera coordinate system relative to the world coordinate system. The matrix M The formula for calculating 3 is as follows: 。 6. The method according to claim 5, characterized in that, In step S4, the feature point pairs are based on a matrix M 3. Methods for performing coordinate transformations include: When the visible light reference image is scaled to 1, each pixel in the image is directly translated to world coordinates. The conversion formula is as follows: in,( x , y , z ) represents the coordinates of a pixel in the world coordinate system. c , r () indicates the coordinate position of a pixel in the reference image. W This indicates the total number of pixels horizontally in the reference image. H This indicates the total number of pixels in the vertical direction of the reference image; Then the feature point pairs are based on the matrix M The formula for calculating the coordinate transformation of 3 is as follows: in,( u , v () represents the pixel coordinates of the feature point in the transformed image. z c This indicates the directional position of the feature point on the camera's Z-axis.

7. The method according to claim 6, characterized in that, In step S4, the method for calculating the gradient features of the corresponding key points after coordinate transformation includes: Calculate the orientation field angles of the key points after coordinate transformation. The formula for calculating the orientation field is the same as that in step S1, but the definitions of dx and dy in the formula are: in,( , )and( , These are feature point pairs after coordinate transformation; Gradient diffusion is performed on the transformed keypoint gradient features, that is, each pixel and its n×n neighborhood, where n is a natural number, are ORed according to the discretized direction. Each keypoint gradient feature contains more than one directional field angle.

8. The method according to claim 7, characterized in that, The n=5.

9. The method according to claim 7, characterized in that, Step S5 specifically includes the following steps: Calculate similarity using the following formula: in, Sa This indicates the orientation field angle of key points in a real-time infrared image. Ta This represents the gradient feature direction field angle of key points after transformation of the visible light reference image. L ( Sa , Ta This is used to calculate the feature similarity between two key points; Using a sliding window with a step size of t, the real-time infrared images are traversed, and the similarity between the gradient features of key points in all regions and the visible light reference image after coordinate transformation is calculated. The similarity calculation formula is as follows: in, k This indicates the total number of key points extracted from the reference image. m This indicates that the key point exists. m The angular characteristics of the field in each direction Sa This indicates the key point orientation field angle in this region of the real-time infrared image, corresponding to the key point in the visible light reference image. Ls This indicates the similarity between the region and the reference image. Ls The lower the value, the higher the similarity. When the similarity of a certain region is less than the threshold K, it means that the target has been successfully matched, and the coordinates of the target's location are output. Otherwise, the next infrared image frame is taken as the current frame, and the process proceeds to step S2.

Images