Terahertz video radar image registration method

Abstract

The invention provides a terahertz video radar image registration method, which comprises the following steps: receiving an image frame sequence of a terahertz video radar, and respectively taking two adjacent frames of images as a reference image and an image to be registered; performing feature detection on the cut reference image and the image to be registered by adopting a KAZE algorithm to obtain feature point sets of the two images and feature description subsets corresponding to the feature point sets; searching the feature descriptor set to obtain matching point pairs; removing the matching point pairs which do not meet the slope constraint, and obtaining a slope set of the matching point pairs which meet the slope constraint; distributing the slope set to a plurality of element intervals based on an element interval optimization strategy, performing rejection operation on the matching point pairs by using an RANSAC algorithm in the element intervals, and calculating to obtain a registration matrix of the rejected matching point pairs; and realizing spatial alignment of the two images by using the registration matrix. According to the method, more feature points can be extracted, and when the mismatching contrast ratio is relatively high, the registration precision of the terahertz radar image can still be kept.

Description

Technical Field

[0001] The invention belongs to the technical field of digital image processing and relates to a terahertz video radar image registration method. Background Art

[0002] Compared to light waves, terahertz waves have excellent penetration and interference resistance, and are unaffected by weather, smoke, and dust. Compared to microwave radar, terahertz radar offers advantages such as wide bandwidth, high resolution, and Doppler sensitivity, making it an important emerging technology in target detection, tracking, and identification. Video radar collects data by continuously transmitting and receiving electromagnetic wave signals to detect information such as the distance, speed, and direction of target objects. Due to signal processing requirements, storage and transmission limitations, and real-time requirements, these signals are processed to generate frame-by-frame image data with varying perspectives and content. Therefore, image registration is required to achieve spatial alignment to support applications such as image fusion, target positioning and tracking, and environmental perception.

[0003] Common image registration methods include region-based, deep learning-based, and feature-based methods. Region-based image registration algorithms, such as mutual information (MI) and normalized cross correlation (NCC), do not rely on feature extraction and are highly resistant to noise, but are sensitive to grayscale variations. Due to the low contrast and lack of significant grayscale variations in terahertz radar images, region-based image registration algorithms struggle to meet registration accuracy requirements. Deep learning-based image registration algorithms utilize neural networks to automatically learn the registration mapping relationship between images. However, due to the high cost of acquiring terahertz radar images, the number of images required for network training is insufficient. Feature-based image registration algorithms achieve registration by extracting salient features from the image. Commonly used algorithms include the scale-invariant feature transform (SIFT) feature detection algorithm, the accelerated robust feature (SURF) detection algorithm, and the nonlinear scale space (KAZE) feature detection algorithm. SIFT and SURF algorithms use Gaussian filtering to construct a linear scale space, which blurs image details, resulting in inaccurate feature point positioning and reduced detection accuracy. The KAZE algorithm uses nonlinear diffusion filtering to construct a nonlinear scale space, alleviating the loss of image detail and edge blurring caused by linear filtering. This avoids information loss in the initial stages and allows for the extraction of more feature points. However, due to the high noise, low contrast, and unclear texture details of terahertz radar images, a large number of false matches are present after feature matching, far outnumbering the number of correct matches. The Random Sample Consensus (RANSAC) algorithm estimates model parameters through random sampling and iteration. When the proportion of false matches is high, RANSAC's performance degrades significantly, leading to model estimation failure.

[0004] Therefore, how to provide a terahertz video radar image registration method that can further improve the registration accuracy is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] In view of this, the present invention proposes a terahertz video radar image registration method, which is based on the KAZE feature extraction algorithm combined with the improved RANSAC algorithm with slope constraint and interval optimization. The matching point pairs are feedforward preprocessed to eliminate erroneous matching pairs and improve the registration accuracy of terahertz radar images.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention discloses a terahertz video radar image registration method. A terahertz video radar imaging system moves at a uniform speed along a straight path and forms an image in a strip mode. The method comprises the following steps:

[0008] S1: Receive the image frame sequence of the terahertz video radar, the rth frame image is used as the reference image, and the r+1th frame image is used as the image to be registered, r≥1;

[0009] S2: Perform edge cropping on the image frame to obtain a cropped image frame, and use the KAZE algorithm to perform feature detection on the cropped reference image and the image to be registered to obtain a set of feature points of the two images and their corresponding feature descriptor sets;

[0010] S3: Search the feature descriptor set of the reference image and the image to be registered to obtain matching initial feature point pairs and obtain a matching point pair set 1;

[0011] S4: Based on the slope constraint between the two matching point pairs, eliminate the matching point pairs that do not satisfy the slope constraint in the matching point pair set 1, obtain the slope values of the matching point pairs that satisfy the slope constraint, and form a slope set;

[0012] S5: allocating the slope set to a number of element intervals based on the element interval optimization strategy, and performing a culling operation on the matching point pairs within the element intervals using the RANSAC algorithm to calculate and obtain the registration matrix of the matching point pairs after culling;

[0013] S6: Mapping the image to be registered to the coordinate system of the reference image using the registration matrix obtained in S5 to achieve spatial alignment of the two images.

[0014] Preferably, the step of performing feature detection on the two images using the KAZE algorithm includes:

[0015] S21: A variable conduction diffusion method and additive operator splitting algorithm are used to construct a pyramid-shaped nonlinear scale space for the image, generating multi-scale image layers;

[0016] S22: Detecting pixel grayscale values in each multi-scale image layer to obtain initial extreme points;

[0017] S23: Use the second-order Taylor expansion to correct the position, scale and grayscale value of the extreme point to obtain the precise location of the extreme point;

[0018] S24: allocating a main direction to the precise positioning extreme point based on the wavelet feature to obtain a feature descriptor including the direction.

[0019] Preferably, the S22 includes the following steps:

[0020] S221: Obtaining neighboring pixels of each pixel, where the neighboring pixels include: pixels in the same multi-scale image layer adjacent to the current pixel, pixels at the same position in the upper and lower layers corresponding to the multi-scale image layer where the current pixel is located, and their adjacent pixels in the same layer;

[0021] S222: Compare the grayscale values of the current pixel point with those of its neighboring pixel points. If the grayscale value of the current pixel point is greater than / less than all of its neighboring pixel points, the current pixel point is an initial extreme value point.

[0022] Preferably, the S24 includes the following steps:

[0023] S241: Select the position coordinates of the current precise positioning extreme value point as the center, calculate the wavelet features of all precise positioning extreme value points in a 60° sector area centered on the position coordinates, and perform Gaussian weighting on the results to obtain the wavelet feature value in the current sector area;

[0024] S242: Rotate the current sector area with a rotation step of 60° and repeat S241 until the 360° range is traversed to obtain the sector area with the largest wavelet eigenvalue, and determine its direction as the main direction of the current precise positioning extreme point.

[0025] Preferably, S3 includes: using an approximate nearest neighbor algorithm to measure the similarity between feature points of the feature descriptor set of the reference image and the image to be registered, and taking feature point pairs that meet a similarity matching threshold as matching point pairs.

[0026] Preferably, the S4 includes: counting the slope value with the highest frequency of occurrence in the element interval, and screening out matching point pairs within a preset range of the slope value.

[0027] Preferably, the S5 comprises the following steps:

[0028] S51: Allocate the slope set to n element intervals, and select matching point pairs that meet the frequency requirement according to the occurrence frequency of the slope values in the element intervals to obtain a second matching point pair set, where n≥1;

[0029] S52: Use the RANSAC algorithm to perform a elimination operation on the second set of matching point pairs, and calculate the registration matrix and registration error of the remaining matching point pairs, and determine whether the registration error is less than the set threshold. If so, retain the registration matrix of the matching point pair and enter S6. If not, increase the number of element intervals n and return to S51.

[0030] Preferably, the least squares method is used in S52 to calculate the registration matrix of the remaining matching point pairs.

[0031] It can be seen from the above technical solution that, compared with the prior art, the beneficial effects of the present invention include:

[0032] Improving image registration accuracy: This method uses the KAZE algorithm to construct a nonlinear scale space, preserving image edges and texture details through variable conductive diffusion, avoiding the blurring effect of traditional Gaussian filtering. This method is particularly suitable for low-contrast, high-noise terahertz radar images. It also incorporates an improved RANSAC algorithm, using slope constraints to eliminate obvious mismatches. Dynamic interval optimization then narrows the selection range, effectively reducing the proportion of mismatches.

[0033] Enhanced system robustness: This paper adopts a multi-stage false match elimination strategy, in which the slope constraint is based on the prior knowledge of the platform's uniform motion (such as the strip imaging mode) to quickly filter out matching pairs that do not conform to the physical model; dynamic interval optimization gradually narrows the slope range and adaptively adjusts the screening strategy to avoid model failure due to excessive initial errors; combined with RANSAC iteration, it gradually approaches the correct model and can still converge stably even in the case of high false matches.

[0034] In summary, compared to traditional SIFT / SURF, this method uses a nonlinear scale space instead of linear Gaussian filtering. This significantly increases the number of feature point detections in terahertz images while effectively improving localization accuracy. Compared to traditional RANSAC, this method introduces slope constraints and dynamic range optimization, maintaining the registration accuracy of terahertz radar images even in the presence of mismatched pairs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. Those skilled in the art can also derive other drawings based on the provided drawings without inventive effort.

[0036] Figure 1 A schematic diagram of a process for terahertz video radar image registration method provided by an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of an improved RANSAC algorithm based on slope constraint and interval optimization provided by an embodiment of the present invention;

[0038] Figure 3 The terahertz video radar image, the cropped reference image, and the image to be registered provided in the embodiment of the present invention;

[0039] Figure 4 A schematic diagram of KAZE algorithm matching provided by an embodiment of the present invention;

[0040] Figure 5 The matching results of the improved RANSAC algorithm with slope constraint and interval optimization provided by the embodiment of the present invention;

[0041] Figure 6 This is a pseudo-color comparison of two images before and after registration provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] The embodiment of the present invention provides a terahertz video radar image registration method. When the terahertz video radar imaging system is imaging in strip mode, the platform moves at a constant speed along a straight path to form a continuous strip-shaped imaging area. Figure 1 As shown, the following steps are included:

[0044] S1: Receive the image frame sequence of the terahertz video radar, the rth frame image is used as the reference image, and the r+1th frame image is used as the image to be registered, r≥1;

[0045] S2: Perform edge cropping on the image frame to obtain a cropped image frame, and use the KAZE algorithm to perform feature detection on the cropped reference image and the image to be registered to obtain a set of feature points of the two images and their corresponding feature descriptor sets;

[0046] S3: Search the feature descriptor set of the reference image and the image to be registered to obtain matching initial feature point pairs and obtain a matching point pair set 1;

[0047] S4: Based on the slope constraint between the two matching point pairs, the matching point pairs that do not satisfy the slope constraint are eliminated from the matching point pair set 1, and the slope values of the matching point pairs that satisfy the slope constraint are obtained to form a slope set;

[0048] S5: Based on the element interval optimization strategy, the slope set is distributed to several element intervals, and the matching point pairs are eliminated using the RANSAC algorithm within the element intervals to calculate the registration matrix of the eliminated matching point pairs;

[0049] S6: Use the registration matrix obtained in S5 to map the image to be registered to the coordinate system of the reference image to achieve spatial alignment of the two images.

[0050] In one embodiment, the image frame edge clipping step in S2 includes: Figure 2 As shown in the figure, a 2048*2048 pixel terahertz radar image is cropped to remove obscure edge information and retain valid information in the center. The resulting cropped image is 1200*500. Figures (a) and (b) are the original reference image and the original image to be registered before cropping, respectively. Figures (c) and (d) are the original reference image and the original image to be registered after cropping, respectively. The images described in this embodiment are specifically terahertz video radar image sequences.

[0051] It should be noted that:

[0052] Due to the limitation of the radar antenna aperture, only a small part in the middle of the image contains valid information. Cropping the image can reduce the number of pixels involved in the registration calculation, reduce the computational complexity and improve the running speed of subsequent steps.

[0053] In one embodiment, the specific steps of performing feature detection on two images using the KAZE algorithm include:

[0054] S21: A pyramid-shaped nonlinear scale space is constructed for the image using the variable conduction diffusion method and the additive operator splitting algorithm to generate a multi-scale image layer. The nonlinear scale space of the image I(x,y) is:

[0055]

[0056] Where i∈[0,N-1] is the i-th multi-scale image layer, N is the total number of images in the scale space; L is the image after Gaussian filtering; I is the unit matrix; t is the diffusion time; A l (L i ) is the conduction matrix on the i-th multi-scale image layer, which is determined by the local gradient of the image and reflects the intensity and direction of diffusion; l∈[1,m] is the image dimension, such as m=2 in a two-dimensional image, corresponding to the x and y directions.

[0057] S22: Detecting pixel grayscale values in each multi-scale image layer to obtain initial extreme points;

[0058] S23: Preliminary extreme points can be generated through S22. Since the pixels and scale space of the image are discrete, the extreme points found need to be further accurately located. The position, scale and grayscale value of the extreme points are corrected using the second-order Taylor expansion to obtain the precise location of the extreme points. The specific execution steps include:

[0059] By fitting a three-dimensional quadratic function, we filter out low-contrast key points and unstable edge points, thus enhancing the stability of feature points. The second-order Taylor expansion of the scaling function L(x, y, σ) is as follows:

[0060]

[0061] Where X = (x, y, σ) T is the offset of the feature point, σ is the scale space factor, and the derivative of the above formula is used to obtain the coordinates of the extreme point position and the corresponding extreme values

[0062]

[0063] Furthermore, due to the small amount of image information, it is necessary to retain as many feature points as possible, set the extreme point threshold, such as the extreme point threshold is 0.0001, and retain the extreme points greater than the threshold, that is,

[0064] S24: Assigning a main direction to accurately locate the extreme point based on the wavelet feature to obtain a feature descriptor containing the direction.

[0065] In this embodiment, S22 includes the following steps:

[0066] S221: Obtaining neighboring pixels of each pixel, where the neighboring pixels include: pixels in the same multi-scale image layer adjacent to the current pixel, pixels at the same position in the upper and lower layers corresponding to the multi-scale image layer where the current pixel is located, and their adjacent pixels in the same layer;

[0067] S222: Compare the grayscale values of the current pixel with those of its neighboring pixels. If the grayscale value of the current pixel is greater than / less than all of its neighboring pixels, the current pixel is considered an initial extreme value point.

[0068] In this embodiment, in a 3×3 pixel grid at each layer in the nonlinear scale space, each pixel is compared with the 8 surrounding pixels in the same layer, plus 9+9=18 pixels in the upper and lower layers, for a total of 26 pixels. If the pixel is greater than (or less than) all neighboring pixels, the point is considered an extreme point.

[0069] In this embodiment, S24 assigns a reference direction to the previously retained feature points based on the local information of the image, so that the feature points have rotation invariance. The steps include:

[0070] S241: Select the position coordinates of the current precise positioning extreme value point as the center, calculate the wavelet features of all precise positioning extreme value points in a 60° sector area centered on the position coordinates, and perform Gaussian weighting on the results to obtain the wavelet feature value in the current sector area;

[0071] S242: Rotate the current sector area with a rotation step of 60° and repeat S241 until the 360° range is traversed to obtain the sector area with the largest wavelet eigenvalue, and determine its direction as the main direction of the current precise positioning extreme point.

[0072] In one embodiment, S3 includes: using an approximate nearest neighbor algorithm to measure the similarity between feature points of the feature descriptor set of the reference image and the image to be registered, and taking feature point pairs that meet the similarity matching threshold as matching point pairs, such as Figure 3 shown.

[0073] In this embodiment, the similarity matching threshold may be set to 20.

[0074] In one embodiment, in S4, since the motion state of the imaging device remains stable and no sudden turns occur, there is only a small offset between the images. By applying slope constraints to the matching point pairs obtained in the above steps, obviously incorrect matching point pairs are eliminated. Specifically, the method includes: calculating the slope value with the highest frequency in the element interval, and filtering out matching point pairs within a preset slope value range, such as Figure 3 As shown in Figure 3, the slopes of the lines between different matching point pairs are different.

[0075] In one embodiment, Figure 4 As shown in FIG, the execution flow of S4-S5 is given. Among them, S5 includes the following steps:

[0076] S51: Assign the slope set to n element intervals, and select matching point pairs that meet the frequency requirement according to the frequency of occurrence of the slope values in the element intervals to obtain a second matching point pair set, where n≥1;

[0077] S52: Use the RANSAC algorithm to perform a elimination operation on the second set of matching point pairs, and calculate the registration matrix and registration error of the remaining matching point pairs, and determine whether the registration error is less than the set threshold. If so, retain the registration matrix of the matching point pair and enter S6. If not, increase the number of element intervals n and return to S51.

[0078] In this embodiment, the least square method is used in S52 to calculate the registration matrix of the remaining matching point pairs.

[0079] In the specific implementation of this embodiment, it is assumed that the reference image is I1 and the image to be registered is I2. The feature point sets of I1 and I2 obtained by matching in the above steps are expressed as:

[0080]

[0081] Where M is the number of matching pairs between the two images; the slope of any matching pair j in the two images is expressed as:

[0082]

[0083] Set the slope threshold between ±10, i.e. |k j |<10, the values that meet the threshold constitute the slope set K q ={k1,k2,...,k q}, 1≤q≤M, q is the number of elements in the set.

[0084] The set K q The elements in the interval are divided into intervals, and the initial number of intervals n = 1, which is to set K q All elements in the interval are classified into one interval, and the slope value k with the highest frequency in the interval is counted, and matching points with slopes in the range of [k-0.5, k+0.5] are selected.

[0085] The RANSAC algorithm is used to further eliminate incorrect matching point pairs. The least squares method is used to calculate the registration matrix for the retained correct matching point pairs. The relative error E of the matching pairs is calculated. If the error is less than the set threshold, the registration matrix between the two images is obtained. Otherwise, the number of intervals n is gradually increased to reduce the slope interval until the calculated error E is less than the threshold.

[0086] E=|P1-P2×TM| 2 ,i=1,2,3,4...;

[0087] Among them, P1 is the matching point in the reference image I1, P2 is the matching point in the image to be registered I2, and TM is the registration matrix obtained based on the least squares method. The matching results of the improved RANSAC algorithm with slope constraint and interval optimization are as follows: Figure 5 shown.

[0088] It should be noted that when the number of intervals n>1, the slope set is distributed to n element intervals using the rule of average distribution.

[0089] like Figure 6 As shown, it is a pseudo-color comparison image of the two images before and after registration after S1-S6 processing, wherein Figure (a) is the superposition effect image of the two images before registration, and Figure (b) is the superposition effect image of the two images after registration. It can be seen that the registration method of the present invention realizes the spatial alignment of the two images.

[0090] The terahertz video radar image registration method provided by the present invention is introduced in detail above. In this embodiment, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

[0091] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in this embodiment may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown in this embodiment, but is intended to conform to the widest scope consistent with the principles and novel features disclosed in this embodiment.

Claims

1. A terahertz video radar image registration method, characterized in that: The terahertz video radar imaging system moves at a constant speed along a straight path and forms an image in a stripe pattern, comprising the following steps: S1: Receive the image frame sequence of the terahertz video radar, the rth frame image is used as the reference image, and the r+1th frame image is used as the image to be registered, r≥1; S2: Perform edge cropping on the image frame to obtain a cropped image frame, and use the KAZE algorithm to perform feature detection on the cropped reference image and the image to be registered to obtain a set of feature points of the two images and their corresponding feature descriptor sets; S3: Search the feature descriptor set of the reference image and the image to be registered to obtain matching initial feature point pairs and obtain a matching point pair set 1; S4: Based on the slope constraint between the two matching point pairs, eliminate the matching point pairs that do not satisfy the slope constraint in the matching point pair set 1, obtain the slope values of the matching point pairs that satisfy the slope constraint, and form a slope set; S5: allocating the slope set to a number of element intervals based on the element interval optimization strategy, and performing a culling operation on the matching point pairs within the element intervals using the RANSAC algorithm to calculate and obtain the registration matrix of the matching point pairs after culling; S6: Mapping the image to be registered to the coordinate system of the reference image using the registration matrix obtained in S5 to achieve spatial alignment of the two images.

2. The terahertz video radar image registration method according to claim 1, characterized in that: The steps of using the KAZE algorithm to perform feature detection on two images include: S21: A variable conduction diffusion method and additive operator splitting algorithm are used to construct a pyramid-shaped nonlinear scale space for the image, generating multi-scale image layers; S22: Detecting pixel grayscale values in each multi-scale image layer to obtain initial extreme points; S23: Use the second-order Taylor expansion to correct the position, scale and grayscale value of the extreme point to obtain the precise location of the extreme point; S24: allocating a main direction to the precise positioning extreme point based on the wavelet feature to obtain a feature descriptor including the direction.

3. The terahertz video radar image registration method according to claim 2, characterized in that: The S22 includes the following steps: S221: Obtaining neighboring pixels of each pixel, where the neighboring pixels include: pixels in the same multi-scale image layer adjacent to the current pixel, pixels at the same position in the upper and lower layers corresponding to the multi-scale image layer where the current pixel is located, and their adjacent pixels in the same layer; S222: Compare the grayscale values of the current pixel point with those of its neighboring pixel points. If the grayscale value of the current pixel point is greater than / less than all of its neighboring pixel points, the current pixel point is an initial extreme value point.

4. The terahertz video radar image registration method according to claim 2, characterized in that: The S24 includes the following steps: S241: Select the position coordinates of the current precise positioning extreme value point as the center, calculate the wavelet features of all precise positioning extreme value points in a 60° sector area centered on the position coordinates, and perform Gaussian weighting on the results to obtain the wavelet feature value in the current sector area; S242: Rotate the current sector area with a rotation step of 60° and repeat S241 until the 360° range is traversed to obtain the sector area with the largest wavelet eigenvalue, and determine its direction as the main direction of the current precise positioning extreme point.

5. The terahertz video radar image registration method according to claim 1, characterized in that: The step S3 includes: using an approximate nearest neighbor algorithm to measure the similarity between feature points of a feature descriptor set of the reference image and the image to be registered, and taking feature point pairs that meet a similarity matching threshold as matching point pairs.

6. The terahertz video radar image registration method according to claim 1, characterized in that: The step S4 includes: counting the slope value with the highest frequency in the element interval, and screening out matching point pairs within a preset range of the slope value.

7. The terahertz video radar image registration method according to claim 1, characterized in that: The S5 comprises the following steps: S51: Distribute the slope set to n element intervals, and select matching point pairs that meet the frequency requirement according to the occurrence frequency of the slope values in the element intervals to obtain a second matching point pair set, where n≥1; S52: Use the RANSAC algorithm to perform a elimination operation on the second set of matching point pairs, and calculate the registration matrix and registration error of the remaining matching point pairs, and determine whether the registration error is less than the set threshold. If so, retain the registration matrix of the matching point pair and enter S6. If not, increase the number of element intervals n and return to S51.

8. The terahertz video radar image registration method according to claim 7, characterized in that: In S52, the least square method is used to calculate the registration matrix of the remaining matching point pairs.

Images