Continuous frame real-time registration method

Abstract

The invention relates to a continuous frame real-time registration method, which comprises the following steps of: 1, registering a current frame and a reference frame to obtain a registration transformation matrix; 2, taking the current frame as a reference frame, and obtaining a new current frame; 3, registering the current frame and the reference frame to obtain an inter-frame transformation matrix, judging whether the current frame meets a continuous frame registration condition or not according to a calculation penetration transformation matrix of the registration transformation matrix and the inter-frame transformation matrix, if so, taking the penetration transformation matrix as the registration transformation matrix of the current frame, and returning to the step 2; and if not, obtaining a new current frame, and returning to the step 1. According to the continuous frame real-time registration method provided by the invention, continuous registration is carried out by utilizing the characteristics of small displacement deviation, large overlapping area, high registration rate and accuracy between the continuous frames, the registration between the current frame and the initial reference frame is more accurate, and the registration accuracy and the registration efficiency of the current frame and the initial reference frame are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and particularly to a method for real-time registration of consecutive frames. Background Art

[0002] Image registration was first proposed by the United States in the 1970s and was mainly applied to flight navigation and weapon guidance in the military field at that time. With the development of science and technology and informatization, image registration technology has been widely used in various fields such as security, medical treatment, transportation, astronomy, and education. The purpose of image registration is to correspond the mutual positional relationships of images, and it often serves as a necessary preprocessing step for applications such as image restoration, image enhancement, target extraction and recognition, and information mining.

[0003] With the continuous development of image registration technology, many image registration methods have been derived, including those based on gray-scale features, spectral features, feature point matching, joint histograms, transformation models, deep learning technology, etc., providing registration solutions for various transformation relationships such as translation, rotation, distortion, scaling, and mirroring between images. Along with the complication of the transformation relationships between images, and currently with the improvement of the image acquisition frame rate and resolution, the requirement for registration accuracy is getting higher and higher. However, the existing image registration methods cannot meet the needs of high-precision registration of consecutive frames. How to quickly and accurately achieve batch registration of consecutive frame image sets has become an important issue in the industry, and this issue has important significance for the further processing and information mining of subsequent images. Summary of the Invention

[0004] Embodiments of the present invention provide a method for real-time registration of consecutive frames. By utilizing the characteristics that the displacement deviation between consecutive frames is small, the overlapping area is large, the registration success rate is high and accurate, a method for real-time registration of consecutive frames with high speed and high precision is designed.

[0005] In order to achieve the above object, embodiments of the present invention provide the following technical solutions:

[0006] A method for real-time registration of consecutive frames includes the following steps:

[0007] Step 1: Register the current frame with the reference frame to obtain a registration transformation matrix;

[0008] Step 2: Use the current frame as the reference frame and obtain a new current frame;

[0009] Step 3: Register the current frame with the reference frame to obtain an inter-frame transformation matrix. Calculate a penetration transformation matrix based on the registration transformation matrix and the inter-frame transformation matrix, and determine whether the current frame meets the consecutive frame registration condition. If it meets, use the penetration transformation matrix as the registration transformation matrix of the current frame and return to Step 2; if it does not meet, obtain a new current frame and return to Step 1.

[0010] Further, the method for selecting the reference frame in step 1: Obtain multiple candidate images, divide each candidate image into multiple regions respectively, obtain the feature points in each region, and the number of feature points obtained in each region does not exceed the upper limit value; Select the candidate image with the best feature point distribution as the pending image; Determine whether the feature point distribution in the pending image meets the preset conditions. If it meets, determine the pending image as the reference frame, otherwise obtain candidate images again.

[0011] Further, step 1 includes: Divide the current frame into multiple regions, and each region is respectively matched with the entire reference frame; Calculate the registration transformation matrix according to the matching point pairs.

[0012] Further, the method for obtaining the inter-frame transformation matrix in step 3 includes: Divide the current frame into multiple current sub-regions, divide the reference frame into multiple reference sub-regions, there is an overlap between the reference sub-regions, and the area of the reference sub-region is larger than the corresponding current sub-region; Each current sub-region is respectively matched with its corresponding reference sub-region; Calculate the inter-frame transformation matrix according to the matching point pairs.

[0013] Further, it also includes screening of the matching point pairs:

[0014] Calculate the basic transformation matrix according to all the matching point pairs; Obtain the basic coordinates of the matched feature points in the current frame after being transformed by the basic transformation matrix, calculate the difference between the basic coordinates and the coordinates of the corresponding feature points in the reference frame. If the difference does not meet the preset conditions, determine this matching point pair as unmatched; For each unmatched feature point in the current frame, search for unmatched feature points in the reference frame within the preset coordinate range around its basic coordinates and calculate the matching degree, and take the feature point with the highest matching degree and greater than the first threshold as its matching point.

[0015] Further, it also includes filtering of the matching point pairs:

[0016] Calculate the coordinate difference of each pair of matching points, respectively set the coordinate difference of each pair of matching points as the benchmark, calculate the difference between the coordinate differences of the remaining pairs of matching points and the benchmark. If the difference meets the preset conditions, associate this matching point pair with the benchmark; Then count the number of matching point pairs associated with each benchmark, and take the matching point pair corresponding to the benchmark with the largest number and the associated matching point pairs as the filtered matching point pairs.

[0017] Further, the determination of whether the current frame meets the continuous frame registration condition includes:

[0018] Set the verification interval; When the verification interval is not reached, determine that the current frame meets the continuous frame registration condition; When the verification interval is reached, verify the current frame, including:

[0019] Obtain the matching point pairs between the current frame and the original reference frame; if the number of the matching point pairs meets the preset condition, determine that the current frame meets the continuous frame registration condition, and calculate the updated penetration transformation matrix according to the matching point pairs.

[0020] Further, the method for obtaining the matching point pairs between the current frame and the original reference frame is as follows:

[0021] Obtain the matching point pairs between the current frame and the original reference frame through the mapping relationship of the matching point pairs in each adjacent frame established in advance; for each feature point that is not matched in the current frame, obtain its initial coordinates after being transformed by the penetration transformation matrix, search for the feature points that are not matched in the original reference frame within the preset coordinate range around the initial coordinates and calculate the matching degree, and use the feature point with the highest matching degree and greater than the second threshold as its matching point.

[0022] Further, if the current frame meets the continuous frame registration condition, obtain the displacement between the current frame and the original reference frame according to the parameters of the penetration transformation matrix; if the displacement is greater than the third threshold, use the current frame as the key node frame, and the verification of the subsequent frames of the key node frame includes: obtaining the matching point pairs between the subsequent frame and the key node frame. If the number of the matching point pairs meets the preset condition, determine that the subsequent frame meets the continuous frame registration condition, calculate the node transformation matrix according to the matching point pairs, and calculate and update the penetration transformation matrix according to the node transformation matrix and the registration transformation matrix of the key node frame; and, if the subsequent frame passes the verification, obtain the displacement between the subsequent frame and the key node frame according to the parameters of the node transformation matrix; if the displacement is greater than the fourth threshold, update the subsequent frame to the key node frame.

[0023] Further, obtain the displacement between the subsequent frame and the original reference frame according to the parameters of the penetration transformation matrix of the subsequent frame; if the displacement is less than the fifth threshold, the verification of the subsequent frame includes:

[0024] Obtain the matching point pairs between the subsequent frame and the original reference frame; if the number of the matching point pairs meets the preset condition, determine that the subsequent frame meets the continuous frame registration condition, and calculate the updated penetration transformation matrix according to the matching point pairs.

[0025] The embodiments of the present invention have the following advantages:

[0026] A continuous frame real-time registration method provided by the present invention does not adopt the existing method of directly and frame-by-frame registering the current frame with the initial reference frame. Instead, it utilizes the characteristics that the displacement deviation between consecutive frames is small, the overlapping area is large, the registration success rate is high and accurate to perform continuous registration between adjacent frames, and accumulatively calculates the registration transformation matrix obtained by registering adjacent frames to obtain the registration transformation matrix between the current frame and the initial reference frame, thereby obtaining a more accurate registration relationship between the current frame and the initial reference frame, greatly improving the registration accuracy between the current frame and the initial reference frame, and improving the registration efficiency. Moreover, this technology also adds the verification and update of the registration transformation matrix to eliminate the cumulative error in the cumulative calculation of the transformation matrix and ensure the accuracy of the registration transformation matrix. This technology also adds the establishment of a feature point penetration queue, which utilizes the continuous accumulation of paired point pairs between adjacent frames, and supplements the search for feature points on this basis to obtain the most accurate paired point pairs between the current frame and the reference frame, thereby improving the verification speed and verification accuracy of matrix verification.

[0027] A continuous frame real-time registration method provided by the present invention also divides the image into multiple regions and limits the upper limit of the acquisition of feature points for each region, making the distribution of feature points detected on the entire image more uniform, and solving the problem that feature points converge in a certain feature-obvious region. And in continuous multi-frame images, it is possible to select the image with the best feature point distribution as the reference frame, thereby ensuring the quality of the reference frame, facilitating subsequent image registration, and further improving the registration accuracy.

[0028] When the continuous frame real-time registration method provided by the present invention registers the reference frame and the current frame, it uses a sub-region matching method, divides the current frame into multiple regions, and each region is respectively registered with the entire reference frame to eliminate the mutual influence between regions during registration, reduce errors, and improve the success rate of correct pairing. For each feature point in the current frame, obtain the coordinates of the feature point after transformation, calculate the distance between its coordinates and the coordinates of the corresponding feature point in the reference frame. If the distance is less than the threshold, it is considered a match; for the un-matched feature points with a distance exceeding the threshold, search for other un-matched feature points in a specific area around them and calculate the matching degree respectively. The feature point with a matching degree greater than the threshold and the highest matching degree is used as its matching point for passing the match. Then, recalculate the transformation matrix based on the points that pass the match, so that the transformation matrix can be more accurate and the registration accuracy can be improved.

[0029] A continuous frame real-time registration method provided by the present invention judges the displacement degree of the current frame relative to the reference frame, sets the current frame that passes the verification of the transformation matrix and has a large displacement as a key node frame, and subsequent frames re-establish the feature point penetration queue and recalculate the transformation matrix based on this key node frame, thereby reducing the influence of the cumulative error of the transformation matrix. Description of the Drawings

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary. For those of ordinary skill in the art, without creative efforts, other implementation drawings can also be obtained based on the provided drawings.

[0031] The structures, proportions, sizes, etc. illustrated in this specification are only used to cooperate with the content disclosed in the specification for those who are familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Therefore, they do not have substantial technical significance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope that can be covered by the technical content disclosed by the present invention.

[0032] Figure 1 It is a method flow chart of a continuous frame real-time registration method provided by an embodiment of the present invention;

[0033] Figure 2 It is a method flow chart for selecting a reference frame in a continuous frame real-time registration method provided by an embodiment of the present invention;

[0034] Figure 3 It is a comparison diagram of the effects of the image processing method for the current image 2 in an embodiment of the present invention. The left figure is the image without dividing the image area, and the right figure is the image divided into 8 * 8 image areas, and the upper limit R of the feature points in each image area is set to 30;

[0035] Figure 4 For Figure 1 It is the registration flow chart between the reference frame and the current frame;

[0036] Figure 5 It is a method flow chart for screening matching point pairs in a continuous frame real-time registration method provided by an embodiment of the present invention;

[0037] Figure 6 For Figure 1 It is the method flow chart for obtaining the inter-frame transformation matrix;

[0038] Figure 7 It is a comparison diagram of the image division between the current frame and the reference frame in an embodiment of the present invention. The left image is the current frame divided into 2 * 2 image areas, and the right image is the reference frame divided into 2 * 2 image areas. The black area in the right image is the overlapping area;

[0039] Figure 8 It is a method flow chart for determining whether the current frame meets the registration conditions in a continuous frame real-time registration method provided by an embodiment of the present invention;

[0040] Figure 9 Schematic diagram of the change of the feature point penetration queue in a continuous frame real-time registration method according to an embodiment of the present invention;

[0041] Figure 10 Schematic diagram of the change of the registration transformation matrix of subsequent frames after adding key node frames in a continuous frame real-time registration method according to an embodiment of the present invention. Specific embodiments

[0042] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] Embodiment 1

[0044] As Figure 1 shown, a continuous frame real-time registration method is characterized by including the following steps:

[0045] Step 1: Register the current frame with the reference frame to obtain a registration transformation matrix;

[0046] Step 2: Take the current frame as the reference frame and obtain a new current frame;

[0047] Step 3: Register the current frame with the reference frame to obtain an inter-frame transformation matrix, calculate a penetration transformation matrix according to the registration transformation matrix and the inter-frame transformation matrix, and determine whether the current frame meets the continuous frame registration condition. If it meets, take the penetration transformation matrix as the registration transformation matrix of the current frame and return to Step 2; if it does not meet, obtain a new current frame and return to Step 1.

[0048] The implementation methods, beneficial effects and extended solutions of the above steps will be further described below in combination with specific embodiments:

[0049] In the said Step 1, the method for determining the reference frame includes:

[0050] As Figure 2 shown, obtain multiple candidate images, divide each candidate image into multiple regions respectively, obtain the feature points in each region, and the number of feature points obtained in each region does not exceed the upper limit value to avoid the situation of feature point clustering. If feature point detection is performed on the entire image, it is very easy to have the situation that feature points cluster and focus on a certain feature obvious area, such as Figure 3As shown in the left figure in the middle, the upper limit of feature points in this figure is 1920. Therefore, in this method, a frame of image is divided into N*N or N*M image regions, and then the feature points of each image region are obtained. In this embodiment, the method for obtaining feature points is not limited, such as using feature point detection methods such as fast, Harris, SIFT, etc. to obtain feature points. The upper limit of feature points obtained for each image region is R, which can make the feature points detected on the entire image more evenly distributed, and solve the problem that feature points converge in a certain feature - obvious region, such as Figure 3 The right figure in Figure 3 divides the image into 8*8 image regions, and the upper limit R of feature points in each image region is set to 30.

[0051] In order to select an image with the best or better image quality as the reference frame, it specifically includes:

[0052] a. Evaluate frame by frame

[0053] Select an alternative image with the best feature - point distribution from multiple alternative images as the pending image. For example, select the alternative image with the largest total number of feature points, or use the regions where the number of feature points in the region is greater than the set value as qualified regions, and select the alternative image with the largest number of qualified regions, etc. It is also possible to statistically analyze the feature - point distribution of the alternative images, and select the alternative image as the pending image according to the mean value or maximum value of the statistically analyzed feature points.

[0054] Then determine whether the feature - point distribution in the pending image meets the preset conditions. If it meets, determine the pending image as the reference frame; otherwise, obtain alternative images again. The judgment method is, for example: compare the number of feature points in the pending image with the feature - point threshold, and determine whether it meets the preset conditions according to the comparison result. The feature - point threshold can be set for the total number of all feature points in the entire image, or for the number of feature points in each region, and then summarize the situations of all regions as the preset conditions. The comparison result can be simply greater than or less than the threshold, or the ratio with the feature - point threshold, etc.

[0055] In this embodiment, the feature - point threshold set for the number of feature points in each region is at least half of the upper limit R of feature points in the image region, preferably two - thirds of the upper limit R of feature points in the image region. For example, if R = 30, then the feature - point threshold = 20. If the number of feature points in the image region is greater than or equal to the feature - point threshold, it means that the quality of this image region is good, the image texture is clear, and the image contrast is high. After accumulating all image regions and taking the average value, use the preset value of this average value as the preset condition for the entire image.

[0056] In this embodiment, the pending image that meets the preset conditions is determined as the reference frame; otherwise, alternative images are obtained again.

[0057] b. Multi-frame evaluation

[0058] If the pending image selected from multiple consecutive alternative images does not meet the preset conditions, multiple alternative images can be retrieved again. If the selected pending image still does not meet the preset conditions, a prompt is returned to prompt the user to adjust the imaging, which may indicate problems such as shooting failure or equipment failure. In this embodiment, the number of alternative images obtained each time is not limited. For example, 20 frames are selected for the first time and 10 frames are selected for the second time, etc.

[0059] In step 1, the method for registering the current frame with the reference frame includes:

[0060] After determining the reference frame, the current frame is divided into multiple regions. The regions divided when selecting the reference frame before can be directly used. In this embodiment, the method of descriptor matching is adopted, and other existing registration methods can also be used.

[0061] In this embodiment, the descriptors of the feature points in the reference frame are calculated. The current frame is divided into N*N or N*M image regions, and the feature points and corresponding descriptors in each image region are obtained respectively. The type of descriptor is not limited in this technology. In this embodiment, the BRIEF descriptor is selected.

[0062] Each region of the current frame is respectively matched with the entire reference frame. Specifically, the descriptors of each image region of the current frame are respectively matched with all the descriptors of the reference frame. In this embodiment, the BFmatcher function is used to perform the matching operation between the feature points, and the paired point pairs between each region of the current frame and the reference frame are obtained.

[0063] The paired point pairs of each region are aggregated to obtain the set matchesAll, and then the registration transformation matrix is calculated according to the matching point pairs in the set matchesAll.

[0064] In this embodiment, the robust least squares method or RANSAC is selected to solve the transformation matrix. The calculation formula of the registration transformation matrix is:

[0065]

[0066] In the formula, (x, y) is the coordinate of the feature point in the reference frame in the paired point pair,

[0067] (xA, yA) is the coordinate of the feature point in the current frame in the paired point pair.

[0068] Let \(H\) be a \(3\times3\) transformation matrix. Since the application scenario of this embodiment is the registration of fundus images and the degree of transformation is small, a \(3\times3\) affine transformation matrix can be used. As known to those skilled in the art, in other application scenarios, other existing transformation matrices can be selected, such as a homography matrix, a perspective transformation matrix, a binary quadratic transformation matrix, a similarity transformation matrix, etc., and the formulas in the embodiment will be adjusted accordingly.

[0069] In order to obtain a more accurate registration transformation matrix and further improve the registration accuracy between the reference frame and the current frame, before calculating the registration transformation matrix, it also includes screening the matching point pairs in the set \(matchesAll\). This screening step is not only performed before calculating the registration transformation matrix, but also can be screened and the transformation matrix can be updated according to needs after each calculation of the transformation matrix during the subsequent entire registration process to improve the accuracy, as Figure 5 shown, the screening step includes:

[0070] Calculate the basic transformation matrix according to all the matching point pairs in the set \(matchesAll\).

[0071] Then obtain the basic coordinates of the matched feature points of the current frame in the set \(matchesAll\) after being transformed by the basic transformation matrix, and calculate the difference between the basic coordinates and the coordinates of the corresponding feature points in the reference frame. If the difference does not meet the preset conditions, then determine that the matching point pair is unmatched. For example, calculate the basic coordinates of a certain current frame feature point in the paired point pair after being transformed by the basic transformation matrix, and find the reference frame feature point matched by this current frame feature point, and calculate the coordinate distance between the basic coordinates and the coordinates of this reference frame feature point. If the coordinate distance is less than the distance threshold, then determine that this group of paired point pairs is matched; otherwise, if the coordinate distance is greater than the distance threshold, then determine that this group of paired point pairs is unmatched. The difference in coordinates can be to calculate the distance between two coordinate points, or to calculate the differences of the horizontal and vertical coordinates respectively, and use the sum or average value of the differences of the horizontal and vertical coordinates as the difference value.

[0072] For each unmatched feature point in the current frame, since the reference frame and the basic coordinates use the same coordinate system, all the unmatched reference frame feature points searched within the preset coordinate range around its basic coordinates are respectively calculated for the matching degree with this current frame feature point, and the reference frame feature point with the highest matching degree and greater than the first threshold is used as the matching point of this current frame feature point.

[0073] After that, a set of all matching point pairs matchesAll’ is obtained. It is also possible to determine whether the number of matching point pairs in the set matchesAll’ is greater than the matching point pair threshold. If the number of matching point pairs is less than the matching point pair threshold, the registration of the current frame and the reference frame fails, and the registration of the current frame of the next frame is entered; if the number of matching point pairs is greater than the matching point pair threshold, the robust least squares method or RANSAC is used to solve the transformation matrix of the set matchesAll’, and the registration transformation matrix between the reference frame and the current frame is obtained.

[0074] In step 2, the current frame is used as a new reference frame, and a new current frame is obtained. The new current frame can be the next frame after the new reference frame.

[0075] In step 3, the method for obtaining the inter-frame transformation matrix between the current frame and the reference frame:

[0076] Since the overlapping area between correct consecutive frames is large, each image region of the current frame (i.e., the new current frame) roughly corresponds to the image region in the reference frame (i.e., the new reference frame). As Figure 6 shown, the reference frame and the current frame adopt the same coordinate system. The current frame is divided into multiple current sub-regions, and the reference frame is divided into multiple reference sub-regions. The area of each reference sub-region is larger than its corresponding current sub-region. Preferably, the reference sub-region contains its corresponding current sub-region and expands around it. Therefore, there are overlapping regions between the reference sub-regions. The range of the overlapping region is 1 / 8 to 1 / 3 of the current sub-region, which can not only meet the required region range for matching, but also reduce the amount of matching data of the current sub-region, and avoid the error influence of regions that are far away on the matching result, thereby improving the matching accuracy and matching efficiency. As Figure 7 shown in the left image, it is divided into 2*2 current sub-regions. Figure 7 shown in the right image, it is divided into 2*2 reference sub-regions. The black region in the right image is the overlapping region between the upper left reference sub-region and the upper right reference sub-region. Each current sub-region is respectively matched with the reference sub-region at its corresponding position. If the overlapping area between the current sub-region and a certain reference sub-region is the largest, the current sub-region and the reference sub-region are in a corresponding relationship, and the matching point pairs between several groups of sub-regions are obtained. Then, the inter-frame transformation matrix is calculated according to the matching point pairs.

[0077] Since there is generally no large-angle or large-displacement deviation between consecutive frames, and there is no significant brightness change, the matching point pairs follow the principle that "the distances between two matching point pairs are close, and the angles of the vectors formed by the two matching point pairs are close". Therefore, the screening of matching point pairs in this technology can achieve brightness invariance and solve the problem of inaccurate registration when there is a large brightness change, a large rotation angle, or a small overlapping area between images. The screening method includes: calculating the coordinate differences of each pair of matching points, setting the coordinate differences of each pair of matching points as the reference respectively, calculating the differences between the coordinate differences of the remaining pairs of matching points and the reference, and if the differences meet the preset conditions, associating the matching point pair with the reference; counting the number of matching point pairs associated with each reference, and taking the matching point pair corresponding to the reference with the largest number and the associated matching point pairs as the screened matching point pairs.

[0078] For example, the coordinate differences are set as the distance distance and / or the vector angle angle between the matching point pairs. In this embodiment, both the distance and the vector angle are calculated. The calculation formulas for the distance distance and the vector angle angle are as follows:

[0079]

[0080]

[0081] In the formula, xA and xB are the abscissas of the two feature points in the matching point pair respectively;

[0082] yA and yB are the ordinates of the two feature points in the matching point pair respectively.

[0083] Calculate the distance and vector angle of each paired point among the matching point pairs of several groups of sub-regions. Respectively, taking the distance and vector angle of each pair of matching point pairs as a reference. For example, if the distance and vector angle of the first group of matching point pairs are set as the reference, then the remaining matching point pairs are the other matching point pairs relative to the first group of matching point pairs. Then, calculate the absolute value of the distance difference between the distance of the other matching point pairs and the distance of the first group of matching point pairs, and calculate the absolute value of the vector angle difference between the vector angle of the other matching point pairs and the vector angle of the first group of paired points respectively. If the absolute value of the distance difference is less than the distance threshold and the absolute value of the vector angle difference is less than the angle threshold, then the difference between the coordinate differences of these two groups of matching point pairs meets the condition. Associate the other matching point pairs with the first group of paired points. After traversing all the other matching point pairs, obtain all the other matching point pairs associated with the first group of paired points. The specific operation of "association" here can be to put these point pairs into a set, or to mark these point pairs, or to record and store their coordinates, etc. "Association" is just a process, and the required result is to compare the number of "associated" points for each group later; Similarly, set the distance and vector angle of the second group of matching point pairs as the reference, then the point pairs other than the second group of matching point pairs are the other matching point pairs, and the other paired points include the first group of paired points. Then, calculate the absolute value of the distance difference between the distance of the other matching point pairs and the distance of the second group of matching point pairs, and calculate the absolute value of the vector angle difference between the vector angle of the other matching point pairs and the vector angle of the second group of paired points respectively. If the absolute value of the distance difference is less than the distance threshold and the absolute value of the vector angle difference is less than the angle threshold, then the difference between the coordinate differences of these two groups of matching point pairs meets the condition. Associate the other matching point pairs with the second group of paired points. After traversing all the other matching point pairs, obtain all the other matching point pairs associated with the second group of paired points. And so on. After traversing all the matching point pairs, count the number of other matching point pairs associated with each reference. Take the matching point pairs corresponding to the reference with the largest number of associated other matching point pairs and the matching point pairs associated with this reference as the filtered matching point pairs to obtain the set matches. If the number of matching point pairs in the set matches is greater than the preset matching point pair threshold, then judge it as valid, and use the robust least squares method or RANSAC to solve the transformation matrix of the set matches, that is, obtain the inter-frame transformation matrix between the new current frame and the new reference frame; On this basis, if the number of matching point pairs in the set matches is less than the matching point pair threshold, then judge it as registration failure, indicating that the current matching is a particularly poor situation. In this case, the validity of the verification is very low, and it will affect the current frame after updating. Therefore, for particularly poor situations, there is no need to spend more time on verification and updating.

[0084] In step 3, the method of calculating the penetration transformation matrix based on the registration transformation matrix and the inter-frame transformation matrix is a prior art. In this embodiment, the penetration transformation matrix is directly obtained by multiplying the registration transformation matrix and the inter-frame transformation matrix. For example:

[0085]

[0086]

[0087]

[0088] In the formula, x and y are the abscissa and ordinate of the point in the reference frame. The transformation matrix H in this embodiment is a 3×3 matrix. Therefore, to meet the requirements of matrix multiplication, although the coordinate points in each frame are two-dimensional coordinates (x, y), a row of "1" needs to be added to become 3 rows; it can be understood that if the transformation matrix adopts other forms of matrices, such as a 2×5 matrix, the representation form of the coordinate points in each frame should also be adjusted accordingly, such as directly using to represent;

[0089] x A and y A are the abscissa and ordinate of the point in frame A, and H is the transformation matrix between the reference frame and frame A;

[0090] x B and y B are the abscissa and ordinate of frame B, and H AB is the transformation matrix between frame A and frame B; according to the principle of penetration by the transformation matrix, H*H AB is the transformation matrix between frame A and frame B.

[0091] And so on, calculate the penetration transformation matrix H M = H N * H NM between the initial reference frame and the current frame M. In specific cases, H M = H * H AB *... * H NM . This method can obtain a more accurate registration transformation matrix between the current frame and the initial reference frame by not directly participating the current frame in the registration of the initial reference frame, but accumulating the transformation matrices between adjacent frames, taking advantage of the characteristics that there are more overlapping features between consecutive frames and the inter-frame registration accuracy is higher.

[0092] Since there will always be errors in the transformation matrix between two frames, no matter how accurate it is, the accumulated error will become larger as the number of consecutive multiplications of the transformation matrix increases. Eventually, the accuracy of the transformation matrix between the current frame and the initial reference frame decreases as the number of frames increases. To improve the accuracy of the transformation matrix, eliminate the accumulated error, and ensure the accuracy of the transformation matrix, it is necessary to determine whether the current frame meets the continuous frame registration condition in step 3. This determination method is the verification and update of the transformation matrix. As Figure 8 shown, it includes:

[0093] Set the verification interval. When the verification interval is not reached, it is determined that the current frame meets the continuous frame registration condition; when the verification interval is reached, the transformation matrix of the current frame is verified.

[0094] Obtain the matching point pairs between the current frame and the original reference frame. By using the pre-established mapping relationship of the matching point pairs in each adjacent frame, the matching point pairs between the current frame and the original reference frame are obtained. The principle of establishing the mapping relationship of the matching point pairs in each adjacent frame is that if point A in one frame matches point B in another frame, and point B matches point C in other frames, then there is a mapping relationship among point A, point B, and point C. Through this mapping relationship, it can be known that point A also matches point C.

[0095] To facilitate the representation of this mapping relationship, in this embodiment, a feature point penetration queue is established to accelerate the verification speed and improve the verification accuracy in the subsequent matrix verification. The construction method of the feature point penetration queue is as follows:

[0096] First, construct a feature point queue from the matching point pairs between the initial reference frame and the first registered current frame. As Figure 9 shown, there are various matching point pairs between the feature points (R3, R4, R5, R6....) of the initial reference frame refer and the feature points (A1, A2, A3, A4....) of the current frame A in. A feature point queue RA (R3 - A1, R4 - A2, R5 - A3, R6 - A4....) is constructed.

[0097] After setting the first registered current frame as the new reference frame, several inter-frame matching point pairs are obtained between the new reference frame and the new current frame. If feature points identical to the inter-frame matching point pairs can be found in the feature point queue, then the inter-frame matching point pairs are incorporated into the feature point queue. After traversing all the inter-frame matching point pairs, a feature point penetration queue is obtained. As Figure 9The feature points (A2, A3, A4....) of the current frame A in the [context] are matched with the feature points (B1, B2, B3....) of the current frame B to obtain inter-frame matching point pairs (A2 - B1, A3 - B2, A4 - B3....). Matching point pairs (R4 - A2, R5 - A3, R6 - A4....) with the same feature points as the inter-frame matching point pairs can be found in the feature point queue,... After merging, (R4 - A2 - B1, R5 - A3 - B2, R6 - A4 - B3) can be obtained, and a penetration pairing point sequence RB (R4 - B1, R5 - B2, R6 - B3....) can also be obtained. If the feature point B6 in the current frame B does not find a pairing point in the reference frame B, the following operations are performed:

[0098] For the unmatched feature points in the current frame, obtain the initial coordinates of the unmatched feature points after being transformed by the penetration transformation matrix, then search for the unmatched feature points in the original reference frame within a preset coordinate range around the initial coordinates and calculate the matching degree. The feature point with the highest matching degree and greater than the second threshold is used as its matching point. Preferably, the second threshold is the same as the first threshold. For example, the feature point B6 in the current frame B is transformed by the penetration transformation matrix to obtain the initial coordinate B6'. The current frame and the reference frame use the same coordinate system. Therefore, the unmatched feature points in the original reference frame can be searched within a preset coordinate range around B6' and the matching degree can be calculated. The preset coordinate range is an area centered on B6' with a range length of length; select the feature point R with the highest matching degree with B6' and a matching degree greater than the second threshold k , and then pair the feature point B6 with the feature point R k to form a matching point pair R k - B6.

[0099] After obtaining all the matching point pairs between the current frame and the original reference frame according to the above penetration plus search method, if the number of the matching point pairs meets the preset conditions, the preset conditions such as the number of the matching point pairs is greater than the matching point pair threshold, or the ratio of the number of the matching point pairs to the total number of feature points is greater than the ratio threshold, etc.; then it is determined that the current frame meets the continuous frame registration condition, and an updated penetration transformation matrix is calculated according to the matching point pairs; if the number of the matching point pairs in the penetration pairing point sequence does not meet the preset conditions, and the number of feature points in the current frame is less than the threshold, then it is determined that the current frame does not meet the continuous frame registration condition, a new current frame is obtained, and the current frame is verified. If the number of the matching point pairs in N consecutive current frames does not meet the preset conditions, and the number of feature points in the current frames is not greater than the threshold, then it is determined that the N current frames do not meet the continuous frame registration condition. Setting a verification interval can improve efficiency. In this embodiment, the frequency of verifying the transformation matrix is not limited. The transformation matrix can be verified after multiple frames of registration at intervals. To obtain the best registration accuracy, it is preferably to verify the transformation matrix frame by frame.

[0100] During the process of image registration, if there is a situation where the current frame has a large displacement relative to the reference frame, the overlapping area between the current frame and the reference frame will become smaller. In this case, it is impossible to verify the transformation matrix. Therefore, based on the condition that the current frame meets the continuous frame registration condition, this embodiment also adds the selection and update of key node frames to handle the possible situation of large displacement, including:

[0101] Obtain the displacement between the current frame and the original reference frame according to the parameters of the penetration transformation matrix. If the displacement is greater than the third threshold, then use the current frame as the key node frame; if the displacement is less than the third threshold, then perform the registration of subsequent frames. For example, judge the displacement size according to parameter c or parameter f in the penetration transformation matrix. If the absolute value of parameter c or parameter f or the square root of the sum of the squares of parameter c and parameter f is greater than the third threshold, it means that the displacement change between the current frame and the reference frame is too large, and then use the current frame as the key node frame.

[0102] Then the verification of the subsequent frames of the key node frame includes:

[0103] Obtain the matching point pairs between the subsequent frame and the key node frame. If the number of the matching point pairs meets the preset condition, then determine that the subsequent frame meets the continuous frame registration condition, calculate the node transformation matrix according to the matching point pairs, and calculate the updated penetration transformation matrix according to the node transformation matrix and the registration transformation matrix of the key node frame. For example, Figure 10 The penetration transformation matrix HD between the current frame D and the initial reference frame H, that is, the registration transformation matrix HD of the current frame D. Since both parameter c and parameter f in the registration transformation matrix HD are greater than the relevant parameter thresholds, the current frame D is updated to the key node frame D, and the registration transformation matrix HD of the key node frame D is retained. Then for the subsequent frames of the key node frame D, that is, the subsequent frame F and the subsequent image frames, re - establish the penetration sequence starting from the key node frame D, that is, the points in the subsequent other frames penetrate to the key node frame D, instead of penetrating to the original reference frame refer. Obtain the matching point pairs between the subsequent frame and the key node frame D to eliminate the cumulative error of feature point penetration. If there is a current frame that is successfully registered with the key node frame D, such as the subsequent frame F, then calculate the node transformation matrix DF between the key node frame D and the subsequent frame F according to the matching point pairs between them, and then calculate and update the penetration transformation matrix HF according to the node transformation matrix DF between the key node frame D and the subsequent frame F and the registration transformation matrix HD of the key node frame D. For example, obtain the penetration transformation matrix HF according to the product of the node transformation matrix DF and the registration transformation matrix HD, HF = HD * DF.

[0104] If the subsequent frame passes the verification performed on the subsequent frames of the key node frame, obtain the displacement between the subsequent frame and the key node frame according to the parameters of the node transformation matrix. If the displacement is greater than the fourth threshold, update the subsequent frame to the key node frame. For example, if the subsequent frame K passes the verification, obtain the displacement between the subsequent frame K and the key node frame D according to the parameter c or parameter f of the node transformation matrix between the subsequent frame K and the key node frame D. If the displacement is greater than the fourth threshold, use the subsequent frame K as the new key node frame K, and retain the registration transformation matrix HK of the key node frame K. The purpose of setting the key node frame is to weaken the influence of cumulative error. Therefore, in this embodiment, it is preferred that this step and the verification of the transformation matrix act together. For example, if the verification interval set above is 0 and the transformation matrix is verified frame by frame, then the key node frame D has passed the verification of the transformation matrix, and the matching point pairs between the current frame D and the original reference frame are obtained, including not only the matching point pairs obtained according to the mapping relationship but also the supplemented matching point pairs obtained by searching. The matrix HD calculated in this way eliminates the cumulative error caused by consecutive multiplications. The same applies to HK, and HM = HK * KM.

[0105] Since the movement between images is random, there will be a situation where the overlapping area between the subsequent frame N and the initial reference frame refer becomes larger when it moves back, that is, the registration requirement between the current frame N and the initial reference frame refer is met. In this case, the previously key node frames with larger displacements are no longer needed. If the key node frames are still used for registration, it may reduce the registration accuracy. Therefore, in this embodiment, the displacement between the current frame and the original reference frame is also obtained according to the parameters of the penetration transformation matrix. If the displacement is less than the fifth threshold, it means that the subsequent frame has moved back and the overlapping area between it and the initial reference frame refer has become larger, and then the verification rule for the subsequent frame of the key node frame will change. For example, according to the parameter c or parameter f in the penetration transformation matrix, if the absolute value of the parameter c or parameter f or the square root of the sum of the squares of the parameters c and f is less than the relevant parameter threshold, it means that the displacement change between the current frame and the reference frame is not significant, and there is no need to set a key node frame. Then, it is necessary to verify whether the current frame meets the continuous registration condition. The verification method includes:

[0106] Obtain the matching point pairs between the subsequent frame and the original reference frame. If the number of the matching point pairs meets the preset condition, determine that the subsequent frame meets the continuous frame registration condition, and calculate the updated penetration transformation matrix according to the matching point pairs.

[0107] Although the present invention has been described in detail with general descriptions and specific embodiments above, based on the present invention, some modifications or improvements can be made, which are obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of the present invention claimed.

Claims

1. A method for real-time registration of consecutive frames, characterized in that, it includes the following steps: Step 1: Register the current frame with the reference frame to obtain a registration transformation matrix; Step 2: Take the current frame as the reference frame and obtain a new current frame; Step 3: Register the current frame with the reference frame to obtain an inter-frame transformation matrix, calculate a penetration transformation matrix based on the registration transformation matrix and the inter-frame transformation matrix, and determine whether the current frame meets the consecutive frame registration condition. If it meets, take the penetration transformation matrix as the registration transformation matrix of the current frame and return to Step 2; If it does not meet, obtain a new current frame and return to Step 1.

2. The method for real-time registration of consecutive frames according to claim 1, characterized in that, the reference frame in Step 1 is specifically selected by the following method: Obtain multiple candidate images, divide each candidate image into multiple regions respectively, obtain the feature points in each region, and the number of feature points obtained in each region does not exceed the upper limit value; Select a candidate image with the best feature point distribution as the pending image; Judge whether the feature point distribution in the pending image meets the preset conditions. If it meets, determine the pending image as the reference frame, otherwise obtain candidate images again.

3. The method for real-time registration of consecutive frames according to claim 1, characterized in that, Step 1 includes: Divide the current frame into multiple regions, and each region is respectively matched with the entire reference frame; Calculate the registration transformation matrix according to the matching point pairs.

4. The method for real-time registration of consecutive frames according to claim 1, characterized in that, in Step 3, registering the current frame with the reference frame to obtain an inter-frame transformation matrix includes: Divide the current frame into multiple current sub-regions, divide the reference frame into multiple reference sub-regions, there is an overlap between the reference sub-regions, and the area of the reference sub-region is larger than the corresponding current sub-region; Each current sub-region is respectively matched with its corresponding reference sub-region; Calculate the inter-frame transformation matrix according to the matching point pairs.

5. The method for real-time registration of consecutive frames according to claim 1, characterized in that, it also includes screening of matching point pairs: Calculate a basic transformation matrix according to all matching point pairs; Obtain the basic coordinates of each matched feature point in the current frame after being transformed by the basic transformation matrix, calculate the difference between the basic coordinates and the coordinates of the corresponding feature point in the reference frame. If the difference does not meet the preset conditions, determine this matching point pair as unmatched; For each unmatched feature point in the current frame, search for unmatched feature points in the reference frame within a preset coordinate range around its basic coordinates and calculate the matching degree, and take the feature point with the highest matching degree and greater than the first threshold as its matching point.

6. The method for real-time registration of consecutive frames according to claim 1, characterized in that, it also includes screening of matching point pairs: Calculate the coordinate difference of each pair of matching points, respectively take the coordinate difference of each pair of matching points as the benchmark, calculate the difference between the coordinate differences of the remaining pairs of matching points and the benchmark. If the difference meets the preset conditions, associate this matching point pair with the benchmark; Count the number of matching point pairs associated with each reference, and use the matching point pairs corresponding to the reference with the largest number and the associated matching point pairs as the filtered matching point pairs.

7. A continuous frame real-time registration method according to claim 1, wherein, the determination of whether the current frame meets the continuous frame registration condition includes: setting a verification interval; when the verification interval has not been reached, determine that the current frame meets the continuous frame registration condition; when the verification interval is reached, verify the current frame, including: obtaining the matching point pairs between the current frame and the original reference frame; if the number of the matching point pairs meets the preset condition, determine that the current frame meets the continuous frame registration condition, and calculate an updated penetration transformation matrix according to the matching point pairs.

8. A continuous frame real-time registration method according to claim 7, wherein, the method for obtaining the matching point pairs between the current frame and the original reference frame is: obtaining the matching point pairs between the current frame and the original reference frame through the mapping relationship of the matching point pairs in each adjacent frame established in advance; for each feature point that is not matched in the current frame, obtain its initial coordinates after being transformed by the penetration transformation matrix, search for the feature points that are not matched in the original reference frame within the preset coordinate range around the initial coordinates and calculate the matching degree, and use the feature point with the highest matching degree and greater than the second threshold as its matching point.

9. A continuous frame real-time registration method according to claim 1, wherein, further comprising: if the current frame meets the continuous frame registration condition, obtain the displacement between the current frame and the original reference frame according to the parameters of the penetration transformation matrix; if the displacement is greater than the third threshold, use the current frame as a key node frame, and the verification of the subsequent frames of the key node frame includes: obtaining the matching point pairs between the subsequent frame and the key node frame; if the number of the matching point pairs meets the preset condition, determine that the subsequent frame meets the continuous frame registration condition, calculate a node transformation matrix according to the matching point pairs, and calculate and update the penetration transformation matrix according to the node transformation matrix and the registration transformation matrix of the key node frame; and, if the subsequent frame passes the verification, obtain the displacement between the subsequent frame and the key node frame according to the parameters of the node transformation matrix; if the displacement is greater than the fourth threshold, update the subsequent frame to a key node frame.

10. A continuous frame real-time registration method according to claim 9, wherein, further comprising: obtaining the displacement between the subsequent frame and the original reference frame according to the parameters of the penetration transformation matrix of the subsequent frame; if the displacement is less than the fifth threshold, the verification of the subsequent frame includes: obtaining the matching point pairs between the subsequent frame and the original reference frame; if the number of the matching point pairs meets the preset condition, determine that the subsequent frame meets the continuous frame registration condition, and calculate an updated penetration transformation matrix according to the matching point pairs.

Images