Image registration method and real-time tracking method

Abstract

The invention relates to an image registration method and a real-time tracking method, and the method comprises the steps: dividing a reference frame and a subsequent frame into a plurality of regions according to the same rule, carrying out the registration of the reference frame and the subsequent frame, and obtaining a region combination (a reference frame region and a subsequent frame region) where each pair of registration points are located; screening out matching point pairs according to the distribution condition of the registration point pairs in all region combinations; and calculating a transformation matrix according to the matching point pairs. As the change of the images between the continuous frames is not large, the method calculates the transformation matrix for registration by using the most representative local area, can avoid interference information brought by the influence of the environment and noise on the partial area in the image, can ensure that the screened matching point pair for registration has high reliability, and can improve the registration accuracy. In addition, calculation is simple and convenient, the rapid processing requirement of real-time registration can be met, and the method has wide application scenes and value.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and particularly relates to an image registration method and a real-time tracking method. Background Art

[0002] Image registration was first proposed by the United States in the 1970s and was mainly applied to flight navigation and weapon guidance in the military field. With the development of science and technology and informatization, image registration technology has been widely used in various fields such as security, medical treatment, transportation, astronomy, and education. The purpose of image registration is to correspond the mutual positional relationships of images, and it often serves as a necessary preprocessing for applications such as image restoration, image enhancement, target extraction and recognition, and information mining.

[0003] With the continuous development of image registration technology, many image registration methods have been derived, including those based on gray-scale features, spectral features, feature point matching, joint histograms, transformation models, deep learning technology, etc., providing registration solutions for various transformation relationships such as translation, rotation, distortion, scaling, and mirroring between images. Along with the complication of the transformation relationships between images, and currently with the improvement of the image acquisition frame rate and resolution, the requirement for registration accuracy is getting higher and higher. Therefore, a high-precision image registration technology is needed.

[0004] Moreover, with the development of video real-time transmission and processing technology, the requirement for the registration speed of consecutive images has also increased. Traditional registration methods are difficult to balance registration accuracy and speed, especially in the aspect of medical images, and cannot meet the requirements of high-precision real-time stable tracking registration. Summary of the Invention

[0005] Embodiments of the present invention provide an image registration method and a real-time tracking method to achieve high-precision image registration.

[0006] To achieve the above object, embodiments of the present invention provide the following technical solutions:

[0007] An image registration method includes the following steps:

[0008] Divide a reference frame and subsequent frames into multiple regions according to the same rule, register the reference frame and subsequent frames, and obtain the region combination (reference frame region, subsequent frame region) where each pair of registered points is located;

[0009] Screen out matching point pairs according to the distribution of registered point pairs in all region combinations;

[0010] Calculate a transformation matrix according to the matching point pairs.

[0011] Further, the screening of the matching point pairs according to the distribution of the registration point pairs in all region combinations includes: obtaining the region combination with the most registration point pairs, and determining the matching point pairs according to the registration point pairs in the region combination and the registration point pairs in the corresponding region combinations around the region combination.

[0012] Further, the screening of the matching point pairs according to the distribution of the registration point pairs in all region combinations includes: obtaining the registration point pairs in all region combinations and the corresponding region combinations around them, where the corresponding region combinations around the region combination include the corresponding region combinations determined by rotating the subsequent frame or the reference frame by at least one angle; determining the current rotation angle according to the number of the registration point pairs, obtaining the region combination corresponding to the current rotation angle, and determining the matching point pairs according to the registration point pairs corresponding to the region combination.

[0013] Further, the screening of the matching point pairs according to the distribution of the registration point pairs in all region combinations includes: selecting the registration point pairs according to the distribution of the registration point pairs in all region combinations; calculating the basic transformation matrix according to the selected registration point pairs; obtaining the basic coordinates of the registered feature points within the selected range in the reference frame after being transformed by the basic transformation matrix, and calculating the difference between the basic coordinates and the coordinates of the corresponding feature points in the subsequent frame. If the difference is less than the first threshold, the registration point pair is determined as a matching point pair.

[0014] Further, the selected range is obtained by the following method: determining the to-be-determined region combination according to the distribution of the registration point pairs in all region combinations; obtaining the representative coordinates of the reference frame region and the representative coordinates of the subsequent frame region in the to-be-determined region combination; calculating the difference between the coordinates of the representative coordinates of the reference frame region after being transformed by the basic transformation matrix and the representative coordinates of the subsequent frame region. If the difference is less than the second threshold, the reference frame region is determined as the selected range.

[0015] Further, after calculating the transformation matrix according to the matching point pairs, it further includes the evaluation of the registration accuracy: obtaining the matched feature points in the reference frame, and dividing the reference frame into multiple evaluation regions; judging whether the number distribution of the feature points in all evaluation regions meets the preset conditions; if not, supplementing the matching point pairs in the registration point pairs before screening.

[0016] A real-time tracking method, after calculating the transformation matrix, transforms the points to be tracked in the reference frame according to the transformation matrix to obtain the coordinates of the points in the subsequent frame.

[0017] Further, after calculating the transformation matrix according to the matching point pairs, it further includes: obtaining the displacement between the subsequent frame and the reference frame according to the parameters of the transformation matrix; if the displacement does not meet the preset conditions, supplementing the reference frame.

[0018] Further, the supplement to the reference frame includes: calculating the coordinates of the subsequent frame transformed into the reference frame according to the transformation matrix, determining whether the coordinates are the coordinates of the feature points in the reference frame, and if not, adjusting the descriptor of the feature points according to the transformation matrix, and taking the adjusted descriptor and feature points as the descriptor and feature points of the reference frame.

[0019] Further, during the tracking of continuous N frames, count the number of times each feature point in the reference frame is matched. If the number of times is less than the third threshold, delete the feature point.

[0020] The present invention has the following advantages:

[0021] An image registration method and a real-time tracking method provided by the present invention divide the reference frame and the subsequent frames into multiple regions according to the same rule, register the reference frame and the subsequent frames, and obtain the region combinations (reference frame region, subsequent frame region) where each pair of registered points is located; screen out the matching point pairs according to the distribution of the registered point pairs in all region combinations; calculate the transformation matrix according to the matching point pairs. Since the change of the image between consecutive frames is small, this method calculates the transformation matrix for registration with the most representative local regions, which can avoid the interference information brought by the influence of the environment and noise on some regions of the image, ensure that the selected matching point pairs for registration have high reliability, thereby improving the registration accuracy, and the calculation is simple, which can meet the fast processing requirements of real-time registration, and has a wide range of application scenarios and values. Description of the Drawings

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, other implementation drawings can be obtained according to the provided drawings without creative efforts.

[0023] The structures, ratios, sizes, etc. shown in this specification are only used to cooperate with the content disclosed in the specification for those who are familiar with this technology to understand and read, and are not used to limit the limited conditions under which the present invention can be implemented. Therefore, they do not have technical substance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope that the technical content disclosed by the present invention can cover.

[0024] Figure 1 is the method flow chart of the embodiment of the present invention;

[0025] Figure 2 is the effect comparison diagram of two feature point acquisition methods;

[0026] Figure 3 It is a schematic diagram of area numbering;

[0027] Figure 4 It is another schematic diagram of area numbering;

[0028] Figure 5 It is a display diagram of the 8-neighborhood between the reference frame and the subsequent frame in the embodiment of the present invention;

[0029] Figure 6 It is a method flowchart for evaluating the registration accuracy in the embodiment of the present invention;

[0030] Figure 7 It is a method flowchart for adjusting the registration point pairs in the embodiment of the present invention. Specific embodiments

[0031] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0032] As Figure 1 shown, an image registration method and a real-time tracking method include the following steps:

[0033] Divide the reference frame and the subsequent frame into multiple regions according to the same rule, register the reference frame and the subsequent frame, and obtain the region combination (reference frame region, subsequent frame region) where each pair of registration points is located; screen out the matching point pairs according to the distribution of the registration point pairs in all region combinations; calculate the transformation matrix according to the matching point pairs.

[0034] Among them, in order to obtain a better registration effect, this embodiment selects an image with the best quality or better quality as the reference frame. And, whether selecting the reference frame or during subsequent registration, if feature point detection is performed on the entire reference frame image, it is very easy to have a situation where feature points gather and focus on a certain feature-rich area, such as Figure 2 shown in the left figure in

[0035] There are 1920 feature points in this figure. To solve the above problems, the reference frame of this method is specifically selected through the following method:

[0036] Obtain multiple candidate images, divide each candidate image into multiple regions respectively, obtain the feature points in each region, and the number of feature points obtained in each region does not exceed the upper limit value;

[0037] Determine whether the distribution of feature points in the image to be determined meets the preset conditions. If it meets, determine the image to be determined as the reference frame; otherwise, obtain alternative images again.

[0038] Specifically, divide each alternative image into N*N or N*M image regions, and then obtain the feature points of each image region. In this embodiment, the method for obtaining feature points is not limited, such as using feature point detection methods like fast, Harris, SIFT, etc. to obtain feature points. Set an upper limit on the number of feature points obtained for each image region, so that the distribution of feature points detected on the entire image is more uniform, solving the problem that feature points converge in a certain feature - obvious region. For example, Figure 2 As shown in the right - hand figure, this figure shows that according to this method, the image is divided into 8*8 image regions, and the upper limit of the number of feature points in each image region is 30, and 1920 feature points are collected. The feature points in the figure are evenly distributed, more representative, and can reflect the features of each region. Using these feature points for registration can improve the accuracy of registration.

[0039] After obtaining the feature points, calculate the descriptors of the feature points. The type of descriptor in this technology is not limited. For example, calculate the centroid of the image region where the feature point is located, use the vector angle from the feature point coordinates to the centroid coordinates as the angle of the feature point, and determine the BRIEF descriptor according to the angle.

[0040] Similarly, during subsequent registration, also divide the subsequent frames into multiple regions, obtain the feature points in each region, and the number of feature points obtained in each region does not exceed the upper limit value; if the total number of feature points extracted does not meet the total threshold, or the number of feature points in multiple regions does not meet the region - specific threshold, it means that the image quality of the current subsequent frame is very poor and cannot be used for accurate registration. Abandon the current subsequent frame and obtain the next subsequent frame. For the convenience of registration, the subsequent - frame descriptors are preferably of the same type as the reference - frame descriptors.

[0041] The registration method between the reference frame and the subsequent frames includes:

[0042] Divide the reference frame and the subsequent frames into multiple regions according to the same rule, and register the reference frame and the subsequent frames. For example, divide both of them into 4*4 regions, and register the descriptors in the subsequent frame with the descriptors in the reference frame. The registration method can adopt existing technologies, such as calling the BFmatcher function provided by OPENCV for registration.

[0043] After the reference frame and the subsequent frames are divided into regions according to the same rule, naturally, reference - frame regions and subsequent - frame regions with one - to - one corresponding positions can be obtained.

[0044] Both the reference frame and the subsequent frames are divided into regions with a size of n*n, and the regional intervals of the abscissa and ordinate of each region are both n. For example, if the overall image sizes of the reference frame and the subsequent frames are both 1024*1024, the reference frame and the subsequent frames are both divided into 32*32 regions, and the size of each region is 32*32. For convenience of recording, each region can be numbered, and the numbering can be carried out according to the row and column rules. As Figure 3 shown, or directly numbered with numbers, as Figure 4 shown. The reference frame and the subsequent frames can be numbered according to the same rules or different rules.

[0045] For each pair of registration points, obtain the region where the coordinates of the reference frame in this pair of registration points are located. For example, if the reference frame point is located in Figure 4 the No. 1 region shown, obtain the region where the coordinates of the subsequent frame in this pair of registration points are located, such as the No. 2 region, and then obtain the region combination (reference frame region, subsequent frame region) where this pair of registration points is located, that is, (1, 2).

[0046] Count the region combinations of all registration point pairs, and screen out the matching point pairs according to the distribution of the registration point pairs in all region combinations. Specifically, the following five method examples (1) to (5) are given, and it is not limited to these five methods:

[0047] (1) Obtain the region combination with the most registration point pairs, and determine the matching point pairs according to the registration point pairs in the region combination;

[0048] The region combination with the most registration point pairs is the region combination that appears the most times among all registration point pairs. For example, if the region combinations of 20 pairs of registration points are all (1, 2), and the number of registration point pairs in other region combinations is within 10, then (1, 2) is the region combination with the most registration point pairs.

[0049] Next, the registration point pairs in this region combination can be directly determined as the matching point pairs; or these selected registration point pairs can be further selected as the final matching point pairs. The further selection method is:

[0050] Calculate the basic transformation matrix according to the registration point pairs in the aforementioned region combination with the most;

[0051] Obtain the basic coordinates after the transformation of the registered feature points within the selected range in the reference frame by the basic transformation matrix, calculate the difference between the basic coordinates and the coordinates of the corresponding feature points in the subsequent frame, and if the difference is less than the first threshold, determine this pair of registration points as the matching point pair.

[0052] If the difference is not less than the first threshold, the registration point pair is determined to be unmatched; the unmatched points can be directly deleted, or the unmatched feature points in the reference frame can be searched within a preset coordinate range around their base coordinates and the matching degree can be calculated, and the feature point with the highest matching degree and greater than the fourth threshold is used as its matching point.

[0053] Among them, the "selected range" can be set by those skilled in the art according to the actual situation. To obtain the most feature points, the "selected range" can be set to the entire image; to improve efficiency, the selected range can be set to the part of the area that was previously used to calculate the basic transformation matrix, such as the area combination with the most registration point pairs; in addition, it can also be set to the area with the most registration point pairs and its surrounding areas.

[0054] To balance the registration accuracy and efficiency, this embodiment also adopts a method for efficiently determining a valuable "selected range", and the selected range is obtained according to the following method:

[0055] Determine the pending area combination according to the distribution of the registration point pairs in all area combinations; in this embodiment, the pending area combination can be set to the area combination where all the registration point pairs in the entire image exist.

[0056] Obtain the representative coordinates of the reference frame area and the representative coordinates of the subsequent frame area in the pending area combination. In this embodiment, the coordinates at the center of the area are used as its representative coordinates;

[0057] Calculate the difference between the coordinates of the representative coordinates of the reference frame area after being transformed by the basic transformation matrix and the representative coordinates of the subsequent frame area. If the difference is less than the second threshold, the reference frame area is determined as the selected range. Among them, the second threshold can be the same as the first threshold, and both are determined by those skilled in the art according to the size of the image and the possible degree of change of the image.

[0058] The beneficial effect of this operation is that first, it calculates whether the representative points of the area conform to the basic transformation matrix. If not, the points in this area are not calculated, saving a large amount of calculation time. The processing idea from the center to the whole image and from the local to the whole not only improves efficiency but also ensures the registration accuracy.

[0059] In this embodiment, the robust least squares method or RANSAC is selected to solve the transformation matrix, and the calculation formula of the transformation matrix:

[0060]

[0061] In the formula, (x, y) are the coordinates of the reference frame feature points in the registration point pair,

[0062] (x A ,y A ) are the coordinates of the subsequent frame feature points in the registration point pair.

[0063] H is a 3*3 transformation matrix. Since the application scenario of this embodiment is the registration of fundus images and the degree of transformation is small, a 3*3 affine transformation matrix can be used. As known to those skilled in the art, in other application scenarios, other existing transformation matrices can be selected, such as a homography matrix, a perspective transformation matrix, a binary quadratic transformation matrix, a similarity transformation matrix, etc., and the formulas in the embodiment will be adjusted accordingly. To match the number of rows and columns of the H matrix, a row of 1 is added after the coordinates x and y in this formula to meet the requirements of multiplication calculation in mathematics. If other forms of H matrices are adopted, the calculation form of this formula will also be adjusted accordingly, not limited to this formula.

[0064] In this embodiment, the points in the reference frame are transformed into the subsequent frame. As known to those skilled in the art, it is also possible to transform the points in the subsequent frame into the reference frame conversely. In this case, the transformation matrix becomes the inverse matrix, and the other parts are basically the same.

[0065] (2) Obtain the region combination with the most registration point pairs, and determine the matching point pairs according to the registration point pairs in the region combination and the registration point pairs in the corresponding surrounding region combination of the region combination.

[0066] Among them, the surrounding region can be the 4-neighborhood, 8-neighborhood, circular region around the original region, or include the 8-neighborhood and add another circle of neighborhoods outside the 8-neighborhood for a total of 21 regions, etc. The size of the surrounding region can be different from the size of the initial division of the reference frame and the subsequent frame, and is set by those skilled in the art according to needs. If it encounters the edge region of the image and some neighborhoods are missing, only the existing neighborhoods are calculated, and the missing parts are ignored. Figure 5 The schematic diagram of the 8-neighborhood is given in [reference]. The regions A1 to A8 in the figure are the 8-neighborhood of the region A0.

[0067] The registration point pairs in the corresponding surrounding region combination of the region combination refer to: one of the registration point pairs is located in a certain surrounding region of the reference frame (such as Figure 5 the left A1 region), and the other point is located in the corresponding surrounding region of the subsequent frame (such as Figure 5 the rightmost upper B1 region in the figure). The A1 region and the B1 region are both the upper left regions in the image, and they correspond to each other in position and number. Therefore, if the region combination with the most registration point pairs is (A0, B0), that is, Figure 5 the left A0 region and Figure 5 the rightmost upper B0 region in the figure, then the registration point pairs located in the (A1, B1) region are the registration point pairs in the corresponding surrounding region combination of the region combination. Similarly, the (A2, B2), (A3, B3)... (A8, B8) in these two figures are all their corresponding surrounding region combinations.

[0068] After that, the region combination with the most registration point pairs and the registration point pairs in the corresponding surrounding region combinations above can be determined as matching point pairs. Alternatively, the matching point pairs can also be further selected based on these registration point pairs, and the further selection method is the same as that in method (1).

[0069] The purpose of adding the registration point pairs in the surrounding regions is to avoid the situation where the number of registration point pairs in the region combination with the most registration point pairs is too small. For example, if the number of registration point pairs is less than 6 and cannot meet the calculation requirements of the transformation matrix, the registration point pairs in the surrounding regions of the central region are added to supplement the registration point pairs. Moreover, the region with the most registration point pairs represents the region with the best matching effect and most suitable for full-image registration in the image, and its surrounding regions are some regions closest to it. The reliability of the registration points in these regions is also relatively strong. Using these points for supplementation will not only not introduce errors, but also increase the accuracy of the transformation matrix calculation due to the increase in the number of points. Therefore, whether the number of registration points is greater than 6 or not, adding the surrounding regions will help improve the registration accuracy.

[0070] (3) Obtain the region combination with the most registration point pairs, and determine the matching point pairs based on the registration point pairs in the region combination and the registration point pairs in the corresponding surrounding region combinations of the region combination. The corresponding surrounding region combinations of the region combination include the corresponding surrounding region combinations determined by rotating the subsequent frame or the reference frame by at least one angle.

[0071] Among them, the rotation angle can be determined according to the characteristics of the images to be registered. For example, in this embodiment, the registration method is applied to the field of eye movement tracking. Since the tracked eyes may rotate, but at a specific acquisition frequency, normal eyes will not rotate by a large angle. In this embodiment, the maximum rotation angle between the acquired reference frame and the subsequent frame is about plus or minus 45 degrees of rotation. Based on this premise, the 8-neighborhoods established in this embodiment are the forward 8-neighborhood, the 8-neighborhood corresponding to 45 degrees forward, and the 8-neighborhood corresponding to 45 degrees backward. Among them, the 8-neighborhood of the reference frame is fixed as the forward 8-neighborhood, and the 8-neighborhood of the subsequent frame corresponds to the above 3 kinds of 8-neighborhoods. Of course, the subsequent frame can also be fixed and the reference frame can be rotated by these 3 angles.

[0072] In this embodiment, the feature points in the forward 8-neighborhood of the reference frame are respectively matched with the feature points in the 3 8-neighborhoods of the subsequent frame, such as Figure 5As shown in the figure, on the left is the forward 8-neighborhood of the reference frame A0 area, on the upper right is the forward 8-neighborhood of the registration point pair for the subsequent frame B0 area, in the middle right is the forward 45-degree rotated 8-neighborhood of the subsequent frame, and at the lower right is the reverse 45-degree rotated 8-neighborhood of the subsequent frame. The numbers and areas have been rotated correspondingly. Therefore, the registration point pairs between the forward 8-neighborhood of the reference frame and the 8-neighborhoods of the subsequent frame at 0 degrees, forward 45 degrees, and reverse 45 degrees are numbered correspondingly, such as A1B1, A7B7, A4B4, etc., excluding non-corresponding areas such as A1B3, A7B6, etc.

[0073] In this embodiment, the matching point pairs are determined according to the registration point pairs obtained at these 3 angles, and the angle with the largest number of registration point pairs among the 3 surrounding rotation angles can be determined, and the registration point pairs included are used as the matching point pairs;

[0074] Alternatively, the basic transformation matrices are calculated respectively with the registration point pairs obtained at these 3 angles, the basic coordinates after the transformation of the respective feature points registered in the surrounding areas of these 3 angles in the reference frame by these 3 basic transformation matrices are obtained, the differences between the basic coordinates and the coordinates of the corresponding feature points in the subsequent frame are calculated, and the angle with the largest number of registration point pairs with the difference less than the first threshold is determined, and the registration point pairs included in this angle are used as the matching point pairs.

[0075] Alternatively, the matching point pairs can also be further selected according to these registration point pairs, and the further selection method is the same as method (1).

[0076] (4) Obtain the registration point pairs in all area combinations and their corresponding surrounding area combinations. The corresponding surrounding area combinations of the area combinations include the corresponding surrounding area combinations determined by rotating the subsequent frame or the reference frame by at least one angle;

[0077] Determine the current rotation angle according to the number of the registration point pairs, and for the area combination corresponding to the current rotation angle, determine the matching point pairs according to the registration point pairs corresponding to the area combination.

[0078] Specifically, in this embodiment, for all area combinations, the 8-neighborhood surrounding areas at 3 angles are obtained for each of them. After traversing the entire image, all the registration point pairs that meet 0 degrees, forward 45 degrees, and reverse 45 degrees are obtained, and the angle with the largest number of registration point pairs is determined. The registration point pairs included in this rotation angle are determined as the matching point pairs. The matching point pairs can also be further selected according to these registration point pairs, and the further selection method is the same as method (1).

[0079] Alternatively, calculate the transformation matrix for the registration point pairs obtained at these three angles respectively, and then make a selection. The selection method is the same as the principle in method (1). Obtain the basic coordinates of each registered feature point within the selected range in the reference frame after being transformed by the basic transformation matrix, calculate the difference between the basic coordinates and the coordinates of the corresponding feature points in the subsequent frame, determine the angle with the largest number of registration point pairs whose differences are less than the first threshold, and determine the registration point pairs included in this rotation angle as the matching point pairs.

[0080] Calculate the transformation matrix based on the matching point pairs. The calculation method is the same as the method for calculating the basic transformation matrix before. Then, in order to ensure the registration accuracy and without affecting the speed, the registration accuracy evaluation is carried out in this embodiment, specifically including: as Figure 6 shown, obtain the matched feature points in the reference frame, divide the reference frame into multiple evaluation regions, and count the number of feature points in each evaluation region; determine whether the distribution of the number of feature points in all evaluation regions meets the preset conditions; if not, supplement the matching point pairs in the registration point pairs before screening.

[0081] The determination method of whether it meets the preset conditions in this embodiment includes: compare the number of feature points in each evaluation region with the feature point threshold, count the number of regions where the number of feature points in the evaluation region is greater than or equal to the feature point threshold. If the number of regions is greater than the fifth threshold, it is determined that it meets the preset conditions. In addition, the region score of the evaluation region where the number of feature points is greater than or equal to the feature point threshold can be set to 1; if the number of feature points in the evaluation region is less than the feature point threshold, the region score of this evaluation region is the number of feature points in this evaluation region divided by the feature point threshold. Then, comprehensively calculate the region scores of all evaluation regions, such as by taking the average value. The method for comprehensively calculating the region scores is not limited in this embodiment. After obtaining the registration accuracy score through comprehensive scoring, compare the registration accuracy score with the evaluation threshold. If the registration accuracy score is higher than or equal to the evaluation threshold, it is determined that this reference frame meets the preset conditions; if the registration accuracy score is lower than the evaluation threshold, it is determined that this reference frame does not meet the preset conditions.

[0082] If the registration accuracy score is less than the evaluation threshold, it indicates that the distribution of matching point pairs is uneven and matching point pairs need to be supplemented. Therefore, retrieve the registration point pairs before screening, that is, all the registration point pairs obtained during the initial registration. Calculate the verification coordinates of the feature point coordinates in the reference frame of the registration point pair after being transformed by the transformation matrix, and then calculate the coordinate distance between the verification coordinates and the feature point coordinates in the subsequent frame of the registration point pair. If the coordinate distance is less than the distance threshold and this registration point pair has not been selected as a matching point pair before, then use this pair of registration point pairs as the supplemented matching point pair. After traversing all the registration point pairs, use all the supplemented matching point pairs to evaluate the registration accuracy again. If the registration accuracy evaluation obtained again still does not meet the requirements, it indicates that the data quality of this subsequent frame is indeed poor, resulting in low reliability of the transformation matrix, and the processing of this frame of image needs to be abandoned and proceed to the processing of the next frame of image; if the registration accuracy evaluation obtained again meets the requirements, then re-obtain the corrected transformation matrix with all the supplemented matching point pairs, and at this time, the reliability of the obtained transformation matrix is relatively high.

[0083] Relying on the above registration method, it can be applied in real-time tracking methods, such as Figure 7 As shown, after calculating the final transformation matrix, transform the points to be tracked in the reference frame according to the transformation matrix to obtain the coordinates of these points in the subsequent frame. This is because the application scenario of real-time tracking requires continuously displaying the real-time registration result on the subsequent frame to reflect the feature information on the reference frame. For example, during real-time tracking of fundus images, it is necessary to mark the positions of fundus lesions or laser treatment planning points in real-time on the current frame. Therefore, with the subsequent frame unchanged, transform the reference frame to the subsequent frame through the transformation matrix;

[0084] If it is applied to real-time image stabilization, on the contrary, with the reference frame unchanged, transform the subsequent frame to the reference frame through the transformation matrix, which is convenient for viewing the stable image in real-time. In the above-mentioned registration method, wherever transformation is involved, it is uniformly changed to transform the subsequent frame, and the transformation matrix formula is also changed to the inverse matrix form.

[0085] In this embodiment, the displacement between the subsequent frame and the reference frame is also obtained according to the parameters of the transformation matrix; if the displacement does not meet the preset conditions, supplement the reference frame.

[0086] Specifically, in this embodiment, the displacement between the subsequent frame and the reference frame is characterized by the parameters c and f in the transformation matrix. If the absolute values of the parameters c and f in the transformation matrix or the square root of the sum of the squares of the parameters c and f are greater than the parameter threshold, it is determined that it does not meet the preset conditions, indicating that there is a relatively large displacement of the current subsequent frame relative to the reference frame, that is, the overlapping part between the subsequent frame and the reference frame is small. In this case, the calculated transformation matrix will have inaccurate problems. Therefore, it is necessary to update the reference frame. This technology adds the supplement of the reference frame feature points to make the matching between the reference frame and the subsequent frame more accurate. Taking the supplement of the reference frame as an example, the specific method includes: calculating the transformation coordinates of the subsequent frame transformed into the reference frame according to the transformation matrix, and determining whether the transformation coordinates are the coordinates of the feature points in the reference frame. If not, adjust the descriptor of the feature point according to the transformation matrix, and use the adjusted descriptor and feature point as the descriptor and feature point of the reference frame. Since the BRIEF descriptor is used in this embodiment and the arrangement of the BRIEF descriptor is determined by the angle, there may be an angular rotation of the subsequent frame relative to the reference frame. Therefore, it is necessary to adjust the descriptor of the supplemented feature points according to the angular change between the subsequent frame and the reference frame, that is, modify the gray centroid angle of the feature point, and finally use the adjusted descriptor and feature point as the descriptor and feature point of the reference frame.

[0087] After calculating the transformation matrix, count the number of times each feature point in the reference frame is matched during the tracking of N consecutive frames. If the number of times is less than the third threshold, it indicates that the value of this reference frame feature point is very low and it belongs to a useless feature point, then delete this feature point. This step can delete worthless feature points such as edge points and noise points, avoid repeated processing of worthless feature points, and achieve the purpose of improving the feature matching speed. If there is a supplement operation for the reference frame feature points and descriptors, start counting N frames again and retain the occurrence times of the previous reference frame feature points.

[0088] An image registration method and a real-time tracking method provided by the present invention divide the reference frame and the subsequent frame into multiple regions according to the same rule, register the reference frame and the subsequent frame, and obtain the region combination (reference frame region, subsequent frame region) where each pair of registration points is located; screen out the matching point pairs according to the distribution of the registration point pairs in all region combinations; calculate the transformation matrix according to the matching point pairs. Since the change of the image between consecutive frames is small, this method calculates the transformation matrix for registration with the most representative local region, which can avoid the interference information brought by the influence of noise in some regions of the image, ensure that the selected matching point pairs for registration have high reliability, thereby improving the registration accuracy, and the calculation is simple, which can meet the fast processing requirements of real-time registration.

[0089] On this basis, after obtaining the transformation matrix in this embodiment, the distances between the feature points of each subsequent frame after transformation according to the transformation matrix and their corresponding feature points in the reference frame are calculated. The points with distances meeting the requirements are supplemented as matching point pairs, and the transformation matrix is recalculated to make the transformation matrix more accurate. A registration accuracy evaluation index is also added to further determine the reliability of the transformation matrix. If the registration accuracy does not meet the requirements, matching point pairs are supplemented. According to the registration logic from small to large and from local to global, the accuracy and reliability of the final transformation matrix are ensured.

[0090] In this embodiment, the surrounding neighborhood window is traversed at multiple angles, and each registration point pair combination that meets the corresponding pairing of the surrounding neighborhood is placed in its corresponding angle set. The optimal transformation matrix is obtained according to the registration point pair conditions of multiple angle sets, realizing better statistical screening of correct registration point pairs and ensuring the reliability of the final transformation matrix.

[0091] This embodiment adds feature point splicing and descriptor splicing operations. On the premise of ensuring reliability, the feature points and descriptors in the reference frame are supplemented so that the reference frame can still be registered well in subsequent frames with large displacements. This technology also uses the judgment of the value of feature points to delete feature points with useless values to improve the matching speed.

[0092] Although the present invention has been described in detail above with general descriptions and specific embodiments, on the basis of the present invention, some modifications or improvements can be made, which are obvious to those skilled in the art. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection required by the present invention.

Claims

1. An image registration method, characterized in that, it includes the following steps: Dividing the reference frame and subsequent frames into multiple regions according to the same rule, registering the reference frame and subsequent frames, and obtaining the region combinations (reference frame region, subsequent frame region) where each pair of registered points is located; Screening out matching point pairs according to the distribution of registered point pairs in all region combinations; Calculating a transformation matrix according to the matching point pairs.

2. The image registration method according to claim 1, characterized in that, the screening out matching point pairs according to the distribution of registered point pairs in all region combinations includes: Obtaining the region combination with the most registered point pairs, and determining the matching point pairs according to the registered point pairs in the region combination and the registered point pairs in the corresponding surrounding region combinations of the region combination.

3. The image registration method according to claim 1, characterized in that, the screening out matching point pairs according to the distribution of registered point pairs in all region combinations includes: Obtaining the registered point pairs in all region combinations and their corresponding surrounding region combinations, and the corresponding surrounding region combinations of the region combination include the corresponding surrounding region combinations determined by rotating the subsequent frame or the reference frame by at least one angle; Determining the current rotation angle according to the number of the registered point pairs, obtaining the region combination corresponding to the current rotation angle, and determining the matching point pairs according to the registered point pairs corresponding to the region combination.

4. The image registration method according to any one of claims 1-3, characterized in that, the screening out matching point pairs according to the distribution of registered point pairs in all region combinations includes: Selecting registered point pairs according to the distribution of registered point pairs in all region combinations; Calculating a basic transformation matrix according to the selected registered point pairs; Obtaining the basic coordinates of each registered feature point within a selected range in the reference frame after being transformed by the basic transformation matrix, calculating the difference between the basic coordinates and the coordinates of the corresponding feature points in the subsequent frame, and if the difference is less than the first threshold, determining the registered point pair as a matching point pair.

5. The image registration method according to claim 4, characterized in that, the selected range is obtained by the following method: Determining a to-be-determined region combination according to the distribution of registered point pairs in all region combinations; Obtaining the representative coordinates of the reference frame region and the representative coordinates of the subsequent frame region in the to-be-determined region combination; Calculating the difference between the coordinates of the representative coordinates of the reference frame region after being transformed by the basic transformation matrix and the representative coordinates of the subsequent frame region, and if the difference is less than the second threshold, determining the reference frame region as the selected range.

6. The image registration method according to claim 1, characterized in that, after calculating the transformation matrix according to the matching point pairs, it further includes registration accuracy evaluation: Obtaining the matched feature points in the reference frame, dividing the reference frame into multiple evaluation regions; judging whether the quantity distribution of the feature points in all evaluation regions meets the preset conditions; If not, supplementing matching point pairs among the registered point pairs before screening.

7. A real-time tracking method applying the image registration method according to any one of claims 1-6, characterized in that, After calculating the transformation matrix, the points to be tracked in the reference frame are transformed according to the transformation matrix to obtain the coordinates of the points in the subsequent frame.

8. A real-time tracking method according to claim 7, wherein, after calculating the transformation matrix according to the matching point pairs, it further includes: obtaining the displacement between the subsequent frame and the reference frame according to the parameters of the transformation matrix; if the displacement does not meet the preset conditions, the reference frame is supplemented.

9. A real-time tracking method according to claim 8, wherein, the supplementing of the reference frame includes: calculating the coordinates of the subsequent frame transformed into the reference frame according to the transformation matrix, determining whether the coordinates are the coordinates of the feature points in the reference frame, if not, adjusting the descriptor of the feature point according to the transformation matrix, and using the adjusted descriptor and the feature point as the descriptor and the feature point of the reference frame.

10. A real-time tracking method according to claim 7, wherein, it includes: counting the number of times each feature point in the reference frame is matched during the tracking of N consecutive frames, and if the number is less than the third threshold, deleting the feature point.

Images