A coarse-to-fine image registration method, system, terminal and readable storage medium based on DFS
By employing the DFS coarse-to-fine image registration method and utilizing the sMLD features of long and short line segments and grid division, the problem of insufficient registration accuracy in MLD features is solved, achieving efficient and accurate image registration.
Patent Information
- Application Number
- CN202210060743.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-01-19
AI Technical Summary
Existing image registration methods based on MLD features suffer from insufficient registration accuracy and high computational complexity. In particular, the effectiveness of the matching algorithm is affected by feature islands and scale transformation.
A coarse-to-fine image registration method based on DFS is adopted. By dividing the linear descriptors (sMLD) into long and short line segments, long line segments are used for coarse matching and short line segments are used for fine matching. Combined with grid partitioning and GMS verification, the matching accuracy and efficiency are improved.
It improves the accuracy and efficiency of image registration by utilizing the data characteristics of long and short line segments and the grid structure to accelerate matching, and further filtering by combining GMS verification to ensure the accuracy of the registration results.
Smart Images

Figure CN114445470B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image stitching, and in particular relates to a coarse-to-fine image registration method, system, terminal and readable storage medium based on DFS. Background Art
[0002] Image registration is generally divided into two processes: feature extraction and feature matching. Feature extraction includes feature detection and descriptor generation. The classic feature extraction method is to perform statistics on the neighborhood information of feature points. In order to effectively express information such as scale and direction, its feature description is usually more complex.
[0003] In the feature matching stage, it is usually carried out from two perspectives: direct matching and indirect matching. Direct matching is to transform the feature matching problem into the problem of correspondence between two point sets, so as to estimate the correspondence between points; indirect matching is to establish the correspondence through geometric conditions and then eliminate the error items. The existing technology is greatly affected by scale and affine transformation.
[0004] MLD feature (proposed by Shi et al., the feature points detected by the detector are connected to multiple feature points through line segments). Compared with other classic point features, MLD feature has stronger robustness. In MLD feature, any feature point on an image can be connected to all other feature points by line segments, so there is a drawback: point feature constructs descriptors by extracting feature point neighborhood information. In fact, the number of feature points determines the number of descriptors, while line feature is different from point feature. Feature points are connected to each other, and the number of descriptors will be exponentially higher. Assuming that there are N feature points, there are N(N-1) / 2 line segments, that is, the same number of descriptors. In order to solve the problem of high time complexity, some people proposed to limit the length of the line between feature points in MLD feature pairs, so that each feature point can be connected to only C line segments on average, where C is a constant, so C×N binary descriptors are actually obtained. The smaller the range is, the smaller the C value is, and the fewer feature descriptors are conducive to reducing storage occupancy and improving matching efficiency. At the same time, constraining the range of line segment length also means constraining the scale invariance of features. However, the MLD feature has the phenomenon of feature islands, which has a certain impact on the effectiveness of the matching algorithm, and the registration accuracy still has room for improvement.
[0005] In summary, the feature point matching method based on graph theory combines global similarity to divide and pair features, which has high computational complexity and the existence of feature islands, which affects the effectiveness of the matching algorithm and leads to the need for further improvement in matching. Summary of the invention
[0006] The present invention aims at the problem of registration accuracy based on MLD features and provides a coarse-to-fine image registration method, system, terminal and readable storage medium based on DFS (Depth-First-Search). The method generates a mesh topology structure in the coarse matching stage by drawing long and short line segments, and uses "long line segments" to filter features; in the fine matching stage, "short line segments" are used to filter features, thereby improving the registration accuracy.
[0007] On the one hand, the present invention provides a coarse-to-fine image registration method based on DFS, which comprises the following steps:
[0008] Step S1: Obtain image feature points of two images to be matched, generate linear descriptors sMLD, and divide long line segments and short line segments;
[0009] Among them, an image feature point in an image is connected with other image feature points by a line segment, and then the corresponding linear descriptor sMLD is generated, and the long line segment and short line segment are divided based on the length of the line segment;
[0010] Step S2: performing rough matching based on the long line segment to obtain matching point pairs, wherein an initial matching point pair is determined; then the initial matching point pair is used as the starting point of the rough matching path, and other matching point pairs on the rough matching path are determined in sequence within the range of the long line segment based on the similarity, wherein the similarity represents the degree of similarity between two linear descriptors sMLD on the two images;
[0011] Step S3: performing fine matching based on the short line segment, wherein the matching point pair obtained by the rough matching is used as the current point pair, and the matching point pairs on the fine matching path are sequentially determined based on the similarity within the short line segment;
[0012] Step S4: Calculate a local homography matrix based on the matching point pairs determined by the precise matching, where the local homography matrix is the registration information of the two images.
[0013] The image registration method provided by the present invention is the preliminary preparation for image stitching. The two images to be registered must be in a continuous adjacent relationship. The present invention divides the linear descriptors sMLD of long line segments and short line segments according to the length of the line segments, and applies them to the rough matching and fine matching processes respectively. Different from the existing MLD feature, which only retains the line segment length to meet the line segment length parameter (L min ,L max ) line segments, the present invention makes full use of the data characteristics of long and short line segments to improve the registration accuracy and efficiency.
[0014] Optionally, the linear descriptor sMLD whose line segment length L satisfies: L∈(l1,l2) is regarded as a long line segment, and the linear descriptor sMLD whose line segment length L satisfies: L<l min The linear descriptor sMLD is regarded as a short line segment, where lmin <<l1,l1,l2,l min The boundary values are divided by the preset line segment lengths.
[0015] Optionally, the method further comprises: dividing the image into G×G cells, where the side length of the cell is between l1,l min After the matching point pair is determined based on the similarity in step S3, the following is further performed: the matching points in the matching point pair are verified using the GMS verification principle, and all matching points in the cells with the result F are filtered out. The verification process is expressed as:
[0016]
[0017]
[0018] Where i and j represent cell i on image 1 and cell j on image 2 respectively, S ij is the matching statistical index between cell i and cell j, i k 、j l are the neighboring cells of cell i and cell j respectively, i 5 、j 5 Corresponding to cell i and cell j, k and j represent the neighborhood, χ ik χ jl Represents cell i k 、j l The number of matching point pairs between them, l represents the neighborhood superscript, τ i is the threshold for checking the matching points in the center cell, α is a constant, and n i is the average value of the matching point pairs within the cell.
[0019] The larger the α, the more reliable it is to reject a large number of wrong matching point pairs, for example, α = 6. When the corresponding cell in the two images to be registered is judged as F, it is considered that the matching points in the cell are not accurate, so all matching points in the corresponding cells in the two images are removed. The average number of cell features n i It can be the average number of matching point pairs in all neighboring cells or the average number of matching point pairs in all neighboring cells of cell i or other settings.
[0020] The present invention divides cells so that the graph structure can be fully utilized to accelerate matching during image matching. In particular, long line segments can search for matching points across the grid; short line segments can search within the grid, thereby improving the calculation accuracy of the local homography matrix.
[0021] Optionally, the implementation process of the rough matching in step S2 is:
[0022] S2-1: Calculate the similarity between the linear descriptor sMLD corresponding to each image feature point on the image Image1 and the linear descriptor sMLD corresponding to each image feature point on the image Image2;
[0023] S2-2: Select a pair of linear descriptors sMLD with the highest similarity, and select an initial matching point pair based on their corresponding image feature points; wherein the endpoints of either end of the pair of linear descriptors sMLD with the highest similarity are used as the initial matching point pair of the two images;
[0024] S2-3: Taking the initial matching point pair as the starting point of the rough matching path, for each linear descriptor sMLD of the current matching point on the image Image1, respectively calculate the similarity between it and all the linear descriptors sMLD of the current matching point on the image Image2, and delete the linear descriptors sMLD on the image Image1 that do not meet the preset similarity condition; for each linear descriptor sMLD of the current matching point on the image Image2, respectively calculate the similarity between it and all the linear descriptors sMLD of the current matching point on the image Image1, and delete the linear descriptors sMLD on the image Image2 that do not meet the preset similarity condition; and delete the linear descriptors sMLD that do not belong to long line segments;
[0025] S2-4: Based on the remaining linear descriptors sMLD of the current matching points in the images Image1 and Image2, a pair of linear descriptors sMLD with the highest similarity is selected as the optimal path, and the other end point of the optimal path is used as the next matching point, and then all matching point pairs on the rough matching path are obtained according to steps S2-3 and S2-4.
[0026] Among them, the preset similarity condition is set by the size of the Hamming distance between the linear descriptors sMLD and the preset threshold. For example, if the Hamming distance between the linear descriptor sMLD of image Image1 and all linear descriptors sMLD of the current matching point on image Image2 is greater than the threshold, it is considered not satisfied; or if the Hamming distances of some numbers are greater than the threshold, it is considered not satisfied.
[0027] Optionally, the implementation process of the precise matching in step S3 is:
[0028] Step S3-1: taking the matching point pair obtained by rough matching as the current point pair, retaining the short line segment of the current point pair, and removing the line segments of other lengths;
[0029] Step S3-2: For each current point pair, calculate the similarity between the associated linear descriptors sMLD, select the line where the pair of linear descriptors sMLD with the highest similarity is located as the optimal path, and determine the next matching point, and cyclically search to obtain all matching point pairs on the precise matching path.
[0030] Alternatively, if the image is divided into G×G cells, the side length of the cell is between l1,l min In step S3, the implementation process of the precise matching further includes:
[0031] During the search process of step S3-2, delete the endpoints that exceed the cell boundaries, or for all matching point pairs on the exact matching path of each set of current point pairs in step S3-2, delete the endpoints that exceed the cell boundaries on the dual graph, and then use the remaining matching point pairs as the current point pairs, and perform a dual graph depth-first search in the manner of steps S3-1 and S3-2 until all matching point pairs are found in the cell.
[0032] Optionally, in step S4, four matching points are selected using the minimum circumscribed convex quadrilateral principle, and then a local homography matrix is calculated;
[0033] The minimum circumscribed convex quadrilateral including all matching points is obtained, and then the four outermost matching points in the minimum circumscribed convex quadrilateral are selected to calculate the homography matrix.
[0034] In a second aspect, the present invention provides a system based on the above-mentioned image registration method, comprising:
[0035] An image feature point acquisition module is used to acquire image feature points of two images to be matched;
[0036] Linear descriptor sMLD generation module, generates linear descriptor sMLD based on image feature points;
[0037] Long-short line segmentation module, used to divide the linear descriptor sMLD of long line segments and short line segments;
[0038] The coarse matching module performs coarse matching based on long line segments to obtain matching point pairs;
[0039] A precise matching module performs precise matching based on short line segments and determines matching point pairs on the precise matching path;
[0040] The local homography matrix calculation module calculates the local homography matrix based on the matching point pairs determined by the precise matching, and the local homography matrix is the registration information of the two images.
[0041] In a third aspect, the present invention provides a terminal, comprising:
[0042] one or more processors;
[0043] a memory storing one or more computer programs;
[0044] The processor calls the computer program to implement:
[0045] A step of a coarse-to-fine image registration method based on DFS.
[0046] In a fourth aspect, the present invention provides a readable storage medium storing a computer program, wherein the computer program is called by a processor to implement:
[0047] A step of a coarse-to-fine image registration method based on DFS.
[0048] Beneficial Effects
[0049] The present invention provides a coarse-to-fine image registration method based on DFS, which divides the linear descriptors sMLD of long line segments and short line segments according to the length of the line segments, and applies them to the coarse matching and fine matching processes respectively. Different from the MLD feature, which only retains the line segment length to satisfy the line segment length parameter (L min ,L max ) line segments, the present invention makes full use of the data characteristics of long and short line segments to improve the registration accuracy and efficiency.
[0050] In a further preferred embodiment, the present invention divides the image into a grid so that long line segments can cross the grid to search for matching points; short line segments are searched within the grid, and the graph structure can be fully utilized to accelerate the matching during the image matching process. At the same time, it is combined with long and short line segments to improve the search accuracy, thereby improving the accuracy of the registration results.
[0051] In a further preferred embodiment, the present invention also combines the image grid with the GMS verification principle to further screen the results of the precise registration to ensure the registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 A flowchart showing an implementation of the image registration method provided in Embodiment 1 of the present invention;
[0053] Figure 2 is a schematic diagram of the sMLD model;
[0054] Figure 3 A schematic diagram showing a mesh topology structure formed by line descriptors;
[0055] Figure 4 It represents the relationship between the endpoints and the descriptors in the sMLD feature proposed by the present invention;
[0056] Figure 5 is the Hamming distance calculation rule;
[0057] Figure 6 A schematic diagram showing a rough matching algorithm proposed in the present invention;
[0058] Figure 7 A schematic diagram showing the precise matching optimization proposed by the present invention;
[0059] Figure 8 A schematic diagram showing the GMS grid score evaluation;
[0060] Fig. 9 The diagram shows the principle diagram of the minimum circumscribed convex quadrilateral principle proposed by the present invention. DETAILED DESCRIPTION
[0061] The present invention provides a coarse-to-fine image registration method based on DFS, which improves the sMLD feature and divides the linear descriptors sMLD of long and short line segments, thereby more accurately realizing the search of registration points, thereby improving both the registration accuracy and the registration efficiency. In particular, the present invention performs grid division on the image. On the one hand, it preferably introduces the GMS verification principle to further screen the precise registration results, thereby ultimately improving the registration accuracy. On the other hand, the long and short line segments are combined with the grid, so that the long line segments can search for matching points across the grid; the short line segments can be searched within the grid, and the graph structure can be fully utilized to accelerate matching and improve registration accuracy during the image matching process. The present invention will be further described below in conjunction with the embodiments.
[0062] Embodiment 1:
[0063] like Figure 1 As shown, an embodiment of the present invention provides a coarse-to-fine image registration method based on DFS, which includes the following steps:
[0064] Step 1: Obtain the image feature points of the two images to be matched, generate the linear descriptor sMLD, and divide the long line segments and short line segments.
[0065] Shi et al. proposed that the line segments formed by connecting feature points in an image are MLD features. The present invention improves the MLD feature and proposes the sMLD feature, which only retains the line length range (L) compared to the MLD feature. min ,L max )=(128,256). In the present invention, the line descriptor with a line length range of L∈(l1,l2) is called a "long line segment", which is used to cross the grid in the matching algorithm; search for matching points; the line length range L<l min The line descriptor of is called a “short line segment”, which is mainly used for searching within the grid in the matching algorithm to improve the calculation accuracy of the local homography matrix. min <<l1,l1,l2,l min The specific value of is selected based on many factors such as size, experiment and accuracy requirements.
[0066] like Figure 2 , use the FAST corner detection algorithm to detect the feature points in the two images respectively, and connect multiple feature points through line segments. The network topology structure formed after connection is as follows Figure 2 sMLD feature refers to a multi-linear description feature formed by connecting a feature point with other feature points in an image. Figure 2 As shown in (b), for each sMLD linear descriptor, assuming that the line segment length is L, n+1 equally divided points P are obtained. r (r=0,1,2,…,n). Calculate the total pixel intensity value of each equally divided point neighborhood a×a size image block, where the image block side length a and line length L satisfy the linear relationship Finally, compare P0 to P n The total pixel intensity values of adjacent image blocks are set to 1 if the latter is greater than the former, otherwise set to 0. Finally, each line segment can obtain an n-dimensional binary descriptor. Based on two n-dimensional binary descriptors, the similarity distance can be calculated. In this embodiment, the Hamming distance is selected to represent the similarity. The smaller the Hamming distance, the higher the similarity. The calculation rule of the Hamming distance is as follows: Figure 5 As shown, the Hamming distance is the number of different values between two vectors. It is usually used to compare two binary descriptors of the same length. By calculating the Hamming distance, the similarity of two binary descriptors can be compared.
[0067] In summary, two image feature points on an image are connected by a line segment to form a linear descriptor sMLD, and each image feature point is associated with a constant number of linear descriptors sMLD. Figure 3 As shown, the feature information obtained by the sMLD feature description algorithm in this embodiment is stored according to a graph structure, so feature matching does not need to be exhaustive or iterative, and the matching efficiency can be improved by a local optimization algorithm.
[0068] Step 2: Divide the image into G×G non-overlapping cells. Cell division can make full use of the graph structure to accelerate the matching process in the image matching process, and use long and short line segments to achieve cross-region search in the cell.
[0069] Step 3: Rough matching: Perform rough matching based on long line segments to obtain matching point pairs. The details are as follows:
[0070] A: Calculate the similarity between the linear descriptor sMLD corresponding to each image feature point on image Image1 and the linear descriptor sMLD corresponding to each image feature point on image Image2.
[0071] S2-2: Select a pair of linear descriptors sMLD with the highest similarity, and select an initial matching point pair based on their corresponding image feature points.
[0072] Assume that there are m feature points in Image1 and n feature points in Image2, and each feature point is connected to a constant number of linear descriptors. Calculate the Hamming distance (brute force matching) between the multiple linear descriptors connected to the m feature points in Image1 and the multiple linear descriptors connected to the n feature points in Image2, sort them according to the size of the Hamming distance, and obtain a pair of linear descriptors with the highest similarity. The image feature points corresponding to the linear descriptors are the initial pair of correct matching points.
[0073] S2-3: Taking the initial matching point pair as the starting point of the rough matching path, for each linear descriptor sMLD of the current matching point on the image Image1, respectively calculate its similarity with all the linear descriptors sMLD of the current matching point on the image Image2, and delete the linear descriptors sMLD on the image Image1 that do not meet the preset similarity condition; for each linear descriptor sMLD of the current matching point on the image Image2, respectively calculate its similarity with all the linear descriptors sMLD of the current matching point on the image Image1, and delete the linear descriptors sMLD on the image Image2 that do not meet the preset similarity condition; and delete the linear descriptors sMLD that do not belong to long line segments;
[0074] S2-4: Select a pair of linear descriptors sMLD with the highest similarity from the remaining linear descriptors sMLD of the current matching points in the images Image1 and Image2 as the optimal path, take the other end point of the optimal path as the next matching point, and then obtain all the matching point pairs of the rough match according to the method of steps S2-3 and S2-4.
[0075] by Figure 6Take (a) as an example for explanation. The first pair of initial matching points (middle points) in ① is the current sMLD feature point pair, which is connected to 6 and 7 linear descriptors in Image1 and Image2 respectively. The initial matching points in Image1 connect to 6 linear descriptors, and each linear descriptor is taken in turn to calculate the Hamming distance with the 7 linear descriptors in Image2. If the Hamming distance is greater than the set threshold thr, the linear descriptor in Image1 is eliminated. The gray line segment in ① is the linear descriptor that is eliminated because the Hamming distance does not meet the threshold condition. Two linear descriptors in Image1 are eliminated by this condition. Similarly, the initial matching points in Image2 connect to 7 linear descriptors, and each linear descriptor is taken in turn to calculate the Hamming distance with the 6 linear descriptors in Image1. If the Hamming distance is greater than the set threshold thr, the linear descriptor in Image2 is eliminated. The gray line segment in the figure is the linear descriptor that is eliminated because the Hamming distance does not meet the condition. Three linear descriptors in Image2 are eliminated by this condition.
[0076] Some linear descriptors (crosses) do not satisfy L∈(l1, l2) and cannot cross the domain grid, so they are also eliminated. In the figure, it is represented as image Image1 eliminating one linear descriptor and image Image2 eliminating one linear descriptor.
[0077] After linear descriptors are eliminated based on the Hamming distance and long line segment restriction, three linear descriptors are retained in Image1 and three linear descriptors are retained in Image2 in the line segments connected by the initial feature points. The Hamming distance is then calculated for the linear descriptors retained by the point pair in the two images, and the pair of linear descriptors with the highest similarity is selected as the optimal path. The endpoint corresponding to the other end of the line segment becomes the new current sMLD matching point (orange point), so that the sMLD matching point is transferred between the two images.
[0078] like Figure 5 As shown in the figure, the coarse matching is done by searching through “long line segments”, so the “positioning points” are evenly distributed in most grids. The fine matching is a precise search of cells, and its function is to obtain a more accurate local homography matrix.
[0079] Step 4: Precise matching: Precise matching based on short line segments. Figure 7 As shown in (b) in the figure, a cell is extracted from Image2 for fine matching. The implementation process is as follows:
[0080] A: First, the "matching point pair" found by rough matching is used as the current point pair, the line segments within the distance range of the "short line segment" are retained, and the line segments of other lengths are discarded.
[0081] B: Calculate the Hamming distance for each set of linear descriptors associated with the point pair between the two images. That is, for each set of current point pairs, calculate the similarity between the linear descriptors sMLD associated with them, select the line where the pair of linear descriptors sMLD with the highest similarity is located as the optimal path, and determine the next matching point, and cyclically search to obtain all matching point pairs on the precise matching path of each point pair.
[0082] C: When performing an in-depth search for sMLD matching points, if the coordinate position of some endpoints exceeds the grid boundary, this endpoint is removed, and the remaining matching point pairs are used as the current point pairs. A dual-graph depth-first search is performed in accordance with steps S3-1 and S3-2 until all matching point pairs are found in the cell.
[0083] In this embodiment, when performing in-depth search, in order to ensure that the feature points at the edge of the grid can be correctly matched, the limit on the number of grids in the search process is relaxed, such as Figure 7 As shown in (c). If the coordinate position of some endpoints exceeds the grid boundary, the endpoint is removed and the dual-graph depth-first search is performed again until all matching point pairs are found in the cell. Among them, the endpoints exceeding the grid boundary do not participate in the matching point search in the unit grid. In other embodiments, it is also possible to choose to delete the endpoints exceeding the cell boundary during the search process of step S3-2.
[0084] D: Use the GMS verification principle to verify the matching points in the matching point pair and filter out all matching points in the cells with the result F.
[0085] like Figure 8 As shown, check cell i 5 and j 5 To determine whether the cell matches, we need to determine whether its neighborhood (the neighborhood is the eight neighborhoods near the cell) matches. In this invention, the neighborhood cell i k and j l The maximum number of matches in (l=1,2,...,9) is used as a statistical indicator, and then the statistical indicator is used to determine the cell i 5 and j 5 Whether it matches.
[0086] Step 5: Based on the matching point pairs determined by precise matching, obtain the minimum circumscribed convex quadrilateral containing all matching points, and then select the 4 outermost matching points in the minimum circumscribed convex quadrilateral to calculate the homography matrix. The larger the proportion of the rectangular area in the original image area, the higher the accuracy of subsequent image registration. Experiments have shown that there is a minimum circumscribed convex quadrilateral that can contain all feature points, so the 4 outermost feature points that constitute this rectangle are taken to calculate the homography matrix.
[0087] It should be understood that the essence of image stitching is to transform pixel coordinate positions using a homography matrix. Therefore, the obtained homography matrix is the registration information of the two images.
[0088] like Fig. 9 As shown, let rectangle C1C2C3C4 be an image to be registered with a midpoint of O. There are two sets of feature points A1A2A3A4 and B1B2B3B4 in the image, which are connected to form two quadrilaterals. The calculated homography matrix is H A and H B . Enlarge the left image to the right image, and let ABC represent the positions that A2B2C2 would originally be transformed to through the homography matrix. Assume that a certain calculation error is introduced at this time. Through the homography matrix H A Perform perspective transformation, point A2 is offset to point D, and point C2 is offset to point G; if the homography matrix H B Performing perspective transformation, point B2 is offset to point E, and point C2 is offset to point F. For image C1C2C3C4, the homography matrix calculated by feature points B1B2B3B4 has stronger anti-interference ability than that calculated by A1A2A3A4, making image registration more accurate. The larger the area of the quadrilateral formed by the four feature points used to calculate the homography matrix accounts for in the image, the higher the registration accuracy. It can be seen that the present invention can improve the registration accuracy by using the minimum circumscribed convex quadrilateral method.
[0089] In summary, the image registration method provided in this embodiment not only improves the sMLD feature, but also performs grid processing, introduces GMS verification, etc. Therefore, the image registration method described in this embodiment is regarded as the best embodiment of the present invention, but it is not the only implementation method. It should be understood that without departing from the concept of the present invention, that is, on the basis of improving the sMLD feature, optimizing the registration method in other aspects or deleting some technical points also falls within the scope of protection of the present invention.
[0090] Embodiment 2:
[0091] This embodiment provides a system based on an image registration method, which includes:
[0092] An image feature point acquisition module is used to acquire image feature points of two images to be matched;
[0093] Linear descriptor sMLD generation module, generates linear descriptor sMLD based on image feature points;
[0094] Long-short line segmentation module, used to divide the linear descriptor sMLD of long line segments and short line segments;
[0095] The coarse matching module performs coarse matching based on long line segments to obtain matching point pairs;
[0096] A precise matching module performs precise matching based on short line segments and determines matching point pairs on the precise matching path;
[0097] The local homography matrix calculation module calculates the local homography matrix based on the matching point pairs determined by the precise matching, and the local homography matrix is the registration information of the two images.
[0098] The implementation process of each functional module unit may refer to the description of the corresponding method.
[0099] The division of the functional module units is only a division of logical functions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated units can be implemented in the form of hardware or software functional units.
[0100] Embodiment 3:
[0101] This embodiment provides a terminal, which includes one or more processors and a memory storing one or more computer programs, wherein the processor calls the computer program to implement:
[0102] Step S1: Obtain image feature points of two images to be matched, generate linear descriptors sMLD, and divide long line segments and short line segments;
[0103] Step S2: performing rough matching based on the long line segment to obtain matching point pairs, wherein an initial matching point pair is determined; then the initial matching point pair is used as the starting point of the rough matching path, and other matching point pairs on the rough matching path are determined in sequence within the range of the long line segment based on the similarity, wherein the similarity represents the degree of similarity between two linear descriptors sMLD on the two images;
[0104] Step S3: performing fine matching based on the short line segment, wherein the matching point pair obtained by the rough matching is used as the current point pair, and the matching point pairs on the fine matching path are sequentially determined based on the similarity within the short line segment;
[0105] Step S4: Calculate a local homography matrix based on the matching point pairs determined by the precise matching, where the local homography matrix is the registration information of the two images.
[0106] For example, the specific implementation method of Example 1 may be followed.
[0107] The terminal also includes: a communication interface for communicating with external equipment and performing data interactive transmission.
[0108] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.
[0109] If the memory, processor and communication interface are implemented independently, the memory, processor and communication interface can be connected to each other through a bus and communicate with each other. The bus can be an industrial standard architecture bus, a peripheral device interconnection bus or an extended industrial standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.
[0110] Optionally, in a specific implementation, if the memory, processor and communication interface are integrated on a chip, the memory, processor and communication interface can communicate with each other through an internal interface.
[0111] For the specific implementation process of each step, please refer to the description of the above method.
[0112] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.
[0113] Embodiment 4:
[0114] This embodiment provides a readable storage medium, which stores a computer program, and the computer program is called by a processor to implement:
[0115] Step S1: Obtain image feature points of two images to be matched, generate linear descriptors sMLD, and divide long line segments and short line segments;
[0116] Step S2: performing rough matching based on the long line segment to obtain matching point pairs, wherein an initial matching point pair is determined; then the initial matching point pair is used as the starting point of the rough matching path, and other matching point pairs on the rough matching path are determined in sequence within the range of the long line segment based on the similarity, wherein the similarity represents the degree of similarity between two linear descriptors sMLD on the two images;
[0117] Step S3: performing fine matching based on the short line segment, wherein the matching point pair obtained by the rough matching is used as the current point pair, and the matching point pairs on the fine matching path are sequentially determined based on the similarity within the short line segment;
[0118] Step S4: Calculate a local homography matrix based on the matching point pairs determined by the precise matching, where the local homography matrix is the registration information of the two images.
[0119] For example, the specific implementation method of Example 1 may be followed.
[0120] For the specific implementation process of each step, please refer to the description of the above method.
[0121] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the controller described in any of the foregoing embodiments, such as a hard disk or memory of the controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0122] Based on this understanding, the technical solution of the present invention, in essence or in other words, the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store program codes.
[0123] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.
Claims
1. A coarse-to-fine image registration method based on DFS, characterized by: The following steps are involved: Step S1: Obtain image feature points of two images to be matched, generate linear descriptors sMLD, and divide long line segments and short line segments; Among them, an image feature point in an image is connected with other image feature points by a line segment, and then the corresponding linear descriptor sMLD is generated, and the long line segment and short line segment are divided based on the length of the line segment; Step S2: performing rough matching based on the long line segment to obtain matching point pairs, wherein an initial matching point pair is determined; then the initial matching point pair is used as the starting point of the rough matching path, and other matching point pairs on the rough matching path are determined in sequence within the range of the long line segment based on the similarity, wherein the similarity represents the degree of similarity between two linear descriptors sMLD on the two images; Step S3: performing fine matching based on the short line segment, wherein the matching point pair obtained by the rough matching is used as the current point pair, and the matching point pairs on the fine matching path are sequentially determined based on the similarity within the short line segment; Step S4: Calculate a local homography matrix based on the matching point pairs determined by the precise matching, wherein the local homography matrix is the registration information of the two images; The implementation process of the rough matching in step S2 is: S2-1: Calculate the similarity between the linear descriptor sMLD corresponding to each image feature point on the image Image1 and the linear descriptor sMLD corresponding to each image feature point on the image Image2; S2-2: Select a pair of linear descriptors sMLD with the highest similarity, and select an initial matching point pair based on their corresponding image feature points; S2-3: Taking the initial matching point pair as the starting point of the rough matching path, for each linear descriptor sMLD of the current matching point on the image Image1, respectively calculate the similarity between it and all the linear descriptors sMLD of the current matching point on the image Image2, and delete the linear descriptors sMLD on the image Image1 that do not meet the preset similarity condition; for each linear descriptor sMLD of the current matching point on the image Image2, respectively calculate the similarity between it and all the linear descriptors sMLD of the current matching point on the image Image1, and delete the linear descriptors sMLD on the image Image2 that do not meet the preset similarity condition; and delete the linear descriptors sMLD that do not belong to long line segments; S2-4: based on the linear descriptors sMLD remaining in the current matching points in the images Image1 and Image2, a pair of linear descriptors sMLD with the highest similarity is selected as the optimal path, and the other end point of the optimal path is used as the next matching point, and then all matching point pairs on the rough matching path are obtained according to the method of steps S2-3 and S2-4; The implementation process of the precise matching in step S3 is: Step S3-1: taking the matching point pair obtained by rough matching as the current point pair, retaining the short line segment of the current point pair, and removing the line segments of other lengths; Step S3-2: For each current point pair, calculate the similarity between the associated linear descriptors sMLD, select the line where the pair of linear descriptors sMLD with the highest similarity is located as the optimal path, and determine the next matching point, and cyclically search to obtain all matching point pairs on the precise matching path.
2. The method according to claim 1, characterized in that: The linear descriptor sMLD whose line segment length L satisfies L∈(l1,l2) is regarded as a long line segment, and the linear descriptor sMLD whose line segment length L satisfies L <l min The linear descriptor sMLD is regarded as a short line segment, where l min <<l1,l1,l2,l min The boundary values are divided by the preset line segment lengths.
3. The method according to claim 1, characterized in that: The method further includes: dividing the image into G×G cells, where the side length of the cell is between l1,l min After the matching point pair is determined based on the similarity in step S3, the following is further performed: the matching points in the matching point pair are verified using the GMS verification principle, and all matching points in the cells with the result F are filtered out. The verification process is expressed as: Where i and j represent cell i on image 1 and cell j on image 2 respectively, S ij is the matching statistical index between cell i and cell j, i k 、j l are the neighboring cells of cell i and cell j respectively, i 5 、j 5 Corresponding to cell i and cell j, k and j represent the neighborhood, χ ik χ jl Represents cell i k 、j l The number of matching point pairs between them, l represents the neighborhood superscript, τ i is the threshold for checking the matching points in the center cell, α is a constant, and n i is the average number of matching point pairs within a cell.
4. The method according to claim 1, characterized in that: If the image is divided into G×G cells, the side length of the cell is between l1,l min In step S3, the implementation process of the precise matching further includes: During the search process of step S3-2, delete the endpoints that exceed the cell boundaries, or for all matching point pairs on the exact matching path of each set of current point pairs in step S3-2, delete the endpoints that exceed the cell boundaries, and then use the remaining matching point pairs as the current point pairs, and perform a dual-graph depth-first search in the manner of steps S3-1 and S3-2 until all matching point pairs are found in the cell.
5. The method according to claim 1, characterized in that: In step S4, four matching points are selected using the minimum circumscribed convex quadrilateral principle, and then the local homography matrix is calculated; The minimum circumscribed convex quadrilateral including all matching points is obtained, and then the four outermost matching points in the minimum circumscribed convex quadrilateral are selected to calculate the homography matrix.
6. A system based on the method according to any one of claims 1 to 5, characterized in that: include: An image feature point acquisition module is used to acquire image feature points of two images to be matched; Linear descriptor sMLD generation module, generates linear descriptor sMLD based on image feature points; Long-short line segmentation module, used to divide the linear descriptor sMLD of long line segments and short line segments; The coarse matching module performs coarse matching based on long line segments to obtain matching point pairs; A precise matching module performs precise matching based on short line segments and determines matching point pairs on the precise matching path; The local homography matrix calculation module calculates the local homography matrix based on the matching point pairs determined by the precise matching, and the local homography matrix is the registration information of the two images.
7. A terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The steps of the method according to any one of claims 1 to 5.
8. A readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
An infrared image splicing method based on coarse-to-fine registration
CN109886878A
Panoramic image splicing method in airport environment
CN111127318A
Wide-viewing-angle quick splicing method for high-resolution images
CN111445389A
