Method for screening matching pairs, device therefor and three-dimensional reconstruction method

Abstract

The application provides a matching pair screening method, a device thereof and a three-dimensional reconstruction method. The matching pair screening method comprises: obtaining K matching pairs, wherein the K matching pairs comprise K first feature points in a first image and K second feature points in a second image which are matched with the K first feature points respectively; when K is greater than an expected matching pair number M, screening the K matching pairs according to a response value mean or a response value sum between the first feature points and the second feature points corresponding to at least two matching pairs in the K matching pairs, to obtain P matching pairs; and determining a final matching pair according to the P matching pairs, wherein K>P, K, P and M are positive integers. The embodiment of the application can reduce the number of final matching pairs while filtering matching pairs with poor quality, ensure that the quality of the first feature points and the second feature points of each matching pair in the screened matching pairs is the highest, and further improve the reliability of the matching result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a matching pair selection method and apparatus, and a three-dimensional reconstruction method. Background Technology

[0002] Current image processing techniques typically employ the approximate nearest neighbor ratio method to calculate initial matching pairs, and then use geometric verification to filter out matching pairs that do not meet the geometric verification criteria, thus obtaining the final matching pairs. However, especially in textured scenes such as farmland or buildings, the matching success rate of feature points from different keyframe images is high, resulting in a still enormous number of final matching pairs, and the quality of the final matching pairs is difficult to guarantee. This severely impacts the speed and accuracy of subsequent 3D reconstruction based on the final matching pairs.

[0003] For example, when this typical matching pair selection method is applied to images captured by drone cameras, as the number of images acquired by the drone camera increases, the number of final matching pairs also increases, leading to a surge in the amount of data required for image reconstruction. This poses a significant challenge to rapid image reconstruction. Therefore, reducing the number of final matching pairs while maintaining their quality is extremely important in image processing technology. Summary of the Invention

[0004] In view of this, this application provides a method and apparatus for screening matching pairs and a three-dimensional reconstruction method to solve the problem that the number of final matching pairs is large and the quality is difficult to guarantee in the prior art.

[0005] The first aspect of this application provides a method for filtering matching pairs. The method includes: obtaining K matching pairs, wherein the K matching pairs include K first feature points in a first image and K second feature points in a second image that match the K first feature points; when K is greater than the desired number of matching pairs M, filtering the K matching pairs based on the average or sum of the response values ​​between the first and second feature points corresponding to at least two matching pairs to obtain P matching pairs; and determining the final matching pairs based on the P matching pairs, where K > P, and K, P, and M are positive integers.

[0006] In one embodiment of this application, the above-mentioned determination of the final matching pair based on P matching pairs includes: when P>M, filtering out the first M matching pairs corresponding to the smaller numerical values ​​of the matching pair distances corresponding to the P matching pairs; and determining the matching pairs corresponding to the first M matching pairs corresponding to the smaller numerical values ​​as the final matching pair.

[0007] In an embodiment of the present application, before screening the K matching pairs based on the mean or sum of response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs, it further includes: dividing the first image into R grids, where R≥M and R is a positive integer; wherein, the screening of the K matching pairs based on the mean or sum of response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs includes: when the first grid in the R grids includes A first feature points and A is a positive integer greater than 1, respectively calculating the mean or sum of response values between the A first feature points and the A second feature points corresponding to the A first feature points to obtain A mean response values or A sums of response values; screening out the matching pairs corresponding to the top L mean response values or the top L sums of response values with larger numerical values according to the numerical sizes of the A mean response values or A sums of response values to obtain P matching pairs, where L<A, and L and A are positive integers.

[0008] In an embodiment of the present application, determining the final matching pair based on the P matching pairs includes: when P≤M, determining the P matching pairs as the final matching pairs.

[0009] In an embodiment of the present application, the screening of the K matching pairs based on the mean or sum of response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs includes: calculating the mean or sum of response values between the first feature points and the second feature points corresponding to each of the K matching pairs to obtain K mean response values or K sums of response values; screening out the matching pairs corresponding to the top P mean response values or the top P sums of response values with larger numerical values according to the numerical sizes of the K mean response values or K sums of response values to obtain P matching pairs.

[0010] In an embodiment of the present application, after obtaining the K matching pairs, the screening method of the matching pairs further includes: when K≤M, determining the K matching pairs as the final matching pairs.

[0011] In an embodiment of the present application, obtaining the K matching pairs includes: extracting S first feature points from the first image; searching for S second feature points matching the S first feature points from the second image based on the matching pair distance to obtain S matching pairs; filtering the S matching pairs through geometric verification to obtain K matching pairs, where K≤S.

[0012] In one embodiment of the present application, after extracting S first feature points from the first image, the method for screening matching pairs further includes: when S is greater than the expected number of feature points N, uniformly screening the S first feature points to screen out C first feature points, where C < S, and S and C are positive integers; among them, the above-mentioned finding S second feature points matching the S first feature points from the second image based on the matching pair distance to obtain S matching pairs includes: finding C second feature points matching the C first feature points from the second image based on the matching pair distance to obtain C matching pairs; among them, the above-mentioned filtering the S matching pairs through geometric verification to obtain K matching pairs includes: filtering the C matching pairs through geometric verification to obtain K matching pairs, where K ≤ C.

[0013] In one embodiment of the present application, the above-mentioned uniformly screening the S first feature points to screen out C first feature points includes: dividing the first image into D grids, where D ≥ N and D is a positive integer; when the second grid among the D grids includes E first feature points and E is a positive integer greater than 1, respectively calculating the response values of the E first feature points; screening out the first feature points corresponding to the first F response values with larger numerical values according to the numerical magnitudes of the response values of the E first feature points to screen out C first feature points, where F < E, and F and E are positive integers.

[0014] In one embodiment of the present application, the first image and the second image are two different images obtained by a photographing device for photographing an object to be reconstructed.

[0015] The second aspect of the present application provides a three-dimensional reconstruction method. The three-dimensional reconstruction method includes: obtaining final matching pairs according to any of the methods for screening matching pairs provided in the first aspect of the present application; reconstructing the three-dimensional point cloud data of the object to be reconstructed according to the final matching pairs.

[0016] The third aspect of the present application provides a device for screening matching pairs. The device for screening matching pairs includes: an acquisition module, configured to acquire K matching pairs, where the K matching pairs include K first feature points in the first image and K second feature points in the second image respectively matching the K first feature points; a screening module, configured to, when K is greater than the expected number of matching pairs M, screen the K matching pairs according to the mean value or the sum of the response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs; a determination module, configured to determine final matching pairs according to the P matching pairs, where K > P, and K, P, and M are positive integers.

[0017] The fourth aspect of this application provides an unmanned device. The unmanned device includes a memory and a processor. The memory stores executable instructions for a computer, and the processor, when executing the executable instructions, implements either a matching pair filtering method as provided in the first aspect of this application or a three-dimensional reconstruction method as provided in the second aspect of this application.

[0018] The fifth aspect of this application provides a computer-readable storage medium. This computer-readable storage medium stores executable instructions for a computer, which, when executed by a processor, implement either a matching pair filtering method as provided in the first aspect of this application or a three-dimensional reconstruction method as provided in the second aspect of this application.

[0019] According to the technical solution provided in the embodiments of this application, by filtering P matching pairs from K matching pairs, the number of matching pairs is reduced, thereby reducing the final number of matching pairs. Furthermore, since a larger response value indicates higher quality feature points, this embodiment considers the average or sum of the response values ​​between the first and second feature points corresponding to at least two matching pairs. This reduces the final number of matching pairs while filtering out poor-quality matching pairs, ensuring that the quality of the first and second feature points in each of the filtered matching pairs is the highest, thus improving the reliability of the matching results. Attached Figure Description

[0020] Figure 1 The diagram shown is a flowchart illustrating a matching pair filtering method provided in an embodiment of this application.

[0021] Figure 2A The diagram shown is a flowchart illustrating a matching pair filtering method provided in another embodiment of this application.

[0022] Figure 2B The image shown is a schematic diagram of the final matching pair.

[0023] Figure 2C The diagram shown is a schematic representation of the final matching pairs selected according to an embodiment of this application.

[0024] Figure 3A The diagram shown is a flowchart illustrating a matching pair filtering method provided in another embodiment of this application.

[0025] Figure 3B The image shown is a schematic diagram illustrating the effect of a first feature point.

[0026] Figure 3C The diagram shown is a schematic representation of the effect of the first feature point selected according to an embodiment of this application.

[0027] Figure 4The diagram shown is a flowchart illustrating a method for selecting a first feature point according to an embodiment of this application.

[0028] Figure 5 The diagram shown is a flowchart illustrating a matching pair filtering method provided in another embodiment of this application.

[0029] Figure 6 The diagram shown is a flowchart of a three-dimensional reconstruction method provided in an embodiment of this application.

[0030] Figure 7 The diagram shown is a structural schematic of a matching pair screening device provided in an embodiment of this application.

[0031] Figure 8 The diagram shown is a block diagram of an unmanned device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] Figure 1 The diagram shown is a flowchart illustrating a matching pair filtering method according to an embodiment of this application. The executing entity of this matching pair filtering method can be a controller, server, or processor, etc., and this application does not specifically limit it. The following example uses a controller as the executing entity. Figure 1 As shown, the method for filtering the matching pairs includes the following steps.

[0034] S110: Obtain K matching pairs. The K matching pairs include K first feature points in the first image and K second feature points in the second image that match the K first feature points respectively.

[0035] In some embodiments, the first image and the second image can be two different keyframe images from a video captured by a shooting device such as a mobile phone, computer, robot, or drone in the same physical space. For example, the first image and the second image can be two consecutive keyframe images, or two different keyframe images spaced apart by a preset time interval. In other embodiments, the first image can be an image captured by a shooting device, and the second image can be an image captured by a device different from or the same as the first image, such as a map image captured by a satellite. This application does not specifically limit the scope of the second image.

[0036] It should be understood that the method for obtaining K matching pairs can be based on optical flow tracing algorithm or descriptor algorithm, and this application does not make specific limitations on this.

[0037] S120: When K is greater than the expected number of matching pairs M, the K matching pairs are filtered according to the average or sum of the response values ​​between the first and second feature points of at least two matching pairs to obtain P matching pairs.

[0038] For example, at least two matching pairs include matching pair 1, matching pair 2, and matching pair 3. Matching pair 1 includes a first feature point 1 and a second feature point 1 that matches the first feature point 1. Matching pair 2 includes a first feature point 2 and a second feature point 2 that matches the first feature point 2. Matching pair 3 includes a first feature point 3 and a second feature point 3 that matches the first feature point 3. In some embodiments, the controller's response value X to the first feature point 1... 11 The response value X of the second feature point 1 12 The average response value is obtained by averaging. The controller's response value X to the first feature point 2 21 The response value X of the second feature point 2 22 The average response value is obtained by averaging. The controller's response value X to the first feature point 3 31 The response value X of the second feature point 2 32 The average response value is obtained by averaging. By comparison and The numerical values ​​are used to filter out one or two matching pairs with larger average response values, thereby filtering K matching pairs to obtain P matching pairs. In other embodiments, the response values ​​of the first feature point and the second feature point of each matching pair can be directly summed, resulting in X sums of response values ​​for matching pair 1, matching pair 2, and matching pair 3, respectively. 1总 =X 11 +X 12 X 2总 =X 21 +X 22 and X 3总 =X 31 +X 32 By comparing the magnitude of the sum of response values ​​X 1总 X 2总 and X 3总, one or two matching pairs with larger total response values are selected, so as to screen the K matching pairs to obtain P matching pairs. As long as the K matching pairs are screened according to the average response value or the total response value corresponding to at least two matching pairs to obtain P matching pairs, and P is less than K, the present application does not make specific limitations on this.

[0039] It should be understood that at least two matching pairs can be two or more matching pairs. For example, at least two matching pairs can be all K matching pairs, or can be Y matching pairs with smaller matching pair distances selected from the K matching pairs, where Y < K. It can also be the matching pairs in the grids containing more than 1 matching pair after meshing the first image. The present application does not make specific limitations on this.

[0040] In some embodiments, before step S120, the controller can execute the step of judging whether K is greater than M. When K is greater than M, step S120 is executed. In some other embodiments, it can also be that the user judges whether K is greater than M. When K is greater than M, the user sends an instruction to the controller, and the controller executes step S120 according to the user's instruction. The present application does not make specific limitations on whether the controller executes the judging step.

[0041] S130: Determine the final matching pairs according to the P matching pairs, where K > P, and K, P, and M are positive integers.

[0042] It should be understood that the controller can directly determine the P matching pairs as the final matching pairs, or can further process the P matching pairs and then determine the processed matching pairs as the final matching pairs. The present application does not make specific limitations on this.

[0043] According to the technical solution provided by the embodiments of the present application, by screening P matching pairs from K matching pairs, the number of matching pairs is reduced, and thus the number of final matching pairs is reduced. In addition, since the larger the response value, the higher the quality of the feature points, in the embodiments of the present application, the average response value or the total response value between the first feature points and the second feature points corresponding to at least two matching pairs is considered, so as to filter out some matching pairs with poor quality while reducing the number of final matching pairs, ensuring that the quality of the first feature points and the second feature points of each matching pair in the screened matching pairs is the highest, and thus improving the reliability of the matching result.

[0044] In an embodiment of the present application, the first image and the second image are two different images obtained by a photographing device for photographing an object to be reconstructed.

[0045] The object to be reconstructed can be a specific object, such as a building or a mountain, or it can be all the target objects within a certain spatial area, such as all the roads, rivers and buildings in a city, or all the attractions and tourists in a scenic area.

[0046] In this embodiment of the application, by setting the first image and the second image to be two different images captured by the imaging device for the object to be reconstructed, it is beneficial to reduce the number of final matching pairs and ensure the quality of the final matching pairs, thereby accelerating the processing speed of the images captured by the imaging device and also accelerating the reconstruction speed of the images to be reconstructed.

[0047] Figure 2A The diagram shown is a flowchart illustrating a matching pair filtering method provided in another embodiment of this application. Figure 2A The embodiment shown is Figure 1 A variation of the illustrated embodiment. For example... Figure 2A As shown, with Figure 1 The difference in the illustrated embodiment is that the matching pair filtering method further includes S119 before step S120, wherein steps S1201 and S1202 correspond to Figure 1 Step S120 in the illustrated embodiment.

[0048] S119: Divide the first image into R grids, where R ≥ M and R is a positive integer;

[0049] For example, to avoid filtering too many matching pairs, the grid size W is initially set to be relatively small, as long as R ≥ M is satisfied. This application does not impose a specific limit on the size of the grid size W. Assuming the horizontal width of the first image is r and the vertical length is c, then the number of grids in each row is rn = r / W, the number of grids in each column is cn = c / W, and the total number of grids is R = rn × cn.

[0050] In some embodiments, after performing step S119, the K first feature points can be directly assigned to the corresponding grids. In other embodiments, after performing step S119, the K first feature points can also be assigned to the corresponding grids according to their coordinate positions in the first image. This application does not impose specific limitations on this.

[0051] It should be understood that R can be any value greater than or equal to M, and the present application does not make specific limitations thereto. Dividing the first image into R grids can be the R grids formed based on Delaunay triangulation; the R grids formed by advancing the front; the R grids formed by quadtree image segmentation; the R grids formed by octree image segmentation. The present application does not make specific limitations thereto. R can be a value directly set by the user himself, or it can be a multiple of M directly set by the user himself, and the value determined by the controller according to this multiple of M. The present application does not make specific limitations thereto.

[0052] S1201: When the first grid within the R grids includes A first feature points and A is a positive integer greater than 1, calculate the mean value or the sum of the response values between the A first feature points and the A second feature points corresponding to the A first feature points respectively, so as to obtain A mean values of the response values or A sums of the response values.

[0053] It should be understood that the first grid can be any grid in the R grids where the number of first feature points is greater than 1, and the present application does not make specific limitations thereto. The value of A in different first grids can be the same or different, and the present application does not make specific limitations thereto.

[0054] S1202: Screen out the top L mean values of the response values corresponding to the matching pairs or the top L sums of the response values corresponding to the matching pairs with larger numerical values according to the numerical sizes of the A mean values of the response values or the A sums of the response values, so as to obtain P matching pairs, where L < A, and L and A are positive integers.

[0055] Specifically, the controller can sort the numerical sizes of the A mean values of the response values or the A sums of the response values, and select the matching pairs corresponding to the top L mean values of the response values with larger numerical values or the matching pairs corresponding to the top L sums of the response values as the screened matching pairs in the first grid. The matching pairs in all the first grids including more than 1 first feature points can be screened in the same way, and the screened matching pairs in all the first grids and the matching pairs in the un-screened grids are combined to obtain P matching pairs.

[0056] It should be understood that L can be 1 or any preset value, as long as it is less than A, and the present application does not make specific limitations thereto. When L = 1, it can be to screen out the matching pair corresponding to the largest mean value of the response values or the matching pair corresponding to the largest sum of the response values, so as to ensure that the quality of the first feature points and the second feature points corresponding to the screened matching pairs in each first grid is the highest.

[0057] Figure 2B Shown is a schematic diagram of the effect of the final matching pairs. Figure 2CThe diagram shown illustrates the effect of the final matched pairs selected according to an embodiment of this application. Figure 2B and Figure 2C The slanted, parallel white lines in the image represent the selected matches.

[0058] By comparison Figure 2B and Figure 2C As can be seen, according to the technical solution provided in the embodiments of this application, by dividing the first image into R grids, and selecting the matching pairs corresponding to the average of the top L response values ​​or the sum of the top L response values ​​based on the magnitude of the average of A response values ​​or the sum of the top L response values, P matching pairs are obtained. This significantly reduces the number of final matching pairs while ensuring that the quality of the first feature point and the second feature point corresponding to the selected matching pair in each first grid is the highest, and also significantly increases the uniformity of the distribution of the final matching pairs, thereby improving the reliability of the matching results.

[0059] In one embodiment of this application, steps S1301 and S1302 may correspond to Figure 1 Step S130 in the illustrated embodiment.

[0060] S1301: When P>M, filter out the M matching pairs with smaller numerical distances based on the numerical values ​​of the matching distances of the P matching pairs.

[0061] Specifically, the distances of the P matching pairs are sorted in ascending order, and the top M matching pairs with smaller values ​​are selected from the P matching pairs. The matching pairs corresponding to the top M matching pairs are determined based on the top M matching pairs with smaller values.

[0062] It should be understood that the matching pair distance can be Euclidean distance or other distances such as absolute distance, as long as the distance between the first feature point and the second feature point that matches the first feature point can be calculated. This application does not impose any specific limitations on this. The smaller the matching pair distance, the higher the similarity of the matching pair.

[0063] In one embodiment of this application, the matching pair distance is the Euclidean distance.

[0064] In this embodiment of the application, Euclidean distance is selected as the matching pair distance. Since Euclidean distance can more accurately reflect the true distance between two feature points, it can ensure that the P matches are further filtered accurately using Euclidean distance, thereby ensuring that the similarity of each matching pair in the final matching pair is the highest.

[0065] S1302: Determine the matching pairs corresponding to the distances of the first M matching pairs with smaller values ​​as the final matching pairs.

[0066] In this embodiment, when P>M, the first M pairs of matching pairs with larger distance values ​​are further selected from the P matching pairs to determine the final matching pairs. This integrates the matching pair distances of the matching pairs to further constrain the P matching pairs. While ensuring that the quality of the first feature point and the second feature point corresponding to the selected matching pair in each first grid is the highest, it also ensures that the similarity of each matching pair in the final matching pair is the highest, and ensures that the number of the final matching pairs after selection is equal to the expected number of matching pairs M, further reducing the number of the final matching pairs.

[0067] In one embodiment of this application, step S1303 corresponds to Figure 1 Step S130 in the illustrated embodiment.

[0068] S1303: When P≤M, determine P matching pairs as the final matching pairs.

[0069] In some embodiments, step S125 may be included before steps S1301 and S1303.

[0070] S125: Determine whether P is greater than M.

[0071] When the judgment result is P>M, execute steps S1301 and S1302; when the judgment result is P≤M, execute step S1303.

[0072] In other embodiments, step S125 may not need to be executed by the execution entity. For example, the user may determine whether P is greater than M and send the user's instruction containing the determination result directly to the execution entity. The execution entity may then directly select to execute steps S1301 and S1302, or execute step S1303, based on the user's instruction. This application does not specifically limit whether the execution entity executes step S125.

[0073] In this embodiment of the application, when P≤M, by determining P matching pairs as the final matching pairs, the excessive screening of K matching pairs is avoided, which would result in a small number of final matching pairs. This helps to control the number of final matching pairs to be close to the expected number of matching pairs, and avoids affecting the quality of the image reconstructed from the final matching pairs due to the small number of final matching pairs.

[0074] In one embodiment of this application, step S110 may be followed by step S116.

[0075] S116: When K≤M, determine the K matching pairs as the final matching pairs.

[0076] In some embodiments, step S113 may be performed before step S116 or S119.

[0077] S113: Determine if K is greater than M.

[0078] In other embodiments, step S113 may not be executed by the execution subject. For example, the user may determine whether K is greater than M and send the user's instruction containing the determination result directly to the execution subject. The execution subject may then directly select to execute step S116 or S119 based on the user's instruction. This application does not specifically limit whether the execution subject executes step S113.

[0079] In this embodiment of the application, when K≤M, by determining K matching pairs as the final matching pairs, the excessive screening of K matching pairs is avoided, which would result in a small number of final matching pairs. This helps to control the number of final matching pairs to be close to the expected number of matching pairs, and avoids affecting the quality of the image reconstructed from the final matching pairs due to the small number of final matching pairs.

[0080] In one embodiment of this application, steps S1103, S1106, and S1109 correspond to Figure 1 Step S110 in the illustrated embodiment.

[0081] S1103: Extract S first feature points from the first image.

[0082] It should be understood that extracting S first feature points from the first image can be based on the Scale Invariant Feature Transform (SIFT) algorithm, or on the Speeded Up Robust Features (SURF) algorithm, or on other algorithms such as the Gradient Location-Orientation Histogram (GLOH) algorithm. This application does not specifically limit the extraction of feature points from the first image.

[0083] S1106: Based on the matching pair distance, find S second feature points in the second image that match the S first feature points to obtain S matching pairs.

[0084] Specifically, the controller searches for S second feature points in the second image that match each of the S first feature points one by one, based on the matching pair distance. One first feature point from the S first feature points and one second feature point from the S matching second feature points are combined to form S matching pairs, thus obtaining S matching pairs. It should be understood that the S matching pairs can also serve as initial matching pairs.

[0085] S1109: Filter S matching pairs through geometric verification to obtain K matching pairs, where K ≤ S.

[0086] It should be understood that the geometric verification can be performed by using the Random Sample Consensus (RANSAC) algorithm, or other methods such as the K-means algorithm and the splitting method can be used to perform geometric verification, as long as the matching pairs that do not meet the geometric verification can be filtered out. The present application does not specifically limit the manner of geometric verification.

[0087] In the embodiment of the present application, S second feature points matching the S first feature points are found from the second image based on the distance of the matching pairs, so as to obtain the initial matching pairs, and the S matching pairs are filtered through geometric verification, and finally K matching pairs are obtained. Therefore, the matching pairs that do not meet the geometric verification are filtered out, and K matching pairs that are evenly distributed and meet the geometric verification are obtained, which is beneficial to improving the accuracy of the subsequent UAV motion estimation based on the final matching pairs and the accuracy of the subsequent image reconstruction based on the final matching pairs.

[0088] Figure 3A The following shows a schematic flowchart of a method for screening matching pairs provided by another embodiment of the present application. Figure 3A The shown embodiment is Figure 2A A variant of the shown embodiment. As Figure 3A shown, the difference from the Figure 2A shown embodiment is that S1105 is further included before step S11061, where step S11061 corresponds to Figure 2A step S1106 in the Figure 2A shown embodiment, and step S11091 corresponds to

[0089] S1105: When S is greater than the expected number of feature points N, perform uniform screening on the S first feature points to screen out C first feature points, where C < S, and S and C are positive integers.

[0090] It should be understood that the S first feature points can be uniformly screened by using grid-based methods such as quadtrees, Delaunay triangulation, and frontal advancement, as long as the C first feature points screened out are evenly distributed. The present application does not specifically limit this.

[0091] For example, the first image is an agricultural scene image taken by a drone camera. In step S1105, the SIFT algorithm is used to extract feature points from the first image. Assume the extraction threshold is t, and the expected number of feature points is N. SIFT feature points are extracted from the first image based on the threshold t, resulting in S first feature points. It should be understood that in scenes with weak texture or many repetitive textures, the value of S is relatively small; in scenes with rich texture, S is relatively large. If S is greater than N, step S1105 is executed; if S is not greater than N, step S1105 can be skipped and step S11061 can be executed directly, where S = C.

[0092] In some embodiments, step S1104 may be performed before step S1105.

[0093] S1104: Determine if S is greater than N.

[0094] In other embodiments, step S1104 may not be executed by the execution subject. For example, the user may determine whether S is greater than N and send the user's instruction containing the determination result directly to the execution subject, which then directly selects to execute step S1103 based on the user's instruction. This application does not specifically limit whether the execution subject executes step S1104. When S is less than or equal to N, step S11061 can be executed directly.

[0095] S11061: Based on the matching pair distance, find C second feature points in the second image that match C first feature points to obtain C matching pairs.

[0096] Specifically, the controller searches for C second feature points in the second image that match each of the C first feature points one by one, based on the matching pair distance. A first feature point from the C first feature points and a second feature point from the C matching second feature points are combined to form C matching pairs, thus obtaining C matching pairs. It should be understood that the C matching pairs can also serve as initial matching pairs.

[0097] For example, when the matching pair distance is the Euclidean distance, the similarity between the first feature point in the first image and the second feature point in the second image can be measured based on the Euclidean distance. For the first feature point f in the first image, a matching point is searched in the second image. In some embodiments, searching for a matching point in the second image can directly use the second feature point in the second image whose Euclidean distance from the first feature point f is less than a preset distance as the matching point. In other embodiments, the method of searching for a matching point in the second image can utilize the approximate nearest neighbor ratio method. Specifically, assume that the second feature point with the highest matching similarity score is f1, the matching score is s1, and the second feature point with the second highest matching score is f2, and the matching score is s2. If f1 < a × f2, where a is a preset ratio parameter, it is considered that the first feature point f1 and the second feature point f2 are successfully matched. Among them, the lower the matching score, that is, the smaller the Euclidean distance, the more similar the matching pair. The embodiments of the present application do not specifically limit the method of searching for C second feature points that match C first feature points from the second image based on the matching pair distance.

[0098] S11091: Filter C matching pairs through geometric verification to obtain K matching pairs, where K ≤ C.

[0099] In some embodiments, there may be matching pairs that do not meet the geometric verification among the C matching pairs. By filtering the C matching pairs through geometric verification, the matching pairs that do not meet the geometric verification are filtered out, and then K matching pairs are obtained, where K < C. In other embodiments, there may also be no matching pairs that do not meet the geometric verification among the C matching pairs. By filtering the C matching pairs through geometric verification, since there are no matching pairs that do not meet the geometric verification to be filtered, then K matching pairs are obtained, where K = C. The value of K only needs to be less than or equal to C, and the embodiments of the present application do not specifically limit the value of K.

[0100] Figure 3B The figure shows a schematic diagram of the effect of a first feature point. Figure 3C The figure shows a schematic diagram of the effect of the first feature point selected according to an embodiment of the present application.

[0101] Currently, in the process of selecting C first feature points from S first feature points, usually the response values of the S first feature points are sorted from large to small, and C first feature points are taken as the initial matching pairs. Although this strategy for feature point selection can ensure that the quality of the C first feature points selected is the highest among the S first feature points, it usually leads to clustering of feature points. The C first feature points (corresponding to Figure 3BThe distribution of the white circles is obviously uneven, which leads to lower accuracy in subsequent drone motion estimation based on the final matching pair, or lower accuracy in subsequent image reconstruction based on the final matching pair.

[0102] By comparison Figure 3B and Figure 3C As shown in the schematic diagram, in this embodiment, when S is greater than the desired number of feature points N, the S first feature points are uniformly filtered, and the filtered C first feature points (corresponding to...) Figure 3C The white circles are clearly more evenly distributed. In addition, by finding C second feature points in the second image that match C first feature points based on the matching pair distance, an initial matching pair is obtained. After filtering the C matching pairs through geometric verification, K matching pairs are finally obtained. Since the C first feature points are evenly distributed, the distribution of the K matching pairs is also more even.

[0103] Figure 4 The diagram shown is a flowchart illustrating a method for selecting a first feature point according to an embodiment of this application. Figure 4 As shown, the method for selecting the first feature point includes steps S11051 to S11053. Steps S11051 to S11053 can correspond to Figure 3A Step S1105 in the illustrated embodiment.

[0104] S11051: Divide the first image into D grids, where D≥N and D is a positive integer.

[0105] It should be understood that the first image can be divided into D grids, where D can be any positive integer greater than N, and this application does not impose any specific limitation on it. D can be a value directly set by the user, or it can be a multiple of N set by the user, and the controller determines the value based on this multiple of N, and this application does not impose any specific limitation on it.

[0106] For example, assuming the grid size is V, the horizontal width of the first image is r, and the vertical length is c, then the number of grids in each row is rn = r / V, the number of grids in each column is cn = c / V, and the total number of grids is... but

[0107] In some embodiments, after performing step S11051, the S first feature points can be directly assigned to the corresponding grids. In other embodiments, after performing step S11051, the S first feature points can also be assigned to the corresponding grids according to their coordinate positions in the first image. This application does not impose specific limitations on this.

[0108] S11052: When the second grid among the D grids contains E first feature points and E is a positive integer greater than 1, calculate the response values of the E first feature points respectively.

[0109] It should be understood that the second grid can be any grid among the D grids where the number of first feature points is greater than 1, and the present application does not make specific limitations thereto. The value of E in different second grids can be the same or different, and the present application does not make specific limitations thereto.

[0110] S11053: Screen out the first feature points corresponding to the top F response values with larger numerical values according to the numerical magnitudes of the response values of the E first feature points, so as to screen out C first feature points, where F < E, and F and E are positive integers.

[0111] Specifically, the controller can sort the numerical magnitudes of the response values of the E first feature points, select the top F response values with larger numerical values, and determine the first feature points corresponding to the top F response values according to the top F response values, and take the first feature points corresponding to the top F response values as the screened first feature points in the second grid. The same method can be used to screen matching pairs for all second grids that contain more than 1 first feature point, and combine the screened first feature points in all second grids and the first feature points in the un-screened grids, so as to obtain C first feature points.

[0112] It should be understood that F can be 1 or any preset numerical value, as long as it is less than E, and the present application does not make specific limitations thereto. When F = 1, it can be to screen out the first feature point corresponding to the largest response value, so as to ensure the highest quality of the first feature points retained in each second grid.

[0113] According to the technical solution provided by the embodiment of the present application, by dividing the first image into D grids and screening out the first feature points corresponding to the top F response values with larger numerical values according to the numerical magnitudes of the response values of the E first feature points, C first feature points with uniform distribution and high quality are screened out. In the embodiment of the present application, since the uniformity of the distribution of the first feature points, the quality of the first feature points, and the quantity and quality of the matching pairs are comprehensively considered, while reducing the quantity of the final matching pairs, high-quality final matching pairs are ensured, which is conducive to improving the accuracy of subsequent UAV motion estimation based on the final matching pairs and the accuracy of subsequent image reconstruction based on the final matching pairs.

[0114] Figure 5 The figure shows a schematic flowchart of a method for screening matching pairs provided by another embodiment of the present application. The embodiment shown in Figure 5 is Figure 1 a variant example of the embodiment shown. As Figure 5 shown, compared with Figure 1The difference in the illustrated embodiment is that steps S1203 and S1204 correspond to Figure 1 Step S120 in the illustrated embodiment.

[0115] S110: Obtain K matching pairs, which include K first feature points in the first image and K second feature points in the second image that match the K first feature points.

[0116] S1203: Calculate the mean or sum of response values ​​between the first and second feature points corresponding to each of the K matching pairs to obtain the mean or sum of the K response values.

[0117] S1204: Based on the magnitude of the mean of the K response values ​​or the sum of the K response values, select the matching pairs corresponding to the mean of the top P response values ​​or the matching pairs corresponding to the sum of the top P response values ​​with the larger values, so as to obtain P matching pairs.

[0118] Specifically, the average or sum of the K response values ​​can be sorted from largest to smallest, and the top P average or sum of the top P response values ​​with the largest values ​​can be selected. Based on the top P average or sum of the top P response values ​​with the largest values, P matching pairs can be determined.

[0119] S130: Determine the final matching pair based on P matching pairs, where K>P, and K, P, and M are positive integers.

[0120] It should be understood that, in the absence of any contradiction in substitution or combination, Figure 2A and Figure 3A The steps in the illustrated embodiment can also be combined with... Figure 5 The embodiments shown may be combined or substituted, and this application does not specifically limit this.

[0121] According to the technical solution provided in the embodiments of this application, P matching pairs are obtained by filtering out the matching pairs corresponding to the average of the top P response values ​​or the sum of the top P response values ​​based on the magnitude of the average of the K response values ​​or the sum of the top P response values. This ensures that the quality of the P matching pairs selected is the highest among the K matching pairs. In this way, the amount of data that needs to be processed to reconstruct images of farmland or buildings is reduced, while the speed and accuracy of 3D reconstruction based on the final matching pairs are improved, and the user experience is also enhanced.

[0122] Figure 6 The diagram shown is a flowchart of a three-dimensional reconstruction method according to an embodiment of this application. The three-dimensional reconstruction method includes the following steps.

[0123] S610: According to... Figures 1 to 5The final matching pair can be obtained by using any of the matching pair filtering methods.

[0124] It should be understood that, according to Figures 1 to 5 Any matching pair filtering method can be used to obtain the final matching pair, which is equivalent to or significantly modified from the matching pair filtering method. This application does not specifically limit this method. The first image can be any image of a scene such as farmland or buildings captured by the camera of the UAV. Other images of scenes such as farmland or buildings captured by the camera of the UAV can also be obtained by using any matching pair filtering method in this embodiment.

[0125] S620: Reconstruct the 3D point cloud data of the object to be reconstructed based on the final matching pair.

[0126] For example, the correspondence between the first and second images can be obtained based on the final matching, initially acquiring the camera's attitude and position information in three-dimensional space. The camera's attitude parameters can then be optimized by combining the GPS position information contained in the first image. Then, multiple images of farmland or building scenes captured by the UAV's camera are used to obtain dense point clouds from these images using methods such as depth fusion. These dense point clouds are then fused to obtain the three-dimensional point cloud data of the object to be reconstructed. It should be understood that, in addition to the exemplified method, other methods can be used to reconstruct the three-dimensional point cloud data of the object to be reconstructed, and this application does not specifically limit these methods.

[0127] It should be understood that the object to be reconstructed can be farmland and buildings, or other scenes, and this application does not make any specific restrictions on this.

[0128] According to the technical solution provided in the embodiments of this application, by utilizing, as Figures 1 to 5 The selection of matching pairs can be achieved by using any of the following methods to obtain the final matching pairs, thereby reducing the number of final matching pairs. Furthermore, since the selection process considers the mean or sum of the response values ​​between the first and second feature points of at least two matching pairs, it reduces the number of final matching pairs while filtering out poor-quality pairs. This is beneficial for selecting higher-quality matching pairs based on the magnitude of the mean or sum of the response values, which in turn helps reduce the amount of data required to reconstruct farmland or building images. It also improves the accuracy of subsequent UAV motion estimation based on the final matching pairs and the accuracy of subsequent image reconstruction based on the final matching pairs.

[0129] Figure 7The diagram shows a schematic of a matching pair filtering device according to an embodiment of this application. The matching pair filtering device 700 includes an acquisition module 710, a filtering module 720, and a determination module 730. The acquisition module 710 acquires K matching pairs, where each K matching pair includes K first feature points in a first image and K second feature points in a second image that match the K first feature points. The filtering module 720, when K is greater than the desired number of matching pairs M, filters the K matching pairs based on the average or sum of the response values ​​between the first and second feature points corresponding to at least two matching pairs, to obtain P matching pairs. The determination module 730 determines the final matching pairs based on the P matching pairs, where K > P, and K, P, and M are positive integers.

[0130] It should be understood that the matching pair screening device in the embodiments of this application can, in addition to achieving Figure 1 In addition to the aforementioned matching pair filtering method, it is also possible to achieve, as well as... Figures 2A to 5 Any of the matching pair filtering methods can also be implemented based on Figures 1 to 5 Any matching pair selection method is equivalent to the matching pair selection method after substitution or obvious modification, and this application does not make specific limitations in this regard.

[0131] According to the technical solution provided in the embodiments of this application, by utilizing an acquisition module and a filtering module, P matching pairs are filtered from K matching pairs, thereby reducing the number of matching pairs and ultimately reducing the number of final matching pairs. Furthermore, since a larger response value indicates higher feature point quality, this embodiment considers the average or sum of the response values ​​between the first and second feature points corresponding to at least two matching pairs. This reduces the final number of matching pairs while filtering out poor-quality matching pairs, ensuring that the quality of the first and second feature points in each of the filtered matching pairs is the highest, thus improving the reliability of the matching results.

[0132] Figure 8 The diagram shown is a block diagram of an unmanned device according to an embodiment of this application. The unmanned device 800 includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement... Figures 1 to 5 Any matching pair filtering method or such Figure 6 The illustrated embodiment is a three-dimensional reconstruction method.

[0133] Reference Figure 8The unmanned device 800 includes a processor 810, which further includes one or more processors, and memory resources represented by memory 820 for storing instructions executable by the processor 810, such as application programs. The application programs stored in memory 820 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 810 is configured to execute instructions to perform either a matching pair screening method or a three-dimensional reconstruction method as described above.

[0134] It should be understood that the unmanned device 800 can be a drone or a drone control device, as long as it has image processing capabilities. This application does not make any specific limitations in this regard.

[0135] The unmanned device 800 may also include a power supply component configured for power management of the electronic device 800, a wired or wireless network interface configured to connect the unmanned device 800 to a network, and an input / output (I / O) interface. The unmanned device 800 can operate on an operating system, such as Windows Server, stored in memory 820. TM Mac OSX TM Unix TM Linux TM FreeBSD TM Or similar.

[0136] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the unmanned device 800, enables the unmanned device 800 to perform a matching pair filtering method or a three-dimensional reconstruction method. The method is executed by an agent program. When the unmanned device 800 performs a matching pair filtering method, the filtering method includes obtaining K matching pairs, wherein the K matching pairs include K first feature points in a first image and K second feature points in a second image that match the K first feature points respectively; when K is greater than the expected number of matching pairs M, filtering the K matching pairs according to the average or sum of the response values ​​between the first and second feature points corresponding to at least two matching pairs to obtain P matching pairs; determining the final matching pair based on the P matching pairs, where K>P, and K, P, and M are positive integers. When the unmanned device 800 performs a three-dimensional reconstruction method, the reconstruction method includes: according to... Figures 1 to 5 The final matching pair is obtained by using any of the matching pair filtering methods; the three-dimensional point cloud data of the object to be reconstructed is reconstructed based on the final matching pair.

[0137] Those skilled in the art will recognize that the algorithmic steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0139] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the apparatus and system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0141] It should also be noted that the combination of the technical features in the embodiments of this application is not limited to the combination methods described in the embodiments of this application or the combination methods described in specific embodiments. All technical features described in this application can be freely combined or combined in any way, unless they contradict each other.

[0142] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for filtering matching pairs, characterized in that, Including: Obtain K matching pairs, where the K matching pairs include K first feature points in a first image and K second feature points in a second image that respectively match the K first feature points; When the K is greater than the expected number M of matching pairs, screen the K matching pairs according to the mean value or the total value of the response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs, so as to obtain P matching pairs; Determine the final matching pairs according to the P matching pairs, where K > P, and K, P, and M are positive integers; The determining the final matching pairs according to the P matching pairs includes: When P > M, screen out the first M matching pairs corresponding to the smallest numerical values of the matching pair distances according to the numerical sizes of the matching pair distances corresponding to the P matching pairs. The smaller the matching pair distance, the higher the similarity between the first feature point and the second feature point in the matching pair; Determine the first M matching pairs corresponding to the smallest numerical values of the matching pair distances as the final matching pairs.

2. The screening method according to claim 1, characterized in that, Before screening the K matching pairs according to the mean value or the total value of the response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs, it further includes: Divide the first image into R grids, where R ≥ M and R is a positive integer; Where, the screening the K matching pairs according to the mean value or the total value of the response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs includes: When the first grid in the R grids includes A first feature points and A is a positive integer greater than 1, calculate the mean value or the total value of the response values between the A first feature points and the A second feature points corresponding to the A first feature points respectively, so as to obtain A mean values of response values or A total values of response values; Screen out the first L matching pairs corresponding to the largest numerical values of the A mean values of response values or the first L matching pairs corresponding to the largest numerical values of the A total values of response values according to the numerical sizes of the A mean values of response values or the A total values of response values, so as to obtain P matching pairs, where L < A, and L and A are positive integers.

3. The screening method according to claim 1, characterized in that, The determining the final matching pairs according to the P matching pairs includes: When P ≤ M, determine the P matching pairs as the final matching pairs.

4. The screening method according to claim 1, characterized in that, The screening the K matching pairs according to the mean value or the total value of the response values between the first feature points and the second feature points corresponding to at least two of the K matching pairs to obtain P matching pairs includes: Calculate the mean value or the total value of the response values between the first feature points and the second feature points corresponding to each of the K matching pairs, so as to obtain K mean values of response values or K total values of response values; Screen out the first P matching pairs corresponding to the largest numerical values of the K mean values of response values or the first P matching pairs corresponding to the largest numerical values of the K total values of response values according to the numerical sizes of the K mean values of response values or the K total values of response values, so as to obtain P matching pairs.

5. The screening method according to claim 1, characterized in that, After obtaining the K matching pairs, it further includes: When K ≤ M, determine the K matching pairs as the final matching pairs.

6. The screening method according to any one of claims 1-5, characterized in that, The obtaining of the K matching pairs includes: Extracting S first feature points from the first image; Finding S second feature points in the second image that match the S first feature points based on the matching pair distance to obtain S matching pairs; Filtering the S matching pairs through geometric verification to obtain the K matching pairs, where K ≤ S.

7. The screening method according to claim 6, after extracting S first feature points from the first image, further includes: When S is greater than the desired number of feature points N, uniformly screening the S first feature points to screen out C first feature points, where C < S, and S and C are positive integers; Among them, the finding of S second feature points in the second image that match the S first feature points based on the matching pair distance to obtain S matching pairs includes: Finding C second feature points in the second image that match the C first feature points based on the matching pair distance to obtain C matching pairs; Among them, the filtering of the S matching pairs through geometric verification to obtain the K matching pairs includes: Filtering the C matching pairs through the geometric verification to obtain the K matching pairs, where K ≤ C.

8. The screening method according to claim 7, characterized in that, The uniformly screening of the S first feature points to screen out C first feature points includes: Dividing the first image into D grids, where D ≥ N and D is a positive integer; When the second grid in the D grids includes E first feature points and E is a positive integer greater than 1, respectively calculating the response values of the E first feature points; Screening out the first feature points corresponding to the top F response values with the largest numerical values according to the numerical sizes of the response values of the E first feature points to screen out C first feature points, where F < E, and F and E are positive integers.

9. The screening method according to any one of claims 1-5, characterized in that, The first image and the second image are two different images obtained by a photographing device for photographing an object to be reconstructed.

10. A three-dimensional reconstruction method, characterized in that, Including: Obtaining the final matching pairs according to a screening method of a matching pair as described in any one of claims 1-9; Reconstructing the three-dimensional point cloud data of the object to be reconstructed according to the final matching pairs.

11. A device for screening matching pairs, characterized in that, Including: An obtaining module, configured to obtain K matching pairs, where the K matching pairs include K first feature points in the first image and K second feature points in the second image that respectively match the K first feature points; A screening module, configured to, when K is greater than the desired number of matching pairs M, screen the K matching pairs according to the mean value or the sum of the response values between the first feature points and the second feature points corresponding to at least two matching pairs among the K matching pairs to obtain P matching pairs; A determining module, configured to determine the final matching pairs according to the P matching pairs, where K > P, and K, P, and M are positive integers; The determining of the final matching pairs according to the P matching pairs includes: When P > M, screening out the top M matching pairs corresponding to the smallest numerical values of the matching pair distances according to the numerical sizes of the matching pair distances corresponding to the P matching pairs. The smaller the matching pair distance, the higher the similarity of the matching pair; The matching pairs corresponding to the distances of the top M matching pairs with the smallest values ​​are determined as the final matching pairs.

12. An unmanned device, comprising a memory and a processor, characterized in that, The memory stores executable instructions for a computer, and when the processor executes the executable instructions, it implements a matching pair screening method as described in any one of claims 1-9 or a three-dimensional reconstruction method as described in claim 10.

13. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When the executable instructions are executed by the processor, they implement a matching pair filtering method as described in any one of claims 1-9 or a three-dimensional reconstruction method as described in claim 10.

Images