Image matching method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose an image matching method, apparatus, device and storage medium. The technical solution provided by the embodiments of the present application obtains the position data corresponding to the image to be matched and the image to be matched, and aggregates the adjacent frames of the image to be matched according to the position data to obtain multiple adjacent frame image groups, and for each adjacent frame image group, performs feature point matching on the image to be matched to obtain multiple matching pairs, and each feature point corresponds to a set number of matching pairs, and further screens matching pairs that meet the annular constraint condition, triangulates the screened matching pairs to obtain spatial points, and screens the spatial points that meet the motion constraint condition, determines the image matching result based on the screened spatial points, and screens out the erroneous matching pairs caused by occlusion and deformation of the common view area between images due to rotation and translation based on the annular constraint condition and the motion constraint condition, thereby improving the accuracy and matching effect of image matching.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image matching method, apparatus, device, and storage medium. Background Art

[0002] At present, the mapping and reconstruction of scenes (such as farmland, buildings, roads, etc.) are generally carried out by collecting images of the scene through drones, extracting feature points of the image data, and performing feature matching based on the feature points between the image data to construct point-to-point association relationships between the images. Based on the association relationships between the image data, motion constraints are constructed and the corresponding three-dimensional structure of the scene is restored.

[0003] Existing image matching solutions generally measure similarity by calculating the feature points between two images, such as the Euclidean distance between the feature points of the two images, and then selecting the feature points for constructing matching pairs based on the shortest Euclidean distance. However, due to the various rotations and displacements of drones during flight, the common view area between the images may be occluded or deformed. This can easily lead to the correct matching pair not being the feature point with the smallest Euclidean distance, resulting in poor image matching results. Summary of the Invention

[0004] The embodiments of the present application provide an image matching method, apparatus, device, and storage medium to solve the technical problem in the prior art that, due to the various rotations and displacements of drones during flight, the correct matching pairs are not the feature points with the minimum Euclidean distance, resulting in poor image matching results. The matching pairs are screened according to the constraint relationship between images to improve the image matching effect.

[0005] In a first aspect, an embodiment of the present application provides an image matching method, comprising:

[0006] Acquire position data corresponding to an image to be matched and the image to be matched, and aggregate adjacent frames of the image to be matched according to the position data to obtain a plurality of adjacent frame image groups;

[0007] In each group of adjacent frame images, feature point matching is performed on the image to be matched to obtain a plurality of matching pairs, where each feature point of the image to be matched corresponds to a set number of matching pairs;

[0008] Based on the set annular constraint condition, selecting matching pairs that meet the annular constraint condition in each adjacent frame image group;

[0009] The filtered matching pairs are triangulated to obtain spatial points, the spatial points are filtered based on set motion constraints, and the image matching results are determined based on the filtered spatial points.

[0010] In a second aspect, an embodiment of the present application provides an image matching device, including an image aggregation module, a feature matching module, a ring constraint module, and a motion constraint module, wherein:

[0011] The image aggregation module is used to obtain the position data corresponding to the image to be matched and the image to be matched, and perform adjacent frame aggregation on the image to be matched according to the position data to obtain multiple adjacent frame image groups;

[0012] The feature matching module is used to perform feature point matching on the image to be matched in each group of adjacent frame images to obtain a plurality of matching pairs, where each feature point of the image to be matched corresponds to a set number of matching pairs;

[0013] The annular constraint module is used to screen matching pairs that meet the annular constraint conditions in each group of adjacent frame images based on the set annular constraint conditions;

[0014] The motion constraint module is used to triangulate the filtered matching pairs to obtain spatial points, filter the spatial points based on the set motion constraint conditions, and determine the image matching results based on the filtered spatial points.

[0015] In a third aspect, an embodiment of the present application provides an image matching device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image matching method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to perform the image matching method as described in the first aspect.

[0019] The embodiment of the present application obtains the position data corresponding to the image to be matched and the image to be matched, and aggregates adjacent frames of the image to be matched according to the position data to obtain multiple adjacent frame image groups, and for each adjacent frame image group, performs feature point matching on the image to be matched to obtain multiple matching pairs, and each feature point corresponds to a set number of matching pairs, and further screens the matching pairs that meet the annular constraint condition, triangulates the screened matching pairs to obtain spatial points, and screens the spatial points that meet the motion constraint condition, determines the image matching result according to the screened spatial points, and screens out the erroneous matching pairs caused by occlusion and deformation of the common view area between images due to rotation and translation based on the annular constraint condition and the motion constraint condition, thereby improving the accuracy and matching effect of image matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of an image matching method provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of another image matching method provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a movement trajectory and aggregated position range provided in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of screening matching pairs based on a ring constraint condition provided in an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a spatial point screening process based on motion constraints provided by an embodiment of the present application;

[0025] Figure 6 is a structural diagram of an image matching device provided in an embodiment of the present application;

[0026] Figure 7 It is a structural diagram of an image matching device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It should also be noted that, for ease of description, only parts related to the present application, not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] Figure 1 A flowchart of an image matching method provided in an embodiment of the present application is given. The image matching method provided in an embodiment of the present application can be executed by an image matching device, which can be implemented by hardware and / or software and integrated into an image matching device.

[0029] The following description is made by taking an image matching device executing an image matching method as an example. Figure 1, the image matching method includes:

[0030] S101: Acquire an image to be matched and position data corresponding to the image to be matched, and aggregate adjacent frames of the image to be matched according to the position data to obtain a plurality of adjacent frame image groups.

[0031] The image to be matched provided in this embodiment is obtained by photographing a camera, wherein the camera can be mounted on a mobile vehicle, and the mobile vehicle drives the camera to move to a target area (e.g., farmland, buildings, roads) where image matching or three-dimensional reconstruction is required, and photographs the target area during the movement. The camera can directly output image data in a picture format as the image to be matched, or it can output image data in a video format, and select image frames from the image data as the image to be matched. The mobile vehicle can be a vehicle, a drone, etc.

[0032] Furthermore, the camera or mobile vehicle is also equipped with a positioning device (such as an RTK positioning module). While capturing the image to be matched (image data), the positioning device also obtains position data in real time and determines the correspondence between the image to be matched and the position data. When capturing images at adjacent positions (not necessarily positions captured at adjacent times, but also positions captured when moving relative to or in the same direction after bending), there is a common view area between the corresponding images to be matched, that is, there is an overlapping or duplicated area between the adjacently positioned images to be matched.

[0033] It is understandable that when the images to be matched (image data) are captured by a camera at different positions, due to the inevitable occurrence of various rotations, displacements, etc. during the movement, the common view area between the images to be matched captured at adjacent positions may be occluded or deformed. This occlusion has a significant impact on the matching of image feature points (descriptors) between images. In extreme cases (such as when repeated textures appear), the impact on feature point matching is even greater. In this case, the correct matching pair may not be the two feature points with the smallest Euclidean distance, but may be the second, third, or Kth smallest feature. The existing method of screening matching pairs based on a fixed threshold is obviously unable to cope with such a situation, resulting in a large number of false matches. This solution first adds the images to be matched at adjacent shooting positions to the same adjacent frame image group, and constructs multi-image matching constraints based on feature points in each adjacent frame image group. The final accurate matching pair is determined based on ring constraint matching and motion constraint matching to obtain a more stable and accurate data association relationship.

[0034] Among them, the adjacent frame image group can be understood as a collection of images to be matched that are shot at similar positions. Exemplarily, after obtaining the position data of the image to be matched and the image to be matched, all the images to be matched are aggregated into adjacent frames according to the position data to obtain multiple adjacent frame image groups, and the position data corresponding to all the images to be matched in each adjacent frame image group are within the set aggregation position range. In one embodiment, an image to be matched has a common viewing area with at least two other images to be matched in the same adjacent frame image group. Among them, an image to be matched can exist in multiple adjacent frame image groups at the same time.

[0035] This scheme aggregates adjacent frames of the images to be matched based on position data, and gathers the images to be matched at adjacent shooting positions into the same adjacent frame image group. The shooting positions and image space positions of the images to be matched in the adjacent frame image group are relatively close, and there are more formula relationships between the images to be matched. There are also a large number of constraints between matching pairs. Screening matching pairs within the adjacent frame image group can effectively reduce the amount of calculation and improve image matching efficiency.

[0036] S102: In each group of adjacent frame images, feature point matching is performed on the image to be matched to obtain a plurality of matching pairs, where each feature point of the image to be matched corresponds to a set number of matching pairs.

[0037] Exemplarily, after obtaining a plurality of adjacent frame image groups, feature points of each image to be matched are extracted in each adjacent frame image group. For example, feature points (descriptors) in the image to be matched are extracted based on the SIFT (Scale-invariant feature transform) algorithm. Generally, multiple feature points can be extracted from one image to be matched. Furthermore, feature point matching is performed in each adjacent frame image group to obtain multiple matching pairs, and each feature point of the image to be matched corresponds to a set number of matching pairs. The set number is greater than 1.

[0038] Among them, the feature extraction algorithm for extracting feature points from the image to be matched can be an LBP feature extraction algorithm, a HOG feature extraction algorithm, a Haar feature extraction operator, a LoG feature extraction algorithm, a Harris corner feature extraction algorithm, a SIFT feature extraction algorithm, a SURF feature extraction algorithm, etc. This embodiment takes the SIFT feature extraction algorithm as an example for illustrative description.

[0039] In the prior art, for two images, a matching pair is generally constructed by taking the two feature points with the smallest Euclidean distance, that is, one feature point corresponds to only one matching pair. This solution retains multiple (set number - 1) matching points outside the minimum Euclidean distance to construct matching pairs, and then screens these matching pairs to determine the correct matching pairs, thereby reducing the situation where the correct matching pair is not the feature point with the smallest Euclidean distance due to rotation and displacement during movement, occlusion, deformation, etc. in the common view area between images.

[0040] In one embodiment, matching pairs can be constructed from matching pairs ranked within a set number according to the order of similarity between matching points. Based on this, when performing feature point matching on the image to be matched to obtain multiple matching pairs, the present solution specifically includes:

[0041] S1021: Determine feature points of the image to be matched, perform feature point matching on the images to be matched within the same adjacent frame image group, and determine multiple candidate matching points for each feature point and corresponding similarities.

[0042] S1022: Sort the candidate matching points corresponding to each feature point according to the similarity, and determine the matching pairs corresponding to the feature point based on the candidate matching points whose sorting results are within a set number.

[0043] Specifically, candidate points are extracted from each image to be matched based on a predetermined feature extraction algorithm. Feature point matching is then performed in each group of adjacent frame images to determine multiple candidate matching points and their corresponding similarities for each feature point. Specifically, for a feature point in an image to be matched within a group of adjacent frame images, the similarity between that feature point and all feature points in all images to be matched within the same group of adjacent frame images is calculated. The candidate matching points corresponding to a feature point in an image to be matched within a group of adjacent frame images are the feature point and all feature points in all images to be matched within the same group of adjacent frame images.

[0044] The similarity between the feature point and the candidate matching point can be expressed by cosine similarity. The higher the cosine similarity, the higher the similarity between the feature point and the candidate matching point. In addition, it can also be expressed by the feature point distance (such as Euclidean distance, Hamming distance, Hammanton distance, etc.). The smaller the feature point distance, the higher the similarity between the feature point and the candidate matching point.

[0045] Furthermore, for each feature point, the candidate matching points are sorted by similarity with respect to the remaining images to be matched in the same adjacent frame image, wherein the higher the similarity, the higher the ranking of the corresponding candidate matching point. After the candidate matching points are sorted, the remaining images to be matched in the same adjacent frame image are screened for candidate matching points whose sorting results fall within a first preset number, and matching pairs are constructed between the screened candidate matching points and the feature point. The product of the first preset number and the remaining images to be matched in the same adjacent frame image is the set number, i.e., the first set number of matching pairs is retained for each feature point.

[0046] S103: Based on the set annular constraint condition, select matching pairs that meet the annular constraint condition in each group of adjacent frame images.

[0047] For example, after obtaining a set number of matching pairs corresponding to each feature point, based on the set ring constraint, matching pairs that meet the ring constraint are screened in each group of adjacent frame images. Among the matching pairs that meet the ring constraint, starting from one matching pair, the matching pairs of other images to be matched in the same group of adjacent frame images are sequentially searched to form a closed loop of matching pairs.

[0048] S104: triangulating the filtered matching pairs to obtain spatial points, filtering the spatial points based on set motion constraints, and determining image matching results based on the filtered spatial points.

[0049] For example, for each group of adjacent frame images, the filtered matching pairs are triangulated to obtain the spatial points corresponding to the two feature points in each matching pair. Furthermore, the spatial points are filtered based on the set motion constraints to obtain spatial points that satisfy the motion constraints, and for which the offset distance between the projection position of the spatial point in the corresponding image to be matched and the corresponding feature point is within a set deviation threshold.

[0050] Furthermore, after the spatial points are screened, the image matching results are determined based on the screened spatial points. That is, the matching pairs corresponding to all remaining spatial points after screening represent the final feature point matching relationships between the corresponding images to be matched. This image matching result can provide a large and accurate training dataset for deep learning methods to estimate feature extraction and matching models, allowing for more precise 3D modeling.

[0051] In the above, by obtaining the position data corresponding to the image to be matched and the image to be matched, and performing adjacent frame aggregation on the image to be matched according to the position data, a plurality of adjacent frame image groups are obtained, and for each adjacent frame image group, feature point matching is performed on the image to be matched to obtain a plurality of matching pairs, and each feature point corresponds to a set number of matching pairs, and the matching pairs that meet the annular constraint conditions are further screened, the screened matching pairs are triangulated to obtain spatial points, and the spatial points that meet the motion constraint conditions are screened, and the image matching results are determined according to the screened spatial points, and based on the annular constraint conditions and the motion constraint conditions, the erroneous matching pairs caused by occlusion and deformation of the common view area between images due to rotation and translation are screened out, thereby improving the accuracy and matching effect of image matching.

[0052] Based on the above embodiments, Figure 2 A flowchart of another image matching method provided in an embodiment of the present application is given, which is a specific implementation of the above-mentioned image matching method. Figure 2 , the image matching method includes:

[0053] S201: Acquire an image to be matched and position data corresponding to the image to be matched.

[0054] S202: Determine the aggregation position ranges corresponding to different to-be-matched images according to the position data and a plurality of set aggregation position ranges, and obtain a plurality of adjacent frame image groups based on the to-be-matched images within each aggregation position range.

[0055] Specifically, multiple aggregation position ranges are pre-set based on the target range. After obtaining the matching images and corresponding position data, adjacent frames of the matching images are aggregated based on the correspondence between the position data of the matching images and the respective aggregation position ranges. There may be overlapping areas between adjacent aggregation position ranges.

[0056] Furthermore, the aggregation position range corresponding to each image to be matched is determined according to the position data, and the images to be matched within each aggregation position range are aggregated to obtain a group of adjacent frame images within each aggregation position range.

[0057] In one embodiment, the image to be matched is captured based on a set movement trajectory, and the aggregation position range is determined based on the movement trajectory. For example, the movement trajectory is set within a target range for image matching or 3D reconstruction as needed, and the mobile vehicle subsequently moves according to the set movement trajectory, while the camera captures the image to be matched during the movement. Furthermore, after determining the movement trajectory, multiple aggregation position ranges for aggregating adjacent frames are determined within the target range according to the movement trajectory.

[0058] Figure 3This is a schematic diagram of a movement trajectory and aggregated position range provided in an embodiment of the present application, such as Figure 3 As shown, L1 is the movement trajectory, the arrow of the movement trajectory L1 is the movement direction, S1 and S2 are two of the aggregation position ranges, and there is an overlapping area between the aggregation position range S1 and the aggregation position range S2. 1 to 20 are the images to be matched taken during the movement according to the movement trajectory L1. After moving and taking the images to be matched according to the movement trajectory, assuming that the position data corresponding to the images to be matched 3, 4, 7, and 8 are within the aggregation position range S1, then the images to be matched in the adjacent frame image group corresponding to the aggregation position range S1 are numbered 3, 4, 7, and 8. Correspondingly, assuming that the position data corresponding to the images to be matched 8, 9, 12, and 13 are within the aggregation position range S1, then the images to be matched in the adjacent frame image group corresponding to the aggregation position range S1 are numbered 8, 9, 12, and 13, where the image to be matched 8 exists in both the aggregation position ranges S1 and S2.

[0059] S203: In each group of adjacent frame images, feature point matching is performed on the image to be matched to obtain a plurality of matching pairs, where each feature point of the image to be matched corresponds to a set number of matching pairs.

[0060] S204: In each adjacent frame image group, starting from a matching pair in one of the images to be matched, sequentially passing through the matching pairs of other images to be matched in the same adjacent frame image group, and determining the matching pairs that form a matching closed loop as the matching pairs that satisfy the ring constraint condition.

[0061] Specifically, for each image to be matched in each adjacent frame image group, traverse the matching pairs corresponding to each feature point, start from the matching pair corresponding to the feature point, and pass through the matching pairs of other images to be matched in the same adjacent frame image group in a head-to-tail manner, looking for a matching pair line that can return to the feature point. The matching pairs corresponding to the matching pair line form a matching closed loop.

[0062] After traversing the matching pairs corresponding to each feature point, the matching pairs that can form a matching closed loop are retained, and the matching pairs that do not exist with the matching closed loop are eliminated. It can be understood that for the matching pairs that cannot form a matching closed loop, they may be incorrect matching pairs caused by occlusion and deformation of the common viewing area between images due to rotation and translation. These incorrect matching pairs can be eliminated to ensure the correctness of the remaining matching pairs. Among the matching pairs after screening, the similarity between the corresponding feature points may be the highest similarity, or it may be the second, third or Kth highest feature. The correctly matched matching pairs are retained through the ring constraint, and a large number of incorrect matching pairs are eliminated, effectively ensuring the correctness of the matching pairs.

[0063] Figure 4This is a schematic diagram of a method for screening matching pairs based on a ring constraint condition provided in an embodiment of the present application. Figure 4 As shown, taking the four images to be matched in a group of adjacent frame images as an example (a1-a4 in the figure), assuming that the number of matching points of each feature point in the adjacent images to be matched is 2, that is, each feature point corresponds to 2 matching pairs in the adjacent images to be matched. For a feature point P in the image a1 to be matched, starting from the feature point P, matching pairs are formed with the two feature points of the image a2 to be matched, each feature point in the image a2 to be matched forms a matching pair with the two feature points in the image a3 to be matched, each feature point in the image a3 to be matched forms a matching pair with the two feature points in the image a4 to be matched, and each feature point in the image a4 to be matched forms a matching pair with the two feature points in the image a1 to be matched (one of which is the feature point P). Therefore, the number of matched feature points changes as follows: 1-2-4-8-16. Thus, the feature point P starting from the image a1 to be matched is finally matched back from the image a4 to be matched, and there are a total of 16 feature points. The circular constraint condition is to find the starting point among these 16 constraints. If the starting point that constitutes the circular matching can be found among the final 16 matched feature points, that is, a closed loop of matching points is formed (such as the closed loop line L2 in the figure), then these matching pairs are retained, and any other matching pairs that do not meet the circular matching constraint are eliminated, thereby removing a large number of false matches.

[0064] S205: triangulating the filtered matching pairs to obtain spatial points, filtering the spatial points based on set motion constraints, and determining image matching results based on the filtered spatial points.

[0065] In one embodiment, among the spatial points that satisfy the motion constraint condition, the offset distance between the projection position of the spatial point in the corresponding image to be matched and the corresponding feature point is within the set deviation threshold. Based on this, Figure 5 The provided schematic diagram of a spatial point screening process based on motion constraints is shown in the following example. When screening spatial points based on the set motion constraints, the following steps are performed:

[0066] S2051: Calculate the offset distance between the projection position of each spatial point in the corresponding image to be matched and the corresponding feature point according to the spatial point and the camera parameters of the corresponding image to be matched.

[0067] Specifically, for each spatial point, the system projects the spatial point onto the image to be matched based on its coordinates and the camera parameters of the image to be matched. The projected position of the spatial point into the image to be matched is determined. Furthermore, the offset distance between the projected position of the spatial point in the image to be matched and the corresponding feature point is calculated.

[0068] The camera parameters include the camera's rotation parameters, translation parameters, and intrinsic parameter matrix, which can be obtained from the camera output. It can be understood that spatial points are obtained by back-projecting feature points into spatial coordinates. If the corresponding matching pair is a correct matching pair, then when the spatial point is projected into the image to be matched, the offset distance between the projected position and the corresponding feature point should be within a set deviation threshold. If the offset distance exceeds the set deviation threshold, the corresponding matching pair can be considered an incorrect matching pair.

[0069] S2052: Determine a spatial point whose offset distance exceeds a set deviation threshold as an erroneous spatial point, and adjust the camera parameters of the image to be matched corresponding to the erroneous spatial point based on a set robust kernel function to constrain the offset distance corresponding to the erroneous spatial point within the set constraint threshold.

[0070] Specifically, for each spatial point, the corresponding offset distance is compared with the set deviation threshold. If the offset distance is within the set deviation threshold, the corresponding matching pair is considered to be a correct matching pair, and the corresponding spatial point is determined to be the correct spatial point, and no constraint processing is performed on the spatial point; if the offset distance is greater than the set deviation threshold, the corresponding matching pair is considered to be an incorrect matching pair, and the corresponding spatial point is an incorrect spatial point.

[0071] Furthermore, when an erroneous spatial point is found, the camera parameters of the image to be matched corresponding to the erroneous spatial point are adjusted based on the set robust kernel function to constrain the offset distance corresponding to the erroneous spatial point to be within the set constraint threshold.

[0072] It needs to be explained that this solution does not eliminate the erroneous spatial points when they are found. Instead, it continuously adjusts the camera parameters based on the set robust kernel function during the motion constraint optimization process, and eliminates the erroneous spatial points only after the motion constraint optimization is completed, thereby reducing the situation where the spatial points are incorrectly judged due to the early elimination of erroneous spatial points.

[0073] S2053: Filter out erroneous spatial points from the spatial points to obtain spatial points that meet the motion constraint conditions.

[0074] Specifically, after completing the motion constraint optimization for all spatial points, the erroneous spatial points are screened out, and the remaining spatial points are the spatial points that meet the motion constraint conditions. The image matching results can be determined based on these screened spatial points.

[0075] In one embodiment, after filtering out a large number of incorrect matching pairs using the ring matching constraint, a small number of matching pairs that do not meet the overall motion constraint remain. After triangulating all matching pairs to obtain spatial points, the SFM algorithm can be used to construct motion constraints for joint optimization of the spatial points. Specifically, the spatial points provided by this solution can be jointly optimized and screened based on the following structure-from-motion formula:

[0076]

[0077] Among them, P is the spatial point, M is the number of spatial points, R and t are the rotation parameters and translation parameters of the camera, K is the intrinsic parameter matrix of the camera, and x is the feature point x corresponding to the spatial point. The superscript 2 in denotes quadratic calculation, and the subscript γ denotes a robust kernel function (e.g., Huber robust kernel function).

[0078] Furthermore, the robust kernel function provided by this solution can be determined based on the following formula:

[0079]

[0080] Where d = xK(RP+t), δ is the set deviation threshold, 2δ|d|-δ 2 To set the constraint threshold.

[0081] It should be explained that since the image to be matched is acquired with a single global exposure, all correct matching pairs have the same motion trend as the moving vehicle (such as a drone), while incorrect matching pairs usually have disordered motion. Therefore, when minimizing the objective function of the motion recovery structure formula (i.e., F(R, t, P)), incorrect matching pairs with disordered motion will produce a very large cost, causing the image matching problem to diverge. Therefore, this solution sets a maximum threshold condition for a single residual term, that is, introduces the Huber Robin kernel function to eliminate these incorrect matching pairs, thereby ensuring the accuracy of image matching and the reliability of the final matching pairs.

[0082] In the above method, position data corresponding to the image to be matched and the image to be matched are obtained, and adjacent frames of the image to be matched are aggregated based on the position data to obtain multiple adjacent frame image groups. For each adjacent frame image group, feature point matching is performed on the image to be matched to obtain multiple matching pairs, with each feature point corresponding to a set number of matching pairs. Matching pairs that satisfy the annular constraint are further screened, and the screened matching pairs are triangulated to obtain spatial points. Spatial points that satisfy the motion constraint are also screened, and image matching results are determined based on the screened spatial points. Based on the annular constraint and the motion constraint, erroneous matching pairs caused by occlusion and deformation of the common view area between images due to rotation and translation are filtered out, thereby improving the accuracy and matching effect of image matching. Furthermore, the annular constraint is used to retain correctly matched matching pairs and eliminate a large number of erroneous matching pairs, effectively ensuring the correctness of the matching pairs. Furthermore, by recording spatial points that exceed a set deviation threshold, matching pairs that do not satisfy the motion constraint are filtered out, thereby obtaining the optimal matching pairs, drone posture, and three-dimensional scene structure. Moreover, when erroneous spatial points are found, they are not eliminated. Instead, the camera parameters are continuously adjusted based on the set robust kernel function during the motion constraint optimization process. The erroneous spatial points are eliminated only after the motion constraint optimization is completed, reducing the situation where spatial points are incorrectly judged due to the early elimination of erroneous spatial points.

[0083] Figure 6 A schematic diagram of the structure of an image matching device provided in an embodiment of the present application is given. Figure 6 The image matching device includes an image aggregation module 61, a feature matching module 62, a ring constraint module 63 and a motion constraint module 64.

[0084] Among them, the image aggregation module 61 is used to obtain the position data corresponding to the image to be matched and the image to be matched, and perform adjacent frame aggregation on the image to be matched according to the position data to obtain multiple adjacent frame image groups; the feature matching module 62 is used to perform feature point matching on the image to be matched in each adjacent frame image group to obtain multiple matching pairs, and each feature point of the image to be matched corresponds to a set number of matching pairs; the annular constraint module 63 is used to screen matching pairs that meet the annular constraint conditions in each adjacent frame image group based on the set annular constraint conditions; the motion constraint module 64 is used to triangulate the screened matching pairs to obtain spatial points, and screen the spatial points based on the set motion constraint conditions, and determine the image matching results based on the screened spatial points.

[0085] In the above, by obtaining the position data corresponding to the image to be matched and the image to be matched, and performing adjacent frame aggregation on the image to be matched according to the position data, a plurality of adjacent frame image groups are obtained, and for each adjacent frame image group, feature point matching is performed on the image to be matched to obtain a plurality of matching pairs, and each feature point corresponds to a set number of matching pairs, and the matching pairs that meet the annular constraint conditions are further screened, the screened matching pairs are triangulated to obtain spatial points, and the spatial points that meet the motion constraint conditions are screened, and the image matching results are determined according to the screened spatial points, and based on the annular constraint conditions and the motion constraint conditions, the erroneous matching pairs caused by occlusion and deformation of the common view area between images due to rotation and translation are screened out, thereby improving the accuracy and matching effect of image matching.

[0086] The embodiment of the present application also provides an image matching device, which can be integrated with the image matching apparatus provided in the embodiment of the present application. Figure 7 This is a schematic diagram of the structure of an image matching device provided in an embodiment of the present application. Figure 7 The image matching device includes: an input device 73, an output device 74, a memory 72, and one or more processors 71; the memory 72 is used to store one or more programs; when the one or more programs are executed by the one or more processors 71, the one or more processors 71 implement the image matching method provided in the above embodiment. The input device 73, the output device 74, the memory 72, and the processor 71 can be connected by a bus or other means. Figure 7 The bus connection is taken as an example.

[0087] The memory 72 is a computing device-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the image matching method described in any embodiment of the present application (for example, the image aggregation module 61, feature matching module 62, annular constraint module 63, and motion constraint module 64 in the image matching device). The memory 72 may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created based on the use of the device, etc. In addition, the memory 72 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 72 may further include a memory remotely located relative to the processor 71, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0088] The input device 73 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 74 may include a display device such as a display screen.

[0089] The processor 71 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 72, that is, implements the above-mentioned image matching method.

[0090] The image matching apparatus, device, and computer provided above can be used to execute the image matching method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0091] An embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the image matching method provided in the above embodiment, the image matching method comprising: obtaining position data corresponding to an image to be matched and the image to be matched, performing adjacent frame aggregation on the image to be matched according to the position data to obtain a plurality of adjacent frame image groups; in each adjacent frame image group, performing feature point matching on the image to be matched to obtain a plurality of matching pairs, each feature point of the image to be matched corresponds to a set number of matching pairs; based on a set annular constraint condition, screening matching pairs that meet the annular constraint condition in each adjacent frame image group; triangulating the screened matching pairs to obtain spatial points, screening the spatial points based on a set motion constraint condition, and determining the image matching result based on the screened spatial points.

[0092] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape drives; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or it may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that can reside in different locations (e.g., in different computer systems connected via a network). The storage medium can store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0093] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application, whose computer-executable instructions are not limited to the image matching method described above, can also execute related operations in the image matching method provided in any embodiment of the present application.

[0094] The image matching apparatus, device and storage medium provided in the above embodiments can execute the image matching method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the image matching method provided in any embodiment of the present application.

[0095] The above are only preferred embodiments of the present application and the technical principles employed. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that are possible for those skilled in the art will not depart from the scope of protection of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments and may include more other equivalent embodiments without departing from the concept of the present application. The scope of the present application is determined by the scope of the claims.

Claims

1. An image matching method, characterized in that: include: Acquire position data corresponding to an image to be matched and the image to be matched, and aggregate adjacent frames of the image to be matched according to the position data to obtain a plurality of adjacent frame image groups; In each group of adjacent frame images, feature point matching is performed on the image to be matched to obtain a plurality of matching pairs, where each feature point of the image to be matched corresponds to a set number of matching pairs; Based on the set ring constraint condition, matching pairs that meet the ring constraint condition are screened in each adjacent frame image group, wherein, among the matching pairs that meet the ring constraint condition, starting from one matching pair, the matching pairs of other to-be-matched images in the same adjacent frame image group are sequentially passed through to form a closed loop of matching pairs; The filtered matching pairs are triangulated to obtain spatial points, the spatial points are filtered based on set motion constraints, and the image matching results are determined based on the filtered spatial points.

2. The image matching method according to claim 1, wherein: The step of aggregating adjacent frames of the image to be matched according to the position data to obtain a plurality of adjacent frame image groups includes: According to the position data and a plurality of set aggregation position ranges, aggregation position ranges corresponding to different to-be-matched images are determined, and a plurality of adjacent frame image groups are obtained based on the to-be-matched images within each aggregation position range.

3. The image matching method according to claim 2, characterized in that: The image to be matched is obtained by shooting based on a set movement trajectory, and the aggregation position range is determined based on the movement trajectory.

4. The image matching method according to claim 1, wherein: The performing feature point matching on the image to be matched to obtain a plurality of matching pairs includes: Determining feature points of the image to be matched, performing feature point matching on the images to be matched within the same adjacent frame image group, and determining multiple candidate matching points for each feature point and corresponding similarities; The candidate matching points corresponding to each feature point are sorted according to the similarity, and matching pairs corresponding to the feature points are determined based on candidate matching points whose sorting results are within a set number.

5. The image matching method according to claim 1, wherein: The step of screening matching pairs that satisfy the set annular constraint conditions in each group of adjacent frame images includes: In each adjacent frame image group, starting from a matching pair in one of the images to be matched, the matching pairs of other images to be matched in the same adjacent frame image group are passed through in sequence, and the matching pairs that form a matching closed loop are determined as matching pairs that satisfy the said ring constraint condition.

6. The image matching method according to claim 1, wherein: Among the spatial points that satisfy the motion constraint condition, the offset distance between the projection position of the spatial point in the corresponding image to be matched and the corresponding feature point is within a set deviation threshold.

7. The image matching method according to claim 6, characterized in that: The screening of the spatial points based on the set motion constraint conditions includes: Calculating the offset distance between the projection position of each spatial point in the corresponding image to be matched and the corresponding feature point according to the spatial point and the camera parameters of the corresponding image to be matched; Determine a spatial point whose offset distance exceeds a set deviation threshold as an error spatial point, and adjust the camera parameters of the image to be matched corresponding to the error spatial point based on a set robust kernel function to constrain the offset distance corresponding to the error spatial point to within the set constraint threshold; Erroneous spatial points are screened out from the spatial points to obtain spatial points that meet the motion constraint conditions.

8. The image matching method according to claim 7, characterized in that: The spatial points are screened based on the following structure-from-motion formula: Among them, P is the spatial point, M is the number of spatial points, R and t are the rotation parameters and translation parameters of the camera, K is the intrinsic parameter matrix of the camera, and x is the feature point x corresponding to the spatial point. The superscript 2 in denotes quadratic calculation, and the subscript γ denotes the robust kernel function.

9. The image matching method according to claim 8, characterized in that: The robust kernel function is determined based on the following formula: Wherein d=xK(RP+t), δ is the set deviation threshold.

10. An image matching device, characterized in that: It includes image aggregation module, feature matching module, ring constraint module and motion constraint module, among which: The image aggregation module is used to obtain the position data corresponding to the image to be matched and the image to be matched, and perform adjacent frame aggregation on the image to be matched according to the position data to obtain multiple adjacent frame image groups; The feature matching module is used to perform feature point matching on the image to be matched in each group of adjacent frame images to obtain a plurality of matching pairs, where each feature point of the image to be matched corresponds to a set number of matching pairs; The ring constraint module is configured to screen, based on a set ring constraint condition, matching pairs that satisfy the ring constraint condition in each adjacent frame image group, wherein, among the matching pairs that satisfy the ring constraint condition, starting from one matching pair, the matching pairs of other to-be-matched images in the same adjacent frame image group are sequentially passed through to form a closed loop of matching pairs; The motion constraint module is used to triangulate the filtered matching pairs to obtain spatial points, filter the spatial points based on the set motion constraint conditions, and determine the image matching results based on the filtered spatial points.

11. An image matching device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image matching method according to any one of claims 1 to 9.

12. A storage medium containing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the image matching method according to any one of claims 1 to 9.

Images