Stereo image feature matching method and device based on classified global energy constraint

Abstract

The embodiment of the present application provides a kind of based on classification global energy constraint stereoscopic image feature matching method and device, the stereoscopic image feature matching method based on classification global energy constraint includes: the feature point on reference image is divided into reference target feature point set and reference background feature point set, and the feature point on image to be matched is divided into target feature point set to be matched and background feature point set to be matched;The reference target feature point set and the target feature point set to be matched are matched, and the reference background feature point set and the background feature point set to be matched are matched, to obtain the feature matching result of stereoscopic image.Some embodiments of the present application divide the feature point on stereoscopic pair into target point set and background point set, and when feature point matching, target point set is only matched with target point set, can effectively reduce the search range of same name point, eliminate matching gross error, improve the efficiency, reliability and stability of matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of feature matching, and in particular, the embodiments of the present application relate to a stereo image feature matching method and device based on classification global energy constraint. BACKGROUND

[0002] By acquiring multi-view stereo images of a target object, an important method for three-dimensional reconstruction of the target object is performed. In order to accurately restore the three-dimensional structure of the target object, high-precision feature matching of the multi-view stereo images is a key technical link. Obtaining a sufficient number of feature point pairs with uniform distribution and reliable precision is a prerequisite for solving the instantaneous accurate position and attitude of the camera imaging of the regional network as a whole. Image feature matching can lay a solid foundation for image three-dimensional reconstruction and can be widely used in three-dimensional restoration of large-scale topography, urban building groups, various complex structural objects indoors and outdoors, and micro targets.

[0003] Traditional stereo image feature matching algorithms only consider the feature similarity between stereo image homonymic points, and feature points satisfying local image maximum consistency constraints are regarded as homonymic points. For example, the most classic and commonly used SIFT algorithm (Scale Invariant Feature Transform), SURF algorithm (Speeded Up Robust Feature), and ZNCC algorithm (Zero-mean Normalized Cross-Correlation) and the like. However, due to the ambiguity of matching (repeated texture, weak texture, same spectrum and different objects, and noise on the surface of a complex target object), the feature most similar to the homonymic point is not necessarily the correct matching point, and there are deficiencies in matching accuracy, efficiency, and reliability. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an image feature matching method and device. Through the method and device of the embodiments of the present application, the target point set can be matched only with the target point set, and the background point set can be matched only with the background point set, thereby effectively reducing the search range of homonymic points.

[0005] In a first aspect, some embodiments of the present application provide an image feature matching method, which comprises: dividing feature points on a reference image into a reference target feature point set and a reference background feature point set, and dividing feature points on a to-be-matched image into a to-be-matched target feature point set and a to-be-matched background feature point set; performing feature point matching on the reference target feature point set and the to-be-matched target feature point set, and performing feature point matching on the reference background feature point set and the to-be-matched background feature point set, to obtain a feature matching result of stereo images.

[0006] Some embodiments of the present application divide feature points on a stereo image pair into two point sets, namely a target point set and a background point set. In feature point matching, the target point set is matched only with the target point set, and the background point set is matched only with the background point set, thereby effectively reducing the search range of corresponding points.

[0007] In some embodiments, the dividing of feature points on the reference image into a reference target feature point set and a reference background feature point set comprises: extracting feature points on the reference image, and performing semantic segmentation on the reference image to obtain a semantic segmentation result; and dividing the feature points on the reference image into the reference target feature point set and the reference background feature point set according to the semantic segmentation result.

[0008] In some embodiments, the dividing of feature points on the reference image into a reference target feature point set and a reference background feature point set comprises: extracting feature points on the reference image, and performing semantic segmentation on the reference image to obtain a semantic segmentation result; and dividing the feature points on the reference image into the reference target feature point set and the reference background feature point set according to the semantic segmentation result.

[0009] In some embodiments, the feature point matching of the reference target feature point set and the target feature point set and the feature point matching of the reference background feature point set and the target feature point set further comprises: constructing an undirected graph of the reference target feature point set and the reference background feature point set on the reference image, respectively, wherein the undirected graph is used to represent the positions of the feature points and the connection relationships between the feature points; constructing a global energy function according to the data in the undirected graph, and solving the optimal solution of the energy function to obtain the feature matching result, respectively.

[0010] In some embodiments of the present application, after the feature point matching of the reference target feature point set and the target feature point set and the feature point matching of the reference background feature point set and the target feature point set, the method further comprises: adjusting an initial feature point matching result according to the principle that the topological relationship between the feature points on the reference image and the topological relationship between the target feature points on the target image satisfy consistency, to obtain the feature matching result, wherein the initial feature point matching result includes a first initial feature point matching result obtained by the feature point matching of the reference target feature point set and the target feature point set, and a second initial feature point matching result obtained by the feature point matching of the reference background feature point set and the target feature point set.

[0011] Some embodiments of the present application perform matching operation based on the topological relationship between feature matching points, which can effectively solve the problem of false matching of traditional local feature matching algorithm.

[0012] In some embodiments of the present application, the adjustment of the initial feature point matching result according to the principle that the topological relationship between the feature points on the reference image and the topological relationship between the to-be-matched feature points on the to-be-matched image is consistent includes: based on the constructed global energy function, the optimal solution is solved according to the principle to complete the adjustment operation of the matching result of the initial feature points, and the feature matching result is obtained, wherein the global energy function includes: a target data item, a background data item and a smoothing item, the target data item is used to calculate the target point matching value when the reference target feature point set and the to-be-matched target feature point set perform feature point matching operation, the background data item is used to calculate the background point matching value when the reference background feature point set and the to-be-matched background feature point set perform feature point matching, and the smoothing item introduces the disparity topological structure as a constraint condition.

[0013] Some embodiments of the present application obtain the matching result by constructing the global energy function in the solving process, introduce the disparity topological result as the smoothing item constraint condition, and the global energy function of some embodiments of the present application includes the data item corresponding to the local feature matching algorithm, the smoothing item introduces the constraint of the disparity topological structure, and further improves the precision and robustness of feature matching.

[0014] In some embodiments, the method further includes: obtaining an undirected graph of the reference image and the to-be-matched image, and obtaining the topological relationship based on the undirected graph, wherein the disparity topological result is obtained through the topological relationship.

[0015] In some embodiments of the present application, the formula of the global energy function is as follows:

[0016]

[0017] wherein, represents the label set assigned to each feature point, represents a feature point in the feature point set on the reference image, represents the label and satisfies , represents the label of the feature point , represents the edge with end points p and q, T represents a Boolean logic operator, and P represents a penalty coefficient, represents the horizontal disparity, represents the vertical disparity, represents the distance smoothing factor, represents the matching cost value corresponding to the feature point p in the case of label being

[0018] Some embodiments of the present application provide an algorithm for quantifying a global energy function, which improves the accuracy of data processing.

[0019] In some embodiments of the present application, the adjustment operation of the matching result for the initial feature points is completed according to the optimal solution of the constructed global energy function, including: calculating the cost of the data item included in the global energy function by using local feature matching, and calculating the penalty of the smoothing item included in the global energy function according to the homonymy point topology structure; obtaining the feature matching result by solving the optimal solution of the global energy function through a greedy algorithm.

[0020] Some embodiments of the present application provide a process for solving the optimal solution of the global energy function.

[0021] In some embodiments of the present application, the matching cost value is determined by the following strategy: using the pixels in the neighborhood window of the two to-be-matched pixels on the reference image and the to-be-matched image, and calculating the similarity degree between the two to-be-matched pixels by a zero-mean normalized similarity measure formula.

[0022] Some embodiments of the present application provide a method for solving the cost value corresponding to the data item.

[0023] In the second aspect, some embodiments of the present application provide a three-dimensional reconstruction method of an object, which performs three-dimensional reconstruction of the object according to the feature matching result obtained by the method of any one of the embodiments of the first aspect.

[0024] In the third aspect, some embodiments of the present application provide a stereo image feature matching device based on a classification global energy constraint, which includes: a target feature point and background feature point division module configured to divide the feature points on the reference image into a reference target feature point set and a reference background feature point set, and divide the feature points on the feature matching image into a to-be-matched target feature point set and a to-be-matched background feature point set; a matching module configured to perform feature point matching on the reference target feature point set and the to-be-matched target feature point set to obtain a feature matching result, and perform feature point matching on the reference background feature point set and the to-be-matched background feature point set to obtain a background matching result.

[0025] In the fourth aspect, some embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the program can implement the method of any one of the embodiments of the first aspect when executed by a processor.

[0026] ​In a fifth aspect, some embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the method according to any of the embodiments of the first aspect when running the program. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 The application scenario provided for the embodiments of the present application is shown in the figure;

[0029] Figure 2 One of the flowcharts of the stereo image feature matching method based on the classification global energy constraint provided for the embodiments of the present application is shown in the figure;

[0030] Figure 3 The second flowchart of the stereo image feature matching method based on the classification global energy constraint provided for the embodiments of the present application is shown in the figure;

[0031] Figure 4 The composition block diagram of the stereo image feature matching device based on the classification global energy constraint provided for the embodiments of the present application is shown in the figure;

[0032] Figure 5 The composition schematic diagram of the electronic device provided for the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0033] The technical solutions of the embodiments of the present application will be described in the following with reference to the drawings in the embodiments of the present application.

[0034] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing description, and cannot be understood as indicating or implying relative importance.

[0035] To at least solve the problem of false matching of traditional local feature matching algorithms, some embodiments of the present application propose a stereo image feature matching method based on classification global energy constraint from the topological relationship between feature matching points. For example, the method uses the recognition result of semantic segmentation to divide the feature points of an object on a stereo image pair into two point sets: a target point set and a background point set. The target point set is matched only with the target point set, and the background point set is matched only with the background point set, so as to effectively reduce the search range of corresponding points. On this basis, some embodiments of the present application also consider the topological relationship between feature points, and consider that the topological relationship between feature points is consistent with the topological relationship between to-be-matched points, so as to convert the feature matching problem into a global energy function optimal solution calculation problem. The global optimal feature matching result is obtained by solving the energy function by using a greedy algorithm.

[0036] Please refer to Figure 1 , Figure 1 The technical scheme of some embodiments of the present application is a schematic diagram of an application scenario, which includes a terminal 10 and a server 20. The terminal 10 includes a first camera 11 and a second camera 12. In some embodiments of the present application, the first camera 11 is a component for shooting a reference image, and the second camera 12 is a component for shooting a to-be-matched image. That is, the first camera 11 is used to shoot a reference image 21, and the second camera 12 is used to shoot a to-be-matched image 22. In some other embodiments of the present application, the first camera 11 and the second camera 12 are both RGB cameras. The server 20 completes stereo image matching based on the reference image and the to-be-matched image from the terminal 10 to obtain a feature matching result, and feeds back the feature matching result to the terminal 10.

[0037] It should be noted that in some embodiments of the present application, the functions of the server 20 can also be performed by the terminal 10, that is, the terminal 10 performs data processing on the reference image and the to-be-matched image obtained by shooting a certain scene to obtain a feature matching result.

[0038] For the sake of convenience, the camera used to shoot the scene graph 21 will be referred to as the main camera in the following.

[0039] The following will be described in combination with Figure 2 the stereo image feature matching method based on classification global energy constraint executed by Figure 1 the server 20 or the terminal 10.

[0040] As Figure 2As shown, some embodiments of the present application provide a stereo image feature matching method based on a classification global energy constraint, which comprises: S101, dividing feature points on a reference image into a reference target feature point set and a reference background feature point set, and dividing feature points on a to-be-matched image into a to-be-matched target feature point set and a to-be-matched background feature point set; and S102, performing feature point matching on the reference target feature point set and the to-be-matched target feature point set, and performing feature point matching on the reference background feature point set and the to-be-matched background feature point set to obtain a feature matching result of the stereo images.

[0041] That is, some embodiments of the present application divide feature points on a stereo image pair into two point sets, i.e., a target point set and a background point set, and match the target point set only with the target point set and match the background point set only with the background point set during feature point matching, thereby effectively reducing the search range of corresponding points.

[0042] The above process is described below.

[0043] In some embodiments of the present application, the process of dividing feature points on a reference image into a reference target feature point set and a reference background feature point set involved in S101 exemplarily comprises: extracting feature points on the reference image, and performing semantic segmentation on the reference image to obtain a semantic segmentation result; and dividing the feature points on the reference image into the reference target feature point set and the reference background feature point set according to the semantic segmentation result.

[0044] In some embodiments of the present application, the process of dividing feature points on a to-be-matched image into a to-be-matched target feature point set and a to-be-matched background feature point set involved in S102 exemplarily comprises: extracting feature points on the to-be-matched image, and performing semantic segmentation on the to-be-matched image to obtain a semantic segmentation result; and dividing the feature points on the to-be-matched image into the to-be-matched target feature point set and the to-be-matched background feature point set according to the semantic segmentation result.

[0045] In some embodiments of the present application, S102 exemplarily comprises: constructing an undirected graph of the reference target feature point set and the reference background feature point set on the reference image, respectively, wherein the undirected graph is used to represent the positions of feature points and the connection relationships between feature points; constructing a global energy function according to the data in the undirected graph, and solving the optimal solution of the energy function to obtain the feature matching result.

[0046] It should be noted that, in order to further improve the accuracy of the matching result to improve the technical defect of low matching precision caused by the existing local matching method, in some embodiments of the present application, after S102, the method of image feature matching further comprises: adjusting the initial feature point matching result according to the principle that the topological relationship between the feature points on the reference image and the topological relationship between the to-be-matched feature points on the to-be-matched image satisfy consistency, to obtain the feature matching result, wherein the initial feature point matching result includes a first initial feature point matching result obtained by performing feature point matching on the reference target feature point set and the to-be-matched target feature point set, and a second initial feature point matching result obtained by performing feature point matching on the reference background feature point set and the to-be-matched background feature point set. Some embodiments of the present application perform matching operation from the topological relationship between the feature matching points, which can effectively solve the problem of false matching of the traditional local feature matching algorithm.

[0047] For example, in some embodiments of the present application, the adjusting of the initial feature point matching result according to the principle that the topological relationship between the feature points on the reference image and the topological relationship between the to-be-matched feature points on the to-be-matched image satisfy consistency comprises: based on the constructed global energy function, the adjustment operation of the matching result of the initial feature point is completed according to the principle to obtain the feature matching result, wherein the global energy function includes: a target data item, a background data item and a smoothing item, the target data item is used to calculate the target point matching value when the feature point matching operation is performed on the reference target feature point set and the to-be-matched target feature point set, the background data item is used to calculate the background point matching value when the feature point matching is performed on the reference background feature point set and the to-be-matched background feature point set, and the smoothing item introduces the disparity topological structure as a constraint condition. Some embodiments of the present application obtain the matching result by constructing the global energy function in the solving process, introduce the disparity topological result as the smoothing item constraint condition, and the global energy function of some embodiments of the present application includes the data item corresponding to the local feature matching algorithm, the smoothing item introduces the constraint of the disparity topological structure, which further improves the precision and robustness of the feature matching.

[0048] The above process is exemplarily described below.

[0049] In order to obtain the target feature point set and the background feature point set, in some embodiments of the present application, S101 comprises: acquiring a stereo image pair, wherein the stereo image pair includes the reference image and the feature matching image; extracting feature points on the reference image and the to-be-matched image respectively; and combining the feature points on the to-be-matched image to form a label space , wherein, The null term indicates that a feature point on the reference image has no corresponding point on the image to be matched. The feature points on the image to be matched are represented; the contours of the targets are reconstructed on the stereo image pair (i.e., undirected graphs of the reference target feature point set and the reference background feature point set are constructed on the reference image); the feature point set V on the reference image is divided into the reference target feature point set VT and the reference background feature point set VB according to the contours of the targets, and the label space is... The feature points are divided into the target feature point set LT and the background feature point set LB. Some embodiments of this application transform the feature point matching problem into a graph computation problem based on the consistency of the topological relationships of corresponding features in stereo images, thus constructing a feature label space.

[0050] In order to utilize the topological relationships between feature points, in some embodiments, the image feature matching method further includes: acquiring an undirected graph of the reference image and the image to be matched, and obtaining the topological relationship based on the undirected graph, wherein the disparity topological result is obtained through the topological relationship.

[0051] For example, in some embodiments of this application, the formula for the global energy function is as follows:

[0052]

[0053] in, This represents the set of labels assigned to each feature point. Represents the set of feature points on the reference image. One of the feature points, Represent the label and satisfy , Representing feature points label, Let T represent an edge with endpoints p and q, T denote a Boolean operator, and P denote a penalty term. Indicates horizontal parallax. Indicates vertical parallax. Represents the distance smoothing factor. This represents the matching cost of feature point p with label l. It should be noted that P is a penalty coefficient, applied to the global energy function calculation when there are inconsistencies in the topological relationships between points, ensuring that the optimal solution of the global energy function best meets the actual needs of stereo image matching. This represents the horizontal parallax between two points, i.e., the distance between their horizontal coordinates. The vertical parallax between two points is represented by the distance between their vertical coordinates. , . The Euclidean distance between the two points p and q is multiplied by a smoothing factor (i.e., distance smoothing factor) to control the consistency and smoothness of the topological relationship between the points.

[0054] The process of solving the global energy function is described below.

[0055] For example, in some embodiments of the present application, the adjustment operation of the matching result for the initial feature points is completed according to the optimal solution of the constructed global energy function, including: calculating the cost of the data item included in the global energy function by using local feature matching, and calculating the penalty of the smoothing item included in the global energy function according to the topological structure of the same name points; obtaining the feature matching result by solving the optimal solution of the global energy function through a greedy algorithm.

[0056] For example, in some embodiments of the present application, the matching cost value is determined by the following strategy: using the pixels in the neighborhood window of the two to-be-matched pixels on the reference image and the to-be-matched image, and calculating the similarity between the two to-be-matched pixels by a zero-mean normalized similarity measure formula.

[0057] The process of feature point matching according to the left image (corresponding to the reference image) and the right image (corresponding to the to-be-matched image) is described below. Figure 3

[0058] As shown in the figure, some embodiments of the present application provide a method of "global energy function image feature matching method based on contour information constraint", which includes: Figure 3

[0059] S202, obtaining a stereo image pair. That is, obtaining a stereo image pair taken of the same target, defining the left image as the reference image and the right image as the to-be-feature-matched image.

[0060] S203, constructing an undirected graph, that is, constructing an undirected graph G(V, E) of the target feature point set and the background feature point set on the reference image, which includes the positions V of the feature points and the connection relationship E between the feature points, that is, all the edges of the triangular mesh formed between the points.

[0061] ​​To realize the global feature matching of multi-view images, it is assumed that the topological structure of the feature points on the left image is similar to the topological structure of the corresponding feature points on the right image, that is, two features on the image that are close in distance are also close in distance of the corresponding corresponding feature points, so as to convert the matching problem into a graph calculation problem. A undirected graph is defined, wherein V represents a set of feature points on the reference image. A triangular mesh is constructed for the set of feature points, and the edges of all triangles together form an edge set. The edge set represents the adjacency relationship of each feature point.

[0062] It should be noted that the step S203 can be executed before S212, and the embodiments of the present application do not limit the specific steps of constructing the undirected graph.

[0063] S204, extracting feature points on the left image.

[0064] S207, extracting feature points on the right image.

[0065] S205, constructing a set of feature points on the left image.

[0066] S204 and S207 are executed to extract feature points on the reference image and the image to be matched, respectively.

[0067] It should be noted that in some embodiments of the present application, feature point extraction on the reference image and the image to be matched can be performed using various local feature point extraction methods, such as Harris, FAST corner detection operator, SIFT, SURF, BRIEF, ORB feature detection operator, etc.

[0068] S208, constructing a label space of feature points on the right image.

[0069] The feature points on the matching image together form a label space, wherein, represents empty, i.e. the feature points on the left image do not have corresponding feature points on the right image, represents the feature points on the right image, and most of the feature points on the reference image can find corresponding feature points in the space .

[0070] S206, reconstructing the target contour on the left image.

[0071] S209, reconstructing the target contour on the right image.

[0072] It can be understood that S206 and S209 are executed to reconstruct the contour of the target on the stereo image pair, respectively.

[0073] S207, constructing a set of target feature points and a set of background feature points on the left image.

[0074] S210, construct the target feature point set and background feature point set in the label space of the right image.

[0075] In other words, S207 divides the feature point set V on the reference image into two subsets: the reference target feature point set VT and the reference background feature point set VB. The corresponding points of all feature points in the reference target feature point set VT must be located in the contour of the image to be matched. S210 divides the label space into two subsets as well: the target feature point set LT to be matched and the background feature point set LB to be matched.

[0076] S211, define the global feature matching energy function (or simply the global energy function).

[0077] Some embodiments of this application define the global feature matching algorithm as a labelling problem, which finds a label for each feature point on the reference image such that the defined global energy function is minimized.

[0078] In some embodiments of this application, the global feature matching algorithm is defined as a labeling problem, which finds a label for each feature point on the reference image such that the defined global energy function is minimized. In the matching of some embodiments of this application, feature points in set VT are searched for corresponding points only in set LT, and feature points in set VB are searched for corresponding points only in set LB. This greatly reduces the search range of corresponding points and improves the reliability and accuracy of feature matching.

[0079] Energy functions for global feature matching defined in some embodiments of this application for:

[0080]

[0081] In the above formula, the first term is the target set data term, the second term is the background set data term, and the third term is the smoothing term.

[0082] in, This represents the set of labels assigned to each feature point; Represents the set of feature points on the reference image One of the feature points; Represent the label, satisfying . Representing feature points The label. This represents an edge with endpoints p and q; T represents a Boolean operator; P represents a penalty term; Indicates horizontal parallax; Indicates vertical parallax; Indicates the distance smoothing factor; denotes the matching cost of the feature point p in the case of label l.

[0083] S212, the cost of the data item is calculated by local feature matching, i.e. the cost of the data item in the target set and the background set in the previous step is calculated by local feature matching method.

[0084] In some embodiments of the present application, The negative number of zero-mean normalized product correlation is used, i.e. the similarity between two matching pixels is calculated by zero-mean normalized similarity measurement formula using the pixels in the neighborhood window of the two matching pixels on the reference image and the image to be matched. denotes:

[0085]

[0086] wherein, In particular, when the feature point on the reference image has no corresponding homonym, and thus the matching cost is equal to a large constant.

[0087] S213, the smoothness penalty is calculated according to the homonym topological result.

[0088] For the smoothness term included in the feature matching energy function, it is assumed that the topological structure between the reference image and the image to be matched should be similar, and the case where the topological structure is not similar is penalized. In some embodiments of the present application, the topological structure is calculated according to the horizontal disparity and the vertical disparity between homonyms, and if the horizontal disparity and the vertical disparity of two adjacent feature points on the reference image differ too much, it is considered that the two adjacent feature points do not satisfy the smoothness constraint and are penalized.

[0089] S214, the global energy function (i.e. the global feature matching energy function) is solved by a greedy algorithm.

[0090] S215, the optimal matching result on the stereo image is generated.

[0091] It can be understood that the optimal solution of the global energy function E is the global feature matching result. Considering that solving E is an NP-hard problem, in the present embodiment, a greedy algorithm is used to quickly calculate the approximate optimal solution of E, thereby obtaining the global feature matching result. In solving the global energy function E, the local optimal solution of each sub-problem is solved, and then the local optimal solutions are combined into an approximate local optimal solution of the global energy function E. That is, the optimal feature matching result on the stereo image is obtained.

[0092] At this point, the feature matching result can be obtained.

[0093] From the above technical solutions, the "global energy function image feature matching method based on contour information constraint" of some embodiments of the present application has the following beneficial effects: 1) By introducing the contour constraints of the target and the background on the stereo image, the search range of the homonym is greatly reduced by using the contour information, and the efficiency and reliability of the image feature matching are improved; 2) By using the topological invariance and image consistency constraints between the feature matching points, the feature matching problem is converted into the calculation problem of the optimal solution of the global energy function. Compared with the traditional stereo image feature matching algorithm, the theory is more rigorous. Among them, the data item corresponds to the local feature matching algorithm, the smooth item introduces the constraint of the disparity topological structure, and the accuracy and robustness of the feature matching are further improved. 3) The determination of the matching points of some embodiments of the present application simultaneously considers the mutual correlation results between all images, which can effectively reduce the false matching. The final result of the matching is the spatial point on the corresponding imaging ray of the reference point, which can provide high-quality feature matching results for subsequent image area network adjustment and three-dimensional reconstruction.

[0094] That is, according to the consistency of the topological relationship of the homonym features of the stereo image, some embodiments of the present application convert the matching problem of the feature points into the calculation problem of the Graph graph, construct the feature Label space; according to the segmentation results of the target and the background area in the stereo image, construct the feature target set and the feature background set of the Label space on the feature points of the reference image and the feature points of the image to be matched, and perform classified matching search; in the stereo image feature matching, the topological invariance and the image consistency constraint are introduced, a global energy function including a target / background data item and a smooth item is constructed, and a method for solving the local optimal solution of the function is constructed.

[0095] Some embodiments of the present application provide a three-dimensional reconstruction method of an object, which obtains the feature matching result according to the method as Figure 2 Or Figure 3 The feature matching result obtained by the method is used for three-dimensional reconstruction of the object.

[0096] Please refer to Figure 4 , Figure 4 The device for image feature matching through the embodiments of the present application is shown, and it should be understood that the device corresponds to the above-mentioned Figure 2 method embodiments, and can perform each step involved in the above-mentioned method embodiments. The specific functions of the device can be referred to the description in the above, and the detailed description is appropriately omitted here to avoid repetition. The device includes at least one software function module which can be stored in the form of software or firmware in the memory or solidified in the operating system of the device. The device for image feature matching includes a target feature point and background feature point division module 110 and a matching module 120.

[0097] The target feature point and background feature point division module 110 is configured to divide the feature points on the reference image into a reference target feature point set and a reference background feature point set, and divide the feature points on the feature matching image into a to-be-matched target feature point set and a to-be-matched background feature point set.

[0098] The matching module 120 is configured to perform feature point matching between the reference target feature point set and the to-be-matched target feature point set, and perform feature point matching between the reference background feature point set and the to-be-matched background feature point set, to obtain a feature matching result of the stereoscopic image.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the apparatus described above can refer to the corresponding process in the foregoing method, which will not be described in detail here.

[0100] Some embodiments of the present application provide a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, can implement the method as Figure 2 or Figure 3 described.

[0101] As Figure 5 shown, some embodiments of the present application provide an electronic device 500, which includes a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520, wherein the processor 520 reads the program from the memory 510 through a bus 530 and executes the program, and can implement the method as Figure 2 or Figure 3 any embodiment described.

[0102] The processor 520 can process digital signals and can include various computing structures. For example, a complex instruction set computer structure, a reduced instruction set computer structure, or a structure implementing a combination of multiple instruction sets. In some examples, the processor 520 can be a microprocessor.

[0103] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution process of the instructions. These instructions and / or data can include code for implementing some or all functions of one or more modules described in embodiments of the present application. The processor 520 of the present disclosure can be used to execute instructions in the memory 510 to implement the method as shown in Figure 2 . The memory 510 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memories well known to those skilled in the art.

[0104] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can also be implemented by other means. The apparatus embodiments described above are only illustrative, for example, the flowcharts and block diagrams in the drawings show the possible implementation architecture, function and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0105] In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0106] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0107] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0108] The above merely provides an example of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and thus, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0109] It should be noted that the relational terms herein such as first and second and the like are used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A stereoscopic image feature matching method based on classification global energy constraint, characterized in that, The stereoscopic image feature matching method comprises: dividing feature points on a reference image into a reference target feature point set and a reference background feature point set, and dividing feature points on a to-be-matched image into a to-be-matched target feature point set and a to-be-matched background feature point set; performing feature point matching on the reference target feature point set and the to-be-matched target feature point set, and performing feature point matching on the reference background feature point set and the to-be-matched background feature point set to obtain a feature matching result of stereoscopic images; obtaining the feature matching result comprises adjusting an initial feature point matching result according to a principle that a topological relationship between feature points on the reference image and a topological relationship between to-be-matched feature points on the to-be-matched image are consistent, to obtain the feature matching result, wherein the initial feature point matching result comprises a first initial feature point matching result obtained by performing feature point matching on the reference target feature point set and the to-be-matched target feature point set, and a second initial feature point matching result obtained by performing feature point matching on the reference background feature point set and the to-be-matched background feature point set.

2. The stereoscopic image feature matching method of claim 1, wherein, The dividing of feature points on the reference image into the reference target feature point set and the reference background feature point set comprises: extracting feature points on the reference image, and performing semantic segmentation on the reference image to obtain a semantic segmentation result; dividing the feature points on the reference image into the reference target feature point set and the reference background feature point set according to the semantic segmentation result.

3. The stereoscopic image feature matching method of claim 1, wherein, The dividing of feature points on the to-be-matched image into the to-be-matched target feature point set and the to-be-matched background feature point set comprises: extracting feature points on the to-be-matched image, and performing semantic segmentation on the to-be-matched image to obtain a semantic segmentation result; dividing the feature points on the to-be-matched image into the to-be-matched target feature point set and the to-be-matched background feature point set according to the semantic segmentation result.

4. The stereoscopic image feature matching method of claim 1, wherein, The performing of feature point matching on the reference target feature point set and the to-be-matched target feature point set, and the performing of feature point matching on the reference background feature point set and the to-be-matched background feature point set comprises: constructing an undirected graph of the reference target feature point set and the reference background feature point set on the reference image respectively, wherein the undirected graph is used to represent positions of feature points and connection relationships between feature points; constructing a global energy function according to data in the undirected graph, and solving an optimal solution of the energy function to obtain the feature matching result respectively.

5. The stereoscopic image feature matching method of claim 4, wherein, The adjusting the initial feature point matching result according to the principle that the topological relations between the feature points on the reference image and the topological relations between the to-be-matched feature points on the to-be-matched image are consistent comprises: based on the global energy function, solving an optimal solution according to the principle to complete the adjusting operation on the matching result of the initial feature points, and obtaining the feature matching result, wherein the global energy function comprises: a target data item, a background data item and a smoothing item, the target data item is used to calculate a target point matching value when the feature point matching operation is performed on the reference target feature point set and the to-be-matched target feature point set, the background data item is used to calculate a background point matching value when the feature point matching operation is performed on the reference background feature point set and the to-be-matched background feature point set, and the smoothing item introduces a disparity topological structure as a constraint condition.

6. The stereoscopic image feature matching method of claim 5, wherein, The method further comprises: obtaining an undirected graph of the reference image and an undirected graph of the to-be-matched image, and obtaining the topological relations based on the undirected graph of the reference image and the undirected graph of the to-be-matched image, wherein the disparity topological result is obtained through the topological relations.

7. The stereoscopic image feature matching method of claim 5, wherein, The formula of the global energy function is as follows: wherein, represents a set of labels assigned to each feature point, represents a set of feature points on the reference image one feature point, represents a label and satisfies , represents a label of a feature point , represents an edge with endpoints p and q, T represents logical true, P represents a penalty coefficient, represents horizontal disparity, represents vertical disparity, represents a distance smoothing factor, represents a matching cost value of a feature point p corresponding to a label , T represents a set of reference target feature points, V B represents a set of reference background feature points, L T represents a set of target feature points to be matched, L B represents a set of background feature points to be matched.

8. The stereoscopic image feature matching method of claim 5, wherein, The adjusting the initial feature point matching result according to the principle that the topological relations between the feature points on the reference image and the topological relations between the to-be-matched feature points on the to-be-matched image are consistent comprises: adopting local feature matching to calculate the cost of the data item included in the global energy function, and calculating the penalty of the smoothing item included in the global energy function according to the topological structure of the same-named points; solving the optimal solution of the global energy function through a greedy algorithm to obtain the feature matching result.

9. The stereoscopic image feature matching method of claim 5, wherein, The matching value is determined through the following strategy: using the pixels in the neighborhood window of the two to-be-matched pixels on the reference image and the to-be-matched image, and calculating the similarity degree between the two to-be-matched pixels through a zero-mean normalized similarity measurement formula.

10. A method of three-dimensional reconstruction of an object, characterized by, An object is reconstructed in three dimensions according to the feature matching result obtained through the method of any one of claims 1-9.

11. A stereo image feature matching device based on classification global energy constraints, characterized in that, The stereoscopic image feature matching device comprises: a target feature point and background feature point dividing module configured to divide the feature points on the reference image into a reference target feature point set and a reference background feature point set, and divide the feature points on the to-be-matched image into a to-be-matched target feature point set and a to-be-matched background feature point set; The matching module is configured to perform feature point matching on the reference target feature point set and the to-be-matched target feature point set, and perform feature point matching on the reference background feature point set and the to-be-matched background feature point set to obtain a feature matching result of the stereoscopic image; the feature matching result is obtained by adjusting an initial feature point matching result according to a principle that a topological relationship between feature points on the reference image and a topological relationship between to-be-matched feature points on the to-be-matched image are consistent, to obtain the feature matching result, wherein the initial feature point matching result includes a first initial feature point matching result obtained by performing feature point matching on the reference target feature point set and the to-be-matched target feature point set, and a second initial feature point matching result obtained by performing feature point matching on the reference background feature point set and the to-be-matched background feature point set.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-9 or the method of claim 10.

13. An information processing apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the program to implement the method of any one of claims 1-9 or the method of claim 10.

Images