A model fitting method based on neighborhood consistency and its application

Abstract

The present invention relates to a model fitting method based on neighborhood consistency and its application, the method comprising the following steps: S1, for input data, obtaining a neighborhood set of each data point through a neighborhood consistency maintaining algorithm; S2, based on the obtained neighborhood set, clustering the data through a clustering algorithm based on neighborhood consistency, merging data with consistency information, and obtaining a clustering result; in the clustering process, evaluating data similarity based on a similarity measurement function of neighborhood relations; S3, distinguishing inliers and outliers belonging to different model instances based on the clustering results, and outputting a data segmentation result. The method is conducive to improving the model fitting performance, and thus improving the data segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a model fitting method based on neighborhood consistency and application thereof. Background Art

[0002] Geometric model fitting methods refer to robustly estimating the parameters and number of model instances in noisy data and segmenting the data into inliers and outliers. Model fitting methods are an important basic work in computer vision and are widely used in many practical application fields, such as image registration, 3D reconstruction, motion segmentation, visual tracking, etc. The performance of most geometric model fitting methods is closely related to model selection. Model fitting methods based on consistency analysis and model fitting methods based on preference analysis are two existing types of model fitting methods that are closely related to model selection.

[0003] First, for model fitting methods based on consistency analysis, model selection is performed by selecting representative model hypotheses as estimated model instances in the parameter space according to the consistency maximization criterion. For example, RansaCov estimates model instances by identifying model hypotheses containing the maximum number of inliers. MSHF uses different weighting functions to select representative model hypotheses for multi-structure model fitting. Hough and its variants can also be regarded as model fitting methods based on consistency analysis, in which model instances are detected as the maximum consistent set in the quantized model hypothesis parameter space. In addition, based on consistency maximization, some methods solve the model fitting problem by using an energy minimization process, such as PEARL. In these model fitting methods, various geometric constraints are incorporated into energy optimization and used as regularization terms. A commonly used geometric constraint is spatial consistency, that is, the inliers of the same structure are often close in space. Geometric prior information is incorporated into these energy minimization-based methods to improve fitting performance. However, such methods usually involve complex optimization strategies and are sensitive to outliers.

[0004] Secondly, for model fitting methods based on preference analysis, the similarities between all data pairs are usually calculated by using preference information, and then the data are clustered into different categories, each corresponding to a model instance. For example, RPA classifies the input data into inliers and outliers of different model instances by using robust preference analysis and low-rank approximation. J-linkage and T-linkage define different data representation functions based on preference information to obtain clustering of inliers. HOMF obtains a dense graph and applies spectral clustering by projecting the relationship between data pairs based on preference information. These model fitting methods can effectively estimate multiple model instances in the data by analyzing the global relationship between all possible data pairs, but they are often affected by a large number of outliers on the fitting results. In order to alleviate the sensitivity of a large number of outliers, some model fitting methods (such as CLSA) first remove a large number of outliers and then cluster the remaining data to estimate multiple model instances, reducing the sensitivity of these methods to outliers in multi-structure model fitting. However, these methods mainly use preference information that is easily disturbed by outliers to describe the global relationship of the input data. In the process of data clustering, these methods often ignore the potential rich information between neighboring data points. Therefore, the global relationships used in such methods are sensitive to outliers and pseudo-outliers during the data clustering stage, which can lead to a serious degradation in model fitting performance.

[0005] The above model fitting methods have achieved good results on general image data. However, as the number of outliers or structures in the data increases, estimating multiple model instances becomes challenging because the data clustering stage is affected by a large number of outliers and pseudo-outliers. In addition, most previous methods only use preference information for data clustering, while the neighborhood consistency information of the data can be used to alleviate the sensitivity to outliers, which is rarely studied in model fitting. Summary of the invention

[0006] The purpose of the present invention is to provide a model fitting method based on neighborhood consistency and its application, which is conducive to improving the model fitting performance and further improving the data segmentation accuracy.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a model fitting method based on neighborhood consistency, comprising the following steps:

[0008] S1. For the input data, the neighborhood set of each data point is obtained through the neighborhood consistency preservation algorithm;

[0009] S2. Based on the obtained neighborhood set, cluster the data by using a clustering algorithm based on neighborhood consistency, merge the data with consistent information, and obtain the clustering result; in the clustering process, the similarity measurement function based on the neighborhood relationship is used to evaluate the data similarity;

[0010] S3. Distinguish the inliers and outliers belonging to different model instances based on the clustering results, and output the data segmentation results.

[0011] Furthermore, the step S1 includes the following steps:

[0012] S101, initializing a neighborhood set through motion pattern consistency;

[0013] S102. Optimize the neighborhood set through residual pattern consistency.

[0014] Furthermore, the implementation method of step S101 is:

[0015] For a given pair of images, suppose there are N input data with matching features where s i =(x i ,y i ) indicates the i-th feature matching, x i and i are two corresponding feature points in the image pair; for feature matching s i =(x i ,y i ), search for two feature points x based on the Euclidean distance i and i The two K nearest neighbors and If there is a data point s j ∈S, the corresponding two feature points x j and j Belong to and Then the data s j and i The movement amplitudes are close to each other, and the data s j and i are regarded as adjacent data with similar motion amplitude; for data point s i ∈S, define its neighborhood set as follows:

[0016]

[0017] in, and |·| represents the number of data points in the neighborhood set; then, the data s is calculated based on the cosine similarity measure i Its adjacent data The motion direction similarity value between them is as follows:

[0018]

[0019] in, Respectively represent data s i and j The motion vector of , <·, ·> and ||·|| represent the inner product and norm respectively; then, based on the motion direction similarity threshold μ i , the motion direction similarity is higher than the threshold μ i The data is kept in For the data point s in the input data i , whose consistent neighborhood set in motion Defined as:

[0020]

[0021] Among them, μ i Yes i and The average value of the similarity of the motion direction between all adjacent data; therefore, for the input data Based on the motion pattern consistency constraint, a series of neighborhood sets are obtained That is, the neighborhood set initialization is realized.

[0022] Furthermore, the implementation method of step S102 is:

[0023] For input data Generate M sets of model hypotheses by sampling from the input data To calculate the residual correlation, the sampling preference function expresses the preference of the data for the model hypothesis as follows:

[0024]

[0025] Among them, r(s i ,θ m ) is the data s calculated using Sampson distance i and the model assumes θ m The residual between m is the estimated inlier scale, δ is the normalization constant; the preference vector f (i) Represents data i and the model assumes a preference between the set Θ, Residual correlation w i,j Associate two data i and j , which is based on two corresponding preference vectors f (i) and f (j) , calculated using cosine similarity:

[0026]

[0027] Among them, w i，j∈[0, 1]; then, the obtained residual correlation is expressed as Where T is each data s i and its neighborhood data The number of all paired data between ; In order to use information theory to obtain the adaptive threshold, let represents the tth element of W and the largest element of W, thus obtaining The prior probability of:

[0028]

[0029] The corresponding entropy is calculated as follows:

[0030]

[0031] If the data corresponds to the correlation The amount of information that is higher than the estimated entropy value E is retained in the initial neighborhood set. Otherwise, the data is filtered out; then, the initial neighborhood set Update to the final neighborhood set as follows:

[0032]

[0033] Furthermore, in step S2, the highly correlated data are clustered into the same inlier cluster by a clustering algorithm based on neighborhood consistency, and the data that is not related to any neighborhood data is classified as outliers, which is implemented as follows:

[0034] Assume the input data is The initial label set is The label Linked Data i ∈S, and initialized to 0; hierarchical clustering is used for clustering, specifically:

[0035] First, it iteratively searches for two data with a neighborhood relationship in the input data; then, it samples the similarity measurement function based on the neighborhood relationship to calculate the data similarity and determine whether the two neighborhood data should be assigned to the same cluster; if the similarity measurement function value is equal to 1, it is considered that the two data are highly correlated, and the labels of the two data are assigned to the same cluster. and Update to the same label value; in this clustering process, highly correlated data are merged into one cluster through the neighborhood relationship of the data. On the contrary, isolated data that is not related to any neighboring data is not assigned to any cluster and is marked as an outlier; finally, iteratively select two neighboring data and calculate their similarity to obtain the clustering result, so as to classify the data points into different model structures or as outliers.

[0036] Furthermore, the neighborhood relationship-based similarity measurement function effectively calculates data similarity by combining multiple relationships between paired data in a neighborhood set; specifically:

[0037] Given a data s i and its corresponding neighborhood set s j The corresponding neighborhood set Similarity measurement function f(s) based on neighborhood relationship i ,s j ,s k , w) is defined as follows:

[0038]

[0039] Among them, 1 s ∈M represents a variable, if s∈M, its variable value is 1, otherwise it is 0; represents another variable, if If true, the variable value is 1, otherwise it is 0; is the parameter in the similarity measure function.

[0040] Furthermore, through the clustering process, the input data will be classified into different clusters and isolated data; the data cluster of the same model instance has a larger number of data points than the cluster mainly composed of outliers; the data in the data cluster with less than a threshold value of data points is regarded as an outlier, and the data cluster with not less than a threshold value of data points is regarded as an inlier cluster, each inlier cluster corresponds to an estimated model instance in the data; the threshold is set to a minimum subset size.

[0041] The present invention also provides an application of the above-mentioned model fitting method based on neighborhood consistency, which is applied to the task of building face segmentation, segmenting inliers and outliers from the three-dimensional point cloud data of the building, and then extracting the geometric features of the building to achieve face segmentation of the building.

[0042] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a model fitting method based on neighborhood consistency and its application, which generates a meaningful neighborhood set for each data point of the input data by using the residual information between the input data and the model assumption and the motion information between the data through the neighborhood consistency maintenance algorithm, thereby robustly processing the outliers and pseudo outliers in the data; at the same time, through the clustering algorithm based on neighborhood consistency, based on the neighborhood relationship of the data and using the hierarchical clustering method, to merge the data points with consistency information to cluster the data, the correct number of clusters can be automatically estimated, and the input data can be accurately classified into outliers and inliers belonging to different model instances; in the clustering process, the similarity metric function based on the neighborhood relationship is used to evaluate the data similarity by analyzing the data residual correlation in the local neighborhood, further reducing the influence of outliers on the model fitting results, and improving the model fitting performance. The method can be widely used in the face segmentation scene of the building, realizing the accurate segmentation of the building face, thereby accurately extracting the wall, roof and other information from the building, and improving the accuracy of the three-dimensional reconstruction of the building. The method can also be applied to motion segmentation. By fitting the model, the inner points that meet the motion model are found, thereby realizing accurate segmentation of the moving object. Therefore, it has a wide range of applications in the fields of video monitoring, motion analysis, etc. The method can also be applied to object feature extraction and object detection in industrial production, such as straight line detection of LCD screen corners to determine whether the verticality of the screen corners meets the requirements. Therefore, the present invention has strong practicality and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a method implementation of an embodiment of the present invention;

[0044] Figure 2 is an implementation example diagram of a clustering algorithm based on neighborhood consistency in an embodiment of the present invention;

[0045] Figure 3 is a fitting result diagram of the basic matrix estimation performed by the method in the AdelaideRMF data set in an embodiment of the present invention;

[0046] Figure 4 It is a fitting result diagram of homography estimation performed by this method in the AdelaideRMF dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] The present invention provides a model fitting method based on neighborhood consistency and its application, so as to solve the problem that the existing model fitting methods do not fully utilize the neighborhood space information and reduce the impact of a large number of outliers on the fitting performance. The method can automatically estimate the number of clusters corresponding to the model instance and accurately distinguish the inliers and outliers belonging to different model instances. On multiple test data, the method can use less computing time and obtain more accurate fitting results than several other competing model fitting methods.

[0051] The model fitting method based on neighborhood consistency provided in this embodiment includes the following steps:

[0052] S1. For the input data, the motion consistency pattern and residual consistency pattern in the local neighborhood are analyzed through the neighborhood consistency preservation algorithm to generate a meaningful neighborhood set for each data point and obtain a reliable neighborhood relationship for data clustering.

[0053] S2. Based on the obtained neighborhood set, the data is clustered by a clustering algorithm based on neighborhood consistency, and the data with consistent information is merged to obtain the clustering result. In the clustering process, the similarity metric function based on the neighborhood relationship is used to evaluate the data similarity, so that the proposed clustering algorithm has stronger robustness to outliers.

[0054] S3. Distinguish the inliers and outliers belonging to different model instances based on the clustering results, and output the data segmentation results.

[0055] This method can automatically estimate the model instance parameters and the corresponding number of model instances, and accurately distinguish the inliers belonging to different model instances from the outliers. Figure 1 The figure shows the implementation flow chart of the method for segmenting data according to the estimated homography matrix.

[0056] 1. Neighborhood consistency maintenance algorithm

[0057] The neighborhood set of data points is obtained through a neighborhood consistency preserving algorithm, which includes two steps: 1) initializing the neighborhood set through motion pattern consistency; 2) optimizing the neighborhood set through residual pattern consistency.

[0058] 1) Initialize the neighborhood set through motion pattern consistency

[0059] For the same object in a set of image pairs, feature matches in local regions usually have similar motion patterns even under large viewpoint changes, because their corresponding feature points in one image are usually located on the same object. Therefore, in local regions, the motion patterns of true matches (i.e., inliers) of the same model instance tend to be consistent, while the motion patterns of false matches (i.e., outliers) tend to be randomly distributed in the image. Based on this observation, this method proposes an effective motion consistency constraint for neighborhood set initialization for robust model fitting.

[0060] For a given pair of images, suppose there are N input data with matching features where s i =(x i ,y i ) represents the i-th feature match (i.e. data), x i and i are two corresponding feature points in the image pair. For feature matching s i =(x i ,y i ), search for two feature points x based on the Euclidean distance i and i The two K nearest neighbors and If there is a data point s j ∈S, the corresponding two feature points x j and j Belong to and Then the data s j and i The movement amplitudes of the data are close to each other. j and i are regarded as adjacent data with similar motion amplitude. i ∈S, define its neighborhood set as follows:

[0061]

[0062] in, and |·| represents the number of data points in the neighborhood set. Then, by further exploiting motion consistency, our goal is to obtain Specifically, the data s is calculated based on the cosine similarity measurement. i Its adjacent data The motion direction similarity value between them is as follows:

[0063]

[0064] in, Respectively represent data s i and j The motion vector of , <·, ·> and ||·|| represent the inner product and norm respectively. Only the data with high similarity in motion direction can be retained in In the example, based on the motion direction similarity threshold μ i , the motion direction similarity is higher than the threshold μ i The data is kept in In this method, the threshold μ is adaptively calculated from the similarity of the data in the neighborhood set. i , to select data with similar motion directions. Therefore, for the data point s in the input data i , whose consistent neighborhood set in motion Defined as:

[0065]

[0066] Among them, μ i Yes i and The average value of the similarity of motion directions between all adjacent data. Examples of Figure 1 Therefore, for the input data Based on the motion pattern consistency constraint, a series of neighborhood sets are obtained That is, the neighborhood set initialization is realized.

[0067] 2) Neighborhood set optimization through residual pattern consistency

[0068] Motion consistency constraints can effectively remove outliers from each initial neighborhood set. However, data from different model instances (also called pseudo outliers) may have consistent motion patterns in real applications. This means that the data in the initial neighborhood set may contain pseudo outliers, which has a negative impact on the construction of significant neighborhood sets. To address this issue, we propose a neighborhood set optimization method based on the residual pattern between the data and model assumptions to remove pseudo outliers in each neighborhood set.

[0069] Generally speaking, the residual distributions between two data points (i.e., two feature matching pairs) and a set of generated model hypotheses belonging to the same model instance tend to be highly correlated. In other words, they usually show consistent residual patterns. And the residual correlation can be calculated based on the cosine similarity measure. Therefore, for neighborhood set optimization, a threshold based on the calculated residual correlation can be used to obtain highly correlated data, thereby removing pseudo outliers in each initial neighborhood set. However, for different input data, it is difficult to manually set an appropriate threshold to optimize the initial neighborhood set. In this proposal, we use information theory to adaptively set the threshold to effectively remove pseudo outliers in the neighborhood set optimization process. Specifically, we first generate a set of model hypotheses from the input data by sampling. Then, based on the residuals between the data and the generated model hypotheses, we calculate the residual correlation between each data and its neighborhood data using the cosine similarity measure. After that, we use information theory to obtain an adaptive threshold to distinguish significant residual correlations from insignificant residual correlations. Finally, data with residual correlations greater than the threshold are retained in the initial neighborhood set. The specific implementation is as follows.

[0070] For input data Generate M sets of model hypotheses by sampling from the input data To calculate the residual correlation, the sampling preference function expresses the preference of the data for the model hypothesis as follows:

[0071]

[0072] Among them, r(s i ,θ m ) is the data s calculated using the Sampson distance i and the model assumes θ m The residual between m is the estimated inlier scale, and δ is a normalization constant. The preference vector f (i) Represents data i and the model assumes a preference between the set Θ, Residual correlation w i,j Associate two data i and j , which is based on two corresponding preference vectors f (i) and f (j) , calculated using cosine similarity:

[0073]

[0074] Among them, w i，j ∈[0, 1]. Then, the obtained residual correlation is expressed as Where T is each data s i and its neighborhood data The number of all pairs of data between . To use information theory to obtain the adaptive threshold, let represents the tth element of W and the largest element of W, so we can get The prior probability of:

[0075]

[0076] The corresponding entropy is calculated as follows:

[0077]

[0078] If the data corresponds to the correlation The amount of information that is higher than the estimated entropy value E is retained in the initial neighborhood set. Otherwise, the data is filtered out; then, the initial neighborhood set Update to the final neighborhood set as follows:

[0079]

[0080] According to formula (9), keep The data in are usually highly consistent in their residual patterns and are more likely to come from the same model instance.

[0081] By preserving the neighborhood consistency from motion and residual patterns, we filter out as many outliers and pseudo outliers as possible in local areas of the data. As a result, we can construct a meaningful set of neighborhoods for each input data and obtain reliable neighborhood relations for robust model fitting.

[0082] 2. Clustering Algorithm Based on Neighborhood Consistency

[0083] After obtaining a meaningful neighborhood set, the next step is to perform a clustering phase to classify the input data into inliers and outliers belonging to different model instances. There are some existing algorithms that cluster data using hierarchical clustering to merge data with similar characteristics. However, these clustering algorithms usually use unreliable neighborhood sets containing many outliers to describe the neighborhood relations of the data, which limits their ability to solve multi-structure fitting problems. In addition, in these clustering algorithms, the similarity metric function used only relies on a single relationship when evaluating the similarity of two data. When there are many outliers in the data, a single relationship shows poor discrimination ability in data clustering, resulting in a serious degradation of clustering performance. To alleviate the above problems, this method proposes a clustering algorithm based on neighborhood consistency, which effectively fits multi-structure data with outliers by combining neighborhood relations and hierarchical clustering processes. In addition, this method also proposes an effective similarity metric function based on neighborhood relations to improve model fitting performance.

[0084] 1) Similarity measurement function based on neighborhood relationship

[0085] This method proposes a new similarity metric function to effectively calculate data similarity by combining multiple relationships between paired data in a neighborhood set.

[0086] Specifically, given a data s i and its corresponding neighborhood set s j The corresponding neighborhood set This method defines a similarity measurement function f(s) based on neighborhood relations. i ,s j ,s k , w) are as follows:

[0087]

[0088] Among them, 1 s∈N Represents a variable, if s∈N, its variable value is 1, otherwise it is 0; represents another variable, if If true, the variable value is 1, otherwise it is 0; is a parameter in the similarity measure function. It is worth noting that two data of the same model instance are usually consistent in residual patterns, and their residual correlation remains high. Therefore, the entropy threshold obtained in the neighborhood optimization stage can be used to indicate the minimum residual correlation allowed between two data. Therefore, after the neighborhood optimization stage, the minimum value in the residual correlation is selected as the similarity measure parameter The default value of .

[0089] Compared with the traditional similarity metric function using a single relationship, the similarity metric function based on neighborhood relationship combines the complementary information of multiple constraints of data in the local area and shows more robust ability in fitting multi-structured data. Therefore, the proposed similarity metric function can effectively cluster highly correlated data, thereby accurately distinguishing inliers and outliers of different model instances.

[0090] 2 Robust clustering for multi-structural model fitting

[0091] The clustering algorithm based on neighborhood consistency proposed in this method aims to merge highly correlated data into the same inlier cluster, while identifying data that are not correlated with any neighborhood data as outliers.

[0092] Assume the input data is The initial label set is The label Linked Data i∈S, and initialized to 0. Based on the hierarchical clustering method, this method iteratively searches for data with neighborhood relationships in the input data and clusters them using the hierarchical clustering method. Then, the proposed similarity measurement function is used to effectively calculate the data similarity.

[0093] Specifically, in the hierarchical clustering process, the neighborhood consistency-based clustering algorithm first obtains two data with neighborhood relationships (i.e., s i and j ). Then, the clustering algorithm based on neighborhood consistency uses a similarity measurement function based on neighborhood relations to determine whether two neighborhood data should be assigned to the same cluster. If the function value of formula (10) is equal to 1, it is considered that the two data are highly correlated, so their labels and are updated to the same label value. In this clustering process, a group of highly related data are merged into one cluster through the neighborhood relationship of the data. In contrast, isolated data that is not related to any neighboring data is not assigned to any cluster and is marked as an outlier. Finally, the clustering algorithm based on neighborhood consistency iteratively selects two neighborhood data and calculates their similarity to obtain the clustering results, thereby classifying the data points into different model structures or as outliers. Based on the neighborhood consistency relationship and the hierarchical clustering process, the clustering algorithm based on neighborhood consistency can adaptively estimate the number of clusters and accurately distinguish between inliers and outliers belonging to different model instances. Figure 2 An example of a clustering algorithm based on neighborhood consistency is shown. Here, starting from data 3, we focus on its neighborhood data 7 and 12. It can be seen that data 3 and 7 and their neighborhood set meet the condition that the similarity metric function value is 1. Therefore, data 7 is assigned to the cluster to which data 3 belongs. In contrast, data 12 is not assigned to the same cluster as data 3 because their relationship does not meet the specified condition. In addition, through data 7, data 4 is finally assigned to the cluster to which data 3 belongs.

[0094] Based on the above clustering process, the input data will be classified into different clusters and isolated data. However, the data clusters of the same model instance usually have a larger number of data points than the clusters mainly composed of outliers. Therefore, we regard the data in the data clusters with less data points than the threshold as outliers. In contrast, the clusters with data points not less than the threshold are regarded as inlier clusters, and each inlier cluster corresponds to an estimated model instance in the data. Here, the threshold is set to the minimum subset size. This is because if the amount of data in the cluster is less than the minimum number of data subsets, it is not enough to estimate the model instance. In addition, in actual scenarios, the data of one model instance may be separated by the data of another model instance. Therefore, according to the above clustering process, the data may be divided into several classes because a part of the data has no neighborhood relationship. Therefore, according to the commonly used data merging method, and based on the residual distribution of the data in each class, the redundant classes are merged to optimize the clusters, and finally the final result of model estimation is obtained by optimizing the redundant classes. Finally, the number of model instances can be obtained from the clustering results, and the parameters of each model instance can be estimated from each cluster.

[0095] In this example, we compared this method with seven popular model fitting methods, including T-Linkage, RCM, RansaCov, RPA, MSHF, HOMF and HRMP. We performed the basic matrix estimation and homography matrix estimation of the eight model fitting methods on the AdelaideRMF dataset containing single-structure and multi-structure data.

[0096] First, the performance of the proposed method and seven popular methods on the AdelaideRMF dataset for image pairs for fundamental matrix estimation is evaluated. Table 1 shows the fitting errors (%) obtained by the eight model fitting methods on 19 sets of images for fundamental matrix estimation in the AdelaideRMF dataset. The best results are indicated in bold. Figure 3 The figure shows some fitting results of the method for estimating the fundamental matrix in five image pairs (a) to (e) of the AdelaideRMF dataset. Each image group shows only one of the two views. Outliers are marked in yellow, and inliers of different structures are marked in different colors.

[0097] Table 1

[0098]

[0099] Table 2

[0100]

[0101] Among them: “#” represents the actual number of model structures in each data, and “%Outlier” represents the percentage of outliers.

[0102] From Table 1 and Figure 3 It can be seen that our method achieves the lowest average fitting error in 15 out of 19 pairs of images. In addition, our method achieves the lowest total average fitting error and total standard deviation among all competing methods. This is because our method uses motion pattern consistency and residual pattern consistency to preserve the local neighborhood relationship of potential inliers and cluster the data, thereby improving the model fitting performance. The results of RPA and HOMF are better than HRMP, but not as accurate as our method. The main reason is that, compared with the proposed method, RPA and HOMF only use residual-based information to obtain the global relationship of the data, which is not robust to outliers. RCM is able to obtain lower average fitting errors than MSHF, T-linkage, and RansaCov in most image pairs. Unlike RCM, which uses spatial consistency constraints, our method studies an effective motion consistency constraint to alleviate the impact of outliers on the fitting of multi-structure models, thereby achieving better fitting results.

[0103] Secondly, the performance of the eight comparison methods is evaluated on the image pairs for homography matrix estimation in the AdelaideRMF dataset. Table 2 shows the fitting errors (%) obtained by the eight model fitting methods on 19 sets of images for homography estimation in the AdelaideRMF dataset. The best results are indicated in bold. Figure 4 The figure shows some fitting results of the homography estimation of the method in five image pairs (a) to (e) of the AdelaideRMF dataset. Each image group shows only one of the two views. Outliers are marked in yellow, and inliers of different structures are marked in different colors.

[0104] From Table 2 and Figure 4It can be seen that the proposed method can achieve a lower average fitting error in most cases among the 19 image pairs. The proposed method also achieves the lowest total average fitting error among all the compared methods. Among the fitting methods based on data clustering (i.e., T-linkage, RPA, HOMF, HRMP, and the proposed method), the proposed method can adaptively estimate the number of model instances in the data and significantly outperforms T-linkage, RPA, and HOMF in most cases. This is because the proposed method combines the neighborhood relationship and the hierarchical clustering process to robustly cluster the data, thereby improving the model fitting accuracy. HRMP can also adaptively estimate the number of model instances, but in most cases it obtains a higher average fitting error than the proposed method. This is because HRMP often incorrectly removes outliers during the outlier removal process, and the remaining data cannot correctly estimate the model instances in the subsequent process. In contrast, the proposed method simultaneously detects outliers and classifies the data into different model instances during the aggregation clustering process for robust model estimation. RansaCov achieves a lower average fitting error on some image pairs, but the number of model instances needs to be manually specified, which seriously affects its practicality in industrial applications. MSHF achieves the second lowest total average fitting error in homography matrix estimation because it can remove most of the irrelevant model assumptions in advance for robust model selection. RCM achieves the lowest average fitting error in some cases, but often fails to fit data contaminated by a large number of outliers, mainly because RCM is sensitive to outliers.

[0105] Therefore, the present invention can be widely used in the face segmentation scene of buildings. By segmenting the inliers and outliers from the three-dimensional point cloud data of the building, the accurate segmentation of the building facets can be achieved, so as to accurately extract the wall, roof and other information from the building, and improve the accuracy of the three-dimensional reconstruction of the building. The method can also be applied to many fields such as image registration, motion segmentation, and industrial detection. For example, when applied to motion segmentation, the inliers that meet the motion model are found through model fitting, so as to achieve accurate segmentation of moving objects, which can be applied to video monitoring, motion analysis and other fields. For another example, when applied to object feature extraction and object detection in industrial production, quality detection of production objects can be achieved. Therefore, the present invention has strong practicality and broad application prospects.

[0106] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0107] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0110] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A model fitting method based on neighborhood consistency, characterized in that: The following steps are involved: S1. For the input data, the neighborhood set of each data point is obtained through the neighborhood consistency preservation algorithm; S2. Based on the obtained neighborhood set, cluster the data using a neighborhood consistency-based clustering algorithm, merge the data with consistent information, and obtain a clustering result; In the clustering process, the similarity metric function based on the neighborhood relationship is used to evaluate the data similarity; S3, distinguishing the inliers and outliers belonging to different model instances based on the clustering results, and outputting the data segmentation results; The step S1 comprises the following steps: S101, initializing a neighborhood set through motion pattern consistency; S102, optimizing the neighborhood set through residual pattern consistency; The implementation method of step S101 is: For a given pair of images, suppose there is N Input data for feature matching ,in Indicates i feature matching, x i and y i are two corresponding feature points in the image pair; for feature matching , respectively search for two feature points based on the Euclidean distance x i and y i Two K Nearest Neighbor and ; If there is a data point , and its corresponding two feature points x j and y j Belong to and , then the data s j and s i The movement amplitudes are close to each other, and the data s j and s i are regarded as adjacent data with similar motion amplitude; for data points , define its neighborhood set as follows: (1) in, and , represents the number of data points in the neighborhood set; then, the data is calculated based on the cosine similarity measure s i Its adjacent data The motion direction similarity value between them is as follows: (2) in, , Respectively represent data s i and s j The motion vector of and Represent the inner product and norm respectively; then, based on the motion direction similarity threshold μ i , the motion direction similarity is higher than the threshold μ i The data is kept in For the data points in the input data s i , whose consistent neighborhood set in motion Defined as: (3) (4) in, μ i yes s i and The average value of the similarity of the motion direction between all adjacent data; therefore, for the input data , based on the motion pattern consistency constraint, a series of neighborhood sets are obtained , that is, the neighborhood set initialization is realized.

2. A model fitting method based on neighborhood consistency according to claim 1, characterized in that: The implementation method of step S102 is: For input data , generated by sampling from the input data M Model Assumptions Set ; To calculate the residual correlation, the sampling preference function represents the preference of the data for the model assumptions, as follows: (5) in, The data is calculated using Sampson distance s i and model assumptions θ m The residual between τ m is the estimated inlier scale, δ is a normalizing constant; the preference vector f (i) Representation data s i and the model assumption set Preferences between ; Residual correlation w i,j Relate two data s i and s j , which is based on two corresponding preference vectors f (i) and f (j) , calculated using cosine similarity: (6) in, ; Then, the obtained residual correlation is expressed as ,in T Each data s i and its neighborhood data The number of all paired data between ; In order to use information theory to obtain the adaptive threshold, let express W No. t Elements The sum of squares W The maximum gap between the elements of The prior probability of: (7) The corresponding entropy is calculated as follows: (8) If the data corresponds to the correlation Higher than the estimated entropy E The amount of information is retained in the initial neighborhood set Otherwise, the data is filtered out; then, the initial neighborhood set Update to the final neighborhood set as follows: (9)。 3. A model fitting method based on neighborhood consistency according to claim 1, characterized in that: In step S2, highly correlated data are clustered into the same inlier cluster by a clustering algorithm based on neighborhood consistency, and data that is not related to any neighborhood data is classified as outliers, which is implemented as follows: Assume the input data is , the initial label set is , where the label Linked Data , and initialized to 0; hierarchical clustering is used for clustering, specifically: First, it iteratively searches for two data with a neighborhood relationship in the input data; then, it samples a similarity measurement function based on the neighborhood relationship to calculate the data similarity and determine whether the two neighborhood data should be assigned to the same cluster; If the similarity measure function value is equal to 1, it is considered that the two data are highly related, and the labels of the two data are and Update to the same label value; in this clustering process, highly correlated data are merged into one cluster through the neighborhood relationship of the data. On the contrary, isolated data that is not related to any neighboring data is not assigned to any cluster and is marked as an outlier; finally, iteratively select two neighboring data and calculate their similarity to obtain the clustering result, so as to classify the data points into different model structures or as outliers.

4. A model fitting method based on neighborhood consistency according to claim 3, characterized in that: The neighborhood relationship-based similarity measurement function effectively calculates data similarity by combining multiple relationships between paired data in a neighborhood set; Specifically: Given data s i and its corresponding neighborhood set ,data s j and its corresponding neighborhood set , and data s k ,in , , similarity measurement function based on neighborhood relationship The definition is as follows: (10) in, Represents a variable, if If the variable value is 1, then it is 0; otherwise, represents another variable, if If true, the variable value is 1, otherwise it is 0; is the parameter in the similarity measure function.

5. The model fitting method based on neighborhood consistency according to claim 3, characterized in that: Through the clustering process, the input data will be classified into different clusters and isolated data; the data clusters of the same model instance have a larger number of data points than the clusters mainly composed of outliers; the data in the data clusters with less than a threshold value of data points are regarded as outliers, while the data clusters with not less than a threshold value of data points are regarded as inliers, and each inlier cluster corresponds to an estimated model instance in the data; the threshold is set to the minimum subset size.

6. The application of a model fitting method based on neighborhood consistency according to any one of claims 1 to 5, characterized in that: The model fitting method based on neighborhood consistency is applied to the task of building face segmentation, which segments inliers and outliers from the three-dimensional point cloud data of the building, and then extracts the geometric features of the building to achieve face segmentation of the building.

Images