Method for constructing positioning database and positioning database construction apparatus

Abstract

A method and apparatus for constructing a location database are disclosed. One embodiment of the present invention describes a method for constructing a location database executed by a location database construction apparatus, comprising the following steps: extracting multiple local features from multiple keyframes of a defined area; identifying individual 3D keypoints in each of the multiple local features that contain information related to three-dimensional position; clustering the multiple local features into multiple clusters based on the individual 3D keypoints; determining representative position information that represents the position of each of the multiple clusters using the individual 3D keypoints of the local features contained in each of the multiple clusters; and storing, for each of the multiple keyframes, a cluster identifier that identifies the multiple clusters and the representative position information of each of the multiple clusters into the location database, wherein each local feature contained in each of the multiple keyframes corresponds to a cluster.

Description

Technical Field

[0001] This invention relates to a method for constructing a location database using clustering of local features, and a location database construction apparatus for performing the method. Background Technology

[0002] Methods for performing positioning in shadowy, localized areas with weak GPS signals include radio wave-based positioning, camera-based positioning, and radar sensor-based positioning.

[0003] Here, radio wave-based positioning includes positioning methods based on base station / AP location and signal strength, namely Cell ID method; positioning methods based on signal arrival time difference or incident angle, namely TDoA / AoA method; and positioning methods based on signal pattern matching of grid units, namely fingerprinting method.

[0004] In radio wave-based positioning methods, camera-based positioning offers relatively high accuracy, but it suffers from drawbacks such as requiring extensive computation for positioning and the need to pre-construct a large-scale 3D map.

[0005] Therefore, in order to more easily use camera-based positioning with high accuracy, it is necessary to explore methods to reduce the data volume of 3D maps. Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] The problem to be solved by this invention is to provide a method for constructing a location database by using clustering of local features, thereby reducing the size of the location database.

[0008] However, the problems to be solved by the present invention are not limited to those mentioned above. Those skilled in the art can clearly understand from the following description other problems not mentioned herein.

[0009] Methods for solving problems

[0010] According to an embodiment of the present invention, a method for constructing a positioning database by a positioning database construction apparatus includes the following steps: extracting multiple local features from multiple keyframes of a defined area; identifying individual 3D keypoints in each of the multiple local features that contain information related to three-dimensional position; clustering the multiple local features into multiple clusters based on the individual 3D keypoints; determining representative position information for each of the multiple clusters using the individual 3D keypoints of the local features contained in each of the multiple clusters; and storing, for each of the multiple keyframes, cluster identifiers that identify the multiple clusters and representative position information for each of the multiple clusters into the positioning database, wherein each local feature contained in each of the multiple keyframes corresponds to a cluster.

[0011] In the step of clustering the above-mentioned multiple local features into multiple clusters, based on the individual 3D key points of each of the above-mentioned multiple local features, the local features representing the same three-dimensional space points within the above-defined region are clustered among the above-mentioned multiple local features.

[0012] The steps of clustering the aforementioned multiple local features into multiple clusters include the following steps: identifying 2D keypoints that indicate the two-dimensional positions of each of the aforementioned multiple local features; using the 2D keypoints of each of the aforementioned multiple local features, determining the individual 3D keypoints of each of the aforementioned multiple local features and the covariance of the individual 3D keypoints; and using the covariance of the individual 3D keypoints of each of the aforementioned multiple local features, clustering the aforementioned multiple local features into at least one cluster.

[0013] It also includes the following steps: identifying individual descriptors, which are used to distinguish the local features contained in each of the multiple clusters from other local features.

[0014] The aforementioned representative location information includes: representative 3D keypoints, which indicate 3D keypoints associated with each of the aforementioned multiple clusters, the representative 3D keypoints being based on individual 3D keypoints of the aforementioned local features; and representative descriptors, which indicate descriptors associated with each of the aforementioned multiple clusters, the representative descriptors being based on individual descriptors of the aforementioned local features.

[0015] The step of clustering the above-mentioned multiple local features into multiple clusters includes the following steps: For the first local feature among the above-mentioned multiple local features, if the first cluster in one or more generated clusters satisfies the preset clustering conditions, the first local feature is included in the first cluster. The clustering conditions include at least one of the following: the distance between individual 3D key points of the first local feature and the representative key point of the first cluster; the disparity angle between the key frame corresponding to each local feature included in the first cluster and the key frame including the first local feature; and the epipolar distance between the local feature included in the first cluster and the first local feature.

[0016] The step of clustering the above-mentioned multiple local features into multiple clusters further includes the following step: for the second local feature among the above-mentioned multiple local features, if there is no cluster that satisfies the above-mentioned preset clustering conditions in the above-mentioned one or more clusters, a second cluster including the above-mentioned second local feature is generated.

[0017] The steps of clustering the above-mentioned multiple local features into multiple clusters include the following steps: clustering the above-mentioned multiple local features into at least one cluster based on the individual 3D key points of each of the above-mentioned multiple local features; and re-clustering the at least one cluster into the above-mentioned multiple clusters based on the individual descriptors of each of the above-mentioned multiple local features.

[0018] The step of re-clustering the at least one cluster into the plurality of clusters includes the following steps: based on the result of applying the individual descriptors of the plurality of local features to preset clustering conditions, the at least one cluster is re-clustered into the plurality of clusters, wherein the clustering conditions include at least one of the distance between the descriptors of the plurality of local features and the representative descriptor of the at least one cluster and the number of local features contained in the at least one cluster.

[0019] The step of re-clustering the at least one cluster into the plurality of clusters further includes the following steps: generating a list of the at least one cluster; re-clustering the local features contained in the first cluster into the plurality of clusters if the local features contained in the first cluster satisfy the clustering conditions; and deleting the first cluster from the list and adding the plurality of re-clustered clusters to the list.

[0020] The aforementioned positioning database includes a keyframe database and a cluster database. The keyframe database stores the global features of each of the aforementioned keyframes and the IDs of the clusters corresponding to the local features contained in each of the aforementioned keyframes. The cluster database stores the representative 3D keypoints and representative descriptors of each of the aforementioned clusters.

[0021] Another embodiment of the present invention provides a positioning database construction apparatus comprising: a memory storing a positioning database construction program that generates data to be constructed in the positioning database; and a processor loading the positioning database construction program from the memory. The processor extracts multiple local features from multiple keyframes of a defined area, identifies individual 3D keypoints containing information related to three-dimensional position for each of the multiple local features, clusters the multiple local features into multiple clusters based on the individual 3D keypoints, determines representative position information representing the position of each of the multiple clusters using the individual 3D keypoints of the local features contained in each of the multiple clusters, and stores cluster identifiers that identify the multiple clusters and representative position information of each of the multiple clusters in the positioning database for each keyframe, wherein each local feature contained in each of the multiple keyframes corresponds to a cluster.

[0022] Another embodiment of the present invention includes a computer-readable recording medium storing a computer program comprising instructions for a processor to execute a method for constructing a positioning database executed by a positioning database construction apparatus, the method comprising the steps of: extracting multiple local features from multiple keyframes of a defined area; identifying individual 3D keypoints of each of the multiple local features containing information related to three-dimensional position; clustering the multiple local features into multiple clusters based on the individual 3D keypoints; determining representative position information representing the position of each of the multiple clusters using the individual 3D keypoints of the local features contained in each of the multiple clusters; and storing, for each of the multiple keyframes, cluster identifiers that identify the multiple clusters and representative position information of each of the multiple clusters into the positioning database, wherein each local feature contained in each of the multiple keyframes corresponds to a cluster.

[0023] Invention Effects

[0024] According to an embodiment of the present invention, only representative 3D key points and representative descriptors are stored in the positioning database according to each cluster of local features, thus saving the database capacity required for positioning. Attached Figure Description

[0025] Figure 1 This is a block diagram illustrating a positioning system according to an embodiment of the present invention.

[0026] Figure 2 This shows an example of a database previously used for camera-based positioning.

[0027] Figure 3 An example of a location database illustrating an embodiment of the present invention is shown.

[0028] Figure 4 This is a block diagram conceptually illustrating the functionality of a location database construction program according to an embodiment of the present invention.

[0029] Figure 5 A method for determining the covariance of 3D keypoints of local features according to an embodiment of the present invention is shown.

[0030] Figure 6 This is a flowchart illustrating a method for clustering local features by the local feature clustering unit according to an embodiment of the present invention.

[0031] Figure 7 This is a flowchart illustrating a method by which the descriptor clustering unit of an embodiment of the present invention re-clusters local features based on condition 1 in the second clustering condition.

[0032] Figure 8 This is a flowchart illustrating a method by which the descriptor clustering unit of an embodiment of the present invention re-clusters local features based on condition 2 in the second clustering condition.

[0033] Figure 9 This is a flowchart illustrating a method by which the descriptor clustering unit of an embodiment of the present invention re-clusters local features based on condition 3 in the second clustering condition. Detailed Implementation

[0034] The advantages and features of this disclosure, as well as the methods for implementing them, will become apparent from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, this disclosure is not limited to the embodiments disclosed below, but can be implemented in a variety of different ways; and the embodiments disclosed herein are provided only to complete the disclosure and to enable those skilled in the art to fully understand the scope of this disclosure; and the scope of this disclosure is defined only by the claims.

[0035] In the description of embodiments according to this disclosure, detailed descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions might unnecessarily obscure the subject matter of this disclosure. Furthermore, the terminology described later is defined in consideration of the functions in the embodiments of this disclosure and may vary depending on the intent or practice of the user or operator. Therefore, definitions should be based on the content throughout this specification.

[0036] Figure 1 This is a block diagram illustrating a positioning system according to an embodiment of the present invention.

[0037] Reference Figure 1 The positioning system 10 may include a positioning database 20, a positioning database construction device 100, and a positioning device 200.

[0038] In this specification, for ease of explanation, the positioning system 10 is described as including a positioning database 20, a positioning database construction device 100, and a positioning device 200, but it is not limited thereto.

[0039] According to one embodiment, the positioning system 10 may include a positioning database 20 and a positioning database construction device 100, in which case the positioning system 10 may be a system for constructing the positioning database 20. Alternatively, according to another embodiment, the positioning system 10 may include a positioning database 20 and a positioning device 200, in which case the positioning system 10 may be a system for performing positioning using the already constructed positioning database 20.

[0040] The positioning database 20 may store data used by the positioning device 200 to determine the location.

[0041] In conventional databases used for camera-based localization, global features, at least one local feature, descriptors for each of the at least one local feature, and 3D coordinates for each of the at least one local feature can be stored for each keyframe.

[0042] For example, further reference Figure 2 , Figure 2 This shows an example of a database previously used for camera-based positioning.

[0043] In the case where the global feature of the first keyframe KF1 is a first global feature, and the first keyframe KF1 includes a first local feature, a second local feature, a third local feature, and a fourth local feature, the database 50 used for camera-based positioning in the past can store the following for the first keyframe KF1: i) the first global feature as a global feature, ii) the first local feature, the second local feature, the third local feature, and the fourth local feature as local features included in the first keyframe, iii) the first descriptor, the second descriptor, the third descriptor, and the fourth descriptor as descriptors related to the first local feature, the second local feature, the third local feature, and the fourth local feature, and iv) the first 3D coordinate, the second 3D coordinate, the third 3D coordinate, and the fourth 3D coordinate as 3D coordinates related to the first local feature, the second local feature, the third local feature, and the fourth local feature.

[0044] Conversely, in the positioning database 20 of the embodiments of the present invention, global features and at least one cluster ID can be included according to each keyframe, and 3D keypoints and descriptors can be mapped according to the IDs of each cluster.

[0045] Therefore, according to the embodiment, the positioning database 20 may include: a keyframe database (not shown) that stores global features of each keyframe and IDs of at least one cluster; and a cluster database (not shown) that stores representative 3D keypoints and representative descriptors for each cluster ID.

[0046] For example, further reference Figure 3 , Figure 3 An example of a location database illustrating an embodiment of the present invention is shown.

[0047] The location database 20 may include a keyframe database 22 and a cluster database 24.

[0048] The keyframe database 22 can store the global features of each keyframe and the IDs of the clusters contained in each keyframe.

[0049] The global feature of the second keyframe KF2 is the second global feature. When the second keyframe KF2 includes the first cluster, the third cluster, and the sixth cluster, the keyframe database 22 can store the second global feature as a global feature, and the IDs of the first cluster, the third cluster, and the sixth cluster as IDs of the clusters included in the second keyframe KF2.

[0050] The cluster database 24 may store representative 3D keypoints and representative descriptors related to all clusters contained in the keyframes used in camera-based localization.

[0051] In the case where the keyframes used in camera-based positioning include a first cluster, a second cluster, a third cluster, a fourth cluster, a fifth cluster, and a sixth cluster, the cluster database 24 may store i) the IDs of the first cluster, the second cluster, the third cluster, the fourth cluster, the fifth cluster, and the sixth cluster as IDs of the clusters included in the keyframe; ii) the first, second, third, fourth, fifth, and sixth representative 3D keypoints representing the 3D keypoints of each of the first, second, third, fourth, fifth, and sixth clusters; and iii) the first, second, third, fourth, fifth, and sixth representative descriptors representing the descriptors of each of the first, second, third, fourth, fifth, and sixth clusters.

[0052] Thus, compared with conventional databases used for camera-based positioning, the positioning database 20 of this embodiment does not store local features and descriptors about local features according to each keyframe, and therefore has the advantage of saving database storage space.

[0053] The positioning database construction device 100 can store the data used by the positioning device 200 to determine its own location in the positioning database 20.

[0054] The location database construction apparatus 100 can determine the global features of each keyframe captured for the location where the location database 20 is to be constructed, and the representative 3D keypoints and representative descriptors of each ID of at least one cluster contained in the keyframe.

[0055] Therefore, the location database construction apparatus 100 may include a processor 110, a transceiver 120, and a memory 130.

[0056] The processor 110 can control the operation of the positioning database construction device 100 as a whole.

[0057] The processor 110 can use the transceiver 120 to transmit data used to build the location database 20 to the location database 20.

[0058] In this specification, the location database construction apparatus 100 is described as a separate device distinct from the location database 20, but it is not limited thereto. That is, according to an embodiment, the location database construction apparatus 100 may include the location database 20, in which case data for constructing the location database 20 is transmitted to the location database 20 via internal signaling.

[0059] The location database construction apparatus 100 can use the transceiver 120 to receive key frames captured of the location where the location database 20 is to be constructed.

[0060] Alternatively, according to an embodiment, the location database construction apparatus 100 may further include a camera (not shown) which captures keyframes.

[0061] The memory 130 may store the location database construction program 300 and the information required when executing the location database construction program 300.

[0062] In this specification, the location database construction program 300 may refer to software that includes instructions programmed in a manner that generates data for constructing the location database 20.

[0063] In order to execute the location database builder 300, the processor 110 may load the location database builder 300 and the information required for executing the location database builder 300 from the memory 130.

[0064] The processor 110 can execute the location database construction program 300 to generate data for constructing the location database 20 using keyframes captured at the locations where the location database 20 is to be constructed.

[0065] Regarding the functions and / or actions of the database builder 300, through... Figure 2 Let me explain in detail.

[0066] In addition, the positioning device 200 can use data stored in the positioning database 20 to determine its own location.

[0067] Therefore, the positioning device 200 may include a processor 210, a transceiver 220, and a memory 230.

[0068] The processor 210 can perform overall control of the operation of the positioning device 200.

[0069] The processor 210 can use the transceiver 220 to receive data from the positioning database 20 for determining the location of the positioning device 200.

[0070] The memory 230 may store a positioning program 240, which includes instructions programmed to determine the position of the positioning device 200, and information required when executing the positioning program 240.

[0071] In order to execute the location program 240, the processor 210 may load the location program 240 and the information required for executing the location program 240 from the memory 230.

[0072] The processor 210 can execute the positioning program 240 to determine its own position.

[0073] Therefore, according to an embodiment, the positioning device 200 may further include a camera (not shown). That is, the positioning device 200 can extract global and local features from images captured using the camera (not shown), and compare the global features of the captured images with the global features of keyframes stored in the positioning database 20 to select similar keyframes. Then, the positioning device 200 can compare the local features and descriptors extracted from the captured images with representative 3D keypoints and representative descriptors of each cluster contained in the selected keyframes to determine its own position.

[0074] Figure 4 This is a block diagram conceptually illustrating the functionality of a location database construction program according to an embodiment of the present invention.

[0075] Reference Figure 1 and Figure 4The location database construction program 300 may include a local feature extraction unit 310, a local feature clustering unit 320, a descriptor clustering unit 330, a representative descriptor determination unit 340, and a location database storage unit 350.

[0076] Figure 4 The local feature extraction unit 310, local feature clustering unit 320, descriptor clustering unit 330, representative descriptor determination unit 340, and location database storage unit 350 shown are conceptual classifications of the functions of the location database construction program 300 for ease of explanation, but are not limited thereto. According to embodiments, the functions of the local feature extraction unit 310, local feature clustering unit 320, descriptor clustering unit 330, representative descriptor determination unit 340, and location database storage unit 350 can be merged / separated, or they can be represented as a series of instructions contained in a single program.

[0077] The local feature extraction unit 310 can extract multiple local features from keyframes obtained using the transceiver 120 or a camera (not shown).

[0078] The aforementioned local features may include 2D keypoints and individual descriptors. The 2D keypoints may refer to the two-dimensional coordinates on the keyframe of the local feature, and the individual descriptors are used to distinguish the local feature from other local features contained in the keyframe. They may refer to multi-dimensional vectors that represent the correlation between the local feature and the pixels surrounding the local feature.

[0079] According to an embodiment, the local feature extraction unit 310 may include a neural network that has been learned to extract local features from keyframes. The local feature extraction unit 310 may input the keyframes into the neural network and output local features.

[0080] Alternatively, according to an embodiment, the local feature extraction unit 310 may use methods such as SuperPoint, R2D2, SIFT, etc. to extract multiple local features from the keyframe.

[0081] The local feature clustering unit 320 can cluster local features representing points in the same three-dimensional space within the aforementioned specific region from the local features contained in each of the multiple keyframes, and determine the representative 3D keypoints of the local feature clusters.

[0082] Here, multiple keyframes can refer to images of the same area taken from different locations, and 3D keypoints can refer to the three-dimensional coordinates (e.g., absolute coordinates) of the three-dimensional space within the aforementioned specific area.

[0083] First, the local feature clustering unit 320 can cluster local features representing points in the same three-dimensional space based on individual 3D key points of local features contained in each of the multiple key frames.

[0084] More specifically, the local feature clustering unit 320 can compare individual 3D key points of local features contained in multiple key frames, and cluster local features whose distance between individual 3D key points is below a preset reference range into a cluster.

[0085] To this end, the local feature clustering unit 320 can determine the individual 3D keypoints of each local feature contained in multiple keyframes by utilizing the 2D keypoints of the local features contained in multiple keyframes and the pose information of the multiple keyframes. The pose information mentioned above may include the absolute position and orientation of the camera that captured the keyframes, and can be represented using 6 degrees of freedom (DOF).

[0086] According to the embodiment, in the absence of determining the pose information of multiple keyframes, the local feature clustering unit 320 can determine the relative pose information of multiple keyframes by using the 2D key points of local features contained in multiple keyframes and the matching between multiple keyframes.

[0087] The local feature clustering unit 320 can determine individual 3D keypoints of local features contained in multiple keyframes by utilizing 3D orientation vectors transformed from 2D keypoints, camera intrinsic information, and relative pose information of multiple keyframes. For example, the local feature clustering unit 320 can use the SfM (Structu re-from-Motion) method to determine the relative pose information of multiple keyframes.

[0088] That is, when the local feature clustering unit 320 calculates the 3D keypoints of each local feature using the 2D keypoints of the local features contained in the multiple keyframes, it can generate pose information for keyframes that are different from each other, depending on the type or settings of the camera that captured the keyframes. Therefore, in order to correct the differences between the various cameras, the local feature clustering unit 320 can also calculate the 3D keypoints of each local feature using the camera's intrinsic information.

[0089] According to another embodiment, when the pose information of multiple keyframes has been determined, the local feature clustering unit 320 can use the determined pose information of multiple keyframes to determine the individual 3D key points of each local feature.

[0090] For example, when using a LiDAR sensor and a camera together to capture the aforementioned multiple keyframes, the local feature clustering unit 320 can obtain the camera's pose information using SLAM (Simultaneous Localization and Mapping) technology, and use the camera's pose information and 3D point cloud to determine individual 3D key points of the local features.

[0091] In order to calculate the distance between individual 3D keypoints, the local feature clustering unit 320 can use the 2D keypoints of the local features to calculate the covariance of the individual 3D keypoints.

[0092] Further reference Figure 5 The local feature clustering unit 320 can use the 2D key point P'c of the local feature and the depth D of the 2D key point P'c relative to the three-dimensional space W to determine the projection point Pc obtained by projecting the 2D key point P'c of the local feature onto the depth D.

[0093] The local feature clustering unit 320 can perform a W2C (World-to-Camera) transformation (R) that transforms the global coordinate system into the camera coordinate system. W2C , t W2C The individual 3D key points Pw are applied to the projection point Pc to determine the local features.

[0094] Here, the projection point Pc and the individual 3D key point Pw represent the same location. The projection point Pc can refer to the coordinates expressed in the camera coordinate system, and the individual 3D key point Pw can refer to the coordinates expressed in the global coordinate system.

[0095] Subsequently, the local feature clustering unit 320 can utilize the depth D of the 2D keypoint P'c and the variance σ of the depth D relative to the local features. D 2 2D keypoints P'c of local features, and the covariance C of 2D keypoints P'c of local features. P'c To determine the covariance C of the projection point Pc Pc .

[0096] The local feature clustering unit 320 can utilize the W2C transformation R W2C The covariance C of the value and the projection point Pc Pc To determine the covariance C of individual 3D keypoints Pw Pw .

[0097] For example, the local feature clustering unit 320 can use the following mathematical formula 1 to determine the covariance C of individual 3D keypoints Pw.Pw .

[0098]

Mathematical Formula 1

[0099]

[0100]

[0101] P W =R W2C ·P C +t W2C

[0102]

[0103]

[0104] Here, u can represent the x-coordinate value of the 2D keypoint P'c of the local feature, and v can represent the y-coordinate value of the 2D keypoint P'c of the local feature.

[0105] Further reference Figure 6 , Figure 6 This is a flowchart illustrating the method by which the local feature clustering unit 320 clusters local features.

[0106] In order to cluster all local features contained in multiple keyframes sequentially, the local feature clustering unit 320 may include the first keyframe, i.e. the first keyframe, in the first cluster (S600).

[0107] Then, the local feature clustering unit 320 can determine whether there are residual local features that have not been clustered (S610). If there are second local features that have not been clustered (S610 "Yes"), it determines whether there are clusters that satisfy the specified first clustering conditions among the generated clusters for the second local feature that is the object of judgment (S620).

[0108] The first clustering condition mentioned above may include the following conditions 1 to 3.

[0109] -Condition 1: The distance between the shared 3D keypoints of the cluster and the 3D keypoints of the second local feature used as the judgment object is below a threshold.

[0110] -Condition 2: The parallax angle between keyframes within a cluster and keyframes including the second local feature is below a threshold.

[0111] -Condition 3: The epipolar distance between the local features and the second local features within the cluster is below the threshold.

[0112] According to the embodiment, the distance in condition 1 is the Mahalanobis distance, which is calculated as shown in the following mathematical formula 2.

[0113]

Mathematical Formula 2

[0114] Mahalanobis Distance=(P1-P2) T ×inv(Cov1+Cov2)×(P1-P2)

[0115] Here, P1 represents the shared 3D keypoints of the cluster, P2 represents the 3D keypoints of the second local feature, Cov1 represents the covariance of the shared 3D keypoints of the cluster, and Cov2 represents the covariance of the second local feature.

[0116] According to the embodiment, if the second local feature for a cluster does not satisfy any of the conditions 1 to 3, the local feature clustering unit 320 may determine that the second local feature for a cluster does not satisfy the first clustering condition without judging whether other conditions are satisfied.

[0117] If there exists a cluster that satisfies the first clustering condition for the second local feature (S620 "Yes"), the local feature clustering unit 320 may include the second local feature in the corresponding cluster (S630).

[0118] Conversely, if there is no cluster that satisfies the first clustering condition for the second local feature ("No" in S620), the local feature clustering unit 320 generates a new cluster and includes the second local feature in the generated cluster (S640).

[0119] If no residual local features exist ("No" in S610), the local feature clustering unit 320 may terminate the clustering of local features (S650). That is, the local feature clustering unit 320 repeatedly performs the above process (S610 to S640) until no residual local features exist, thereby including the local features in the existing clusters or generating new clusters.

[0120] The descriptor clustering unit 330 can re-cluster local features contained in the same cluster based on individual descriptors of local features.

[0121] This is because, even if local features represent the same points in three-dimensional space, if the keyframes are taken at locations far apart, the individual descriptors of the local features will be very different. When merging local features with different individual descriptors, the merging performance will decrease.

[0122] Therefore, the descriptor clustering unit 330 can re-cluster local features that satisfy the second clustering condition among the local features contained in the same cluster.

[0123] The second clustering condition mentioned above can be one of the following three.

[0124] -Condition 1: The distance between an individual descriptor of a local feature and the representative descriptor of the cluster is below a threshold (less than the threshold).

[0125] -Condition 2: The number of local features contained in the cluster is below the baseline value (less than the baseline value).

[0126] -Condition 3: The distance between the representative descriptors of the self-cluster is below the threshold (less than the threshold) and the number of local features contained in the cluster is below the baseline value (less than the baseline value).

[0127] Here, a descriptor may refer to a descriptor representing a cluster that includes local features, and according to an embodiment, it may be the average of the clusters of local features.

[0128] The following describes the method by which the descriptor clustering unit 330 re-clusters local features based on any one of the conditions 1 to 3 of the second clustering condition.

[0129] First, further reference Figure 7 , Figure 7 This is a flowchart of a method by which the descriptor clustering unit 330 re-clusters local features based on condition 1 in the second clustering condition.

[0130] In order to re-cluster local features, the descriptor clustering unit 330 can generate a list of clusters generated by the local feature clustering unit 320 (S700).

[0131] The descriptor clustering unit 330 can determine whether there is a residual cluster in the generated list (S710). If there is a first cluster as a residual cluster (S710 "yes"), it determines whether the first cluster satisfies the first condition in the second clustering condition (S720).

[0132] That is, the descriptor clustering unit 330 can re-cluster local features in the local features contained in the first cluster where the distance between the individual descriptors of the local features and the representative descriptor of the first cluster exceeds a reference value (above the reference value).

[0133] If, among the local features contained in the first cluster, there are local features whose individual descriptors are more than or equal to the distance between them and the representative descriptor of the first cluster, the descriptor clustering unit 330 can re-cluster the local features contained in the first cluster into two or more clusters (S730).

[0134] According to an embodiment, the descriptor clustering unit 330 can use the k-means clustering algorithm to re-cluster the local features contained in the first cluster into two or more clusters.

[0135] Alternatively, according to an embodiment, the descriptor clustering unit 330 may re-cluster local features in the first cluster where the distance between an individual descriptor and the representative descriptor is less than the reference value (below the reference value) into a second cluster, and re-cluster local features where the distance between an individual descriptor and the representative descriptor of the cluster exceeds the reference value (above the reference value) into a third cluster.

[0136] Then, the descriptor clustering unit 330 can delete the first cluster from the list and append the re-clustered clusters to the list (S740).

[0137] However, according to the embodiment, when local features whose distance between individual descriptors and representative descriptors is less than the reference value are re-clustered into a second cluster, and local features whose distance between individual descriptors and representative descriptors of a cluster exceeds the reference value are re-clustered into a third cluster, the descriptor clustering unit 330 may not add the second cluster to the list, but only add the third cluster to the list.

[0138] Conversely, if the distance between an individual descriptor of a local feature contained in the first cluster and the representative descriptor of the first cluster is below the reference value (less than the reference value) ("No" in S720), the descriptor clustering unit 330 may remove the first cluster from the list (S750).

[0139] If no residual clusters exist ("No" in S710), the descriptor clustering unit 330 may terminate the re-clustering of local features (S760). That is, the descriptor clustering unit 330 repeatedly performs the above process (S710 to S750) until no residual clusters exist, thereby re-clustering the local features.

[0140] In addition, further reference Figure 8 , Figure 8 This is a flowchart illustrating a method by which the descriptor clustering unit 330 re-clusters local features based on condition 2 in the second clustering condition.

[0141] The descriptor clustering unit 330 can generate a list of clusters generated by the local feature clustering unit 320 in order to re-cluster local features (S800).

[0142] The descriptor clustering unit 330 can determine whether there is a residual cluster in the generated list (S810). If there is a first cluster as a residual cluster (S810 "yes"), it can determine whether the first cluster satisfies condition 2 in the second clustering condition (S820).

[0143] If the number of local features contained in the first cluster exceeds a threshold (above the threshold) (S820 "Yes"), the descriptor clustering unit 330 can re-cluster the local features contained in the first cluster into two or more clusters (S830). According to the embodiment, the descriptor clustering unit 330 can use the k-means clustering algorithm to re-cluster the local features contained in the first cluster into two or more clusters.

[0144] Then, the descriptor clustering unit 330 can delete the first cluster from the list and append the re-clustered clusters to the list (S840).

[0145] Conversely, if the number of local features contained in the first cluster is below the threshold (less than the threshold) ("No" in S820), the descriptor clustering unit 330 can remove the first cluster from the list (S850).

[0146] If no residual clusters exist ("No" in S810), the descriptor clustering unit 330 may terminate the re-clustering of local features (S860). That is, the descriptor clustering unit 330 repeatedly performs the above process (S810 to S850) until no residual clusters exist, thereby re-clustering the local features.

[0147] Finally, further reference Figure 9 , Figure 9 This is a flowchart illustrating the method by which the descriptor clustering unit 330 re-clusters local features based on condition 3 in the second clustering condition.

[0148] The descriptor clustering unit 330 can generate a list of clusters generated by the local feature clustering unit 320 in order to re-cluster local features (S900).

[0149] The descriptor clustering unit 330 can determine whether there is a residual cluster in the generated list (S910). If there is a first cluster as a residual cluster (S910 "yes"), it determines whether the first cluster satisfies condition 3 in the second clustering condition (S920).

[0150] If the number of local features contained in the first cluster exceeds a threshold (above the threshold) or if the number of local features contained in the first cluster exceeds a threshold (above the threshold) ("Yes" in S920), the descriptor clustering unit 330 can re-cluster the local features contained in the first cluster into two or more clusters (S930).

[0151] According to an embodiment, the descriptor clustering unit 330 can use the k-means clustering algorithm to re-cluster the local features contained in the first cluster into two or more clusters.

[0152] Then, the descriptor clustering unit 330 can delete the first cluster from the list and append the re-clustered clusters to the list (S940).

[0153] Conversely, if the number of local features contained in the first cluster is below the threshold (less than the threshold) ("No" in S920), the descriptor clustering unit 330 can remove the first cluster from the list (S950).

[0154] If no residual clusters exist ("No" in S910), the descriptor clustering unit 330 may terminate the re-clustering of local features (S960). That is, the descriptor clustering unit 330 repeatedly performs the above process (S910 to S950) until no residual clusters exist, thereby re-clustering the local features.

[0155] In the descriptor clustering unit 330, local features are re-clustered, thereby generating multiple clusters.

[0156] The representative descriptor determination unit 340 can determine the representative 3D key points and representative descriptors of multiple clusters generated based on the re-clustering of local features.

[0157] The descriptor determination unit 340 can determine the representative 3D key points of the clusters before re-clustering as the representative 3D key points of the re-clustered clusters.

[0158] That is, when the first cluster is divided into a second cluster and a third cluster by the re-clustering of the descriptor clustering unit 330, the representative 3D key points of the first cluster can be determined as the representative 3D key points of the second cluster and the representative 3D key points of the third cluster.

[0159] In addition, the representative descriptor determination unit 340 can determine the representative descriptor of the re-clustered cluster by taking individual descriptors of the local features contained in the re-clustered cluster.

[0160] According to an embodiment, the representative descriptor determination unit 340 can determine the average of the individual descriptors of the local features contained in the re-clustered clusters as the representative descriptor of the re-clustered clusters.

[0161] Alternatively, according to another embodiment, the representative descriptor determination unit 340 may calculate the average value of individual descriptors of local features contained in the re-clustered clusters, and determine the descriptor closest to the average value among the individual descriptors of local features contained in the re-clustered clusters as the representative descriptor of the re-clustered clusters.

[0162] The positioning database storage unit 350 can store the representative 3D key points and representative descriptors of the re-clustered clusters determined by the representative descriptor determination unit 340 in the positioning database 20.

[0163] That is, the positioning database storage unit 350 can store the global features of the key frame and the ID of the cluster corresponding to each local feature contained in the key frame for each key frame, and store the representative 3D key point and representative descriptor for each cluster.

[0164] Therefore, in the past, the global features, multiple local features, and descriptors of multiple local features of each keyframe were stored in the database for each keyframe, which required a considerable amount of database capacity for localization. However, according to the method of the embodiment of the present invention, the representative 3D keypoints and representative descriptors are stored in the localization database 20 according to each cluster, which can save the amount of database capacity required for localization.

[0165] Each flowchart of this disclosure can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the steps of the flowchart. These computer program instructions can also be stored in a computer-usable or computer-readable medium that can direct the computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-usable or computer-readable medium can produce an article of writing that includes instructions for implementing the functions specified in the boxes of the flowchart. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide processing for implementing the functions specified in the boxes of the flowchart.

[0166] Each step in the flowchart can represent a module, code segment, or code section comprising one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the boxes may not occur in the order shown in the figures. For example, two boxes shown consecutively may actually execute substantially simultaneously, or the two boxes may sometimes execute in reverse order depending on the functions involved.

[0167] The above description is merely an exemplary description of the technical scope of this disclosure, and those skilled in the art will understand that various changes and modifications can be made without departing from the original characteristics of this disclosure. Therefore, the embodiments disclosed in this disclosure are intended to illustrate, not limit, the technical scope of this disclosure, and the technical scope of this disclosure is not limited by the embodiments. The scope of protection of this disclosure should be interpreted based on the appended claims, and it should be understood that all technical scopes included within their equivalents are included within the scope of protection of this disclosure.

Claims

1. A method for constructing a location database, the method being performed by a location database construction device, the method comprising the following steps: Extract multiple local features from multiple keyframes of the designated area; Identify the individual 3D keypoints that contain information related to the 3D position for each of the aforementioned local features; Based on the aforementioned individual 3D key points, the aforementioned multiple local features are clustered into multiple clusters; By utilizing individual 3D keypoints of the local features contained in each of the aforementioned clusters, representative positional information representing the positions of each of the aforementioned clusters is determined; and For each of the aforementioned keyframes, the cluster identifiers of the various clusters and the representative location information of each cluster are respectively stored in the aforementioned positioning database. The local features contained in each of the aforementioned keyframes correspond to a cluster. The steps to cluster the above-mentioned multiple local features into multiple clusters include the following steps: Identify the 2D keypoints that indicate the 2D positions of each of the aforementioned local features; Using the 2D keypoints of each of the aforementioned local features, determine the individual 3D keypoints of each of the aforementioned local features and the covariance of the individual 3D keypoints; and By utilizing the covariance of individual 3D keypoints of each of the aforementioned local features, these local features are clustered into multiple clusters. The above representative location information includes: Representing 3D keypoints, indicating 3D keypoints associated with each of the aforementioned clusters, these representative 3D keypoints are based on individual 3D keypoints of the aforementioned local features; and A representative descriptor, which indicates the descriptor associated with each of the aforementioned clusters, is based on the individual descriptors of the aforementioned local features.

2. The method for constructing a location database according to claim 1, wherein, In the step of clustering the above-mentioned multiple local features into multiple clusters, based on the individual 3D key points of each of the above-mentioned multiple local features, the local features representing the same three-dimensional space points within the above-defined region are clustered among the above-mentioned multiple local features.

3. The method for constructing a location database according to claim 1, wherein, It also includes the following steps: Individual descriptors are identified, which are used to distinguish the local features contained in each of the multiple clusters from other local features.

4. The method for constructing a location database according to claim 1, wherein, The steps to cluster the above-mentioned multiple local features into multiple clusters include the following steps: For the first local feature among the aforementioned multiple local features, if the first cluster among the generated clusters satisfies the preset clustering conditions, the first local feature is included in the first cluster. The clustering conditions mentioned above include at least one of the following: the distance between an individual 3D keypoint of the first local feature and the representative keypoint of the first cluster; the parallax angle between the keyframes corresponding to each local feature included in the first cluster and the keyframes including the first local feature; and the epipolar distance between the local feature included in the first cluster and the first local feature.

5. The method for constructing a location database according to claim 4, wherein, The step of clustering the above-mentioned multiple local features into multiple clusters also includes the following steps: For the second local feature among the above-mentioned multiple local features, if there is no cluster that satisfies the above-mentioned preset clustering conditions in the above-mentioned one or more clusters, a second cluster including the above-mentioned second local feature is generated.

6. The method for constructing a location database according to claim 1, wherein, The steps to cluster the above-mentioned multiple local features into multiple clusters include the following steps: Based on the individual 3D key points of each of the above-mentioned local features, the above-mentioned local features are clustered into at least one cluster; and Based on the individual descriptors of the aforementioned local features, the at least one cluster is re-clustered into the aforementioned multiple clusters.

7. The method for constructing a location database according to claim 6, wherein, The steps of re-clustering at least one of the above clusters into multiple clusters include the following steps: Based on the results of applying the individual descriptors of each of the aforementioned local features to preset clustering conditions, the at least one cluster is re-clustered into the aforementioned multiple clusters. The clustering conditions mentioned above include at least one of the following: the distance between the descriptors of each of the above local features and the representative descriptor of the above at least one cluster; and the number of local features contained in the above at least one cluster.

8. The method for constructing a location database according to claim 7, wherein, The step of re-clustering at least one of the above clusters into multiple clusters further includes the following steps: Generate a list of at least one of the above clusters; If the local features contained in the first cluster of at least one of the above clusters satisfy the above clustering conditions, the local features contained in the first cluster are re-clustered into multiple clusters; and Remove the first cluster from the list above, and add the multiple clusters that have been re-clustered to the list above.

9. The method for constructing a location database according to claim 1, wherein, The aforementioned location database includes a keyframe database and a cluster database. The aforementioned keyframe database stores the global features of each of the aforementioned keyframes and the cluster IDs corresponding to the various local features contained in each of the aforementioned keyframes. The aforementioned cluster database stores representative 3D keypoints and representative descriptors for each of the aforementioned clusters.

10. A location database construction apparatus, comprising: A memory that stores a location database building program that generates data to be built into the aforementioned location database; and The processor loads the location database construction program from the aforementioned memory. The processor described above executes the location database construction program described above. Multiple local features are extracted from multiple keyframes of the designated area. Identify the individual 3D keypoints that contain information related to the 3D position for each of the aforementioned local features. Based on the aforementioned individual 3D key points, the aforementioned multiple local features are clustered into multiple clusters. By utilizing individual 3D keypoints of the local features contained in each of the aforementioned clusters, representative positional information that represents the position of each of the aforementioned clusters is determined. For each of the aforementioned keyframes, the cluster identifiers of the various clusters and the representative location information of each cluster are stored in the aforementioned positioning database. Each local feature contained in each of the aforementioned keyframes corresponds to a cluster. The processor identifies 2D keypoints indicating the 2D locations of the various local features. Using the 2D keypoints of each of the aforementioned local features, we determine the individual 3D keypoints of each of the aforementioned local features and the covariance of each individual 3D keypoint. By utilizing the covariance of individual 3D keypoints of each of the aforementioned local features, these local features are clustered into multiple clusters. The above representative location information includes: Representing 3D keypoints, indicating 3D keypoints associated with each of the aforementioned clusters, these representative 3D keypoints are based on individual 3D keypoints of the aforementioned local features; and A representative descriptor, which indicates the descriptor associated with each of the aforementioned clusters, is based on the individual descriptors of the aforementioned local features.

11. The location database construction apparatus according to claim 10, wherein, Based on the individual 3D key points of each of the aforementioned local features, the processor clusters the local features representing the same three-dimensional space within the aforementioned specified region.

12. The location database construction apparatus according to claim 10, wherein, The processor also identifies individual descriptors used to distinguish local features contained in each of the multiple clusters from other local features.

13. The location database construction apparatus according to claim 10, wherein, The processor described above clusters the aforementioned local features into at least one cluster based on the individual 3D key points of each of the multiple local features. Based on the individual descriptors of the aforementioned local features, the at least one cluster is re-clustered into the aforementioned multiple clusters.

14. The location database construction apparatus according to claim 13, wherein, The processor re-clusters the at least one cluster into the multiple clusters based on the result of applying the individual descriptors of each of the multiple local features to preset clustering conditions. The clustering conditions mentioned above include at least one of the following: the distance between the descriptors of each of the above local features and the representative descriptor of the above at least one cluster; and the number of local features contained in the above at least one cluster.

15. The location database construction apparatus according to claim 14, wherein, The processor generates a list of at least one of the aforementioned clusters. If the local features contained in the first cluster of at least one of the above clusters satisfy the above clustering conditions, the local features contained in the first cluster are re-clustered into multiple clusters. Remove the first cluster from the list above, and add the multiple clusters that have been re-clustered to the list above.

16. The location database construction apparatus according to claim 10, wherein, The aforementioned location database includes a keyframe database and a cluster database. The aforementioned keyframe database stores the global features of each of the aforementioned keyframes and the cluster IDs corresponding to the various local features contained in each of the aforementioned keyframes. The aforementioned cluster database stores representative 3D keypoints and representative descriptors for each of the aforementioned clusters.

Images