Method of constructing positioning db using clustering of local features and apparatus for constructing positioning db

Abstract

A method for constructing a positioning DB performed by a positioning DB construction device according to an embodiment of the present invention may include: a step of clustering a plurality of local features into a plurality of clusters based on individual 3D key points of each of a plurality of local features extracted from a plurality of key frames that have captured a predetermined area; a step of determining a representative 3D key point and a representative descriptor of each of the plurality of clusters using individual 3D key points and individual descriptors of local features included in each of the plurality of clusters; and a step of storing, for each of the plurality of key frames, the ID of the cluster, the representative 3D key point, and the representative descriptor corresponding to each of the local features included in each of the plurality of key frames in the positioning DB.

Description

Technology Field

[0001] The present invention relates to a method for constructing a positioning database using clustering of local features and a device for constructing a positioning database that performs the same. Background Technology

[0002] Methods for performing positioning in shadowed areas where GPS signals are weak include radio wave-based positioning, camera-based positioning, and LiDAR-based positioning.

[0003] Here, radio wave-based positioning includes the Cell ID method, which is a positioning method based on base station / AP location and signal strength; the TDoA / AoA method, which is a positioning method based on the difference in arrival time or angle of incidence of the signal; and the Fingerprinting method, which is a positioning method based on grid-unit signal pattern matching.

[0004] Among radio wave-based positioning methods, camera-based positioning has relatively high positioning accuracy, but it has the disadvantage of requiring a large amount of computation for positioning and the need to build a 3D map with a large data capacity in advance.

[0005] Therefore, in order to more easily use camera-based positioning with high positioning accuracy, it is necessary to seek a method to reduce the data capacity of the 3D map. The problem to be solved

[0006] The problem that the present invention aims to solve is to provide a method for reducing the size of a positioning database by constructing a positioning database using clustering of local features.

[0007] However, the problems that the present invention aims to solve are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below. means of solving the problem

[0008] A method for constructing a positioning DB performed by a positioning DB construction device according to an embodiment of the present invention may include: a step of clustering a plurality of local features into a plurality of clusters based on individual 3D key points of each of a plurality of local features extracted from a plurality of key frames that have captured a predetermined area; a step of determining a representative 3D key point and a representative descriptor of each of the plurality of clusters using individual 3D key points and individual descriptors of local features included in each of the plurality of clusters; and a step of storing, for each of the plurality of key frames, the ID of the cluster, the representative 3D key point, and the representative descriptor corresponding to each of the local features included in each of the plurality of key frames in the positioning DB.

[0009] The step of clustering the plurality of local features into a plurality of clusters can cluster local features among the plurality of local features that point to the same point in three-dimensional space within the predetermined area, based on each individual 3D key point of the plurality of local features.

[0010] The step of clustering the plurality of local features into a plurality of clusters may include: determining the individual 3D keypoint of each of the plurality of local features and the covariance of the individual 3D keypoint using the 2D keypoint of each of the plurality of local features; and clustering the plurality of local features into at least one cluster using the covariance of the individual 3D keypoint of each of the plurality of local features.

[0011] The step of clustering the plurality of local features into a plurality of clusters includes, for a first local feature among the plurality of local features, a step of including the first local feature in the first cluster when the first cluster among one or more generated clusters satisfies a pre-set clustering condition, and the clustering condition may include at least one of the distance between an individual 3D key point of the first local feature and a representative key point of the first cluster, a parallax angle between a keyframe corresponding to each of the local features included in the first cluster and a keyframe including the first local feature, and an epipolar distance between the local features included in the first cluster and the first local feature.

[0012] The step of clustering the plurality of local features into a plurality of clusters may further include the step of generating a second cluster including the second local feature when, for the second local feature among the plurality of local features, there is no cluster satisfying the previously set clustering condition among the one or more clusters generated.

[0013] The step of clustering the plurality of local features into a plurality of clusters may include: clustering the plurality of local features into at least one cluster based on individual 3D keypoints of each of the plurality of local features; and re-clustering the at least one cluster into the plurality of clusters based on individual descriptors of each of the plurality of local features.

[0014] The step of re-clustering the at least one cluster into the plurality of clusters includes the step of re-clustering the at least one cluster into the plurality of clusters based on the result of applying an individual descriptor of each of the plurality of local features to a pre-set clustering condition, and the clustering condition may include at least one of the distance between the descriptor of each of the plurality of local features and the representative descriptor of the at least one cluster and the number of local features included in the at least one cluster.

[0015] The step of re-clustering the at least one cluster into the plurality of clusters may further include: generating a list for the at least one cluster; re-clustering the local features included in the first cluster into the plurality of clusters when the local features included in the first cluster among the at least one clusters satisfy the clustering condition; and deleting the first cluster from the list and adding the re-clustered plurality of clusters to the list.

[0016] The positioning DB above includes a keyframe DB and a cluster DB, and the keyframe DB stores the global features of each of the plurality of keyframes and the ID of a cluster corresponding to each of the local features included in each of the plurality of keyframes, and the cluster DB may store a representative 3D keypoint and a representative descriptor of each of the plurality of clusters.

[0017] A positioning DB construction device for constructing a positioning DB according to another embodiment of the present invention includes: a memory in which a positioning DB construction program for generating data to be constructed in the positioning DB is stored; and a processor for loading the positioning DB construction program from the memory. The processor executes the positioning DB construction program to cluster the plurality of local features into a plurality of clusters based on individual 3D key points of each of the plurality of local features extracted from a plurality of key frames that have captured a predetermined area, and determines a representative 3D key point and a representative descriptor of each of the plurality of clusters using individual 3D key points and individual descriptors of the local features included in each of the plurality of clusters, and for each of the plurality of key frames, stores the ID of the cluster, the representative 3D key point, and the representative descriptor corresponding to each of the local features included in each of the plurality of key frames in the positioning DB.

[0018] A computer-readable recording medium storing a computer program according to another embodiment of the present invention may include: a step of clustering a plurality of local features into a plurality of clusters based on individual 3D key points of each of a plurality of local features extracted from a plurality of key frames that have captured a predetermined area; a step of determining a representative 3D key point and a representative descriptor of each of the plurality of clusters using individual 3D key points and individual descriptors of local features included in each of the plurality of clusters; and a step of storing, for each of the plurality of key frames, the ID of the cluster, the representative 3D key point, and the representative descriptor corresponding to each of the local features included in each of the plurality of key frames in the positioning DB, and may include an instruction for a processor to perform a method of constructing a positioning DB performed by a positioning DB construction device. Effects of the invention

[0019] According to an embodiment of the present invention, since only representative 3D keypoints and representative descriptors are stored for each cluster of local features in the positioning DB, the capacity of the DB required for positioning can be saved. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram showing a positioning system according to an embodiment of the present invention. Figure 2 shows an example of a DB for conventional camera-based positioning. Figure 3 shows an example of a positioning DB according to an embodiment of the present invention. FIG. 4 is a block diagram conceptually illustrating the functions of a positioning DB construction program according to an embodiment of the present invention. FIG. 5 illustrates a method for determining the covariance of 3D keypoints of local features according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a method in which a local feature clustering unit according to an embodiment of the present invention clusters local features. FIG. 7 is a flowchart illustrating a method in which a descriptor clustering unit according to an embodiment of the present invention re-clusters local features based on condition 1 among the second clustering conditions. FIG. 8 is a flowchart illustrating a method in which a descriptor clustering unit according to an embodiment of the present invention re-clusters local features based on condition 2 among the second clustering conditions. FIG. 9 is a flowchart illustrating a method in which a descriptor clustering unit according to an embodiment of the present invention re-clusters local features based on condition 3 among the second clustering conditions. Specific details for implementing the invention

[0021] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0022] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined considering their functions in the embodiments of the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0023] FIG. 1 is a block diagram showing a positioning system according to an embodiment of the present invention.

[0024] Referring to FIG. 1, the positioning system (10) may include a positioning DB (20), a positioning DB construction device (100), and a positioning device (200).

[0025] In this specification, for convenience of explanation, the positioning system (10) is described as including a positioning DB (20), a positioning DB construction device (100), and a positioning device (200), but is not limited thereto.

[0026] According to an embodiment, the positioning system (10) may include a positioning DB (20) and a positioning DB construction device (100), wherein the positioning system (10) may be a system for constructing the positioning DB (20). Additionally, according to another embodiment, the positioning system (10) may include a positioning DB (20) and a positioning device (200), wherein the positioning system (10) may be a system for performing positioning using the previously constructed positioning DB (20).

[0027] The positioning DB (20) can store data for the positioning device (200) to determine the position.

[0028] In a database for conventional camera-based positioning, a global feature, at least one local feature, a descriptor for each of the at least one local feature, and 3D coordinates for each of the at least one local feature can be stored for each keyframe.

[0029] For example, referring further to FIG. 2, FIG. 2 shows an example of a DB for conventional camera-based positioning.

[0030] In the case where the global feature of the first keyframe (KF1) is the first global feature and the first keyframe (KF1) includes the first local feature, the second local feature, the third local feature, and the fourth local feature, the DB (50) for conventional camera-based positioning may store, for the first keyframe (KF1), i) the first global feature as the global feature, ii) the first local feature, the second local feature, the third local feature, and the fourth local feature as the local features included in the first keyframe, iii) the first descriptor, the second descriptor, the third descriptor, and the fourth descriptor as the descriptors for each of the first local feature, the second local feature, the third local feature, and the fourth local feature, and iv) the first 3D coordinate, the second 3D coordinate, the third 3D coordinate, and the fourth 3D coordinate as the 3D coordinates for each of the first local feature, the second local feature, the third local feature, and the fourth local feature.

[0031] On the other hand, the positioning DB (20) according to the embodiment of the present invention may include global features and at least one cluster ID for each keyframe, and representative 3D keypoints and representative descriptors may be mapped for each cluster ID.

[0032] Accordingly, according to the embodiment, the positioning DB (20) may include a keyframe DB (not shown) in which global features per keyframe and the ID of at least one cluster are stored, and a cluster DB (not shown) in which representative 3D keypoints and representative descriptors per cluster ID are stored.

[0033] For example, referring further to FIG. 3, FIG. 3 shows an example of a positioning DB according to an embodiment of the present invention.

[0034] The positioning DB (20) may include a keyframe DB (22) and a cluster DB (24).

[0035] In the keyframe DB (22), for each keyframe, the global features of each keyframe and the IDs of the clusters included in each keyframe can be stored.

[0036] The global feature of the second keyframe (KF2) is the second global feature, and when the second keyframe (KF2) includes the first cluster, the third cluster, and the sixth cluster, the second global feature as the global feature and the ID of the first cluster, the ID of the third cluster, and the ID of the sixth cluster as the ID of the cluster included in the second keyframe (KF2) may be stored in the keyframe DB (22) for the second keyframe (KF2).

[0037] The cluster DB (24) may store representative 3D keypoints and representative descriptors for all clusters included in the keyframes used for camera-based positioning.

[0038] When the keyframes used for camera-based positioning include the first cluster, the second cluster, the third cluster, the fourth cluster, the fifth cluster, and the sixth cluster, the cluster DB (24) includes i) IDs of the clusters included in the keyframes, such as the ID of the first cluster, the ID of the second cluster, the ID of the third cluster, the ID of the fourth cluster, the ID of the fifth cluster, and the ID of the sixth cluster; ii) representative 3D keypoints of each of the first cluster, the second cluster, the third cluster, the fourth cluster, the fifth cluster, and the sixth cluster, such as the first representative 3D keypoint, the second representative 3D keypoint, the third representative 3D keypoint, the fourth representative 3D keypoint, the fifth representative 3D keypoint, and the sixth representative 3D keypoint; and iii) representative descriptors of each of the first cluster, the second cluster, the third cluster, the fourth cluster, the fifth cluster, and the sixth cluster, such as the first representative descriptor, the second representative descriptor, the third representative descriptor, the fourth representative descriptor, the fifth representative descriptor, and the sixth representative The descriptor can be saved.

[0039] As such, compared to a conventional camera-based positioning DB, the positioning DB (20) according to the embodiment of the present invention does not store local features and descriptors for local features per keyframe, so there is an advantage of saving DB storage space.

[0040] The positioning DB construction device (100) can store data used by the positioning device (200) to determine its own location in the positioning DB (20).

[0041] The positioning DB construction device (100) can determine global features per keyframe of a location to be built for positioning DB (20), and representative 3D keypoints and representative descriptors per ID of at least one cluster included in the keyframe.

[0042] To this end, the positioning DB construction device (100) may include a processor (110), a transceiver (120), and a memory (130).

[0043] The processor (110) can control the overall operation of the positioning DB construction device (100).

[0044] The processor (110) can use the transceiver (120) to transmit data to the positioning DB (20) for building the positioning DB (20).

[0045] In this specification, the positioning DB construction device (100) is described as a separate device distinct from the positioning DB (20), but is not limited thereto. That is, according to an embodiment, the positioning DB construction device (100) may include the positioning DB (20), and in such case, data for constructing the positioning DB (20) can be transmitted to the positioning DB (20) through internal signaling.

[0046] The positioning DB construction device (100) can receive a key frame of a location to be built for positioning DB (20) by using a transceiver (120).

[0047] Alternatively, according to an embodiment, the positioning DB construction device (100) may further include a camera (not shown), and the positioning DB construction device (100) may capture a keyframe using the camera.

[0048] The memory (130) can store information necessary for the execution of the positioning DB construction program (300) and the positioning DB construction program (300).

[0049] In the present specification, the positioning DB construction program (300) may refer to software that includes instructions programmed to generate data for constructing the positioning DB (20).

[0050] The processor (110) can load the positioning DB construction program (300) and information necessary for the execution of the positioning DB construction program (300) from memory (130) in order to execute the positioning DB construction program (300).

[0051] The processor (110) can execute a positioning DB construction program (300) to generate data for constructing a positioning DB (20) using a keyframe that captures the location where the positioning DB (20) is to be constructed.

[0052] The functions and / or operations of the positioning DB construction program (300) will be examined in detail through FIG. 2.

[0053] Additionally, the positioning device (200) can determine its own location using data stored in the positioning DB (20).

[0054] To this end, the positioning device (200) may include a processor (210), a transceiver (220), and a memory (230).

[0055] The processor (210) can control the overall operation of the positioning device (200).

[0056] The processor (210) can receive data for determining the location of the positioning device (200) from the positioning DB (20) using the transceiver (220).

[0057] The memory (230) can store a positioning program (240) including instructions programmed to determine the position of the positioning device (200) and information required for the execution of the positioning program (240).

[0058] The processor (210) can load the positioning program (240) and information necessary for the execution of the positioning program (240) from memory (230) in order to execute the positioning program (240).

[0059] The processor (210) can determine its position by executing a positioning program (240).

[0060] To this end, according to an embodiment, the positioning device (200) may further include a camera (not shown). That is, the positioning device (200) can extract global features and local features from an image captured using a camera (not shown), and select similar keyframes by comparing the global features of the captured image with the global features of keyframes stored in the positioning DB (20). Subsequently, the positioning device (200) can determine its position by comparing the local features and descriptors extracted from the captured image with the representative 3D keypoint and representative descriptor of each cluster included in the selected keyframes.

[0061] FIG. 4 is a block diagram conceptually illustrating the functions of a positioning DB construction program according to an embodiment of the present invention.

[0062] Referring to FIGS. 1 and 4, the positioning DB construction program (300) may include a local feature extraction unit (310), a local feature clustering unit (320), a descriptor clustering unit (330), a representative descriptor determination unit (340), and a positioning DB storage unit (350).

[0063] The local feature extraction unit (310), local feature clustering unit (320), descriptor clustering unit (330), representative descriptor determination unit (340), and positioning DB storage unit (350) illustrated in FIG. 4 are conceptual divisions of the functions of the positioning DB construction program (300) to easily explain the functions of the positioning DB construction program (300), but are not limited thereto. According to embodiments, the functions of the local feature extraction unit (310), local feature clustering unit (320), descriptor clustering unit (330), representative descriptor determination unit (340), and positioning DB storage unit (350) can be merged or separated and may be implemented as a series of commands included in a single program.

[0064] The local feature extraction unit (310) can extract multiple local features from a keyframe obtained using a transceiver (120) or a camera (not shown).

[0065] The above local feature may include a 2D keypoint and individual descriptors. The 2D keypoint refers to a 2D coordinate on the keyframe of the local feature, and the individual descriptor is used to distinguish the local feature from other local features included in the keyframe and may refer to a multidimensional vector in which the correlation between the local feature and pixels surrounding the local feature is expressed as a vector.

[0066] According to an embodiment, the local feature extraction unit (310) may include a neural network that has been trained to extract local features from a keyframe, and the local feature extraction unit (310) may input the keyframe into the neural network to output local features.

[0067] Alternatively, according to an embodiment, the local feature extraction unit (310) may extract a plurality of local features from a keyframe using methods such as SuperPoint, R2D2, SIFT, etc.

[0068] The local feature clustering unit (320) can cluster local features that point to (represent) a point in the same 3D space within the specific area among the local features included in each of the multiple keyframes, and determine a representative 3D key point of the local feature cluster.

[0069] Here, multiple keyframes each refer to images of the same area captured at different locations, and 3D keypoints may refer to 3D coordinates (e.g., absolute coordinates) of a 3D space within the specific area.

[0070] First, the local feature clustering unit (320) can cluster local features that point to the same point in 3D space among the local features included in each of the multiple keyframes based on individual 3D keypoints of local features included in each of the multiple keyframes.

[0071] More specifically, the local feature clustering unit (320) can compare individual 3D keypoints of local features included in each of the multiple keyframes and cluster local features whose distance between individual 3D keypoints is less than or equal to a preset standard range into one cluster.

[0072] To this end, the local feature clustering unit (320) can determine individual 3D keypoints of each local feature included in multiple keyframes by using 2D keypoints of local features included in multiple keyframes and pose information of multiple keyframes. The pose information includes the absolute position and direction of the camera that captured the keyframe and can be expressed using 6 Degrees of Freedom (DOF).

[0073] According to an embodiment, if the pose information of a plurality of keyframes is not determined, the local feature clustering unit (320) can determine the relative pose information of a plurality of keyframes by using 2D keypoints of local features included in the plurality of keyframes and matching between the plurality of keyframes.

[0074] The local feature clustering unit (320) can determine individual 3D keypoints of local features included in multiple keyframes by using a 3D direction vector transformed from a 2D keypoint, the camera's instinct information, and relative pose information of multiple keyframes. For example, the local feature clustering unit (320) can determine the relative pose information of multiple keyframes by using the Structure-from-Motion (SfM) method.

[0075] That is, when the local feature clustering unit (320) calculates 3D keypoints of each local feature using 2D keypoints of local features included in the plurality of keyframes, the pose information of the keyframes may be generated differently depending on the type or settings of the camera that captured the keyframes. Therefore, in order to correct the differences between cameras, the local feature clustering unit (320) may also calculate 3D keypoints of each local feature by using the camera's instinct information.

[0076] According to another embodiment, when pose information of a plurality of keyframes is determined, the local feature clustering unit (320) can determine individual 3D keypoints of each local feature using the determined pose information of the plurality of keyframes.

[0077] For example, when the plurality of keyframes are captured using a LiDAR sensor and a camera together, the local feature clustering unit (320) can obtain the camera's pose information through a Simultaneous Localization and Mapping (SLAM) technique and can determine individual 3D keypoints of local features using the camera's pose information and a 3D point cloud.

[0078] The local feature clustering unit (320) can calculate the covariance of individual 3D key points using 2D key points of local features to calculate the distance between individual 3D key points.

[0079] Referring further to FIG. 5, the local feature clustering unit (320) can determine a projection point (Pc) by projecting the 2D key point (P'c) of the local feature by a depth (D) using the 2D key point (P'c) of the local feature and the depth (D) of the 2D key point (P'c) in 3D space (W).

[0080] The local feature clustering unit (320) converts the global coordinate system into a camera coordinate system using the W2C (World-to-Camera) transformation (R W2C , t W2C By applying ) to the projection point (Pc), individual 3D keypoints (Pw) of the local feature can be determined.

[0081] Here, the projection point (Pc) and the individual 3D keypoint (Pw) refer to the same point, where the projection point (Pc) refers to coordinates expressed in the camera coordinate system, and the individual 3D keypoint (Pw) refers to coordinates expressed in the global coordinate system.

[0082] Subsequently, the local feature clustering unit (320) has a depth (D) of the 2D keypoint (P'c) and a variance (σ) of the depth (D) of the local feature. D 2), 2D keypoint of local features (P'c), covariance of 2D keypoint of local features (C P'c Using ), the covariance (C) of the projection point (Pc) Pc ) can be determined.

[0083] The local feature clustering unit (320) is a W2C transformation (R W2C The covariance (C) of the ) value and the projection point (Pc) Pc Using ), the covariance (C) of individual 3D keypoints (Pw) Pw ) can be determined.

[0084] For example, the local feature clustering unit (320) applies the following mathematical formula 1 to the covariance (C) of individual 3D keypoints (Pw). Pw ) can be determined.

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Here, u represents the x-coordinate value of the 2D keypoint (P'c) of the local feature, and v represents the y-coordinate value of the 2D keypoint (P'c) of the local feature.

[0091] Referring further to FIG. 6, FIG. 6 is a flowchart illustrating how a local feature clustering unit (320) clusters local features.

[0092] In order to sequentially cluster all local features included in multiple keyframes, the local feature clustering unit (320) may include the first keyframe, which is the first keyframe to perform clustering, in the first cluster (S600).

[0093] Afterwards, the local feature clustering unit (320) determines whether there are any remaining local features that have not been clustered (S610), and if there are any second local features that have not been clustered (S610 'yes'), it can determine whether there are any clusters among the previously generated clusters that satisfy a predetermined first clustering condition for the second local features that are the subject of determination (S620).

[0094] The above first clustering condition may include the following conditions 1 to 3.

[0095] - Condition 1: The distance between the cluster's common 3D keypoint and the 3D keypoint of the second local feature being evaluated is below the threshold.

[0096] - Condition 2: The parallax angle of keyframes within the cluster and keyframes containing second local features is below the threshold.

[0097] - Condition 3: The epipolar distance between local features within the cluster and the second local feature is below the threshold.

[0098] According to the example, the distance of condition 1 is the Mahalanobis distance and can be calculated as shown in Equation 2 below.

[0099]

[0100] Here, P1 represents the common 3D keypoint of the cluster, P2 represents the 3D keypoint of the second local feature, Cov1 represents the covariance of the common 3D keypoint of the cluster, and Cov2 represents the covariance of the second local feature.

[0101] According to an embodiment, if the second local feature does not satisfy any of conditions 1 to 3 for the cluster, the local feature clustering unit (320) does not determine whether other conditions are satisfied and can determine that the second local feature does not satisfy the first clustering condition for the cluster.

[0102] If there is a cluster that satisfies the first clustering condition for the second local feature (S620 'yes'), the local feature clustering unit (320) can include the second local feature in the corresponding cluster (S630).

[0103] On the other hand, if there is no cluster satisfying the first clustering condition for the second local feature ('No' in S620), the local feature clustering unit (320) can create a new cluster and include the second local feature in the cluster that created it (S640).

[0104] If there are no remaining local features ('No' in S610), the local feature clustering unit (320) may terminate the clustering of local features (S650). That is, the local feature clustering unit (320) may repeat the above processes (S610 to S640) until there are no remaining local features, thereby including local features in existing clusters or creating new clusters.

[0105] The descriptor clustering unit (330) can re-cluster local features included in the same cluster based on individual descriptors of local features.

[0106] This is because, even if local features point to points in the same 3D space, if the locations where keyframes were taken are far apart, the individual descriptors of the local features may differ significantly, and when merging local features with different individual descriptors, the performance of the merger may be degraded.

[0107] Accordingly, the descriptor clustering unit (330) can cluster local features that satisfy the second clustering condition among the local features included in the same cluster.

[0108] The above second clustering condition can be one of the following three.

[0109] - Condition 1: The distance between the individual descriptors of local features and the representative descriptor of the cluster is below the threshold

[0110] - Condition 2: The number of local features included in the cluster is less than or equal to the threshold value

[0111] - Condition 3: The distance from the cluster's representative descriptor is below the threshold, and the number of local features included in the cluster is below the reference value.

[0112] Here, a representative descriptor is a descriptor representing a cluster containing local features, and, depending on the embodiment, may be the average of the clusters of local features.

[0113] Hereinafter, a method for re-clustering local features by the descriptor clustering unit (330) based on any one of conditions 1 to 3 of the second clustering condition will be described.

[0114] First, referring further to FIG. 7, FIG. 7 is a flowchart illustrating a method in which a descriptor clustering unit (330) re-clusters local features based on condition 1 among the second clustering conditions.

[0115] The descriptor clustering unit (330) can generate a list of clusters generated by the local feature clustering unit (320) to re-cluster local features (S700).

[0116] The descriptor clustering unit (330) determines whether there are any remaining clusters in the generated list (S710), and if the first cluster exists as a remaining cluster ('yes' in S710), it can determine whether the first cluster satisfies the first condition among the second clustering conditions (S720).

[0117] That is, the description clustering unit (330) can re-cluster local features among the local features included in the first cluster, where the distance between the individual descriptor of the local feature and the representative descriptor of the first cluster is greater than (greater than) a reference value.

[0118] Among the local features included in the first cluster, if there are local features where the distance between the individual descriptor of the local feature and the representative descriptor of the first cluster exceeds (greater than) a reference value (e.g., 'yes' in S720), the descriptor clustering unit (330) can re-cluster the local features included in the first cluster into two or more clusters (S730).

[0119] According to an embodiment, the descriptor clustering unit (330) can re-cluster local features included in the first cluster into two or more clusters using a k-means clustering algorithm.

[0120] Alternatively, according to an embodiment, the description clustering unit (330) may re-cluster local features among the local features included in the first cluster into a second cluster where the distance between an individual descriptor and a representative descriptor is less than or equal to a reference value, and re-cluster local features where the distance between an individual descriptor and a representative descriptor of the cluster is greater than or equal to a reference value into a third cluster.

[0121] Afterwards, the descriptor clustering unit (330) can delete the first cluster from the list and add the re-clustered clusters to the list (S740).

[0122] However, according to the embodiment, when local features where the distance between an individual descriptor and a representative descriptor is less than or equal to a reference value are re-clustered into a second cluster, and local features where the distance between an individual descriptor and a representative descriptor of a cluster is greater than or equal to a reference value are re-clustered into a third cluster, the descriptor clustering unit (330) may not add the second cluster to the list, but may add only the third cluster to the list.

[0123] On the other hand, if the distance between the individual descriptors of local features included in the first cluster and the representative descriptor of the first cluster is less than or equal to a reference value ('No' in S720), the descriptor clustering unit (330) can remove the first cluster from the list (S750).

[0124] If no remaining clusters exist ('No' in S710), the descriptor clustering unit (330) may terminate the re-clustering of local features (S760). That is, the descriptor clustering unit (330) may re-cluster local features by repeating the above processes (S710 to S750) until no remaining clusters exist.

[0125] Also, referring further to FIG. 8, FIG. 8 is a flowchart illustrating a method in which a descriptor clustering unit (330) re-clusters local features based on condition 2 among the second clustering conditions.

[0126] The descriptor clustering unit (330) can generate a list of clusters generated by the local feature clustering unit (320) to re-cluster local features (S800).

[0127] The descriptor clustering unit (330) determines whether there are any remaining clusters in the generated list (S810), and if the first cluster exists as a remaining cluster ('yes' in S810), it can determine whether the first cluster satisfies condition 2 among the second clustering conditions (S820).

[0128] If the number of local features included in the first cluster exceeds a threshold (e.g., 'Yes' in S820), the descriptor clustering unit (330) can re-cluster the local features included in the first cluster into two or more clusters (S830). According to an embodiment, the descriptor clustering unit (330) can re-cluster the local features included in the first cluster into two or more clusters using a k-means clustering algorithm.

[0129] Afterwards, the descriptor clustering unit (330) can delete the first cluster from the list and add the re-clustered clusters to the list (S840).

[0130] On the other hand, if the number of local features included in the first cluster is below a threshold ('No' in S820), the descriptor clustering unit (330) can remove the first cluster from the list (S850).

[0131] If no remaining clusters exist ('No' in S810), the descriptor clustering unit (330) may terminate the re-clustering of local features (S860). That is, the descriptor clustering unit (330) may re-cluster local features by repeating the above processes (S810 to S850) until no remaining clusters exist.

[0132] Finally, referring further to FIG. 9, FIG. 9 is a flowchart illustrating how the descriptor clustering unit (330) re-clusters local features based on condition 3 among the second clustering conditions.

[0133] The descriptor clustering unit (330) can generate a list of clusters generated by the local feature clustering unit (320) to re-cluster local features (S900).

[0134] The descriptor clustering unit (330) determines whether there are any remaining clusters in the generated list (S910), and if the first cluster exists as a remaining cluster ('yes' in S910), it can determine whether the first cluster satisfies condition 3 among the second clustering conditions (S920).

[0135] If the number of local features included in the first cluster exceeds the threshold (or is greater than or equal to the threshold) or if the number of local features included in the first cluster exceeds the threshold (or is greater than or equal to the threshold) (S920 'Yes'), the descriptor clustering unit (330) can re-cluster the local features included in the first cluster into two or more clusters (S930).

[0136] According to an embodiment, the descriptor clustering unit (330) can re-cluster local features included in the first cluster into two or more clusters using a k-means clustering algorithm.

[0137] Afterwards, the descriptor clustering unit (330) can delete the first cluster from the list and add the re-clustered clusters to the list (S940).

[0138] On the other hand, if the number of local features included in the first cluster is below a threshold ('No' in S920), the descriptor clustering unit (330) can remove the first cluster from the list (S950).

[0139] If no remaining clusters exist ('No' in S910), the descriptor clustering unit (330) may terminate the re-clustering of local features (S960). That is, the descriptor clustering unit (330) may re-cluster local features by repeating the above processes (S910 to S950) until no remaining clusters exist.

[0140] As local features are re-clustered in the descriptor clustering unit (330), multiple clusters can be generated.

[0141] The representative descriptor determination unit (340) can determine representative 3D keypoints and representative descriptors of multiple clusters generated according to the re-clustering of local features.

[0142] The representative descriptor determination unit (340) can determine the representative 3D keypoint of the cluster prior to reclustering as the representative 3D keypoint of the reclustered clusters.

[0143] That is, when the first cluster is divided into the second cluster and the third cluster according to the re-clustering of the descriptor clustering unit (330), the representative 3D key point of the first cluster can be determined as the representative 3D key point of the second cluster and the representative 3D key point of the third cluster.

[0144] Additionally, the representative descriptor determination unit (340) can determine the representative descriptor of the reclustered cluster by merging the individual descriptors of local features included in the reclustered cluster.

[0145] According to an embodiment, the representative descriptor determination unit (340) can determine the representative descriptor of the reclustered cluster by averaging the individual descriptors of local features included in the reclustered cluster.

[0146] Alternatively, according to another embodiment, the representative descriptor determination unit (340) may calculate the average value of individual descriptors of local features included in the reclustered cluster and determine the descriptor closest to the average value among the individual descriptors of local features included in the reclustered cluster as the representative descriptor of the reclustered cluster.

[0147] The positioning DB storage unit (350) can store the representative 3D keypoints and representative descriptors of the re-clustered clusters determined by the representative descriptor determination unit (340) in the positioning DB (20).

[0148] That is, the positioning DB storage unit (350) can store the ID of the cluster corresponding to each of the global features of the keyframe and the local features included in the keyframe for each keyframe, and can store the representative 3D keypoint and representative descriptor for each cluster.

[0149] Accordingly, in the conventional case, the global features of a keyframe, multiple local features, and descriptors of multiple local features are all stored in the DB for each keyframe, and the DB capacity required for positioning is considerable. However, according to the method according to the embodiment of the present invention, representative 3D keypoints and representative descriptors are stored in the positioning DB (20) for each cluster, so the DB capacity required for positioning can be saved.

[0150] Combinations of each block of the block diagram attached to the present invention and each step of the flowchart may be performed by computer program instructions. Since these computer program instructions may be loaded into an encoding processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, the instructions performed through the encoding processor of the computer or other programmable data processing equipment create means for performing the functions described in each block of the block diagram or each step of the flowchart. Since these computer program instructions may also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory may also produce a manufactured item containing instruction means for performing the function described in each block of the block diagram or each step of the flowchart. Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that execute a computer or other programmable data processing equipment by performing a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0151] Additionally, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). Also, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps described in succession may actually be performed substantially simultaneously, or the blocks or steps may be performed in reverse order according to the corresponding function.

[0152] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0153] 10: Positioning System 20: Positioning DB 100: Positioning DB Construction Device 200: Positioning device 300: Positioning DB Construction Program 310: Local Feature Extraction Unit 320: Local Feature Clustering Unit 330: Descriptor clustering part 340: Descriptor merge section 350: Positioning DB Storage Unit

Claims

Claim 1 A method for constructing a positioning database performed by a positioning database construction device comprises: a step of clustering the plurality of local features into a plurality of clusters based on individual 3D keypoints of each of the plurality of local features extracted from a plurality of keyframes capturing a predetermined area; a step of determining a representative 3D keypoint and a representative descriptor of each of the plurality of clusters using individual 3D keypoints and individual descriptors of the local features included in each of the plurality of clusters; and a step of storing, for each of the plurality of keyframes, the cluster ID, representative 3D keypoint, and representative descriptor corresponding to each of the local features included in each of the plurality of keyframes in the positioning database, wherein the step of clustering the plurality of local features into a plurality of clusters comprises a step of determining the individual 3D keypoint of each of the plurality of local features and the covariance of the individual 3D keypoint using 2D keypoints of each of the plurality of local features. A method for constructing a positioning DB comprising the step of clustering the plurality of local features into at least one cluster using the covariance of each individual 3D keypoint of the plurality of local features. Claim 2 In claim 1, the step of clustering the plurality of local features into a plurality of clusters is a method for constructing a positioning DB by clustering local features among the plurality of local features that point to the same three-dimensional space within a predetermined area, based on each individual 3D key point of the plurality of local features. Claim 3 delete Claim 4 A method for constructing a positioning DB according to claim 1, wherein the step of clustering the plurality of local features into the plurality of clusters includes, for a first local feature among the plurality of local features, the step of including the first local feature in the first cluster when the first cluster among one or more previously generated clusters satisfies a pre-set clustering condition, and the clustering condition includes at least one of the distance between an individual 3D key point of the first local feature and a representative key point of the first cluster, the parallax angle between a keyframe corresponding to each of the local features included in the first cluster and a keyframe including the first local feature, and the epipolar distance between the local features included in the first cluster and the first local feature. Claim 5 A method for constructing a positioning DB according to claim 4, wherein the step of clustering the plurality of local features into a plurality of clusters further comprises the step of generating a second cluster including the second local feature when, for a second local feature among the plurality of local features, there is no cluster satisfying the previously set clustering condition among the previously generated one or more clusters. Claim 6 A method for constructing a positioning DB according to claim 1, wherein the step of clustering the plurality of local features into a plurality of clusters comprises: a step of clustering the plurality of local features into at least one cluster based on an individual 3D key point of each of the plurality of local features; and a step of re-clustering the at least one cluster into the plurality of clusters based on an individual descriptor of each of the plurality of local features. Claim 7 In claim 6, the step of re-clustering the at least one cluster into the plurality of clusters includes the step of re-clustering the at least one cluster into the plurality of clusters based on the result of applying the individual descriptors of each of the plurality of local features to a pre-set clustering condition, wherein the clustering condition comprises at least one of the distance between the descriptor of each of the plurality of local features and the representative descriptor of the at least one cluster and the number of local features included in the at least one cluster, thereby constructing a positioning DB. Claim 8 In claim 7, the step of re-clustering the at least one cluster into the plurality of clusters further comprises: generating a list for the at least one cluster; if the local features included in the first cluster among the at least one clusters satisfy the clustering condition, re-clustering the local features included in the first cluster into the plurality of clusters; and deleting the first cluster from the list and adding the re-clustered plurality of clusters to the list, thereby constructing a positioning DB. Claim 9 A method for constructing a positioning DB according to claim 1, wherein the positioning DB includes a keyframe DB and a cluster DB, wherein the keyframe DB stores the global features of each of the plurality of keyframes and the cluster IDs corresponding to each of the local features included in each of the plurality of keyframes, and the cluster DB stores the representative 3D keypoints and representative descriptors of each of the plurality of clusters. Claim 10 A positioning DB construction device for constructing a positioning DB, comprising: a memory storing a positioning DB construction program that generates data to be constructed in the positioning DB; The method comprises a processor that loads the positioning DB construction program from the memory, wherein the processor executes the positioning DB construction program to cluster the plurality of local features into a plurality of clusters based on individual 3D key points of each of the plurality of local features extracted from a plurality of key frames capturing a predetermined area, determines a representative 3D key point and a representative descriptor of each of the plurality of clusters using the individual 3D key points and individual descriptors of the local features included in each of the plurality of clusters, and stores the cluster ID, representative 3D key point, and representative descriptor corresponding to each of the local features included in each of the plurality of key frames in the positioning DB for each of the plurality of key frames, wherein the method determines the individual 3D key points of each of the plurality of local features and the covariance of the individual 3D key points using the 2D key points of each of the plurality of local features, and clusters the plurality of local features into at least one cluster using the covariance of the individual 3D key points of each of the plurality of local features. Device. Claim 11 A computer program stored on a computer-readable recording medium, wherein the computer program comprises instructions for causing the processor to perform a method according to any one of claims 1 to 2 and claims 4 to 9 when executed by a processor.

Images