3D line group data-based visual localization method and device for executing same

Abstract

A visual localization method according to one disclosed embodiment involves: generating at least two fixed points from three-dimensional point cloud data; generating three-dimensional line group data by connecting a three-dimensional point included in the three-dimensional point cloud data and one of the generated fixed points; receiving feature points of an input image; clustering a plurality of lines included in the three-dimensional line group data on the basis of the at least two fixed points, sampling two line group clusters from among the clustered line group clusters, and estimating, on the basis of the sampled line group clusters and the feature points, the orientation of a camera capturing the input image.

Description

3D line-of-sight data-based visual localization method and device for executing the same

[0001] The present invention relates to a visual localization method based on 3D line data, which can increase the computational speed of a visual localization algorithm while preventing personal information leakage, and to a device for executing the same.

[0002] With the rise of augmented reality (AR), virtual reality (VR), mixed reality (MR), autonomous vehicles, and autonomous guided robots, driven by the Fourth Industrial Revolution, the demand for more precise user location estimation technology is increasing. Conventional global navigation satellite systems (GNSS), such as GPS, struggle with indoor applications and have margins of error, making them unsuitable for use in these industrial environments. Visual localization technology is gaining attention as a potential replacement.

[0003] Visual localization technology identifies the precise location and pose of a user's device on a spatial map based on camera images. With the widespread adoption of camera sensors, it can achieve high location accuracy at relatively low cost. Products equipped with visual localization technology transmit query images to a cloud server, compare them with the stored 3D point cloud spatial map, and estimate the user's camera position and pose. However, feature information is stored within the 3D point cloud map stored in the cloud server, so if the spatial map is leaked, the point cloud data can be synthesized (reconstructed) into a realistic image through a deep learning model (e.g., Inverse Structure-from-Motion, InvSfM). Therefore, if the 3D map stored in the cloud server is leaked, the user's sensitive personal information may be leaked.

[0004] To address these issues, a technique for estimating the position and pose of a user device was proposed (previous paper 1) by utilizing a geometrically hidden 3D spatial map, i.e., a randomly oriented 3D straight line passing through the points, instead of a 3D point cloud map. However, privacy concerns were raised again when it was discovered that this technique could also reconstruct a point cloud from randomly oriented line data (previous paper 2).

[0005] In the preceding paper 3 published in 2023, a method for generating a 3D line cloud map was proposed by introducing a method of randomly selecting two points and connecting them with a line, and it showed a higher hiding performance compared to the previously disclosed 3D uniform line cloud map.

[0006] [Previous Paper 1] P. Speciale et al., "Privacy preserving image queries for camera localization", ICCV, 2019.

[0007] [Previous Paper 2] K. Chelani et al., "How Privacy-Preserving Are Line Clouds? Recovering Scene Details From 3D Lines", ICCV, 2021.

[0008] [Previous Paper 3] C. Lee et al., "Paired-Point Lifting for Enhanced Privacy-Preserving Visual Localization", ICCV, 2023.

[0009] Meanwhile, prior art patent 1 has been disclosed for a method of estimating a camera's pose based on pre-series data. Prior art patent 1 proposes a method for generating pre-series data, specifically connecting anchor points to all 3D points. However, prior art patent 1 does not specifically disclose how to establish anchor points, and if camera pose estimation is performed using arbitrary anchor points, estimation performance deteriorates.

[0010] [Prior Patent 1] US 10,964,053 B2

[0011] In order to solve these problems, the disclosed embodiment proposes a method for setting fixed points that protects privacy information by lowering the accuracy of decomposition into point cloud data, while at the same time not lowering the performance of camera pose estimation, a method for generating point cloud data using the same, and a device for executing the same.

[0012] A visual localization method according to one disclosed embodiment includes: generating at least two fixed points from three-dimensional point cloud data; generating three-dimensional line cloud data by connecting a three-dimensional point included in the three-dimensional point cloud data with one of the generated fixed points; receiving feature points of an input image; clustering a plurality of lines included in the three-dimensional line cloud data based on the at least two fixed points, sampling two line cloud clusters among the clustered line cloud clusters, and estimating a pose of a camera that captured the input image based on the sampled line cloud clusters and the feature points.

[0013] Generating the above fixed points may include: storing a setting command for the number of the fixed points to be generated; generating a number of clusters equal to the number of the fixed points from the three-dimensional point cloud data; extracting a center point of the generated cluster; and generating the extracted center point as the fixed point.

[0014] Generating the anchor point may include: setting a three-dimensional space region in which the anchor point exists from the three-dimensional point cloud data; sampling candidate anchor points; and generating the anchor point based on whether the candidate anchor point exists within the three-dimensional space region.

[0015] Setting the three-dimensional space region may include calculating three base axes and variances for the base axes based on principal component analysis of the three-dimensional point cloud data; and setting the three-dimensional space region based on the calculated variances.

[0016] Setting the above 3D space region may include calculating a center of gravity of a 3D point included in the 3D point cloud data; and setting the 3D space region by setting the distance between the center of gravity and the 3D point as a radius.

[0017] Sampling the candidate fixed point may include selecting an arbitrary 3D point from the 3D point cloud data or sampling the candidate fixed point based on a 3D coordinate value generated through random number generation.

[0018] Generating the above 3D point cloud data may include pairing one 3D point randomly selected from among the 3D points included in the 3D point cloud data with one of the at least two fixed points; generating a line connecting the paired 3D point and the fixed point; and deleting the 3D point from which the line was generated.

[0019] Generating the above 3D point cloud data may include dividing the 3D point cloud data into a plurality of subspace regions having a preset size; pairing all 3D points included in one of the subspace regions with one of the at least two fixed points; generating a line connecting the paired 3D points and the fixed point; and deleting the 3D point from which the line was generated.

[0020] The above-mentioned line cluster may be characterized in that the one fixed point intersects at least five lines.

[0021] According to another embodiment of the present disclosure, a server comprises: a processor; a program for operating the processor; and a memory for storing three-dimensional point cloud data received from an external source; wherein the processor generates at least two fixed points from the three-dimensional point cloud data, connects one of the three-dimensional points included in the three-dimensional point cloud data and the generated fixed points to thereby generate three-dimensional line cloud data, clusters a plurality of lines included in the three-dimensional line cloud data based on the at least two fixed points, samples two line cloud clusters among the clustered line cloud clusters, and estimates a pose of a camera that captured the input image based on the sampled line cloud clusters and feature points of the input image.

[0022] The processor can receive a command to set the number of fixed points, generate clusters equal to the number of fixed points from the three-dimensional point cloud data, extract a center point of the generated cluster, and generate the extracted center point as the fixed point.

[0023] The processor can set a three-dimensional space region in which a fixed point exists from the three-dimensional point cloud data, sample candidate fixed points, and generate the fixed point based on whether the candidate fixed point exists within the three-dimensional space region.

[0024] The processor can calculate three base axes and a variance for the base axes based on principal component analysis of the three-dimensional point cloud data, and set the three-dimensional space region based on the calculated variance.

[0025] The above processor can set the three-dimensional space area by calculating the center of gravity of a three-dimensional point included in the three-dimensional point cloud data and setting the distance between the center of gravity and the three-dimensional point as a radius.

[0026] The above processor can select any 3D point from the 3D point cloud data or sample the candidate fixed point based on a 3D coordinate value generated through random number generation.

[0027] The processor can pair one 3D point randomly selected from among the 3D points included in the 3D point cloud data with one of the at least two fixed points, generate a line connecting the paired 3D point and the fixed point, and delete the 3D point from which the line was generated.

[0028] The processor may divide the 3D point cloud data into a plurality of subspace regions having a preset size, pair all 3D points included in one of the subspace regions with one of the at least two fixed points, generate a line connecting the paired 3D points and fixed points, and delete the 3D point from which the line was generated.

[0029] According to another embodiment of the present disclosure, a system comprises: a user terminal for capturing an input image; and a server for communicating with the user terminal; wherein the server generates at least two fixed points from three-dimensional point cloud data, connects a three-dimensional point included in the three-dimensional point cloud data with one of the generated fixed points, thereby generating three-dimensional line cloud data, clustering a plurality of lines included in the transmitted three-dimensional line cloud data based on the at least two fixed points, and sampling at least two line cloud clusters among the clustered line cloud clusters; and the user terminal detects feature points of the input image, and estimates a pose of a camera that captured the input image based on the detected feature points, the at least two fixed points set as centers of pinhole camera models, and the sampled line cloud clusters.

[0030] The visual localization method based on 3D line-of-sight data disclosed herein and the device for executing the same exhibit higher computational speed than the visual localization algorithm of a single image using conventional line-of-sight data, and thus can be applied to various products requiring real-time computation.

[0031] In particular, the disclosed embodiment is applicable to products in which real-time visual localization for immediate interaction with the surrounding environment is essential, such as in the fields of autonomous driving and robotics, and can maintain high spatial information security by preventing attempts to restore 3D point cloud data from a 3D line map.

[0032] FIG. 1 is a schematic diagram illustrating a system for executing the disclosed visual localization method.

[0033] FIG. 2a is a control block diagram of a system according to one disclosed embodiment.

[0034] FIG. 2b is a control block diagram of a system according to another disclosed embodiment.

[0035] Figure 3 is a complete flowchart of the disclosed visual localization method.

[0036] Figure 4 is a flowchart specifically explaining the first embodiment during the fixed point generation step.

[0037] Figure 5 is a flowchart specifically explaining the second embodiment during the fixed point generation step.

[0038] Figure 6 is a drawing for explaining a first embodiment of setting a three-dimensional space area.

[0039] Figure 7 is a drawing for explaining a second embodiment of setting a three-dimensional space area.

[0040] Figure 8 is a drawing to explain the basis for setting a three-dimensional space area.

[0041] FIG. 9 is a flowchart specifically explaining the steps for sampling candidate fixed points included in FIG. 5.

[0042] Figure 10 is a diagram illustrating an example of generating point cloud data based on point cloud data.

[0043] Figure 11 is a flowchart explaining a specific method for generating military data.

[0044] Figure 12 is a diagram for explaining a first embodiment of generating military data.

[0045] Figure 13 is a diagram for explaining a second embodiment of generating military data.

[0046] Figure 14 is a flowchart illustrating a specific method for performing pose estimation of a camera.

[0047] Figure 15 is a drawing specifically explaining the disclosed visual localization method.

[0048] Figure 16 is a diagram for comparing the line data generated through each embodiment based on the same input image.

[0049] Figure 17 is a table illustrating the effectiveness of the disclosed visual localization method.

[0050] Like reference numerals refer to like elements throughout the specification. This specification does not describe all elements of the embodiments, and any general information within the technical field to which the present invention pertains or any information that overlaps between the embodiments is omitted.

[0051] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0052] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0053] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0054] Additionally, terms such as "~part", "~device", "~block", "~absence", and "~module" may refer to a unit that processes at least one function or operation. For example, the terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) / ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0055] The symbols attached to each step are used to identify each step and do not indicate the order of the steps, and the steps may be performed in a different order than stated unless the context clearly indicates a specific order.

[0056] Hereinafter, an embodiment of an electronic device and a control method thereof according to one aspect will be described in detail with reference to the attached drawings.

[0057] FIG. 1 is a schematic diagram illustrating a system for executing the disclosed visual localization method.

[0058] Referring to FIG. 1, a system (1) for executing a visual localization method according to one embodiment disclosed may include a user terminal (10), a server (20), and a communication network (2) connecting the user terminal (10) and the server (20).

[0059] The user terminal (10) obtains an image (4, hereinafter referred to as an input image (or query image)) captured by a camera (11, see 6 of FIG. 2). The user terminal (10) can transmit the input image or a feature point detected from the input image to the server (20). According to one embodiment, the server (20) can perform pose estimation of the camera (11) based on the input image transmitted by the user terminal (10). According to another embodiment, the user terminal (10) can estimate the pose of the camera (11) on its own after exchanging data with the server (20).

[0060] The server (20) stores 3D point cloud data in advance and converts the 3D point cloud data into 3D line cloud data. According to one embodiment, the server (20) can estimate the pose of the camera (11) that captured the input image (4) by matching the feature points of the input image transmitted by the user terminal (10) with the 3D line cloud data. The server (20) transmits the estimation result (5) back to the user terminal (10).

[0061] The user terminal (10) can be implemented as a computer or portable terminal that can connect to a communication network (2). Here, the computer includes, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a WEB browser, and the portable terminal includes, for example, a wireless communication device that ensures portability and mobility, such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone, etc., and all kinds of handheld-based wireless communication devices, and wearable devices such as glasses, contact lenses, or head-mounted-device (HMD).

[0062] The communication network (2) is a channel for transmitting the position estimation result (5) of the position data or the camera (11) between the user terminal (10) and the server (20). The position estimation result (5) is a map coordinate including latitude and longitude, and may be expressed in a format such as decimal degrees (DD), degrees, minutes, seconds (DMS), degrees and decimal minutes (DMM), etc.

[0063] The server (20) is a configuration that stores a large amount of map data that can perform pose estimation of a camera. The server (20) may be implemented in the form of a cloud server to which an unlimited number of users can request access in a cloud computing environment. The server (20) may be configured with a processor (25, see FIG. 2) that converts a large amount of 3D point cloud data into 3D line cloud data and performs pose estimation of the camera (11) through the 3D line cloud data, and a memory (23, see FIG. 2) that stores the aforementioned large amount of data, and may also be implemented in a form that can be directly connected to the 3D point cloud data through an external processor (20-1), etc.

[0064] FIG. 2a is a control block diagram of a system according to one embodiment of the disclosed method.

[0065] Referring to FIG. 2a, a user terminal (10) according to an embodiment disclosed may include, in hardware, a camera (11) for capturing an input image, a communication unit (12) for communicating with a communication unit (22) of a server (20) via a communication network (2), and an output unit (14) for displaying an input image captured by the camera (11) or outputting a pose estimation result of the camera (11) transmitted by the server (20).

[0066] Specifically, the camera (11) may include various photographing means such as a CMOS (Complementary Metal-Oxide Semiconductor) image sensor and a CCD (Charge-Coupled Device) image sensor.

[0067] The communication unit (12) may include one or more components that enable communication with the communication network (2), and may include, for example, at least one of a short-range communication module, a wired communication module, and a wireless communication module.

[0068] The short-range communication module may include various short-range communication modules that transmit and receive signals using a wireless communication network at a short distance, such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module.

[0069] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0070] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), and LTE (Long Term Evolution), in addition to a WiFi module and a WiBro (Wireless broadband) module.

[0071] The output unit (14) outputs a display that displays an input image and a pose estimation result (5) of a camera (11) transmitted by a server (20). To this end, the output unit (14) may be provided as a digital light processing (DLP) panel, a plasma display panel, a liquid crystal display (LCD) panel, an electroluminescence (EL) panel, an electrophoretic display (EPD) panel, an electrochromic display (ECD) panel, a light emitting diode (LED) panel, or an organic light emitting diode (OLED) panel, but is not limited thereto.

[0072] The user terminal (10) may include various configurations in addition to the configuration illustrated in FIG. 2, and is not limited to the names referring to the configurations.

[0073] The server (20) includes an input unit (21) for receiving a user's input command, a communication unit (22) for performing communication with a user terminal (10), a memory (23) for storing input images received by the communication unit (22) or storing large amounts of data and algorithms required to execute the disclosed visual localization method, and a processor (25) for controlling each component of the server (20).

[0074] Specifically, the input unit (21) receives a setting command for the number of fixed points required to generate 3D line data. To this end, the input unit (11) is connected to the server (20) and may include hardware devices such as various buttons, switches, a keyboard, a mouse, a trackball, various levers, handles, or sticks that can receive setting commands from a user. In addition, the input unit (11) may include a GUI (Graphical User Interface), i.e., a software device, such as a touch pad. The touch pad may be implemented as a touch screen panel (TSP) and may form a mutual layer structure with the display.

[0075] The memory (23) may be implemented as at least one of non-volatile memory elements such as cache, ROM (Read Only Memory), PROM (Programmable ROM), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and flash memory, or volatile memory elements such as RAM (Random Access Memory), or storage media such as a hard disk drive (HDD) or CD-ROM, but is not limited thereto. The memory (23) may be implemented as a separate chip from the processor (25) described below, or may be implemented as a single chip with the processor (25).

[0076] The processor (25) is a configuration that performs the disclosed visual localization method while controlling the hardware included in the server (20). To this end, the processor (25) may mean a data processing device built into hardware that has a physically structured circuit to perform a function expressed by a code or command included in a program. The processor (25) is an example of a data processing device built into hardware, and may include, but is not limited to, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), and the like. The processor (25) may include one or more processors.

[0077] The processor (25) can be divided into a fixed point generation unit (26) that generates at least two fixed points from three-dimensional point cloud data in a software manner, a three-dimensional line cloud data generation unit (27) that generates three-dimensional line cloud data by connecting one of the three-dimensional points included in the three-dimensional point cloud data and the generated fixed points, and a visual localization unit (28) that estimates the posture of the camera (11) that captured the input image based on the generated three-dimensional line cloud data. The processor (25) transmits the posture estimation result generated by the visual localization unit (28) to the user terminal (10) through the communication unit (22).

[0078] A specific description of the operation of the software-separated processor (25) will be described later through other drawings.

[0079] Meanwhile, the server (20) may omit some of the configurations described above in FIG. 2A. For example, the server (20) may be composed of only a processor (25) and memory (23), or may be provided with multiple chips implementing the processor (25) and memory (23).

[0080] Fig. 2b is a control block diagram of a system according to another disclosed embodiment. Description of configurations overlapping with those illustrated in Fig. 2a is omitted.

[0081] Referring to FIG. 2b, a user terminal (10) according to another disclosed embodiment may include, in hardware, a camera (11) for capturing an input image, a communication unit (12) for communicating with a communication unit (22) of a server (20) via a communication network (2), a processor (15) for controlling the overall operation of the user terminal (10), performing image processing of an input image captured by the camera (11) and receiving 3D group data transmitted by the server (20) to perform pose estimation of the camera (11), an output unit (14) for displaying an input image captured by the processor (15) or outputting a pose estimation result performed by the processor (15), and a memory (13) for storing various data necessary for the operation of the processor (15).

[0082] Unlike the embodiment described in FIG. 2A, the system (1) according to the embodiment of FIG. 2B can have the processor (15) perform a visual localization process, i.e., pose estimation of the camera (11). To this end, the processor (15) can receive 3D line data through the server (20). The processor (15) can perform pose estimation of the camera by extracting feature points from an input image captured by the camera (11) and then matching the extracted feature points with the 3D line data received from the server (20). The method for performing pose estimation of the camera in this way is the same as the operation method of the visual localization unit (28) of the server (20) described above in FIG. 2A, and the only difference is the subject performing visual localization.

[0083] Meanwhile, the user terminal (10) according to the embodiment of FIG. 2b may be provided with the processor (15) and the memory (13) as the same chip, may further include other configurations other than the configuration shown in FIG. 2b, or may omit some configurations.

[0084] Below, a visual localization method that can be implemented in both the embodiments of FIG. 2a and FIG. 2b is described in detail.

[0085] Figure 3 is a complete flowchart of the disclosed visual localization method.

[0086] Referring to FIG. 3, a system (1) performing the disclosed visual localization method loads three-dimensional point cloud data (100).

[0087] Here, the 3D point cloud data is a large-capacity map data prepared in advance to determine where the input image captured by the camera (11) for which the pose is to be estimated was captured. The 3D point cloud data is stored in advance in the memory (23) of the server (20), and can be loaded from the memory (23) when a request is received from the user terminal (10).

[0088] The system (1) generates at least two fixed points from three-dimensional point cloud data (200).

[0089] The disclosed visual localization method must generate at least two anchor points, and it is also possible to generate three or more anchor points. Furthermore, when generating at least two anchor points, the system (1) sets a three-dimensional spatial region in which the anchor points may exist and samples candidate anchor points within the three-dimensional spatial region. The specific method for generating anchor points is described in detail below with reference to the drawings in FIG. 4 and below.

[0090] The system (1) generates 3D line cloud data by connecting at least two generated fixed points and 3D points included in 3D point cloud data (300).

[0091] Specifically, the system (1) generates 3D line data by pairing fixed points and 3D points and then generating lines based on the paired points. A specific method for generating 3D line data is described in detail below with reference to the drawings in FIG. 11 and below.

[0092] Extract feature points of the input image (400).

[0093] Here, the feature point is a result detected by the user terminal (10) from the input image. According to the embodiment of Fig. 2a, the user terminal (10) can generate feature information in the process of detecting feature points and transmit the feature information to the server (20).

[0094] When the pose estimation of the camera (11) is performed on the user terminal (10) according to the embodiment of FIG. 2b, the server (20) does not receive feature points from the user terminal (10), but instead transmits the generated 3D line data to the user terminal (10). After receiving the 3D line data, the user terminal (10) extracts feature points of the input image. The user terminal (10) performs the following operations based on the extracted feature points and the received 3D line data.

[0095] The system (1) clusters the generated 3D line data and estimates the pose of the camera (11) based on the clustered line data (500).

[0096] A specific method for estimating the pose of a camera (11) based on clustered group data is described in detail below with reference to Fig. 14.

[0097] Figure 4 is a flowchart specifically explaining the first embodiment during the fixed point generation step.

[0098] Referring to FIG. 4, the system (1) receives an input command for a user to set the number of fixed points (210).

[0099] Specifically, the system (1) can receive input commands from the user regarding the number of fixed points to be created through various methods. However, the system (1) can receive input commands from the user regarding the creation of at least two fixed points.

[0100] When the user transmits an input command for the number of fixed points, the system (1) creates N clusters based on a clustering algorithm (211).

[0101] Here, N corresponds to the number of fixed points input by the user. The N clusters can be generated using a typical clustering algorithm (e.g., K-means clustering algorithm).

[0102] The system (1) extracts the center points of N clusters (212) and determines the extracted center points as fixed points (213).

[0103] Figure 5 is a flowchart specifically illustrating a second embodiment of the fixed point generation step. Figure 6 is a diagram illustrating a first embodiment of setting a three-dimensional space region, and Figure 7 is a diagram illustrating a second embodiment of setting a three-dimensional space region. Figure 8 is a diagram illustrating the basis for setting a three-dimensional space region. To avoid redundant explanations, these are described together below.

[0104] Referring to FIG. 5, the system (1) sets a three-dimensional space area where a fixed point exists (220).

[0105] Unlike the first embodiment (Fig. 4), the step (200) of generating a fixed point preliminarily generates a candidate region, i.e., a three-dimensional space region, in which candidate fixed points can be generated, and generates candidate fixed points that become candidates for fixed points within the three-dimensional space region. The system (1) may include the first embodiment (see Fig. 6) and the second embodiment (see Fig. 7) in setting the three-dimensional space region.

[0106] Referring to FIG. 6, the system (1) can perform (221-2) principal component analysis (PCA) of 3D point cloud data (221-1). Here, PCA (Principle Component Analysis) analysis is a technique used for dimensionality reduction and is used in machine learning, data mining, statistical analysis, and noise removal. In other words, the system (1) changes high-dimensional data included in 3D point cloud data into low-dimensional data.

[0107] Specifically, the system (1) calculates three basis axes (v1, v2, v3) and variances (σ1, σ2, σ3) for each axis through PCA analysis. The system (1) can set a region spaced 2 sigmas (2xσ1, 2x σ2, 2xσ3) apart from the center of gravity for each basis axis as a three-dimensional space region (221-3).

[0108] Meanwhile, in the embodiment of FIG. 6, the setting standard for the 3D space area is shown as 2 sigma, but it is not necessarily limited to this and can be changed to various values ​​such as 1.96 sigma.

[0109] Referring to Fig. 7, the system (1) can calculate the center of gravity (centroid) and radius (r) of 3D points included in 3D point cloud data (221-1) (222-2). Specifically, the system (1) can calculate the distance between the center of gravity and 3D points, set the farthest distance from the center of gravity as the radius (r), and set the area within twice the radius (2r) from the center of gravity as the 3D space area.

[0110] Meanwhile, in the embodiment of FIG. 7, the setting standard for the three-dimensional space area is shown as twice the radius, but it is not necessarily limited to this and can be changed to various values.

[0111] As the distance between at least two fixed points increases, the generated line data may become more parallel to each other. In this case, the accuracy of the pose estimation of the camera (11) may decrease. Referring to graph a) of Fig. 8, it can be confirmed that as the distance ratio between two fixed points increases, the rotation and displacement errors (translation errors) increase.

[0112] In addition, as the distance between anchor points increases, the accuracy of restoration of point cloud data from 3D line cloud data can increase. Referring to Fig. 8b), it can be confirmed that as the distance between two anchor points (anchor point 1 and anchor point 2) in the ring-shaped 3D point cloud data increases, the restored point cloud data becomes more similar to the original 3D point cloud. Therefore, by setting the possible existence area of ​​anchor points to a spatial area that is not too far from the 3D point cloud, the disclosed system (1) can lower the accuracy of reverse restoration (protect personal privacy) while maintaining the accuracy of camera pose estimation.

[0113] Referring again to FIG. 5, the system (1) samples candidate fixed points (230).

[0114] The method of generating candidate anchor points is specifically described through FIGS. 9 and 10.

[0115] The system (1) determines whether the sampled candidate fixed point exists within a set three-dimensional space area (240).

[0116] If the candidate anchor point is not included in the set 3D space area (No of 240), the system (1) discards the previous candidate anchor point and generates a new candidate anchor point (230).

[0117] If the candidate fixed point is included within the set 3D space area (Yes of 240), the system (1) determines the candidate fixed point as the fixed point for generating the group data (250).

[0118] FIG. 9 is a flowchart specifically explaining the steps for sampling candidate fixed points included in FIG. 5.

[0119] Referring to FIG. 9, the system (1) can select any 3D point from among the point clouds of 3D point cloud data included in a set 3D space area (231). In addition, the system (1) can generate a virtual point based on a 3D coordinate value through random number generation from among the point clouds of 3D point cloud data included in the set 3D space area (232).

[0120] The 3D points generated in step 231 or step 232 are sampled as candidate fixed points (233).

[0121] Here, the coordinates of the virtual point in step 232 are random points generated as a result of random number generation. The system (1) determines whether the generated virtual point exists within a three-dimensional space region (step 240 of FIG. 5). If the virtual point does not exist within the three-dimensional space region, the virtual point is discarded and a new virtual point is generated through random number generation again. If the virtual point is included within the three-dimensional space region, the system (1) determines it as a candidate fixed point.

[0122] Figure 10 is a diagram illustrating an example of generating line data based on point cloud data.

[0123] If only one candidate anchor point is set, visual localization may encounter a problem of increased displacement error. System (1) sets multiple virtual points included in a 3D point cloud data or a 3D spatial region as candidate anchor points, and solves the problem of increased displacement error by setting at least two anchor points.

[0124] In the embodiment of Fig. 10, unlike the embodiment of Fig. 9, clusters are set (233-2) through the K-means clustering algorithm, and the center points (center 1, center 2) within the clusters can be created as fixed points. The system (1) creates group data based on two fixed points (233-3).

[0125] In the case of the embodiment of Fig. 10, since the cluster always exists within the three-dimensional space region mentioned in Fig. 5 through the K-means clustering algorithm, there may be no need to set a three-dimensional space region separately.

[0126] Figure 11 is a flowchart illustrating a specific method for generating pre-order data. Figure 12 is a diagram illustrating a first embodiment of generating pre-order data. Figure 13 is a diagram illustrating a second embodiment of generating pre-order data. To avoid redundant explanations, they are described together below.

[0127] Referring to Fig. 11, the system (1) determines at least two fixed points (301).

[0128] The method of determining the fixed point is performed in the manner described through FIGS. 4 to 10.

[0129] After at least two fixed points are generated, the system (1) can pair a randomly selected three-dimensional point with one of the at least two fixed points (310).

[0130] Referring to FIG. 12, the system (1) generates at least two fixed points (312) from three-dimensional point cloud data (311).

[0131] The drawings in Figure 12 are drawn in two dimensions for ease of understanding, but each drawing is made in three dimensions.

[0132] The system (1) randomly pairs three-dimensional points and fixed points (313) and generates line data connecting the paired three-dimensional points and fixed points (314).

[0133] Specifically, the system (1) generates a line connecting a first fixed point and an arbitrary point (311-1) of the 3D point cloud data. In addition, the system (1) generates a line connecting a second fixed point and another arbitrary point (311-2) of the 3D point cloud data. At this time, each arbitrary point (311-1, 311-2) can be determined randomly.

[0134] The system (1) repeats the aforementioned process to connect all 3D points and fixed points within the 3D point cloud data (311). The line cloud thus generated has the geometric characteristic of lines generated from different fixed points intersecting each other.

[0135] Referring back to FIG. 9, unlike step 310, the system (1) may divide the 3D point cloud data into subspace regions with preset sizes (320), and may pair all 3D points included in one of the plurality of divided subspace regions with one of the at least two fixed points (321).

[0136] Here, a subspace region is a 3D region that divides 3D point cloud data (or 3D point cloud spatial data), and the size (or volume) and shape of the region can vary. For example, a subspace region can be formed by voxels that evenly divide 3D point cloud spatial data, but this is not limited to this. In other words, a subspace region can also be formed by dividing 3D space unevenly.

[0137] When the system (1) divides 3D point cloud data into subspace regions, fixed point pairing can be performed using voxels as shown in Fig. 13.

[0138] The system (1) can divide 3D point cloud data (321) into voxel-based space (322).

[0139] Specifically, the system (1) can distinguish 3D point cloud data by displaying the 3D point cloud assigned to the voxel with different colors and brightness, as shown in 323 of Fig. 13.

[0140] The system (1) pairs each three-dimensional point included in each area divided into voxels with each fixed point (324).

[0141] Unlike step 310 of Fig. 11, which randomly connects fixed points and 3D points, the system (1) can also pair each fixed point based on a voxel. The line cloud generated in this way has the geometric characteristic that lines generated from different fixed points intersect each other, and the additional geometric characteristic that adjacent points are connected to the same fixed point has the effect of lowering the accuracy of the back-reconstruction into point cloud data.

[0142] Referring again to FIG. 11, the system (1) creates a line by connecting the three-dimensional points and fixed points for which pairing has been completed (330), and deletes the three-dimensional points from which the line was created (340), thereby finally creating line data.

[0143] Fig. 14 is a flowchart illustrating a specific method for performing camera pose estimation. Fig. 15 is a drawing specifically illustrating the disclosed visual localization method.

[0144] Referring first to Fig. 14, the system (1) loads three-dimensional line data (510).

[0145] Here, the 3D line group data is line group data generated through fixed point generation and pairing of fixed points (step 300 of Fig. 3). The generated line group data is stored through memory (23).

[0146] When the server (20) performs pose estimation of the camera (11) according to the disclosed embodiment (Fig. 2a), the server (20) uses the line-of-sight data generated through pairing of fixed points. However, when the user terminal (10) performs pose estimation of the camera (11) according to another embodiment (Fig. 2b), the server (20) loads the 3D line-of-sight data stored through the memory (23) or the like and transmits the generated line-of-sight data to the user terminal (10).

[0147] The system (1) extracts feature points of the input image (520).

[0148] Here, the feature points may include not only the feature points of the input image but also feature information. The feature information may include both line features and color information. According to one disclosed embodiment, when the server (20) performs pose estimation of the camera (11), the user terminal (10) extracts feature points from the input image and transmits them to the server (20). According to another disclosed embodiment, when the user terminal (10) performs pose estimation of the camera (11), the user terminal (10) extracts feature points and executes the following steps.

[0149] The system (1) matches the correspondence between feature points and lines included in the line group data (530).

[0150] Specifically, the system (1) performs one-to-one matching between feature points extracted from an input image and straight lines (lines) included in the line group data using a feature desciptor.

[0151] As will be described below, straight lines matched with feature points are included in one of several line clusters, and each feature point can have a correspondence with a fixed point of the line cluster that includes the matched straight line. Therefore, system (1) can estimate the camera pose using a position estimation algorithm between virtual perspective cameras (hereinafter, corresponding to fixed points).

[0152] The system (1) clusters a line group according to a fixed point among a plurality of lines matched with feature points (540), and the server (20) samples two line group clusters from among the clustered results (550).

[0153] Specifically, the system (1) performs clustering based on at least two or more fixed points. Referring to Fig. 15, the system (1) clusters at least five or more lines from at least two or more fixed points (the first fixed point and the second fixed point). The system (1) performs visual localization based on the camera (11) viewpoint and at least two or more clustered fixed point viewpoints, i.e., three viewpoints.

[0154] For the pose estimation of a camera (11) using a 3D pre-array space map, the prior art proposed a P6L (Point-6-Line) algorithm based on a relative position estimation method between generalized camera models. Unlike the prior art, the disclosed system (1) is based on a relative position estimation method of a pinhole camera model instead of a general camera model, thereby significantly improving the computation speed while maintaining localization accuracy.

[0155] Referring again to FIG. 14, the system (1) estimates the pose of the camera (11) based on the sampled group of lines (560).

[0156] Specifically, the system (1) considers each fixed point (the first and second fixed points in Fig. 15) as the center of a virtual perspective (or pinhole) camera model. That is, when applying the sampled line cluster to the algorithm of the pinhole camera model, the system (1) performs camera pose estimation based on each fixed point set in the clustering process. Through this, the disclosed system (1) can improve the camera pose estimation time from the existing 3 frames / sec (fps) to at least 100 frames / sec.

[0157] Figure 16 is a diagram for comparing the line data generated through each embodiment based on the same input image.

[0158] Referring to FIG. 16, drawings 610, 620, and 630 are the results (611, 612, and 613) of the 3D point cloud data (601) of the input image being restored using a conventional technique. Specifically, 610 is the result of restoration without transformation of the point cloud data, 620 is the result of restoration through prior art paper 1, and 630 is the result of restoration through prior art paper 3. In particular, in 630, only 50% of the 3D point cloud is restored, but the geometric error of the restored point cloud has a small error compared to the original.

[0159] Using conventional technology, the result of geometric restoration into 3D point cloud data (601) is visually recognizable, and in the case of 3D uniform line cloud data, detailed image synthesis of spatial information is possible, so there is a limitation in spatial information security.

[0160] 640 is the result of inverse restoration using the line data generated by pairing the 3D point and the fixed point, and 650 is the result of inverse restoration using the line data generated by pairing the 3D area (voxel) and the fixed point.

[0161] It can be confirmed that the fixed point-based line cloud data for the disclosed visual localization method, when restored to point cloud data, has a large geometric error and induces misrestorement in the line density-based geometric restoration due to the high line density around the fixed point.

[0162] Figure 17 is a table illustrating the effectiveness of the disclosed visual localization method.

[0163] The x-axis of the table in Fig. 17 distinguishes between the present invention and the prior art, and the y-axis represents the inference speed (fps). This comparative experiment shows the restoration operation speed of images captured indoors (Cambridge) and outdoors (energy landscape).

[0164] Referring to Figure 17, the prior art (uniform line cloud) using line data had inversion speeds of 5.09 and 1.1 fps, and the PPL technique disclosed in prior art paper 3 showed computational speeds of 5.34 and 1.79 fps. In contrast, the inversion restoration (Ray cloud) using line data of the present invention recorded 85.76 and 38.11 fps, showing a computational speed increase rate of up to 35 times.

[0165] The disclosed 3D line-of-sight data-based visual localization method and the device for executing the same exhibit higher computational speeds than conventional single-image visual localization algorithms using line-of-sight data, making them applicable to a variety of products requiring real-time computation. In particular, the disclosed embodiment is applicable to products requiring real-time visual localization for immediate interaction with the surrounding environment, such as autonomous driving and robotics, and can maintain high spatial information security by preventing attempts to restore 3D point cloud data from a 3D line-of-sight map.

[0166] Those skilled in the art should understand that the present invention can be implemented in other specific forms without changing the technical spirit or essential characteristics thereof, and therefore, the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included within the scope of the present invention.

Claims

1. Generate at least two fixed points from 3D point cloud data; By connecting a 3D point included in the above 3D point cloud data and one of the generated fixed points, 3D line cloud data is generated; Extract feature points from the input image; A visual localization method comprising: clustering a plurality of lines included in the three-dimensional line group data based on at least two fixed points, sampling two line group clusters among the clustered line group clusters, and estimating the pose of a camera that captured the input image based on the sampled line group clusters and the feature points.

2. In paragraph 1, Creating the above fixed point is, Save the setting command for the number of fixed points to be generated; Generate clusters equal to the number of fixed points from the above 3D point cloud data; Extract the center point of the above-generated cluster; A visual localization method comprising: generating the extracted center point as the fixed point.

3. In paragraph 1, Creating the above fixed point is, A 3D spatial region in which a fixed point exists is set from the above 3D point cloud data; Sample candidate anchor points; A visual localization method comprising: generating the anchor point based on whether the candidate anchor point exists within the three-dimensional space region; 4. In paragraph 3, Setting the above three-dimensional space area is: Based on principal component analysis of the above 3D point cloud data, three basic axes and variances for the basic axes are calculated; A visual localization method comprising: setting the three-dimensional space region based on the calculated variance; 5. In paragraph 3, Setting the above three-dimensional space area is: Calculate the center of gravity of a 3D point included in the above 3D point cloud data; A visual localization method comprising: setting the three-dimensional space region by setting the distance between the center of gravity and the three-dimensional point as a radius; 6. In paragraph 3, Sampling the above candidate anchor points is: Select any 3D point from the above 3D point cloud data, or A visual localization method comprising sampling the candidate fixed points based on three-dimensional coordinate values ​​generated through random number generation.

7. In paragraph 1, Creating the above 3D line data is: Pairing one 3D point randomly selected from among the 3D points included in the 3D point cloud data with one of the at least two fixed points; Generate lines connecting paired 3D points and anchor points; A visual localization method comprising: deleting a three-dimensional point that generated a line; 8. In paragraph 1, Creating the above 3D line data is: Divide the above 3D point cloud data into a plurality of subspace regions with preset sizes; Pairing all three-dimensional points contained in one of the above subspace regions with one of the at least two fixed points; Generate lines connecting paired 3D points and anchor points; A visual localization method comprising: deleting a three-dimensional point that generated a line; 9. In paragraph 1, The above-mentioned military group is, A visual localization method characterized in that the above one fixed point intersects at least five lines.

10. Processor; A program for operating the above processor and a memory for storing 3D point cloud data received from the outside; The above processor, Generate at least two fixed points from the above 3D point cloud data, By connecting a 3D point included in the above 3D point cloud data and one of the generated fixed points, a 3D line cloud data is generated, Clustering a plurality of lines included in the three-dimensional line group data based on at least two fixed points, Sample two of the above clustered clusters, A server that estimates the pose of a camera that captured the input image based on the sampled group of lines and the feature points of the input image.

11. In paragraph 10, The above processor, Receive a command to set the number of fixed points, create a cluster as many as the number of fixed points from the 3D point cloud data, extract the center point of the created cluster, and create the extracted center point as the fixed point; or A server that sets a 3D space region in which a fixed point exists from the 3D point cloud data, samples candidate fixed points, and generates the fixed points based on whether the candidate fixed points exist within the 3D space region.

12. In the above 11 items, The above processor, Based on the principal component analysis of the above 3D point cloud data, three basic axes and a variance for the basic axes are calculated, and the 3D space area is set based on the calculated variance, or By calculating the center of gravity of the 3D point included in the 3D point cloud data and setting the distance between the center of gravity and the 3D point as a radius, the 3D space area is set, A server that selects a random 3D point from the above 3D point cloud data or samples the candidate fixed point based on a 3D coordinate value generated through random number generation.

13. In paragraph 10, The above processor, Pairing one 3D point randomly selected from among the 3D points included in the 3D point cloud data with one of the at least two fixed points, Generate lines connecting paired 3D points and anchor points, A server that deletes the 3D points that created the line.

14. In paragraph 10, The above processor, Divide the above 3D point cloud data into multiple subspace regions with preset sizes, Pairing all three-dimensional points contained in one of the above subspace regions with one of the at least two fixed points; Generate lines connecting paired 3D points and anchor points, A server that deletes the 3D points that created the line.

15. User terminal for capturing input images; and A server that performs communication with the user terminal; The above server, Generate at least two fixed points from 3D point cloud data, By connecting a 3D point included in the above 3D point cloud data and one of the generated fixed points, a 3D line cloud data is generated, Clustering a plurality of lines included in the transmitted 3D line group data based on at least two fixed points, and sampling at least two line group clusters among the clustered line group clusters, The above user terminal, A system for detecting feature points of the input image and estimating the pose of a camera that captured the input image based on the detected feature points, at least two fixed points set as the center of a pinhole camera model, and the sampled line cluster.

Images