Device and method for performing localization based on panoramic images

Abstract

A position estimation device that performs position estimation based on a panoramic image according to one embodiment of the present invention includes a memory in which a position estimation program is stored and a processor that executes the position estimation program, wherein the position estimation program extracts line segment information from an input 360-degree panoramic image to generate a query image, and compares the query image with a pre-prepared 3D line segment map for a target space to estimate the position of the point where the query image was captured within the 3D line segment map. Furthermore, the 3D line segment map includes information on line segments determined by the intersection of each surface constituting the target space in the form of a 3D map, and the query image includes line segments determined by the intersection of each surface constituting the target space included in the input 360-degree panoramic image.

Description

Technology Field

[0001] The present invention relates to a position estimation apparatus and method for performing position estimation based on a panoramic image. Background Technology

[0002] Localization technology is a technique that estimates the position and orientation (pose) of a camera on a 3D map and is essential in fields such as augmented reality (AR), virtual reality (VR), robotics, and autonomous vehicles. Existing localization methods primarily use visual descriptors-based approaches, which generate a 3D map and estimate the location by comparing the image's feature points with the map. However, these methods have limitations, such as large map sizes, high computational costs, privacy infringement issues, and vulnerability to environmental changes.

[0003] First, visual feature point-based technologies require significant memory and computational resources for map generation and storage because they must extract and store either hand-crafted feature points, such as SIFT (Scale-Invariant Feature Transform), or learned feature points, such as SuperPoints. Additionally, technologies that estimate location by extracting a single global feature vector per image utilize Vision Foundation Models like NetVLAD or GPT-4, which require a large amount of computation for artificial neural network calculations. Consequently, current technologies cause a problem where data processing costs increase significantly in large-scale environments. Furthermore, client-server based location estimation systems are vulnerable to privacy violations because user images must be transmitted to the server, posing a risk of exposing sensitive visual information. Moreover, existing visual feature point-based technologies suffer from degraded accuracy under conditions such as changes in lighting, repetitive structures, and environmental variations. Finally, technologies that perform location estimation using point clouds obtained from 3D scans as maps place a burden on map storage due to the need to store a large volume of points.

[0004] To overcome these limitations, a new localization method based on panoramic images and 3D line segment maps is required. Panoramic images provide a 360-degree field of view, allowing them to contain broader spatial information, and are more robust to scene changes or repetitive structures compared to conventional single-camera images. Furthermore, extracting geometric information such as lines and intersections from panoramic images offers several advantages over existing visual feature point-based methods. In particular, by utilizing 3D line segment maps for localization, the amount of memory required for map storage can be significantly reduced compared to localization methodologies that use existing visual feature points or point clouds as maps.

[0005] In consideration of this, the present invention proposes a position estimation device and method that overcome the limitations of existing visual feature point-based position estimation methods based on panoramic images and 3D line segment maps. Prior art literature

[0006] Korean Registered Patent No. 10-2516450 (Title of Invention: Map Generation Method and Image-Based Positioning System Using the Same) The problem to be solved

[0007] The present invention aims to solve the aforementioned problems and has the technical objective of providing a position estimation device and method that estimate the accurate position and direction (pose) of a camera on a three-dimensional line segment map by utilizing a panoramic image captured by a camera.

[0008] However, the technical problem that this embodiment aims to solve is not limited to the technical problem described above, and other technical problems may exist. means of solving the problem

[0009] As a technical means for solving the aforementioned technical problem, a position estimation device that performs position estimation based on a panoramic image according to the first aspect of the present invention includes a memory in which a position estimation program is stored and a processor that executes the position estimation program, wherein the position estimation program extracts line segment information from an input 360-degree panoramic image to generate a query image, and compares the query image with a pre-prepared 3D line segment map for a target space to estimate the position of the point where the query image was taken within the 3D line segment map, wherein the 3D line segment map includes information of line segments determined by the intersection of each surface constituting the target space in the form of a 3D map, and the query image includes line segments determined by the intersection of each surface constituting the target space included in the input 360-degree panoramic image.

[0010] In addition, a method for estimating a position based on a panoramic image using a position estimation device according to the second aspect of the present invention comprises: a step of generating a query image by extracting line segment information from an input 360-degree panoramic image; and a step of estimating the position of a point where the query image was captured within a 3D line segment map by comparing a 3D line segment map prepared in advance for a target space with the query image, wherein the 3D line segment map includes information on line segments determined by the intersection of each face constituting the target space in the form of a 3D map, and the query image includes line segments determined by the intersection of each face constituting the target space included in the input 360-degree panoramic image. Effects of the invention

[0011] According to the means for solving the problem of the present invention described above, by estimating the position using only 2D line segment information and 3D line segment maps of a panoramic image, memory usage is drastically reduced, computational load is reduced, and efficient position estimation is made possible.

[0012] In addition, since only 2D line segment information extracted from panoramic images is utilized, when performing location estimation in a cloud service, only the location information of the line segments is shared instead of the entire image, thereby resolving the issue of user privacy protection. Brief explanation of the drawing

[0013] FIG. 1 is a block diagram illustrating the configuration of a position estimation device according to one embodiment of the present invention. Figure 2 shows a panoramic image and a three-dimensional line segment map. FIG. 3 illustrates the process of a position estimation device according to one embodiment of the present invention extracting a line segment from a panoramic image. FIGS. 4, FIGS. 5a, FIGS. 5b, and FIGS. 5c are drawings for explaining a method for selecting candidate locations based on a line segment and intersection distance function according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a method for correcting position through the intersection of line segments and direction matching according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating a detailed process of a position estimation device according to an embodiment of the present invention estimating a position based on a panoramic image. FIG. 8 is a flowchart illustrating a method for performing position estimation based on a panoramic image of a position estimation device according to one embodiment of the present invention. Specific details for implementing the invention

[0014] The present invention will be described in detail below with reference to the attached drawings. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings. In order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and the size, form, and shape of each component shown in the drawings may be varied in various ways. Identical or similar parts throughout the specification are denoted by identical or similar reference numerals.

[0015] Suffixes such as "module" and "part" for components used in the following description are assigned or used interchangeably solely for the sake of ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed in this specification.

[0016] Throughout the specification, when it is stated that a part is "connected (connected, contacted, or coupled)" to another part, this includes not only cases where they are "directly connected (connected, contacted, or coupled)," but also cases where they are "indirectly connected (connected, contacted, or coupled)" with other members interposed therebetween. Furthermore, when it is stated that a part "includes (provides, or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for additional "included (provided, or provided)" of other components.

[0017] Terms indicating ordinal numbers, such as first, second, etc., used in this specification are used solely for the purpose of distinguishing one component from another and do not limit the order or relationship of the components. For example, the first component of the present invention may be named the second component, and similarly, the second component may be named the first component.

[0018] FIG. 1 is a block diagram illustrating the configuration of a position estimation device according to one embodiment of the present invention.

[0019] The location estimation device (100) includes a memory (120) and a processor (130), and may further include a communication module (110) and a database (140). The location estimation device (100) performs the operation of estimating the location of a point where a panoramic image was taken within a pre-prepared three-dimensional line segment map using an input 360-degree panoramic image.

[0020] To this end, the location estimation device (100) may be implemented as a computer or portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a laptop, desktop, or notebook equipped with a web browser, and the portable terminal may include, for example, all types of handheld-based wireless communication devices such as smartphones, tablet PCs, and smartwatches, as wireless communication devices that ensure portability and mobility. Additionally, the location estimation device (100) may function as a server that provides an estimated location of the point where the panoramic image was taken within a three-dimensional line segment map for a 360-degree panoramic image received from the outside. When the location estimation device (100) operates as a server, the server may operate in a cloud computing service model such as SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In addition, servers can be deployed in forms such as private clouds, public clouds, or hybrid clouds.

[0021] A network refers to a connection structure capable of exchanging information among individual nodes, such as terminals and devices, and includes local area networks (LAN), wide area networks (WAN), the internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC), and LiFi.

[0022] The communication module (110) receives a 360-degree panoramic image of the target space captured by a camera. The target space includes a three-dimensional indoor or outdoor space defined by boundary lines or intersection lines where walls, ceilings, floors, or other surfaces of a structure meet. The communication module (110) may include a device that includes hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.

[0023] The memory (120) stores a position estimation program. At this time, the position estimation program extracts line segment information from an input 360-degree panoramic image to generate a query image, and compares the query image with a pre-prepared 3D line segment map for the target space to estimate the location of the point where the query image was taken within the 3D line segment map. Here, the 3D line segment map contains information of line segments determined by the intersection of each surface constituting the target space in the form of a 3D map, and the query image contains line segments determined by the intersection of each surface constituting the target space included in the input 360-degree panoramic image.

[0024] At this time, memory (120) should be interpreted as a general term for a non-volatile storage device that retains stored information even when power is not supplied, and a volatile storage device that requires power to retain stored information. Memory (120) can perform the function of temporarily or permanently storing data processed by the processor (130). Memory (120) may include magnetic storage media or flash storage media in addition to a volatile storage device that requires power to retain stored information, but the scope of the present invention is not limited thereto.

[0025] The processor (130) executes a location estimation program stored in memory (120) and, as a result of the execution, outputs the location of a point captured within a three-dimensional line segment map for a panoramic image.

[0026] In one example, the processor (130) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the invention is not limited thereto.

[0027] The database (140) can store various data for estimating the location of a point where a 360-degree panoramic image of a camera or a query image within a 3D line segment map is captured, which is received through the communication module (110). Additionally, the database (140) stores location information of the captured point estimated by a location estimation program, and based on this location information of the captured point, it can be utilized in various applications in the fields of AR / VR technology and robotics to estimate the spatial location of a user and place a virtual object.

[0028] First, a method for extracting existing line segment information will be explained with reference to FIGS. 2 and FIGS. 3.

[0029] Figure 2 shows a panoramic image and a three-dimensional line segment map.

[0030] The 360-degree panoramic image illustrated in FIG. 2(a) is a photograph with a 360-degree field of view and, unlike a standard camera photograph, can capture overall visual information around the camera. The 3D line segment map illustrated in FIG. 2(b) is generated from a point cloud that captures the 3D structure of an indoor space and represents the indoor space through the position and orientation of line segments. The present invention can perform location estimation in indoor and outdoor spaces by utilizing such 360-degree panoramic images and 3D line segment maps. For example, a 3D line segment map can be prepared by applying a 3D line segment estimation algorithm to a point cloud generated using a 3D scanner (Matterport, Faro3D) or image-based reconstruction technology (COLMAP) for a target space. Since such a 3D line segment estimation algorithm utilizes previously known technology, a detailed description thereof is omitted.

[0031] FIG. 3 illustrates the process of a position estimation device according to one embodiment of the present invention extracting a line segment from a panoramic image.

[0032] Referring to Fig. 3, the position estimation program extracts line segment information from an input 360-degree panoramic image to generate a query image.

[0033] A location estimation program can divide a panoramic image into multiple small field-of-view images and extract 2D line segment information from each divided field-of-view image. For example, a panoramic image can be divided into small field-of-view images of 90 degrees × 90 degrees. Each of these divided field-of-view images is treated as a small planar image, and a portion of the spherical coordinate system can be converted to an orthogonal coordinate system and projected onto the planar image. Then, line segment information can be detected in each field-of-view image using a Line Segment Detector (LSD) or a similar algorithm. In this process, the start and end points of the detected line segments can be represented in the planar coordinate system (x, y) of each field-of-view image. Through this, the location and orientation of the line segments within the image can be clearly defined.

[0034] In addition, the position estimation program can generate a query image by aligning 2D line segment information extracted from each field of view image into the planar spherical coordinate system of the panoramic image. For example, the start and end points of the line segments detected in each field of view image can be converted to the original spherical coordinate system. Subsequently, all converted line segments are aligned and integrated on the spherical coordinate system, and the integrated line segment information is projected back onto the planar panoramic image coordinate system to finally generate a 2D query image in which all line segments are aligned.

[0035] Meanwhile, existing camera-based localization methods have two major drawbacks. First, existing methodologies utilize feature explanation vectors such as SIFT or SuperPoint, or color information from dense point clouds; however, this approach has the problem of requiring a large amount of storage capacity for maps. In contrast, since the present invention performs localization using only line segment information instead of such feature points or color information, it enables stable and efficient localization even with map sizes that are significantly smaller than those of existing methodologies. Second, among the many existing localization methods, those utilizing global feature vectors from Vision Foundation Models (such as NetVLAD or GPT-4) require neural networks trained on large-scale image data. In this case, not only is a massive amount of computation required for neural network training, but neural network inference is also necessary during the localization process, which limits its application in situations where computation is limited, such as in AR / VR. However, because the present invention utilizes only the geometric information of line segments without the use of additional neural networks, localization is possible with only a small amount of computation.

[0036] A position estimation program compares a pre-prepared 3D line map with a query image in the target space to estimate the location of the point where the query image was captured within the 3D line map. For example, the position estimation program may perform a position search process that samples various camera positions on a 3D line map and selects positions with similar line segment patterns by comparing them with line segments on a panoramic image, and a position refinement process that estimates the final location by performing alignment between the 3D line map and 2D line map information for the selected candidate locations.

[0037] Below, the location search process will be explained in detail.

[0038] FIGS. 4, FIGS. 5a, FIGS. 5b, and FIGS. 5c are drawings for explaining a method for selecting candidate locations based on a line segment and intersection distance function according to an embodiment of the present invention.

[0039] A location estimation program can perform preprocessing to classify line segments included in a 3D line segment map or line segments included in a query image according to three different principal directions, and to classify intersection points specified by the intersection of line segments of different principal directions.

[0040] A position estimation program can calculate a first vector containing the shortest distances from an arbitrary point on a sphere to each line segment included in a query image and the shortest distances from an arbitrary point to intersection points included in a query image, respectively, and a second vector containing the shortest distances from any camera poses included in a 3D line segment map to each line segment included in a 3D line segment map and the shortest distances from camera poses to intersection points included in a 3D line segment map, respectively, and based on the first vector and the second vector, select multiple locations on a 3D line segment map where the difference between each shortest distance is the smallest as candidate locations.

[0041] For example, a location estimation program can calculate the distance between a point and each line segment based on a line segment distance function, and calculate the distance between a point and each intersection point based on an intersection point distance function. For example, as shown in FIG. 4, the intersection point distance function and the line segment distance function calculated from the 2D and 3D line segments and the intersection points of the line segments, respectively, can be visualized. In this case, the line segment distance function calculates the distance from any point to the nearest line segment, and the intersection point distance function calculates the distance from any point to the nearest intersection point. Each distance function evaluates the similarity between a 2D query image and a 3D line segment map and can be used in the location search and location correction processes.

[0042] Specifically, the 2D line segment distance function and the 2D intersection point distance function are defined as in Equation 1.

[0043] [Formula 1]

[0044]

[0045] Here, x is an arbitrary point in the spherical coordinate system, L2D is a set of 2D line segments extracted from the panoramic image, and P 2D is the set of intersection points of two-dimensional line segments.

[0046] In addition, any location on a 3D map is represented by a rotation matrix R and a displacement matrix t, and the 3D line segment distance function and the 3D intersection point distance function are defined as in Equation 2. In this case, the 3D distance function is defined by projecting the line segments and their intersection points in 3D onto a sphere.

[0047] [Formula 2]

[0048]

[0049] Here, x is an arbitrary point in the spherical coordinate system, L 2D is a set of 3D line segments extracted from a 3D line segment map, and P 3D is the set of intersection points of three-dimensional line segments, and is a projection function that converts 3D coordinates into the coordinate system of a panoramic image.

[0050] Then, by comparing the distance function values ​​extracted from the query image with the distance function values ​​calculated from the 3D line segment map, K candidate locations with the smallest difference between the two values ​​can be selected. These selected candidate locations are used as input values ​​in the next step, the location correction process.

[0051] FIG. 5a is a diagram illustrating a two-dimensional intersection distance function according to an embodiment of the present invention.

[0052] For example, referring to FIG. 5a, the present invention can detect intersection points between line segments in a query image. These intersection points are structural features (e.g., corners of walls, intersections of ceilings and walls) and are used as important data in the position estimation process. Subsequently, the shortest distance is selected by calculating the distance from a point (x) on an arbitrary sphere to an intersection point (p), and a 2D intersection point distance function ( Defines ). This function defines the set of intersection points P at point x. 2D The shortest distance can be calculated for all points within.

[0053] Next, a distance map can be generated by calculating the distance function values ​​from every point in the query image to the intersection point. For example, the distance map visualizes the distance values ​​from each point to the nearest intersection point using color, where red represents the area where the nearest intersection point is located and blue represents the area where the intersection point is far away.

[0054] FIG. 5b is a diagram illustrating a two-dimensional line segment distance function according to an embodiment of the present invention.

[0055] For example, referring to FIG. 5b, the present invention can detect line segments in a query image. These line segments include structural features of the environment (e.g., geometric features such as walls, ceilings, floors, etc.). Subsequently, a set of line segments (L) at a point (x) on an arbitrary sphere 2D Calculate the distance to each line segment within ) to select the shortest distance, and based on this, a 2D line segment distance function ( Defines ).

[0056] Next, a distance map can be generated by calculating the distance function values ​​from every point in the query image to each line segment. For example, the distance map visualizes the distance from each point to the nearest line segment using color, where red represents the area close to the line segment and blue represents the area far from the line segment.

[0057] FIG. 5c is a diagram illustrating a three-dimensional distance function according to an embodiment of the present invention.

[0058] For example, referring to FIG. 5c, the present invention can project a 3D line segment map and intersection information into a spherical coordinate system by applying a rotation matrix (R) and a displacement matrix (t). This process converts the 3D line segments and intersection points into line segments and points on a sphere according to the position and orientation of the camera, and each projected line segment ( ) and projected intersection ( It can be expressed as ). Subsequently, to evaluate how well the intersection point of the 3D line segment at a specific pose (R, t) matches the 2D query image, the 3D line segment distance function ( ) and 3D intersection distance function( ) can be calculated.

[0059] Next, the present invention can generate a distance map based on calculated three-dimensional distance function values. For example, the distance map represents the distance from each point to the nearest line segment or intersection point using color, where red indicates an area with high matching accuracy (near distance) and blue indicates an area with low matching accuracy (far distance). Then, by comparing the two-dimensional distance map and the three-dimensional distance map, the location with the smallest difference in distance values ​​can be selected as the initial candidate location.

[0060] Below, the position correction process will be explained in detail.

[0061] FIG. 6 is a diagram illustrating a method for correcting position through the intersection of line segments and direction matching according to an embodiment of the present invention.

[0062] The position estimation program performs position correction on candidate positions and can estimate the position with the smallest loss function value as the final position. For example, referring to Fig. 6, the position estimation program can convert line segment and intersection information of a 2D query image and a 3D line segment map into a spherical coordinate system. Then, it can compare the distance between the intersection point of the 2D query image projected onto the sphere and the intersection point projected from the 3D line segment map, and perform line segment intersection matching to minimize the difference in distance between the intersection points. Next, it can compare the line segment direction of the 2D query image and the line segment direction projected from the 3D line segment map in the spherical coordinate system to perform line segment direction matching to minimize the difference in angle between the line segments. That is, for each of the K candidate positions selected during the position search process, position correction is performed through intersection matching and line segment direction matching, and finally, the position with the smallest loss function value can be estimated as the final position.

[0063] FIG. 7 is a flowchart illustrating a detailed process of a position estimation device according to an embodiment of the present invention estimating a position based on a panoramic image.

[0064] For example, referring to FIG. 7, the location estimation program can generate a grid by dividing a 3D line segment map along three axes (S110), and calculate the line segment distance function and intersection distance function values ​​at the center point of each grid to select the location with the smallest difference in function values ​​as a candidate location (S120). For example, step S110 can obtain a bounding box of the 3D line segment map, then generate a grid by uniformly cutting along the x, y, and z axes based on it, and set the center point of each grid as a candidate location. Step S120 can calculate the values ​​of the 2D distance function and the 3D distance function at the center point of each grid and select K locations with the smallest difference between the two function values.

[0065] In addition, the position estimation program obtains the derivative of the loss function representing the difference in the intersection points and directions of the line segments for the candidate positions (S130), repeatedly performs position correction using gradient descent (S140), and can estimate the position with the smallest loss function value among the corrected positions as the final position (S150).

[0066] For example, in step S130, a rotation matrix R representing the k-th position among the K selected positions. k and displacement matrix t k The displacement matrix can be corrected using a loss function such as Equation 3, which optimizes the alignment between the 2D intersection point and the 3D intersection point.

[0067] [Equation 3]

[0068]

[0069] Here m 2D , m 3D is current location R i , t i These are the matched points whose distance on the sphere is less than or equal to a certain threshold when the three-dimensional intersection points are projected.

[0070] Next, in step S130, the current position R i , t i Projecting 3D line segments from d to determine the direction of the matched line segments 2D , d 3D In this case, the rotation matrix can be corrected using a loss function such as Equation 4.

[0071] [Equation 4]

[0072]

[0073] In other words, the derivative of the rotational loss function and the derivative of the displacement loss function You can obtain.

[0074] In step S140, gradient descent is applied based on the gradients of the rotation loss function and the displacement loss function to the rotation matrix R of Equation 5 i and displacement matrix t i The location of can be updated.

[0075] [Formula 5]

[0076]

[0077] Here, λ is the step size of the gradient descent algorithm. By performing such updates a total of M times, a corrected position can be obtained for each selected position. Subsequently, the loss function value can be measured again for each corrected position.

[0078] Finally, the corrected position with the smallest loss function value measured in step S150 can be determined as the final position estimate.

[0079] Table 1 shows the results of a position estimation experiment of the present invention on the OmniScenes dataset consisting of 7 rooms, and the performance of the present invention was verified through such an experiment.

[0080] Location estimation method Location estimation accuracy (%) Average map size (GB) CosPlace [7] + LightGlue [8] 0.66 20.85 SFRS [9] + LightGlue [8] 0.60 21.07 The present invention 0.76 0.407

[0081] The present invention can perform stable location estimation while maintaining a small map size in various indoor environments. In experiments, accuracy was evaluated based on the error between the location estimation result and the actual ground truth. The accuracy of the location estimation was expressed as the ratio of results with an error of 0.1 m or less and an angle difference of 5 degrees or less. In addition, the 3D map size used for location estimation was measured and compared. Experimental results confirmed that the present invention achieves high location estimation accuracy compared to existing panorama-based location estimation methodologies while using the smallest 3D map size.

[0082] In the following, the description of identical configurations among the configurations illustrated in FIGS. 1 to 7 described above will be omitted.

[0083] FIG. 8 is a flowchart illustrating a method for performing position estimation based on a panoramic image of a position estimation device according to one embodiment of the present invention.

[0084] Referring to FIG. 8, a method for estimating a location based on a panoramic image using a location estimation device of the present invention includes the step of generating a query image by extracting line segment information from an input 360-degree panoramic image (S210), and the step of estimating the location of a point where the query image was captured within the 3D line segment map by comparing the query image with a 3D line segment map prepared in advance for a target space (S220). At this time, the 3D line segment map includes information on line segments determined by the intersection of each face constituting the target space in the form of a 3D map, and the query image includes line segments determined by the intersection of each face constituting the target space included in the input 360-degree panoramic image.

[0085] Step S220 may include a step of classifying line segments included in a 3D line segment map or line segments included in a query image according to three different principal directions, and a step of performing preprocessing to classify intersection points specified by the intersection of line segments of different principal directions. In addition, Step S220 may estimate possible camera rotation poses through combinatorial association of the principal directions on the panorama and the principal directions on the line segment map. For example, the three principal directions of the query image and the 3D line segment map are matched through combination, and possible combinations can be generated including each direction and sign (±). Subsequently, based on the generated combinations, the difference in direction between the panorama and the 3D map is calculated to produce a rotation matrix representing the camera's rotation state. The accuracy of the calculated rotation matrix is ​​evaluated using line segments and intersection points, and the combination with the smallest error and that best reflects the actual environment can be finally selected. Based on the selected rotation matrix, the camera's rotation pose is estimated, which allows for the alignment of the panorama and the 3D map and precise estimation of the camera's position. This contributes to dramatically improving position estimation performance by significantly reducing the search area of ​​the algorithm.

[0086] Additionally, step S220 may include the step of calculating a first vector containing the shortest distances from an arbitrary point on a sphere to each line segment included in a query image and the shortest distances from the arbitrary point to the intersection points included in the query image, respectively; the step of calculating a second vector containing the shortest distances from any camera poses included in a 3D line segment map to each line segment included in a 3D line segment map and the shortest distances from the camera poses to the intersection points included in a 3D line segment map, respectively; and the step of selecting a plurality of locations on a 3D line segment map where the difference between each shortest distance is the smallest as candidate locations based on the first vector and the second vector.

[0087] Step S220 may include a step of calculating the distance between a point and each line segment based on a line segment distance function, and a step of calculating the distance between a point and each intersection point based on an intersection point distance function.

[0088] Step S220 may include a step of performing position correction on candidate locations and estimating the location with the smallest loss function value as the final location.

[0089] The method for estimating location based on panoramic images described above may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, as well as removable and inseparable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and inseparable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0090] A person skilled in the art to which the present invention pertains will understand that, based on the foregoing description, other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0091] 100: Position estimation device 110: Communication module 120: Memory 130: Processor 140: Database

Claims

Claim 1 A position estimation device that performs position estimation based on a panoramic image, comprising a memory storing a position estimation program and a processor executing said position estimation program, wherein the position estimation program extracts line segment information from an input 360-degree panoramic image to generate a query image, and compares the query image with a pre-prepared 3D line segment map for a target space to estimate the position of a point where the query image was captured within the 3D line segment map, wherein the 3D line segment map includes information of line segments determined by the intersection of each surface constituting the target space in the form of a 3D map, and the query image includes line segments determined by the intersection of each surface constituting the target space included in the input 360-degree panoramic image. Claim 2 A position estimation device according to claim 1, wherein the position estimation program classifies a line segment included in the three-dimensional line segment map or a line segment included in the query image according to three different principal directions, and performs preprocessing to classify intersection points specified by the intersection of line segments of different principal directions. Claim 3 A position estimation device according to claim 1, wherein the position estimation program calculates a first vector including the shortest distances from an arbitrary point on a sphere to each line segment included in the query image and the shortest distances from the arbitrary point to the intersection points included in the query image, respectively, and a second vector including the shortest distances from any camera poses included in the three-dimensional line segment map to each line segment included in the three-dimensional line segment map and the shortest distances from the camera poses to the intersection points included in the three-dimensional line segment map, respectively, and, based on the first vector and the second vector, selects a plurality of positions on the three-dimensional line segment map where the difference between each shortest distance is the smallest as candidate positions. Claim 4 In paragraph 3, the position estimation device is one in which the position estimation program calculates the distance between a point and each line segment based on a line segment distance function and calculates the distance between a point and each intersection point based on an intersection point distance function. Claim 5 In paragraph 3, the position estimation device, wherein the position estimation program performs position correction on the candidate positions and estimates the position with the smallest loss function value as the final position. Claim 6 A method for estimating a location based on a panoramic image using a location estimation device, comprising: a step of generating a query image by extracting line segment information from an input 360-degree panoramic image; and a step of estimating the location of a point where the query image was captured within the 3D line segment map by comparing the query image with a 3D line segment map prepared in advance for a target space, wherein the 3D line segment map includes information on line segments determined by the intersection of each face constituting the target space in the form of a 3D map, and the query image includes line segments determined by the intersection of each face constituting the target space included in the input 360-degree panoramic image. Claim 7 A method for estimating a location according to claim 6, wherein the step of estimating the location comprises classifying a line segment included in the three-dimensional line segment map or a line segment included in the query image according to three different principal directions, and performing a preprocessing step of classifying intersection points specified by the intersection of line segments of different principal directions. Claim 8 A method for estimating a position according to claim 6, wherein the step of estimating the position comprises: a step of calculating a first vector including, respectively, the shortest distances from an arbitrary point on a sphere to each line segment included in the query image and the shortest distances from the arbitrary point to the intersection points included in the query image; a step of calculating a second vector including, respectively, the shortest distances from any camera poses included in the three-dimensional line segment map to each line segment included in the three-dimensional line segment map and the shortest distances from the camera poses to the intersection points included in the three-dimensional line segment map; and a step of selecting a plurality of positions on the three-dimensional line segment map where the difference between each shortest distance is the smallest as candidate positions based on the first vector and the second vector. Claim 9 A method for estimating a position according to claim 8, wherein the step of estimating the position includes the step of calculating the distance between a point and each line segment based on a line segment distance function, and the step of calculating the distance between a point and each intersection point based on an intersection point distance function. Claim 10 A position estimation method according to claim 8, wherein the step of estimating the position includes performing position correction on the candidate positions and estimating the position with the smallest loss function value as the final position. Claim 11 A non-transient computer-readable recording medium having a computer program recorded thereon for performing a position estimation method according to any one of paragraphs 6 through 10.

Images