Method and apparatus for registering ar images to three-dimensional map generated on basis of two-dimensional street view images

Abstract

Provided is a data processing method including: acquiring two-dimensional input images of a space captured by a user terminal and a three-dimensional map generated based on two-dimensional street view images of the space; and registering, to the three-dimensional map, three-dimensional points generated based on matching between the input images.

Description

Method and device for aligning AR images to a 3D map generated based on 2D street view images

[0001] The embodiments relate to a method and device for matching AR images that continuously capture a space to a 3D map generated based on 2D street view images of the space, and to a method and device for matching 3D points generated based on matching between AR images to the 3D map.

[0002] Street view images typically provided through map services represent 360-degree images (e.g., spherical images) of a space (e.g., an outdoor environment). These street view images include location information, such as GPS information, and direction information at the time of shooting, and can be a source for constructing a 3D map of the space represented by the street view images.

[0003] One way to create a 3D model, such as a 3D map, from 2D images is to use the Structure-from-Motion (SfM) algorithm.

[0004] 3D maps generated using the SfM algorithm from 2D images such as these street view images can be used for visual localization (VL), etc.

[0005] Meanwhile, from the perspective of improving the performance of visual localization using the 3D map generated in this way, it is required to align the continuous 2D images captured in space by the user terminal (i.e., the poses of the images) to the 3D map.

[0006] Korean Patent Publication No. 10-2020-0094644 (published on August 7, 2020) discloses a learning method and a learning device for updating an HD map by reconstructing a 3D space using depth prediction information for each object and class information for each object obtained through V2X information fusion technology.

[0007] The information described above is for the purpose of understanding only and may contain contents that do not form part of the prior art.

[0008] A data processing method can be provided, which includes obtaining a 3D map generated based on 2D input images (AR images) of a space captured by a user terminal and 2D street view images of the space, and aligning 3D points generated based on matching between the input images to the 3D map.

[0009] A method and device can be provided for generating a 3D map by optimizing 3D points generated through matching between street view images using a SfM (structure from motion) algorithm, estimating a pose of an input image based on the matching between an input image and a street view image, and generating 3D points based on the matching between the input images, while optimizing the 3D points for matching with the generated 3D map.

[0010] In one aspect, a data processing method performed by a computer system is provided, comprising: a step of obtaining two-dimensional input images of a space captured by a user terminal; a step of obtaining a three-dimensional map of the space generated based on two-dimensional street view images of the space; and a step of matching first three-dimensional points of the space generated based on matching between the input images to the three-dimensional map.

[0011] The above input images are images continuously captured in the space by the user terminal, and each input image of the input images may include location information, camera parameter information, and VIO (Visual-Inertial Odometry) information associated with each input image.

[0012] The above data processing method further includes a step of performing feature matching between a first input image among the input images and at least one first two-dimensional street view image corresponding to the first input image among the street view images, and the step of performing feature matching may include a step of filtering out incorrectly matched feature pairs among matching feature pairs between the first input image and the first two-dimensional street view image obtained according to the feature matching.

[0013] The filtering step may include: determining the first street view image, which is the target of the feature matching with the first input image, based on location information of the first input image; and the filtering step may include estimating an essential matrix between the first input image and the first street view image; and filtering out incorrectly matched feature pairs among the matching feature pairs using the essential matrix.

[0014] The above data processing method may further include a step of estimating a pose of the first input image based on the feature matching result between the first input image and the first street view image and 3D points corresponding to the first street view image on the 3D map.

[0015] The estimated pose of the first input image is a candidate initial pose of the input images, and the step of estimating the pose may include: estimating the candidate initial pose based on the feature matching result and 3D points corresponding to the first street view image using a PnP (Perspective-N-Points) algorithm; estimating the first poses of the input images by applying VIO information of each input image of the input images to the candidate initial pose; estimating the second poses of the input images based on the feature matching result between each input image and a street view image corresponding to each input image among the street view images using the PnP algorithm and 3D points corresponding to the street view images on the 3D map; and determining whether the candidate initial pose is appropriate based on a comparison between the first poses and the second poses.

[0016] Using the RANSAC (Random Sample Consensus) technique, pose estimation is repeatedly performed on the input images, so that the initial poses of the input images can be estimated.

[0017] The data processing method further includes: a step of estimating an initial pose of the input images; a step of estimating poses of the input images by applying VIO information of each input image of the input images to the initial pose; and a step of determining the first 3D points representing points matched between the adjacent input images by performing feature matching between adjacent input images among the input images and applying triangulation to the adjacent input images, wherein the matching step can match the first 3D points to the 3D map based on the poses, the 3D points on the 3D map, and the first 3D points.

[0018] The step of aligning may include optimizing the poses, the first 3D points, and the scales of the input images by optimizing at least one of a first reprojection error associated with the 3D map, a second reprojection error associated with the input images, a distance error associated with the 2D map of the space, an odometry error associated with scales of the input images, and a scale error between the input images and the 3D map.

[0019] The optimization of at least one error may be to minimize the at least one error using an algorithm based on a nonlinear least squares method.

[0020] The first reprojection error may represent a reprojection error between the first 3D points and 3D points on the 3D map, the second reprojection error may represent a reprojection error between the first 3D points and each of the input images, the distance error may represent a distance error between the first 3D points and an outline included in the 2D map, the odometry error may represent an error between a pose change amount between the poses and a change amount of odometry information between the input images, and the scale error may represent an error between a scale of each of the input images and a scale of the 3D map.

[0021] The above 3D map may be generated from the street view images using a SfM (structure from motion) algorithm.

[0022] The above 3D map is generated by a 3D map generation method based on the street view images, and the 3D map generation method may include: a step of performing feature matching between at least two street view images among the street view images; a step of applying triangulation to the at least two street view images to determine second 3D points representing points matched between the at least two street view images according to the feature matching; and a step of generating the 3D map based on the second 3D points.

[0023] The step of performing the feature matching may include a step of filtering out incorrectly matched feature pairs among the matching feature pairs between the at least two street view images obtained according to the feature matching, and the filtering step may include a step of estimating an essential matrix between the at least two street view images; and a step of filtering out incorrectly matched feature pairs among the matching feature pairs using the essential matrix.

[0024] The step of performing the feature matching and the step of determining the second 3D points included in the above 3D map generation method are performed for each tile of a plurality of tiles dividing the space, and the step of generating the 3D map can generate the 3D map by aligning the second 3D points determined for each tile.

[0025] The method for generating a three-dimensional map may further include a step of optimizing the second three-dimensional points for each tile and adjusting the scale of each tile based on at least one of first additional information including an outline included in a two-dimensional map of the space, second additional information including DEM (Digital Elevation Model) information or DSM (Digital Surface Model) information for the space, and third additional information including an ortho image for the space, and the step of generating the three-dimensional map may include a step of aligning the optimized second three-dimensional points belonging to a first tile among the tiles and the optimized second three-dimensional points belonging to a second tile among the tiles.

[0026] The method for generating a 3D map may further include a step of estimating a pose of each street view image of the street view images; and a step of optimizing the pose of each street view image and the second 3D points by optimizing at least one error among a first error between the estimated pose and a pose indicated by displacement information of each street view image, a second error which is a projection error between the second 3D points and each street view image, a third error which is a distance error between the second 3D points and an outline included in a 2D map of the space, and a fourth error between each street view image and a virtual GCP (Ground-Control Point) from at least one of DEM (Digital Elevation Model) information, DSM (Digital Surface Model) information, and an orthoimage for the space.

[0027] The optimization of at least one error may be to minimize the at least one error using an algorithm based on a nonlinear least squares method.

[0028] In another aspect, a computer system is provided, comprising at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor obtains two-dimensional input images of a space captured by a user terminal, obtains a three-dimensional map of the space generated based on two-dimensional street view images of the space, and matches first three-dimensional points of the space generated based on matching between the input images to the three-dimensional map.

[0029] By matching continuous AR images captured by a user terminal with a 3D map generated based on 2D street view images using the SfM algorithm, the pose of continuous AR images can be accurately estimated.

[0030] By being able to accurately estimate the pose of consecutive AR images captured by a user terminal, a crowd-sourced visual localization database or server can be built, and the performance of visual localization in situations where the view point changes significantly can be improved.

[0031] Additionally, through alignment between AR images and 3D maps, abundant data can be secured for training image retrieval models, (local) feature matching models, etc.

[0032] FIG. 1 illustrates a data processing method for matching input images (AR images) and a 3D map based on street view images, performed by a computer system, according to one embodiment.

[0033] FIG. 2 illustrates a computer system that performs a data processing method for matching input images and a 3D map based on street view images, according to one embodiment.

[0034] FIG. 3 is a flowchart illustrating a data processing method for aligning input images (AR images) and a 3D map based on street view images according to one embodiment.

[0035] FIG. 4 is a flowchart illustrating a method for estimating the initial poses of input images and, through optimization, aligning 3D points generated based on matching between input images to a 3D map, according to an example.

[0036] FIG. 5 is a flowchart illustrating a method for generating a 3D map based on street view images, according to an example.

[0037] FIG. 6 is a flowchart illustrating a method for optimizing 3D points and poses for generating a 3D map based on street view images, according to an example.

[0038] Figure 7 illustrates a feature matching method between an input image and a street view image according to an example.

[0039] Figure 8 illustrates a feature matching method between input images according to an example.

[0040] FIG. 9 illustrates a method for determining three-dimensional points of matching points using triangulation for images according to an example.

[0041] Fig. 10 is a flowchart illustrating a method for estimating the initial pose of input images according to an example.

[0042] Figures 11 and 12 illustrate a method for aligning an input image with a 3D map by optimizing poses, 3D points, and scales of input images, according to an example.

[0043] Figure 13 shows the results of comparing 3D maps before and after optimization of 3D points generated based on matching between street view images according to an example.

[0044] FIGS. 14 and 15 illustrate a method for generating a 3D map by optimizing 3D points generated based on matching between street view images, according to an example.

[0045] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0046]

[0047] FIG. 1 illustrates a data processing method for matching input images (AR images) and a 3D map based on street view images, performed by a computer system, according to one embodiment.

[0048] Referring to Fig. 1, a method for matching a 3D map (20) of a space and an AR image (30), which is an input image from a user terminal that photographs the space, is described.

[0049] A space may represent an outdoor environment, a composite space combining indoor and outdoor elements, or, in some embodiments, an indoor space.

[0050] A three-dimensional map (20) of a space may include a three-dimensional model of the space generated based on two-dimensional street view images that captured the space. In other words, the three-dimensional map (20) may be generated by modeling the space in three dimensions based on a plurality of street view images that captured the space. A Structure-from-Motion (SfM) algorithm may be used to generate the three-dimensional map (20) from these street view images. A method of generating the three-dimensional map (20) from street view images using SfM will be described in more detail with reference to FIGS. 5 and 6, which will be described later.

[0051] The input image (30) may be used interchangeably with the AR image (30) in the detailed description to be described later. That is, the input image (30) may be a photographed image of a space displayed on an AR (Augmented Reality) view executed on a user terminal, and may be referred to as an AR image (30). The input image (30) is an image captured through a camera of a user terminal (not shown), and may be a two-dimensional image that photographs a space. The input image (30) is obtained by continuously photographing a space, and may represent a plurality of input images. In the detailed description to be described later, the input image (30) may also represent input images (30). Each of the input images (30) may represent one frame image.

[0052] In an embodiment, these multiple input images (30) can be aligned with a 3D map (20) generated based on a plurality of street view images. The alignment here may be data alignment, for example, aligning or matching 3D points generated based on matching between input images (30) onto the 3D map (20). Alternatively, the poses of the input images (30) may be aligned with the 3D map (20) (by aligning the above 3D points onto the 3D map (20).

[0053] According to the alignment of these input images (30) to the 3D map (20), in the embodiment, the pose of consecutive AR images (30) captured by the user terminal can be accurately estimated. Therefore, the data obtained through the alignment of the input images (30) to the 3D map (20) of the embodiment can be used to build a crowd-sourced visual localization database or server, and can be used to improve the performance of visual localization in situations where the view point changes significantly. In addition, the data obtained through the alignment of the input images (30) to the 3D map (20) can be utilized to secure data for learning an image-retrieval model, a (local) feature matching model, etc.

[0054] A data processing method for matching input images (30) of an embodiment to a three-dimensional map (20) can be performed by a computer system (hereinafter, a computer system (100) to be described later with reference to FIG. 2). That is, the computer system (100) can obtain two-dimensional input images (30) that capture a space by a user terminal, and can also obtain a three-dimensional map (20) for the space generated based on two-dimensional street view images for the space, and can match the input images (30) to the three-dimensional map (20). That is, the computer system (100) can match the input images (30) to the three-dimensional map (20) by performing operations and calculations for matching three-dimensional points for the space generated based on matching between the input images (30) to the three-dimensional map (20).

[0055] The computer system (100) may be a server or a computer system constituting a server. The computer system (100) may obtain input images (30) by communicating with a user terminal, and may also obtain a three-dimensional map (20) by communicating with a computer system (or another server) that provides street view images and constructs and generates a three-dimensional map (20). Meanwhile, depending on the embodiment, the computer system (100) may be configured to provide street view images and construct and generate a three-dimensional map (20).

[0056] The specific details of the data processing method for aligning input images (30) to a 3D map (20) will be described in more detail with reference to FIGS. 2 to 15, which will be described later.

[0057]

[0058] FIG. 2 illustrates a computer system that performs a data processing method for matching input images and a 3D map based on street view images, according to one embodiment.

[0059] The computer system (100) may be an electronic device configured to perform a data processing method for matching input images (30) of the embodiment to a three-dimensional map (20). The computer system (100) may be an electronic device for executing an application, software, or program for performing such a data processing method.

[0060] For example, the computer system (100) may be a server or a computer system constituting a server, or may be a separate computer system for performing the above-described data processing method. Meanwhile, the user terminal that generates the input images (30) acquired by the computer system (100) may be a personal computer (PC), a laptop computer, a laptop computer, a smartphone, a tablet, an Internet of Things (IoT) device, or a wearable computer. The user terminal may include a camera for capturing the input images (100).

[0061] As illustrated in FIG. 2, the computer system (100) may include, as components, a memory (110), a processor (120), a communication interface (130), and an input / output interface (140). The input / output interface (140) may communicate with an input / output device (150) within the computer system (100) or separate from the computer system (100).

[0062] The memory (110) is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer system (100) as a separate permanent storage device distinct from the memory (110). In addition, the memory (110) may store an operating system and at least one program code. These software components may be loaded into the memory (110) from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components may be loaded into the memory (110) of the computer system (100) based on a computer program that is installed by files received over a network (160).

[0063] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via the memory (110) or the communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a storage device such as the memory (110).

[0064] The communication interface (130) may provide a function for the computer system (100) to communicate with other devices via a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer system (100) according to program codes stored in a recording device such as a memory (110) may be transmitted to other devices via the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer system (100) via the communication interface (130) of the computer system (100) via the network (160). The signals, commands, data, etc. received via the communication interface (130) may be transmitted to the processor (120) or the memory (110), and the files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer system (100) may further include.

[0065] The communication method through the communication interface (130) is not limited, and may include not only a communication method utilizing a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, broadcasting network) that the network (160) may include, but also short-range wired / wireless communication between devices. For example, the network (160) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. In addition, the network (160) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0066] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, a keyboard, a camera, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input / output interface (140) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (150) may also be configured as a single device with the computer system (100).

[0067] Additionally, in other embodiments, the computer system (100) may include fewer or more components than those illustrated in FIG. 2. However, most conventional components need not be explicitly depicted. For example, the computer system (100) may be implemented to include at least some of the input / output devices (150) described above, or may further include other components such as a transceiver, a camera, various sensors, a database, and the like.

[0068] The processor (120) of the computer system (100) may be configured to perform steps for performing a data processing method for obtaining input images (30) and a three-dimensional map (20) and matching the input images (30) and the three-dimensional map (20), as described below.

[0069] Meanwhile, the computer system (100) may be a server including a computer system for performing the above-described data processing method, or may be a part of a server. The computer system (100) may obtain input images (30) by communicating with a user terminal, and may also obtain a three-dimensional map (20) by communicating with a computer system (or another server) that provides street view images and constructs and generates a three-dimensional map (20). Depending on the embodiment, the computer system (100) may be configured to provide street view images and construct and generate a three-dimensional map (20).

[0070] In the detailed description to be provided below, for convenience of explanation, the above steps are described as being performed by the computer system (100), and operations performed by the processor (120) or other components of the computer system (100), or operations performed by an application / program executed by the processor (120), may be described as operations performed by the computer system (100) for convenience of explanation.

[0071] Above, the technical features described above with reference to Fig. 1 can also be applied to Fig. 2, so redundant descriptions are omitted.

[0072]

[0073] FIG. 3 is a flowchart illustrating a data processing method for aligning input images (AR images) and a 3D map based on street view images according to one embodiment.

[0074] Referring to FIG. 3, steps for performing a data processing method for matching input images (30) and a three-dimensional map (20) by a computer system (100) are described.

[0075] In step (310), the computer system (100) can obtain two-dimensional input images (30) that capture a space using a user terminal. Each input image (30) is an image captured using a camera of the user terminal, and may be a captured image of a space displayed on an AR (Augmented Reality) view running on the user terminal. Each input image (30) may be a two-dimensional image that captures a space. The input images (30) may be acquired by continuously capturing a space. For example, each of the input images (30) may represent one frame image.

[0076] In this way, the input images (30) may be images continuously captured in space by the user terminal. At this time, each input image of the input images (30) may include location information, camera parameter information, and VIO (Visual-Inertial Odometry) information associated with each input image. The location information may be, for example, GPS information, and may include rotation information of the camera as a parameter of the camera that captured each input image. For example, the camera parameter information may include 6 DoF information as degree of freedom information of the camera. The VIO information may include, for example, pose transformation information (e.g., pose transformation matrix) of each input image and the next (i.e., the next captured) input image as Visual-Inertial Odometry coordinate information associated with each input image.

[0077] In step (320), the computer system (100) can obtain a three-dimensional map (20) of the space generated based on two-dimensional street view images of the space. The three-dimensional map (20) can include a three-dimensional model of the space generated based on two-dimensional street view images that captured the space. In other words, the three-dimensional map (20) can be generated by modeling the space in three dimensions based on a plurality of street view images that captured the space. A Structure-from-Motion (SfM) algorithm can be used to generate the three-dimensional map (20) from these street view images. As illustrated in FIG. 1, the three-dimensional map (20) can be composed of three-dimensional points corresponding to the space. Meanwhile, the computer system (100) can further obtain two-dimensional street view images used to generate the three-dimensional map (20).

[0078] In step (350), the computer system (100) can align the input images (30) to the 3D map (20). For example, the computer system (100) can align the first 3D points for the space generated based on the matching between the input images (30) to the 3D map (20). Alternatively, the computer system (100) can align the (respective) poses of the input images (30) to the 3D map (20). In other words, the (respective) poses of the input images (30) can be accurately estimated through the 3D map (20).

[0079] The first 3D points are generated based on matching between input images (30) and may be 3D points in the space.

[0080] A specific method for generating the first three-dimensional points for the above alignment is described in more detail with reference to FIG. 4 (i.e., step (460) of FIG. 4) described below, and FIGS. 8 and 9.

[0081] For the above matching, as in step (330), the computer system (100) can first perform feature matching between the input image (30) and the street view image (used to generate the 3D map (20). That is, the computer system (100) can perform feature matching between a first input image (or each input image) among the input images (30) and at least one first 2D street view image corresponding to the first input image among the street view images used to generate the 3D map (20). The first input image can be any one of the input images (30), and step (330) can be performed for each of the input images (30). The first 2D street view image(s) can be determined based on positional information (i.e., comparison of positional information) between the input images (30) and the street view images. For example, street view image(s) having location information at a certain distance or a certain radius with respect to the location information of the first input image can be determined as the first 2D street view image. Feature matching between these images can be point matching between the images. That is, feature matching can be matching feature points (e.g., vertices of objects in the image, points on edges, etc.) of the first input image with corresponding feature points (e.g., vertices of corresponding objects in the image, points on corresponding edges, etc.) on the first (2D) street view image. For this feature matching, the computer system (100) can first perform feature extraction for the first input image and the first street view image corresponding thereto. That is, the computer system (100) can extract feature points from the first input image and the first street view image corresponding thereto. Each feature point can represent a pixel (pixel coordinate) on the image.

[0082] The computer system (100) can perform the feature matching by identifying feature points on the first street view that match feature points extracted from the first input image. Through this feature matching, a relationship (i.e., a 2D-3D correspondence) between the first input image and a 3D map (20) generated based on the first street view image can be defined. This defined relationship can represent a 2D-3D relationship between pixel coordinates corresponding to feature points of the first input image and corresponding pixel coordinates on the 3D map (20).

[0083] The first input image and the first street view image on which feature matching is performed may be a 360-degree image (i.e., a spherical image) or a spherical image cropped into the form of a general image.

[0084] Meanwhile, as in step (335), the computer system (100) may perform a task of filtering out incorrectly matched feature pairs in the feature matching described above. In other words, the computer system (100) may filter out incorrectly matched feature pairs among the matching feature pairs between the first input image and the first 2D street view image obtained according to the feature matching. The 'matching feature pair' may represent a pair of matching feature points between the first input image and the first 2D street view image. The filtering may be to exclude incorrectly matched feature pair(s) from the feature matching result.

[0085] For example, at step (336), the computer system (100) can estimate an essential matrix between the first input image and the first street view image.

[0086] At step (336), the computer system (100) can filter out incorrectly matched feature pairs among the initially determined matching feature pairs using the estimated essential matrix.

[0087] In this way, noise during feature point matching can be filtered out.

[0088] The essential matrix can be a 3*3 matrix representing the pose change between the (first input) image and the (first street view) image. In other words, the essential matrix can be a 3*3 matrix representing the coordinates of a point in the images captured by two cameras, where it is located in the image of the other camera.

[0089] For example, in an embodiment, using these essential matrices, a line can be formed connecting a feature point on a query image (input image) with a specific point on a reference image (first street view image), and filtering can be performed (excluding the line if there is no corresponding specific point) by determining whether there is a corresponding feature point on the line.

[0090] In this way, feature matching can be performed between the input images (30) and the street view images constituting the 3D map (20), and inappropriate matching point pairs during feature matching can be excluded from subsequent matching.

[0091] Figure 7 illustrates a feature matching method between an input image and a street view image according to an example.

[0092] In the illustrated example, a first input image (710) and a first street view image (720) are illustrated, and a line (730) representing a pair of matching feature points between the first input image (710) and the first street view image (720) is further illustrated. As illustrated, the feature points to be matched may include vertices of buildings, etc. The illustrated example may represent the result of feature matching after the aforementioned filtering is performed or before the filtering is performed. For convenience of notation, the line (730) in FIG. 7 may not accurately represent the pair of matching feature points between the first input image (710) and the first street view image (720). Therefore, unlike illustrated, in reality, the line (730) representing the pair of matching feature points between the first input image (710) and the first street view image (720) (after the aforementioned filtering) may accurately connect the points that are matched between the two images.

[0093] Both the first input image (710) and the first street view image (720) may be two-dimensional images. The first street view image (720) is illustrated as a spherical image, which is a 360-degree image; however, a cropped image of the spherical image, contrary to the illustration, may also be used for feature matching.

[0094] In step (340), the computer system (100) can estimate poses of the input images (30) using the results of the above feature matching. These estimated poses can be used to more accurately match the input images (30) to the 3D map (20).

[0095] For example, the computer system (100) can estimate the pose of the first input image based on the feature matching result (e.g., after filtering by the aforementioned steps (336 and 337)) between the first input image and the first street view image and 3D points corresponding to the first street view image on the 3D map (20).

[0096] For example, the computer system (100) can estimate the initial pose of the input images (30), i.e., the pose of the first input image of the input images (30), and can estimate the poses of the input images (30) (i.e., the pose of each input image (30)) by applying the aforementioned VIO information to this initial pose.

[0097] These estimated initial poses and poses can be used as information to more accurately align input images (30) to a 3D map (20).

[0098] A specific method for estimating the initial pose and poses of input images (30) and a data processing method for aligning the input images (30) to a 3D map (20) are described in more detail with reference to FIG. 4 described below.

[0099] Above, the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIGS. 3 and 7, so redundant descriptions are omitted.

[0100]

[0101] FIG. 4 is a flowchart illustrating a method for estimating the initial poses of input images and, through optimization, aligning 3D points generated based on matching between input images to a 3D map, according to an example.

[0102] In Fig. 4, a specific method for estimating the initial pose and poses of input images (30) and a data processing method for aligning the input images (30) to a 3D map (20) are more specifically illustrated.

[0103] A method for measuring the initial pose and poses of input images (30) through step (400) is described in more detail.

[0104] The pose of the first input image estimated in step (340) may be a candidate initial pose of the input images (30).

[0105] In step (410), the computer system (100) can estimate the candidate initial pose based on the feature matching result between the first input image and the first street view image and the 3D points corresponding to the first street view image on the 3D map (20). For example, the computer system (100) can estimate the candidate initial pose using the PnP (Perspective-N-Points) algorithm. That is, using the PnP algorithm, if the 3D points and the 2D projection image are known, the pose of the camera can be estimated, and thus, the candidate initial pose, which is the pose of the first input image, can be estimated as described above.

[0106] In step (420), the computer system (100) can estimate the first poses of the input images (30) by applying the VIO information of each input image of the input images (30) to the estimated candidate initial pose. The VIO information includes pose transformation information of each input image and the next (i.e., the next captured) input image, so that the poses of the input images (30) can be estimated by step (420).

[0107] In step (430), the computer system (100) can estimate second poses of the input images (30) based on a feature matching result between each input image of the input images (30) and a street view image corresponding to each input image among the street view images used to generate the 3D map (20) (i.e., constituting the 3D map (20)) and 3D points corresponding to the corresponding street view images on the 3D map, using a PnP algorithm. In other words, the second poses can be poses of the input images (30) calculated using the PnP algorithm.

[0108] In step (440), the computer system (100) can determine whether the candidate initial pose is suitable based on a comparison between the first poses and the second poses. The computer system (100) can determine the candidate initial pose as suitable if the degree of agreement between the first poses and the second poses is above a certain level, for example, if the number of poses corresponding to inliers is above a certain level.

[0109] In step (450), the computer system (100) can determine the candidate initial pose as the initial pose of the input images (30) if the candidate initial pose is suitable, and can determine the poses of the input images (30) of the first poses.

[0110] Meanwhile, if the candidate initial pose is not suitable, the computer system (100) can estimate the candidate initial pose for input images other than the first input image and repeat the above-described steps (410 to 440).

[0111] For example, the computer system (100) can estimate the initial pose of the input images (30) by repeatedly performing pose estimation on the input images (30) using the RANSAC (Random Sample Consensus) technique, and thus can estimate the poses of the input images (30).

[0112] In relation to this, with reference to FIG. 10, a method for estimating the initial pose of input images (30) is described in more detail. FIG. 10 is a flowchart illustrating a method for estimating the initial pose of input images according to an example.

[0113] In the embodiment, in order to achieve rapid convergence and stable optimization of data alignment in alignment of input images (30) to a three-dimensional map (20), it is necessary to accurately estimate the initial pose of the input images (30).

[0114] The RANSAC technique can be used to estimate these initial poses. For example, the 3D pose of each input image can be estimated using the feature matching results between each input image and the street view image and the 3D points on the 3D map (20). At this time, the VIO information (i.e., odometry data) included in each input image, i.e., acquired together with each input image, can be used to estimate the initial pose of the input images (30).

[0115] Specifically, as in step (1010), the computer system (100) can obtain a feature matching result (2D-2D feature matching result) between each of the input images (AR images) (30) and the street view images.

[0116] At step (1020), the computer system (100) can randomly select a matching pair. In other words, the computer system (100) can select a pair of an arbitrary input image and a street view image matching it.

[0117] In steps (1030) and (1040), the computer system (100) can determine a 2D-3D correspondence of a pair of selected images, and can estimate the pose of the input image of the pair of selected images as an initial pose (i.e., a candidate initial pose) of the input images (30) using a PnP algorithm.

[0118] At step (1050), the computer system (100) can estimate poses of the input images (30) using the estimated initial pose and VIO information (i.e., odometry data). These poses may correspond to the first poses described above.

[0119] Meanwhile, a pose for each of the input images (30) can be estimated in the same manner as in step (1030), and accordingly, the estimated poses of the input images (30) can correspond to the second poses described above.

[0120] At step (1060), the computer system (100) can compare the first poses and the second poses.

[0121] In step (1070), the computer system (100) can determine the number of poses corresponding to inliers as a result of the comparison of poses. For example, the computer system (100) can count cases where the difference between the poses as a result of the comparison of the corresponding first pose with the second pose is within a certain range.

[0122] In step (1080), the computer system (100) can determine whether the number of inliers exceeds a predetermined number (N), and if the number of inliers exceeds the predetermined number (N) in step (1090), the initial pose determined in step (1040) can be determined as a suitable initial pose of the input images (30). Meanwhile, if the initial pose determined in step (1040) is not suitable, a pair of a random input image and a street view image matching it can be selected again, and steps (1030 to 1080) can be repeated.

[0123] As described above, the computer system (100) can determine the initial pose of the input images (30) and estimate the poses of the input images (30) by applying the VIO information of each input image of the input images (30) to the initial pose.

[0124] Estimating this accurate initial pose is crucial because it serves as a state value for solving the nonlinear least-squares problem of minimizing reprojection errors, which will be described later. As explained, the initial pose can be estimated using the RANSAC technique. The method described above can be summarized using AR image terms instead of input image terms as follows.

[0125] That is, the computer system (100) can obtain the pose of each AR image using the aforementioned 2D-3D relationship and Perspective-N-Points (PnP) algorithm, and can convert the estimated pose for each AR image into a pose at a specific time point of the AR dataset (i.e., the input images (30)) using odometry information between the AR images. The computer system (100) can then convert the converted pose at a specific time point into a pose for each AR image using odometry, and can determine whether the estimated pose is appropriate by calculating an error with the pose estimated through Perspective-N-Points (PnP) in each AR image. Here, an initial value recognition judgment (a judgment using a simple threshold value) can be applied. The computer system (100) can estimate the initial pose of the most suitable AR dataset by recursively counting the number of AR images judged to be suitable (i.e., the number of poses; the number of inliers) and repeating the process to obtain the largest number of inliers.

[0126] Meanwhile, in step (460), the computer system (100) can perform feature matching between adjacent input images among the input images (30), and can determine 3D points representing points that are matched between the adjacent input images by applying triangulation to the adjacent input images (e.g., two adjacent input images). The 3D points thus determined can be the first 3D points described above with reference to FIG. 3.

[0127] For feature matching between input images, the aforementioned feature matching between input images and street view images can be applied similarly. Similarly, the aforementioned filtering can be applied similarly to feature matching between input images. Duplicate explanations in this regard will be omitted.

[0128] In relation to this, Fig. 8 illustrates a feature matching method between input images according to an example.

[0129] In the illustrated example, among the input images, a second input image (810) and an adjacent third input image (820) are illustrated, and a line (830) representing a pair of matching feature points between the second input image (810) and the third input image (820) is further illustrated. As illustrated, the feature points to be matched may include vertices of buildings, etc. The illustrated example represents the result of feature matching after the aforementioned filtering is performed or before the filtering is performed. Both the second input image (810) and the third input image (820) may be two-dimensional images. For convenience of notation, the line (830) in FIG. 8 may not accurately represent a pair of matching feature points between the second input image (810) and the third input image (820). Therefore, unlike what is shown, in reality, the line (830) representing the matching feature point pair between the second input image (810) and the third input image (820) (after the aforementioned filtering) may accurately connect the matching points between the two images.

[0130] Meanwhile, FIG. 9 illustrates a method for determining a three-dimensional point of a matching point using triangulation for images according to an example.

[0131] In Fig. 9, a second input image (910) and a third input image (920) corresponding to a second input image (810) and an adjacent third input image (820) are illustrated. As illustrated, a three-dimensional point P (930) corresponding to points (P1, P2) matching each other in the second input image (910) and the third input image (920) can be generated using triangulation.

[0132] In this triangulation method, the value of a 3D point can be calculated using the epipolar constraint. The epipolar constraint means that a large plane, the epipolar plane, is created by connecting the focal points of the cameras that captured the two images and the 3D points. Here, the epipolar plane can be determined using the actual 3D points, but it can also be determined through the plane equation obtained using the feature points of the detected images and the 3D vector equation obtained using the center point of each image. If the plane obtained in this way is projected onto each image captured by the camera, an epipolar line can be obtained. By using this epipolar line and the 3D vector equation of the feature points, the 3D point can be estimated inversely.

[0133] For reference, the essential matrix mentioned above can also be derived from the epipolar constraint.

[0134] In step (350), the computer system (100) can align the first 3D points for the space generated based on matching between input images (30) and triangulation to the 3D map (20). For example, the computer system (100) can align the first 3D points to the 3D map (20) based on the estimated poses of the input images (30), the 3D points on the 3D map (20), and the first 3D points.

[0135] At this time, in step (350), the computer system (100) can optimize the estimated poses, the first 3D points, and the scales of the input images (30) by optimizing errors resulting from various factors.

[0136] Above, the technical features described above with reference to FIGS. 1 to 3 and FIG. 7 can also be applied to FIGS. 4 and 8 to 10, so redundant descriptions are omitted.

[0137]

[0138] Figures 11 and 12 illustrate a method for aligning an input image with a 3D map by optimizing poses, 3D points, and scales of input images, according to an example.

[0139] As described above, the computer system (100) can optimize the estimated poses of the input images (30), the first 3D points determined from the input images (30), and the scales of the input images (30) by optimizing errors resulting from various factors.

[0140] For example, the computer system (100) can optimize at least one of a first reprojection error associated with the 3D map (20), a second reprojection error associated with the input images (30), a distance error associated with the 2D map of space, an odometry error associated with the scales of the input images (30), and a scale error between the input images (30) and the 3D map (20) to align the first 3D points to the 3D map (20), and the optimization can optimize the estimated poses of the input images (30), the first 3D points determined from the input images (30), and the scales of the input images (30).

[0141] Optimization of the above error can be achieved by minimizing the error using an algorithm based on non-linear least squares.

[0142] Alternatively, the computer system (100) can optimize the poses, the first 3D points, and the scales through error optimization using a graph-based method, as illustrated in FIGS. 11 and 12. That is, the computer system (100) can estimate the desired poses of the input images (30) by optimizing the errors by expressing the estimated poses of the input images (30), the 3D points on the 3D map (20), the first 3D points, etc. as nodes, and expressing the factors of errors due to various factors as links, respectively. The computer system (100) can then estimate the desired poses of the input images (30) and align the first 3D points to the 3D map (20) by optimizing the errors.

[0143] In relation to this, FIGS. 11 and 12 can represent before and after optimizing errors, respectively.

[0144] Referring to FIGS. 11 and 12, the above-described error optimization method is described in more detail.

[0145] In the illustrated graphs (1100, 1200), a node (1110) may represent each of the estimated poses of the input images (30). A node (1120) may represent each of the first 3D points determined by matching between the input images (30) (i.e., determined by triangulation). A node (1130) may represent each of the 3D points on a 3D map (20) generated based on street view images. A node (1140) may be a node designated to correct a planar translation error (in the x-axis and y-axis directions) when capturing each of the input images (30). A link (1180) may represent a factor for determining this planar translation error.

[0146] Meanwhile, the first reprojection error may represent a reprojection error between the first 3D points (1120) and (corresponding) 3D points (1130) on the 3D map (20). The link (1150) may represent a factor for determining the first reprojection error.

[0147] The second reprojection error may represent a reprojection error between each of the first 3D points (1120) and the input images (30). That is, the second reprojection error may represent a reprojection error between the first 3D points generated through triangulation between adjacent input images and the corresponding input images.

[0148] The above distance error may represent a distance error between the first 3D points (1120) and an outline (1190) included in a 2D map of space. The 2D map is a 2D map separate from the 3D map (20), and may be, for example, a commercial 2D map provided through a map provision service. The outline (1190) may represent an outline of a feature (object such as a building or facility) included in the 2D map. The 3D map (20) (i.e., the 3D points (1130) on the 3D map (20)) may be matched to the outline (1190) of the 2D map. The link (1160) may represent a factor for determining this distance error. The distance error may also be referred to as a 2D line - 3D point distance error.

[0149] The above odometry error may represent an error between the amount of pose change between poses (1110) of input images (30) and the amount of change in odometry information between input images (30). The link (1170) is a factor for determining the above odometry error, and a pose considering the scale (i.e., the difference in scale between the input images (30)) may be a factor (or between factor) for determining the error. The link (1170) may represent a pose transformation matrix between poses (1110).

[0150] The above scale error may represent an error between the scale of each of the input images (30) and the scale of the 3D map (20). In addition, the illustrated scale factor (S) may be a factor for determining this scale error. The scale factor (S) may represent the difference in scale between the odometry scale obtained when acquiring each input image and the 3D map (20) (i.e., street view SfM).

[0151] That is to say, in the embodiment, a reprojection error (second reprojection error) between the first 3D points acquired through triangulation for the input images (30) and the input images (30); a reprojection error (first reprojection error) between the 3D points from the 3D map (20) which is the street view SfM and the input images (30) (i.e., the first 3D points); and a distance error between the first 3D points and the outline of the 2D map can be defined, and further, an odometry error considering the scale between consecutive input images can be defined. Here, the odometry error is an error defined between consecutive input images, and may be an error for the difference between the pose variation of the poses of the input images (30) estimated during optimization and the variation of the odometry information acquired when acquiring the dataset of the images (30). Additionally, a scale error is defined to indicate the scale difference between the input images (30) and the 3D map (20) which is a street view SfM, so that optimization considering the scale difference can be performed.

[0152] In an embodiment, after all of the above errors are defined, the computer system (100) can perform optimization for the first 3D points, poses of the input images (30), and odometry scales of the input images (30) using a non-linear least squares solver.

[0153] According to this optimization, a matching result as illustrated in FIG. 12 can be obtained. In the illustrated example, it can be confirmed that the first 3D points (1120) of FIG. 11 are aligned with the 3D map (20) (and the outline (1190) of the 2D map), and the poses (1110) are also optimized.

[0154] Above, the technical features described above with reference to FIGS. 1 to 4 and FIGS. 7 to 10 can also be applied to FIGS. 11 and 12, so redundant descriptions are omitted.

[0155]

[0156] Below, we describe in more detail how to create a 3D map (20) from street view images of a space.

[0157] FIG. 5 is a flowchart illustrating a method for generating a 3D map based on street view images, according to an example.

[0158] The 3D map (20) generated based on the aforementioned street view images may be generated from the street view images using the SfM (structure from motion) algorithm.

[0159] In an embodiment, by generating a 3D map (20) from street view images, a pre-scanning operation for generating the 3D map (20) may not be required, and thus, time and cost efficiency for generating the 3D map (20) may be increased. In addition, the position and pose of inaccurate street view images may be corrected through the generation of the 3D map (20).

[0160] Steps (510 to 542) may represent a method for generating a 3D map based on these street view images. For convenience of explanation, the computer system (100) is described below as performing this 3D map generation method. However, the 3D map generation method may be performed by a computer system other than the computer system (100).

[0161] In step (510), the computer system (100) may acquire street view images of a space. The street view images may be, for example, images provided through a road view service, and each may be a 360-degree image (spherical image). Each street view image may include location information (GPS information) and direction information (i.e., camera direction information when capturing the street view image).

[0162] In an embodiment, location information and rotation information (such as GPS and IMU data) of a street view image are used to create a 3D map (20), thereby enabling implementation of a 3D map of actual scale.

[0163] In step (520), the computer system (100) may perform feature matching between at least two street view images among the acquired street view images. The street view images to be subjected to feature matching may be adjacent street view images (i.e., street view images captured consecutively). Additionally, the street view images to be subjected to feature matching may be images obtained by cropping street view images that are 360-degree images (spherical images).

[0164] For feature matching between street view images, the aforementioned feature matching between input images and street view images, or between input images, can be applied similarly. Similarly, the aforementioned filtering can be applied similarly to feature matching between street view images. Therefore, any redundant explanations can be omitted.

[0165] For example, in step (525), when the computer system (100) performs feature matching, the computer system (100) may filter out incorrectly matched feature pairs among matching feature pairs between at least two street view images obtained through feature matching between the at least two street view images. At this time, in step (526), ​​the computer system (100) may estimate an essential matrix between the at least two street view images. In step (527), the computer system (100) may filter out incorrectly matched feature pairs among the matching feature pairs using the essential matrix.

[0166] The essential matrix can be a 3*3 matrix representing the pose change between street view images. In other words, the essential matrix can be a 3*3 matrix representing the coordinates of a point in an image captured by two cameras, where it is located in the image captured by the other camera.

[0167] For example, in an embodiment, using these essential matrices, a line can be formed connecting a feature point on a query image (a street view image) to a specific point on a reference image (another street view image), and filtering can be performed (excluding the line if there is no corresponding specific point) by determining whether there is a corresponding feature point on the line.

[0168] In this way, feature matching can be performed between street view images, and inappropriate matching point pairs during feature matching can be excluded in the subsequent generation of a 3D map (20).

[0169] In step (530), the computer system (100) can determine second 3D points representing points that are matched between the at least two street view images (i.e., matched points after filtering) according to the feature matching by applying triangulation to the at least two street view images on which feature matching has been performed. With respect to the method of determining 3D points based on triangulation, the description of the method of generating 3D points corresponding to the matching points of the input images (810, 820; 910, 920) described above with reference to FIG. 9 can be similarly applied. Therefore, redundant descriptions can be omitted. The 3D points generated in step (530) can be named second 3D points. The computer system (100) can use location information (e.g., GPS information) and direction information (e.g., shooting angle information of the street view image acquired from the IMU) of each street view image to generate a three-dimensional point identical to the actual scale.

[0170] At step (540), the computer system (100) can generate a three-dimensional map (20) based on the second three-dimensional points generated by applying this triangulation method.

[0171] Meanwhile, the above-described steps (510 to 530, or 520 to 530) may be performed for each tile of a plurality of tiles that divide a space for which a three-dimensional map (20) is to be created. In other words, the three-dimensional map (20) may be created by matching (combining) the tasks performed for each tile that divides the space.

[0172] For example, the steps (520 to 527) of performing the aforementioned feature matching included in the 3D map generation method and the step (530) of determining second 3D points may be performed for each tile of a plurality of tiles dividing the space. The computer system (100) may generate a 3D map (20) by aligning the second 3D points determined for each tile.

[0173] In an embodiment, parallel processing per tile can be performed to generate a 3D map (20), and scalability can be secured.

[0174] Meanwhile, in determining the second 3D points of step (530) (or, for the determined second 3D points), an optimization task can be performed.

[0175] For example, the computer system (100) can optimize the second 3D points determined for each tile and adjust the scale of each tile based on at least one of the following additional information: first additional information including an outline included in a 2D map of the space; second additional information including DEM (Digital Elevation Model) information or DSM (Digital Surface Model) information for the space; and third additional information including an ortho image (or aerial image) for the space.

[0176] The computer system (100) can generate a three-dimensional map (20) by, in step (542), aligning optimized second three-dimensional points belonging to a first tile among the tiles with optimized second three-dimensional points belonging to a second tile among the tiles.

[0177] A more specific method for optimizing the second 3D points will be described in more detail with reference to FIGS. 6, 13, and 15, which will be described later.

[0178] Above, the technical features described above with reference to FIGS. 1 to 4 and FIGS. 7 to 12 can also be applied to FIG. 5, so redundant descriptions are omitted.

[0179]

[0180] FIG. 6 is a flowchart illustrating a method for optimizing 3D points and poses for generating a 3D map based on street view images, according to an example.

[0181] Referring to FIG. 6, a method for optimizing 3D points (i.e., second 3D points) constituting a 3D map (20) to generate a 3D map (20) will be described in more detail.

[0182] In relation to this, Fig. 13 shows the results of comparing 3D maps before and after optimization of 3D points generated based on matching between street view images according to an example.

[0183] Compared to the 3D map (1310) before optimization, the 3D map (1320) after optimization can be confirmed to better match the actual space. That is, in the 3D map (1320) after optimization, the outlines of objects (buildings, etc.) within the space can be more clearly defined. The computer system (100) can optimize 3D points by optimizing errors arising from various factors, and thus can ultimately generate a 3D map (20) such as the 3D map (1320) after optimization.

[0184] In step (610), the computer system (100) can estimate the pose of each street view image of the street view images. For example, the computer system (100) can estimate the pose of each street view image by using displacement information (i.e., location information (e.g., GPS information) and direction information (e.g., shooting angle information of the street view image acquired from the IMU)) of each street view image. However, since the elevation information included in the displacement information included in the street view image is not accurate, DEM information or DSM information can be used as such elevation information. In other words, the computer system (100) can estimate the pose of each street view image (i.e., 'displacement information-based pose') by using the displacement information of each street view image, but using the displacement information in which the elevation information of each street view image has a replaced value based on the DEM information or DSM information.

[0185] Additionally, the computer system (100) can estimate the pose of each street view image based on the second 3D points determined in association with each street view image. Accordingly, the estimated pose can be referred to as a "street view-based pose."

[0186] Considering the possibility of errors in the location and direction information at the time of creation of the street view image, 'street view-based pose' can be considered more reliable than 'displacement information-based pose'.

[0187] At step (620), the computer system (100) can optimize the pose and second 3D points of the street view image by utilizing additional information and / or optimizing errors due to various factors. Based on this optimization, a final 3D map (20) can be generated.

[0188] For example, the computer system (100) can optimize the pose of each street view image and the second 3D points by optimizing at least one error among a first error between the estimated street view-based pose for each street view image and the displacement information-based pose indicated by displacement information (position information and direction information, for example, angle information obtained from GPS information and IMU, etc.) of each street view image, a second error which is a projection error (i.e., a spherical projection error) between the second 3D points and each street view image, a third error which is a distance error between the second 3D points and an outline included in a 2D map of the space, and a fourth error between each street view image and a virtual GCP (Ground-Control Point) from at least one of DEM (Digital Elevation Model) information, DSM (Digital Surface Model) information, and orthophoto (or aerial image) for the space.

[0189] Optimization of the above error can be achieved by minimizing the error using an algorithm based on non-linear least squares.

[0190] As mentioned above, the 'street view-based pose' can be considered more reliable than the 'displacement information-based pose', and taking this into account, the pose of each street view image can be optimized.

[0191] The 2D map is a 2D map separate from the 3D map (20), and may be, for example, a commercial 2D map provided through a map provision service. The outline may represent the outline of a feature (object such as a building or facility) included in the 2D map. The 3D map (20) (i.e., 3D points on the 3D map (20)) may be matched to the outline of the 2D map.

[0192] A virtual GCP (Ground-Control Point) can be determined based on at least one of DEM information, DSM information, and orthophoto (or aerial image), and the error of the street view image based on this virtual GCP can be utilized for optimization of the second 3D points. The virtual GCP can be, for example, the outline of a feature existing in the aerial image (e.g., the outline of a building, a road vehicle, a crosswalk, or other facilities). The virtual GCP can correspond to the corresponding feature in the street view image.

[0193] Below, a method for optimizing the pose of each street view image and the second 3D points through error optimization using a graph-based method, as illustrated in FIGS. 14 and 185, by using a computer system (100), is described in more detail. The computer system (100) can optimize the pose of each street view image and the second 3D points by expressing the 'street view-based pose' and the 'displacement information-based pose' of each street view image, the second 3D points, virtual GCPs, etc. as nodes, and expressing factors of errors due to various factors as links, thereby optimizing the errors, thereby generating a 3D map (20).

[0194] FIGS. 14 and 15 illustrate a method for generating a 3D map by optimizing 3D points generated based on matching between street view images, according to an example.

[0195] Figures 14 and 15 can show before and after optimizing errors, respectively.

[0196] In the illustrated graphs (1400, 1500), node (1410) may represent a street view-based pose as an estimated pose of each street view image. Node (1440) may represent a displacement information-based pose as an estimated pose of each street view image. Node (1420) may represent a virtual GCP determined based on at least one of an orthophoto (aerial photograph), DEM information, and DSM information. Node (1430) may represent the second 3D point described above.

[0197] As described above, a first error between an estimated street view-based pose (1410) for each street view image and a displacement information-based pose (1440) of each street view image can be optimized. Link (1480) can represent a factor for determining the first error. In addition, a second error, which is a projection error (i.e., a spherical projection error) between the second 3D points (1430) and each street view image, can be optimized. Link (1450) can represent a factor for determining the second error (i.e., a spherical coordinate projection error factor). In addition, a third error, which is a distance error between the second 3D points (1430) and an outline (1490) included in a 2D map, can be optimized. Link (1470) can represent a factor for determining the third error. Additionally, a fourth error between each street view image and a virtual GCP (Ground-Control Point) determined from at least one of DEM information, DSM information, and orthophoto (or aerial image) can be optimized. Link (1460) can indicate a factor for determining the fourth error (i.e., a GCP point spherical projection error factor).

[0198] In an embodiment, after all of the above errors are defined, the computer system (100) can optimize the pose of each street view image and the second 3D points by minimizing the errors using a non-linear least squares solver, thereby generating a 3D map (20).

[0199] According to this optimization, a result as illustrated in Fig. 15 can be obtained. In the illustrated example, it can be confirmed that the 3D points (1430) of Fig. 14 are aligned with the outline (4190) of the 2D map, and the pose (1410) of each street view image is also optimized.

[0200] Above, the technical features described above with reference to FIGS. 1 to 5 and FIGS. 7 to 12 can also be applied to FIGS. 6 and 13 to 15, so redundant descriptions are omitted.

[0201]

[0202] Below, the SfM technique (SfM algorithm) applicable to the generation of a 3D map (20) in an embodiment is described in more detail.

[0203] The SfM technique can be composed of the following steps:

[0204] First, feature extraction can be performed to detect corner points or other characteristic points (pixels) from target images. During this feature extraction, descriptors that describe each pixel (each point) can be generated together. The characteristics of this feature extraction can be similarly applied to feature extraction for feature matching described above. After all feature extraction is performed, feature matching can be performed between target images. The feature matching process can be a process of finding the most similar pair of matching points between images using the descriptors. If matching point pairs are generated using only descriptors, a significant number of mismatched pairs can be detected. Therefore, filtering can be performed to filter out such mismatches. For example, a geometric verification technique can be used for filtering. This can be a technique for filtering out mismatched pairs that violate the geometric transformation relationship between two images, such as by estimating a homography, essential matrix, or fundamental matrix. Other filtering techniques may include ratio tests and cross-check methods.

[0205] After these matching point pairs are generated, 3D points can be generated using the triangulation method described above.

[0206] The generated 3D points may not have accurate values ​​due to the pose or matching error of the image, so an optimization process can be performed. That is, optimization can be performed to correct the pose of the image and the 3D points by utilizing the matching point pair information and 3D points obtained from multiple views. To perform the optimization, the error (error) can be defined first. The defined error can be, for example, the reprojection error. This can be calculated as the difference between the pixel value of the actually detected feature and the projected pixel value when the 3D point is projected onto the image plane using the image pose and camera parameters. The 3D point and image pose values ​​can be corrected so that this error is minimized. Once the reprojection error and state values ​​(image pose, 3D point, camera parameters, etc.) for all detected matching feature pairs are defined, optimization can be performed by solving this as a nonlinear least squares problem. For example, a recursive Gauss-Newton method can be utilized to solve the above problem.

[0207] Meanwhile, as an example, the characteristics of a possible SfM technique (SfM algorithm) applied to create a 3D map (20) with street view images are described.

[0208] As mentioned above, street view images can be spherical images (360-degree images). For example, street view images can be repeatedly perspective cropped at specific angles to ensure proper feature extraction. Feature extraction can be performed on each of these cropped images, which can then be converted into pixel coordinates that match the spherical image. Feature matching can be performed using the extracted features, and filtering can be performed, for example, using an essential matrix. That is, matching point pairs that do not match the essential matrix can be filtered.

[0209] Additionally, as described above, displacement information of each street view image is used to generate displacement information-based poses, but the elevation information in the displacement information can be replaced using DEM or DSM information. Using this displacement information-based pose and matching feature pairs, a 3D point (i.e., a second 3D point) can be generated using triangulation. The generated second 3D point can be generated using GPS information and DEM or DSM information and can have a metric scale.

[0210] Meanwhile, the pose and second 3D points of the street view image can be optimized to more reliably ensure metric scale. Optimization can be performed by performing feature matching between the street view image and the orthoimage and DEM or DSM information. Using the orthoimage and DEM or DSM information, accurate 3D point values ​​on the Earth's surface can be obtained for each image pixel. By feature matching between the orthoimage and an image corresponding to the street view image floor, 3D points corresponding to the feature points in the street view image can be obtained. Optimization can be performed to reduce the reprojection error between the 3D points obtained in this way and the street view image feature points. Additionally, by utilizing a 2D map, an outline of the 2D map can be obtained, and corresponding 3D points can be selected. The difference between the selected 3D points and the outline can be defined as an error, and optimization can be performed to minimize the error. After the error is defined, it can be minimized by solving a nonlinear least squares problem, thereby completing the optimization. Through this, a three-dimensional map (20) of the embodiment can be obtained.

[0211] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0212] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0213] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0214] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0215] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In a data processing method performed by a computer system, A step of acquiring two-dimensional input images capturing a space by a user terminal; A step of obtaining a three-dimensional map of the space generated based on two-dimensional street view images of the space; and A step of aligning the first 3D points for the space generated based on the matching between the input images to the 3D map. A method of processing data, comprising:

2. In paragraph 1, The above input images are images continuously captured in the space by the user terminal, A data processing method, wherein each input image of the above input images includes position information, camera parameter information, and VIO (Visual-Inertial Odometry) information associated with each input image.

3. In paragraph 1, A step of performing feature matching between a first input image among the input images and at least one first 2D street view image corresponding to the first input image among the street view images. Including more, The steps for performing the above feature matching are: A step of filtering out incorrectly matched feature pairs among the matching feature pairs between the first input image and the first two-dimensional street view image obtained through the above feature matching. A method of processing data, comprising:

4. In paragraph 3, The first street view image, which is the target of the feature matching with the first input image, is determined based on the location information of the first input image, The above filtering step is, A step of estimating an essential matrix between the first input image and the first street view image; and A step of filtering out incorrectly matched feature pairs among the matching feature pairs using the above essential matrix. A method of processing data, comprising:

5. In paragraph 3, A step of estimating a pose of the first input image based on the feature matching result between the first input image and the first street view image and 3D points corresponding to the first street view image on the 3D map. A method of processing data, further comprising:

6. In paragraph 5, The estimated pose of the first input image is a candidate initial pose of the input images, The step of estimating the above pose is, A step of estimating the candidate initial pose based on the feature matching result and 3D points corresponding to the first street view image using the PnP (Perspective-N-Points) algorithm; A step of estimating first poses of the input images by applying VIO information of each input image of the input images to the candidate initial pose; A step of estimating second poses of the input images based on feature matching results between each input image and a street view image corresponding to each input image among the street view images and 3D points corresponding to the street view images on the 3D map using the PnP algorithm; and A step of determining whether the candidate initial pose is suitable based on a comparison between the first poses and the second poses. A method of processing data, comprising:

7. In paragraph 6, A data processing method in which pose estimation is repeatedly performed on the input images using the RANSAC (Random Sample Consensus) technique, thereby estimating the initial poses of the input images.

8. In paragraph 1, A step of estimating the initial pose of the above input images; A step of estimating poses of the input images by applying VIO information of each input image of the input images to the initial pose; and A step of performing feature matching between adjacent input images among the above input images and applying triangulation to the adjacent input images, thereby determining the first 3D points representing points matching between the adjacent input images. Including more, The above matching steps are: A data processing method for aligning the first 3D points to the 3D map based on the above poses, the 3D points on the 3D map, and the first 3D points.

9. In paragraph 8, The above matching steps are: A step of optimizing the poses, the first 3D points, and the scales of the input images by optimizing at least one of a first reprojection error associated with the 3D map, a second reprojection error associated with the input images, a distance error associated with the 2D map of the space, an odometry error associated with scales of the input images, and a scale error between the input images and the 3D map. A method of processing data, comprising:

10. In paragraph 9, A data processing method, wherein the optimization of at least one error comprises minimizing the at least one error using an algorithm based on a nonlinear least squares method.

11. In paragraph 9, The above first reprojection error represents the reprojection error between the first 3D points and the 3D points on the 3D map, The above second reprojection error represents the reprojection error between the first 3D points and each of the input images, The above distance error represents the distance error between the first 3D points and the outline included in the 2D map, The above odometry error represents the error between the pose change amount between the poses and the change amount of odometry information between the input images. A data processing method, wherein the above scale error represents an error between the scale of each of the input images and the scale of the three-dimensional map.

12. In paragraph 1, A data processing method wherein the above three-dimensional map is generated from the street view images using a SfM (structure from motion) algorithm.

13. In paragraph 12, The above 3D map is generated by a 3D map generation method based on the above street view images, The above 3D map creation method is, A step of performing feature matching between at least two street view images among the above street view images; A step of determining second 3D points representing points that are matched between the at least two street view images according to the feature matching by applying triangulation to the at least two street view images; and A step of generating the three-dimensional map based on the second three-dimensional points above. A method of processing data, comprising:

14. In paragraph 13, The steps for performing the above feature matching are: A step of filtering out incorrectly matched feature pairs among the matching feature pairs between the at least two street view images obtained through the above feature matching. Including, The above filtering step is, a step of estimating an essential matrix between at least two street view images; and A step of filtering out incorrectly matched feature pairs among the matching feature pairs using the above essential matrix. A method of processing data, comprising:

15. In paragraph 13, The step of performing the feature matching and the step of determining the second 3D points included in the above 3D map generation method are performed for each tile of the plurality of tiles dividing the space, The steps for generating the above 3D map are: A data processing method for generating a three-dimensional map by aligning the second three-dimensional points determined for each tile.

16. In paragraph 15, The above 3D map creation method is, A step of optimizing the second 3D points for each tile and adjusting the scale of each tile based on at least one of the first additional information including an outline included in a two-dimensional map of the space, the second additional information including DEM (Digital Elevation Model) information or DSM (Digital Surface Model) information for the space, and the third additional information including an ortho image for the space. Including more, The steps for generating the above 3D map are: A step of aligning the optimized second 3D points belonging to the first tile among the above tiles and the optimized second 3D points belonging to the second tile among the above tiles. A method of processing data, comprising:

17. In paragraph 13, The above 3D map creation method is, A step of estimating the pose of each street view image of the above street view images; and A step of optimizing the pose of each street view image and the second 3D points by optimizing at least one error among a first error between the estimated pose and the pose indicated by the displacement information of each street view image, a second error which is a projection error between the second 3D points and each street view image, a third error which is a distance error between the second 3D points and an outline included in a two-dimensional map of the space, and a fourth error between each street view image and a virtual GCP (Ground-Control Point) from at least one of DEM (Digital Elevation Model) information, DSM (Digital Surface Model) information, and orthoimage for the space. A method of processing data, further comprising:

18. In paragraph 17, A data processing method, wherein the optimization of at least one error comprises minimizing the at least one error using an algorithm based on a nonlinear least squares method.

19. A non-transitory computer-readable recording medium storing a computer program for executing the method of paragraph 1 on the computer system.

20. In computer systems, At least one processor configured to execute computer-readable instructions contained in memory Including, At least one processor of the above, Obtain two-dimensional input images that capture space through a user terminal, Obtain a 3D map of the space generated based on 2D street view images of the space, A computer system that aligns first 3D points for the space generated based on matching between the input images to the 3D map.

Images