Method and system for aligning 3D models with Street View data

Abstract

The present disclosure relates to a method for aligning a 3D model with street view data, the method being performed by at least one processor, and includes receiving a 3D model of a specific area including 3D geometric information and texture information expressed in absolute coordinate positions, receiving a plurality of street view images taken at a plurality of nodes within the specific area and street view data of the specific area including absolute coordinate position information for the plurality of street view images with a first accuracy, and estimating, based on the 3D model and the street view data, the absolute coordinate position information and direction information for the plurality of street view images with a second accuracy, where the second accuracy is higher than the first accuracy.

Description

[Technical Field]

[0001] The present disclosure relates to methods and systems for aligning 3D models with Street View data, and more particularly to methods and systems for aligning 3D models with Street View data by estimating location and orientation information for Street View imagery using 3D models that include highly accurate location information. [Background technology]

[0002] With the development of information technology, map information services have become commercially available. One area of ​​map information services is the provision of street view images. For example, a map information service provider can acquire images of a real space and then provide the images taken at a specific point on an electronic map as street view images of the point.

[0003] Street View images contain high-quality texture information of the ground, buildings, structures, etc. in real space, providing a user experience that makes them feel as if they are looking around at a specific point in real space. However, Street View images have the drawbacks of lacking accurate location information and not including 3D geometric information.

[0004] On the other hand, aerial photograph-based 3D models have the advantage of including 3D geometric information and accurate absolute coordinate position information. However, aerial photograph-based 3D models have the drawback of low quality texture information. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure provides a method for aligning a 3D model with street view data to solve the above-mentioned problems, a computer-readable non-transitory recording medium having instructions recorded thereon, and an apparatus (system). [Means for solving the problem]

[0006] The present disclosure may be embodied in numerous ways, including as a method, an apparatus (system), or a computer-readable non-transitory storage medium having instructions recorded thereon.

[0007] According to one embodiment of the present disclosure, a method for aligning a 3D model with Street View data, performed by at least one processor, includes: receiving a 3D model of a specific area, the 3D model including 3D geometric information and texture information expressed in absolute coordinate positions; receiving a plurality of Street View images taken at a plurality of nodes within the specific area, and Street View data of the specific area, the Street View data including absolute coordinate position information with a first accuracy for the plurality of Street View images; and estimating absolute coordinate position information and direction information with a second accuracy for the plurality of Street View images based on the 3D model and the Street View data, wherein the second accuracy is higher than the first accuracy.

[0008] A non-transitory computer-readable recording medium is provided that stores instructions for executing a method according to an embodiment of the present disclosure on a computer.

[0009] An information processing system according to an embodiment of the present disclosure includes: a communications module; a memory; and at least one processor coupled to the memory and configured to execute at least one computer-readable program stored in the memory, the at least one program including: receiving a 3D model of a specific area including 3D geometric information and texture information expressed in absolute coordinate positions; receiving a plurality of street view images taken at a plurality of nodes within the specific area and street view data of the specific area including absolute coordinate position information for the plurality of street view images with a first accuracy; and estimating absolute coordinate position information and direction information for the plurality of street view images with a second accuracy based on the 3D model and the street view data, wherein the second accuracy is higher than the first accuracy. [Effects of the Invention]

[0010] According to one embodiment of the present disclosure, by aligning a 3D model including 3D geometric information and highly accurate location and direction information with street view data including high-quality texture information, it is possible to provide a variety of services that utilize all of the 3D geometric information, highly accurate location and direction information, and high-quality texture information.

[0011] According to some embodiments of the present disclosure, matching information between Street View images and 3D models can be automatically extracted without any additional control point surveying or tagging work, thereby efficiently using limited resources.

[0012] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure belongs (referred to as an "ordinary engineer") from the description in the claims. [Brief explanation of the drawings]

[0013] Embodiments of the present disclosure will now be described with reference to the accompanying drawings, in which like reference numerals indicate like elements, but are not limited to the drawings, as follows: [Figure 1] FIG. 1 illustrates an example method for aligning a 3D model with street view data, according to one embodiment of the present disclosure. [Figure 2] 1 is a schematic diagram illustrating a configuration in which an information processing system is communicably connected to a plurality of user terminals according to an embodiment of the present disclosure. [Figure 3] 1 is a block diagram illustrating an internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating an example of a three-dimensional model according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates an example of acquiring street view data according to an embodiment of the present disclosure. [Figure 6]FIG. 10 is a diagram illustrating an example of extracting map matching points and / or map matching lines by performing feature matching between a 3D model and street view data according to an embodiment of the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating an example of extracting multiple feature point correspondence sets by performing feature matching between multiple street view images included in street view data according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram illustrating an example in which 3D geometric information included in a 3D model is projected onto a street view image using the street view image and estimated high-precision location information, according to one embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating an example method for aligning a 3D model with street view data, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, specific contents for implementing the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, specific descriptions of well-known functions and configurations will be omitted if they may obscure the gist of the present disclosure.

[0015] In the accompanying drawings, identical or corresponding components are denoted by the same reference numerals. In addition, in the following description of the embodiments, duplicated descriptions of identical or corresponding components will be omitted. However, omission of a description of a component does not mean that the component is not included in any of the embodiments.

[0016] The advantages and features of the disclosed embodiments, as well as methods for achieving them, will become apparent from the following examples, taken in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be embodied in various other forms. The present embodiments are provided solely to complete the disclosure and fully convey the scope of the invention to those skilled in the art.

[0017] The terms used in this specification will be briefly explained, and the disclosed embodiments will be specifically described. The terms used in this specification are currently commonly used and general terms that have been selected as much as possible while taking into consideration the functions of the present disclosure. However, these terms may change depending on the intentions of engineers in the relevant field, legal precedents, the emergence of new technologies, etc. In addition, in certain cases, the applicant may arbitrarily select terms, and in such cases, the meanings of these terms will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined based on the meanings of the terms and the overall content of the present disclosure, rather than simply by the names of the terms.

[0018] In this specification, the singular expression includes the plural expression unless the context clearly dictates otherwise. Furthermore, the plural expression includes the singular expression unless the context clearly dictates otherwise. Throughout this specification, when a part includes one element, this does not mean that it may further include other elements, but does not exclude other elements, unless otherwise specified.

[0019] Furthermore, the terms "module" and "module" used in this specification refer to software or hardware components, and the "module" or "module" may perform either function. However, the term "module" or "module" is not limited to software or hardware. A "module" or "module" may be configured to be provided on an addressable storage medium or to execute one or more processors. Thus, as an example, a "module" or "module" may include components such as software components, object-oriented software components, class components, and task components, as well as at least one of processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. Components and "modules" or "modules" may be combined into fewer components and "modules" or "modules" so that the functionality provided by them can be further separated into additional components, "modules," or "modules."

[0020] According to one embodiment of the present disclosure, a "module" or "unit" may be implemented as a processor and memory. "Processor" should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, "processor" may refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. "Processor" may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration. Additionally, "memory" should be broadly interpreted to include any electronic component capable of storing electronic information. "Memory" may refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage devices, registers, etc. Memory is in electronic communication with a processor if the processor can read information from and / or store information in the memory. Memory that is integrated into a processor is in electronic communication with the processor.

[0021] In the present disclosure, a "system" may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may be configured with one or more server devices. As another example, a system may be configured with one or more cloud devices. As yet another example, a system may be configured and operated by both a server device and a cloud device.

[0022] In this disclosure, "display" may refer to any display device associated with a computing device, for example, any display device capable of displaying any information / data controlled by or provided by the computing device.

[0023] In the present disclosure, "each of a plurality of A's" or "each of a plurality of A's" may refer to each of all the components included in the plurality of A's, or may refer to each of some of the components included in the plurality of A's.

[0024] In this disclosure, "street view data" may refer to data including not only road view data, which includes images and location information taken from the roadway, but also foot view data, which includes images and location information taken from the sidewalk. "Street view data" may further include images and location information taken from any point outdoors (or indoors looking outdoors), in addition to roadways and sidewalks.

[0025] 1 illustrates an example method for aligning a 3D model 110 with street view data 120, according to one embodiment of the present disclosure. An information processing system may acquire / receive the 3D model 110 and street view data 120 for a particular area.

[0026] The 3D model 110 may include 3D geometric information expressed in absolute coordinate positions and corresponding texture information. Here, the position information included in the 3D model 110 may be information with higher accuracy than the position information included in the street view data 120. Furthermore, the texture information included in the 3D model 110 may be information with lower quality (e.g., lower resolution) than the texture information included in the street view data 120. According to an embodiment, the 3D geometric information expressed in absolute coordinate positions may be generated based on an aerial photograph taken of a specific area from above the specific area.

[0027] The 3D model 110 for a particular area may include, for example, 3D building models 112, a Digital Elevation Model (DEM) 114, a true ortho image 116 for the particular area, a road layout, a road DEM, etc. An example of a 3D model 110 utilized in the present disclosure will be described in more detail below with reference to FIG.

[0028] The street view data 120 may include a plurality of street view images captured at a plurality of nodes within a specific area, and absolute coordinate location information for each of the plurality of street view images. Here, the location information included in the street view data 120 may be less accurate than the location information included in the 3D model 110, and the texture information included in the street view images may be higher quality (e.g., higher resolution) than the texture information included in the 3D model 110. For example, the location information included in the street view data 120 may be location information acquired using a GPS device when capturing street view images at the nodes. Location information acquired using a vehicle's GPS device may have an error of approximately 5 to 10 meters. Furthermore, the street view data may include direction information (i.e., image capture direction information) for each of the plurality of street view images. An example of a method for acquiring the street view data 120 used in the present disclosure will be described in detail below with reference to FIG. 5.

[0029] The information processing system may perform map matching 130 between the 3D model 110 and the street view data 120. Specifically, the information processing system may perform feature matching between texture information included in the 3D model 110 and a plurality of street view images included in the street view data 120. To perform map matching 130, the information processing system may convert at least some of the plurality of street view images included in the street view data 120 into top view images. As a result of map matching 130, a plurality of map matching points / map matching lines 132 may be extracted.

[0030] A map matching point may indicate a correspondence pair between one point in a street view image and one point in the 3D model 110. Various types of map matching points may be used depending on the type of 3D model 110 used in map matching 130, the location of the point, etc. For example, a map matching point may include at least one of a ground control point (GCP), which is a point correspondence pair on the ground in a specific area, a building control point (BCP), which is a point correspondence pair on a building in a specific area, or a structure control point, which is a point correspondence pair on a structure in a specific area. In addition to the above-mentioned ground, building, and structure, map matching points may be extracted from any region of the street view image and the 3D model 110.

[0031] A map matching line may indicate a corresponding pair of a line in a street view image and a line in the 3D model 110. Various types of map matching lines may be used depending on the type of 3D model 110 used in map matching 130, the position of the line, etc. For example, the map matching line may include at least one of a ground control line (GCL), which is a corresponding pair of lines on the ground in a specific area, a building control line (BCL), which is a corresponding pair of lines on a building in a specific area, a structure control line, which is a corresponding pair of lines on a structure in a specific area, or a lane control line, which is a corresponding pair of lines on a lane in a specific area. In addition to the ground, buildings, structures, and lanes described above, map matching lines may be extracted from any region of the street view image and the 3D model 110.

[0032] An example of an information processing system performing feature matching between the 3D model 110 and the street view data 120 to extract map matching points and / or map matching lines will be described in detail below with reference to FIG. 6 .

[0033] The information processing system may also perform feature matching 150 between a plurality of street view images to extract a plurality of feature point correspondence sets 152. According to one embodiment, for robust feature matching, feature matching 150 between a plurality of street view images may be performed using at least a portion of 3D model 110. For example, feature matching 150 between street view images may be performed using 3D building models 112 included in 3D model 110. An example of the information processing system performing feature matching 150 between a plurality of street view images included in street view data 120 to extract a plurality of feature point correspondence sets 152 will be described in detail below with reference to FIG. 7 .

[0034] The information processing system may then perform absolute coordinate position information and direction information estimation 160 for the plurality of street view images based on at least one of the plurality of map matching points / lines 132 and at least a portion of the plurality of feature point correspondence sets 152. For example, the processor may perform absolute coordinate position information and direction information estimation 160 for the plurality of street view images using a bundle adjustment technique. According to one embodiment, the estimated absolute coordinate position information and direction information 162 is information in an absolute coordinate system representing the 3D model 110 and may be six-degree-of-freedom (DoF) parameters. The absolute coordinate position information and direction information 162 estimated through this process may be data with higher accuracy than the absolute coordinate position information and direction information included in the street view data 120.

[0035] According to one embodiment of the present disclosure, by aligning a 3D model 110 including 3D geometric information and highly accurate positional and directional information with street view data 120 including high-quality texture information, various services can be provided that utilize all of the 3D geometric information, highly accurate positional and directional information, and high-quality texture information.

[0036] 2 is a schematic diagram illustrating a configuration in which an information processing system 230 according to an embodiment of the present disclosure is communicatively connected to multiple user terminals 210_1, 210_2, and 210_3. As illustrated, the multiple user terminals 210_1, 210_2, and 210_3 may be connected to the information processing system 230, which can provide a map information service, via a network 220. Here, the multiple user terminals 210_1, 210_2, and 210_3 may include terminals of users receiving the map information service. Furthermore, the multiple user terminals 210_1, 210_2, and 210_3 may be automobiles that capture street view images at nodes. In one embodiment, the information processing system 230 may include one or more server devices and / or databases, or one or more distributed computing devices and / or distributed databases based on a cloud computing service, that can store, provide, and execute computer-executable programs (e.g., downloadable applications) and data related to providing the map information service.

[0037] The map information service provided by the information processing system 230 may be provided to a user through an application or a web browser installed in each of the user terminals 210_1, 210_2, and 210_3. For example, the information processing system 230 may provide information corresponding to a street view image request, an image-based location recognition request, or the like received from the user terminals 210_1, 210_2, and 210_3 through an application or the like, or perform corresponding processing.

[0038] A plurality of user terminals 210_1, 210_2, and 210_3 can communicate with the information processing system 230 through the network 220. The network 220 may be configured to enable communication between the plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be configured as a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless network such as WLAN (Wireless LAN), Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include a communication method utilizing a communication network that the network 220 may include (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, a satellite network, etc.), as well as short-range wireless communication between the user terminals 210_1, 210_2, and 210_3.

[0039] 2, a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3 are shown as examples of user terminals, but are not limited thereto, and the user terminals 210_1, 210_2, and 210_3 may be any computing device capable of wired and / or wireless communication and on which an application or a web browser, etc., can be installed and executed. For example, the user terminal may include an AI speaker, a smartphone, a mobile phone, a navigation system, a computer, a laptop, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a tablet PC, a game console, a wearable device, an IoT (Internet of Things) device, a VR (Virtual Reality) device, an AR (Augmented Reality) device, a set-top box, etc. Also, although FIG. 2 shows three user terminals 210_1, 210_2, and 210_3 communicating with the information processing system 230 through the network 220, this is not limited thereto, and other numbers of user terminals may be configured to communicate with the information processing system 230 through the network 220.

[0040] According to one embodiment, the information processing system 230 may receive data related to a street view image request, including information about a specific location on an electronic map, from the user terminals 210_1, 210_2, and 210_3. The information processing system 230 may then transmit a street view image for the specific location on the electronic map to the user terminals 210_1, 210_2, and 210_3 based on the received information about the specific location on the electronic map. Additionally or alternatively, the information processing system 230 may receive data related to an image-based location recognition request, including an image or picture taken at the specific location, from the user terminals 210_1, 210_2, and 210_3. The information processing system 230 may then estimate precise location information and direction information for the location where the image or picture was taken based on the received image or picture, and transmit the estimated location information and direction information to the user terminals 210_1, 210_2, and 210_3. In addition, the information processing system 230 can transmit various service-related data based on data created by integrating the three-dimensional model with the street view data to the user terminals 210_1, 210_2, and 210_3.

[0041] FIG. 3 is a block diagram illustrating the internal configuration of a user terminal 210 and an information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing an application, a web browser, or the like, and capable of wired / wireless communication, and may include, for example, the mobile phone terminal 210_1, the tablet terminal 210_2, and the PC terminal 210_3 of FIG. 2 . As illustrated, the user terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input / output interface 318. Similarly, the information processing system 230 may include a memory 332, a processor 334, a communication module 336, and an input / output interface 338. As illustrated in FIG. 3 , the user terminal 210 and the information processing system 230 may be configured to communicate information and / or data over the network 220 using their respective communication modules 316 and 336. Additionally, the input / output device 320 may be configured to input information and / or data to the user terminal 210 and / or output information and / or data generated from the user terminal 210 via the input / output interface 318 .

[0042] The memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 may include a permanent mass storage device such as a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, a non-transitory mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the user terminal 210 or the information processing system 230 as a permanent storage device separate from the memory. The memories 312 and 332 may also store an operating system and at least one program code (e.g., code for an application installed and run on the user terminal 210).

[0043] Such software components may be loaded from a computer-readable recording medium separate from the memory 312, 332. Such separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, but may also include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. As another example, the software components may be loaded into the memory 312, 332 through a communication module rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory 312, 332 based on a computer program installed by a file provided over the network 220 by a developer or a file distribution system that distributes application installation files.

[0044] The processors 314, 334 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processors 314, 334 by the memory 312, 332 or the communication modules 316, 336. For example, the processors 314, 334 may be configured to execute instructions received by program code stored in a storage device, such as the memory 312, 332.

[0045] The communication modules 316 and 336 may provide a configuration or function for the user terminal 210 and the information processing system 230 to communicate with each other via the network 220, and may provide a configuration or function for the user terminal 210 and / or the information processing system 230 to communicate with other user terminals or other systems (e.g., a separate cloud system, etc.). For example, a request or data (e.g., data related to a Street View image request for a specific area) generated by the processor 314 of the user terminal 210 via program code stored in a storage device such as the memory 312 may be transmitted to the information processing system 230 via the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 may be received by the user terminal 210 via the communication module 316 of the user terminal 210 via the communication module 336 and the network 220. For example, the user terminal 210 may receive data related to Street View imagery for a specific area from the information processing system 230.

[0046] The input / output interface 318 may be a means for interfacing with the input / output device 320. For example, the input device may include a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, a mouse, etc., and the output device may include a display, a speaker, a haptic feedback device, etc. As another example, the input / output interface 318 may be a means for interfacing with a device that integrates input and output configurations or functions, such as a touchscreen. For example, when the processor 314 of the user terminal 210 processes instructions of a computer program loaded in the memory 312, a service screen configured using information and / or data provided by the information processing system 230 or another user terminal may be displayed on the display through the input / output interface 318. Although FIG. 3 illustrates the input / output device 320 as not being included in the user terminal 210, this is not limiting and the input / output device 320 may be configured as a single device together with the user terminal 210. Furthermore, the input / output interface 338 of the information processing system 230 may be a means for interfacing with an input or output device (not shown) that may be connected to or included in the information processing system 230. Although the input / output interfaces 318 and 338 are shown in FIG. 3 as elements configured separately from the processors 314 and 334, the present invention is not limited thereto, and the input / output interfaces 318 and 338 may be configured to be included in the processors 314 and 334.

[0047] The user terminal 210 and the information processing system 230 may include more components than those shown in FIG. 3 . However, it is not necessary to explicitly show most of the conventional components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the input / output devices 320 described above. The user terminal 210 may also include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, a database, etc. For example, if the user terminal 210 is a smartphone, it may include components typically included in a smartphone, such as an acceleration sensor, a gyro sensor, an image sensor, a proximity sensor, a touch sensor, an illuminance sensor, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration. According to one embodiment, the processor 314 of the user terminal 210 may be configured to run an application that provides a map information service. In this case, code related to the application and / or program may be loaded into the memory 312 of the user terminal 210.

[0048] While a program for an application that provides a map information service is running, the processor 314 may receive text, images, pictures, voice, and / or actions entered or selected through an input device, such as a touch screen, keyboard, camera including an audio sensor and / or image sensor, or microphone, connected to the input / output interface 318, and may store the received text, images, pictures, voice, and / or actions in the memory 312 or provide them to the information processing system 230 via the communication module 316 and the network 220. For example, the processor 314 may receive a user input requesting a street view image for a specific area and provide it to the information processing system 230 via the communication module 316 and the network 220.

[0049] The processor 314 of the user terminal 210 may be configured to manage, process, and / or store information and / or data received from the input / output device 320, other user terminals, the information processing system 230, and / or multiple external systems. The information and / or data processed by the processor 314 may be provided to the information processing system 230 via the communication module 316 and the network 220. The processor 314 of the user terminal 210 may transmit and output information and / or data to the input / output device 320 via the input / output interface 318. For example, the processor 314 may display the received information and / or data on a screen of the user terminal.

[0050] The processor 334 of the information processing system 230 may be configured to manage, process, and / or store information and / or data received from multiple user terminals 210 and / or multiple external systems. The information and / or data processed by the processor 334 may be provided to the user terminal 210 via the communication module 336 and the network 220.

[0051] 4 is a diagram illustrating an example of a 3D model according to an embodiment of the present disclosure. The 3D model for a specific region may include 3D geometric information expressed in absolute coordinate positions and corresponding texture information. The 3D geometric information included in the 3D model for the specific region does not need to include information about all areas of the specific region. For example, the 3D model may include only geometric information and corresponding texture information for some areas of the specific region, such as the ground, buildings, and structures.

[0052] As a specific example, the three-dimensional model may be a model generated based on a digital elevation model 410 containing geometric information about the ground surface of a specific area and a corresponding precise orthoimage 420.

[0053] As another example, the three-dimensional model may be a model generated based on a digital elevation model 410 containing geometric information relative to the ground of a particular area, and a corresponding plurality of aerial photographs and their absolute coordinate position information and orientation information 430. In one embodiment, a precise orthoimage can be generated based on the plurality of aerial photographs and their absolute coordinate position information and orientation information 430.

[0054] As yet another example, the 3D model may be a model generated based on a 3D mesh model 440 including geometric information about buildings in a specific area and atlas data 450 including texture information corresponding to the 3D mesh model. For example, the 3D mesh model may be a 3D triangular mesh model.

[0055] As yet another example, the 3D model may be a model generated based on a 3D mesh model 440 containing geometric information for buildings in a specific area, a plurality of corresponding aerial photographs, and absolute coordinate position information and direction information 430 for each aerial photograph.

[0056] As yet another example, the 3D model may be a model generated based on a 3D mesh model including geometric information on structures such as signboards, traffic lights, etc. in a specific area and corresponding texture information.

[0057] 5 is a diagram illustrating an example of acquiring street view data according to an embodiment of the present disclosure. The street view data may include a plurality of street view images taken at a plurality of nodes within a specific area 520, and absolute coordinate position information for the plurality of street view images. Here, the plurality of nodes may be virtual nodes arranged at predefined intervals (e.g., every 5 m, every 10 m, etc.) within a road 522 of the specific area 520.

[0058] The multiple street view images may be acquired from multiple images of the specific area 520 taken from various directions while driving a vehicle 510 equipped with at least one camera 512 along roads 522 in the specific area 520. For example, the camera 512 may be four fisheye cameras. According to one embodiment, each street view image may be a 360° panoramic image generated based on multiple images taken from various directions at each node. For example, the street view image for each node may be a 360° panoramic image generated by stitching multiple images taken from various directions at each node. In this way, the street view image is generated based on images of the space within the specific area taken at a relatively close distance and may include high-quality texture information.

[0059] Meanwhile, the location information for the plurality of street view images included in the street view data may be location information with a relatively low degree of accuracy. For example, the location information for the plurality of street view images included in the street view data may be location information at the node where each street view image was captured, acquired by a GPS device installed in the vehicle 510. The location information acquired using the GPS device installed in the vehicle 510 may have an error of about 5 to 10 meters.

[0060] 6 is a diagram illustrating an example of extracting map matching points 632 and / or map matching lines 634 by performing feature matching between a 3D model and street view data according to an embodiment of the present disclosure. The information processing system may extract multiple map matching points 632 and / or multiple map matching lines 634 by performing map matching between the 3D model and street view data. For example, the information processing system may convert a street view image 610 into a top-view image, and perform feature matching between texture information 620 of a 3D model corresponding to the street view image 610 (e.g., an orthoimage or aerial image of the 3D model) and the top-view converted street view image 612, thereby extracting multiple map matching points 632 and / or multiple map matching lines 634. Here, the texture information 620 of the 3D model corresponding to the street view image 610 may be obtained using absolute coordinate position information associated with the street view image 610.

[0061] Each map matching point 632 may represent a corresponding pair of a point (e.g., (u1, v1)) in the street view image 610 and a point (e.g., (x1, y1, z1)) in the 3D model. The representation of each map matching point 632 representing a point in the street view image 610 may vary depending on the format of the street view image 610 (e.g., equirectangular format, cubic format, projective format, etc.).

[0062] The map matching points 632 may be categorized in various ways depending on the type of 3D model used for map matching, the location of the points, etc. For example, the map matching points 632 may include at least one of ground control points, which are point-corresponding pairs on the ground in a specific area, building control points, which are point-corresponding pairs on buildings in a specific area, or structure control points, which are point-corresponding pairs on structures in a specific area. In addition to the ground, buildings, and structures described above, the map matching points 632 may be extracted from any region of the street view image 610 and the 3D model.

[0063] Each map matching line 634 may represent a corresponding pair of one line in the street view image 610 and one line in the three-dimensional model. Any representation method (e.g., two points, a line equation, one point and a direction vector, etc.) may be used to represent the line. For example, each map matching line 634 may include a corresponding pair of two points (u1, v1) and (u2, v2) in the street view image 610 and two points (x1, y1, z1) and (x2, y2, z2) in the three-dimensional model.

[0064] The map matching lines 634 may be of various types depending on the type of 3D model used for map matching, the position of the lines, etc. For example, the map matching lines 634 may include at least one of ground control lines, which are a pair of lines corresponding to the ground in a specific area, building control lines, which are a pair of lines corresponding to buildings in a specific area, structure control lines, which are a pair of lines corresponding to structures in a specific area, or lane control lines, which are a pair of lines corresponding to lanes in a specific area. In addition to the ground, buildings, structures, and lanes described above, the map matching lines 634 may be extracted from any region of the street view image 610 and the 3D model.

[0065] Conventional methods require direct ground point surveying to estimate absolute coordinate position information for an image, or require workers to manually tag ground point locations on images using standard ground points provided by the government. However, ground point surveying and tagging are inefficient and involve increased costs as the scope of work expands. The method disclosed herein can automatically extract matching information between a Street View image 610 and a 3D model without additional ground point surveying or tagging, thereby efficiently using limited resources. Furthermore, by using map matching lines 634 in addition to map matching points 632, more accurate map matching can be achieved even in areas where map matching points 632 are difficult to extract or have poor extraction quality, such as highways and urban thoroughfares.

[0066] 7 is a diagram illustrating an example of extracting multiple feature point correspondence sets by performing feature matching between multiple street view images 710, 720, and 730 included in street view data according to one embodiment of the present disclosure. The information processing system can extract multiple feature point correspondence sets by performing feature matching between the multiple street view images 710, 720, and 730. The method for performing feature matching between the multiple street view images 710, 720, and 730 is not limited to a specific method, and any feature matching method (e.g., conventional features such as Scale Invariant Feature Transform (SIFT), deep-based features such as SuperPoint / Glue, and R2D2) may be used.

[0067] For example, the information processing system may extract multiple feature point correspondence sets by performing feature matching between multiple street view images 710, 720, and 730 for adjacent nodes 712, 722, and 732. Here, the nodes may be virtual nodes arranged at predefined intervals (in the example of FIG. 7 ) within a road 700 in a specific area. As a specific example, the information processing system may extract multiple feature point correspondence sets by performing feature matching between a first street view image 710 for a first node 712, a second street view image 720 for a second node 722, and a third street view image 730 for a third node 732 within the road 700 in the specific area. Each feature point correspondence set may represent a correspondence set (e.g., {(u1, v1), (u2, v2), ..., (uun, vn)} of points estimated to be the same location, extracted from at least some of the multiple street view images 710, 720, and 730.

[0068] According to one embodiment, for robust feature matching, at least a portion of a 3D model may be used to perform feature matching between the plurality of street view images 710, 720, and 730. For example, the information processing system may perform feature matching between the plurality of street view images 710, 720, and 730 using at least a portion of a 3D mesh model of a building.

[0069] 7 shows feature matching between three Street View images, but is not limited to this. For example, feature matching may be performed consecutively between two adjacent Street View images, or feature matching may be performed between four or more Street View images.

[0070] 8 is a diagram illustrating an example of a street view image 810 according to an embodiment of the present disclosure, in which 3D geometric information included in a 3D model is projected onto the street view image using estimated high-precision position information and direction information. According to one embodiment, highly accurate absolute coordinate position information and direction information for the street view image 810 may be estimated using the process described in FIG. 1. Here, the estimated position information and direction information is information in an absolute coordinate system that represents a 3D model of a specific area, and may be a 6-degree-of-freedom parameter.

[0071] The information processing system can align the 3D model with the Street View data based on the highly accurate absolute coordinate position information and orientation information for the estimated plurality of Street View images, and can project the 3D geometric information included in the 3D model onto the Street View images based on the highly accurate absolute coordinate position information and orientation information for the estimated plurality of Street View images.

[0072] As a result of matching 3D models with Street View data, data containing 3D geometric information, highly accurate location and direction information, and high-quality texture information can be generated, which can be used to provide a variety of services.

[0073] For example, as shown in Fig. 8, it is possible to provide a street view image 820 in which street information generated by projecting 3D geometric information included in a 3D model onto an existing street view image 810 is displayed in color. In addition, it is possible to provide various services that utilize the matching results, such as a realistic street view rendering service that utilizes 3D geometric information, an automatic POI (Point of Interest) enhancement service for street view images that utilizes street information, and an image-based location recognition (Visual Localization) service for outdoor environments.

[0074] 9 is a flowchart illustrating a method 900 according to an embodiment of the present disclosure. In the method 900, a processor (e.g., at least one processor of an information processing system) may receive a 3D model of a specific area including 3D geometric information and texture information expressed in absolute coordinate positions (S910). According to an embodiment, the 3D model may be generated based on aerial photographs, and the texture information included in the 3D model may be lower quality texture information compared to the street view imagery included in the street view data.

[0075] The 3D model of a specific area including 3D geometric information and texture information expressed in absolute coordinate positions may be a 3D model of a portion of the specific area. For example, the 3D model may be a model generated based on a digital elevation model and a precise orthoimage of the specific area. As another example, the 3D model may be a model generated based on a digital elevation model of the specific area, multiple aerial photographs, and absolute coordinate position information and orientation information for each aerial photograph. As yet another example, the 3D model may be a model generated based on atlas data including a 3D mesh model of buildings in the specific area and texture information corresponding to the 3D mesh model. As yet another example, the 3D model may be a model generated based on a 3D mesh model of buildings in the specific area, multiple aerial photographs, and absolute coordinate position information and orientation information for each aerial photograph.

[0076] The processor may also receive street view data for the specific area, the street view data including a plurality of street view images (e.g., 360° panoramic images) taken at a plurality of nodes within the specific area and absolute coordinate position information with a first accuracy for the plurality of street view images (S920). According to one embodiment, the plurality of nodes may be nodes arranged at predefined intervals within roads in the specific area.

[0077] The processor may then estimate absolute coordinate position information and direction information with a second accuracy for the plurality of street view images based on the 3D model and the street view data (S930), where the second accuracy may be higher than the first accuracy.

[0078] For example, to estimate the location information and direction information, the processor may perform feature matching between the 3D model and the street view data to extract at least one of a plurality of map matching points or a plurality of map matching lines. In one embodiment, the processor may convert at least some of the plurality of street view images into top-view images and perform feature matching between the 3D model and the top-view converted street view images to extract the map matching points and / or map matching lines.

[0079] A map matching point may indicate a correspondence pair between one point in a Street View image and one point in a 3D model. Map matching points may be categorized into various types depending on the type of 3D model, the location of the point, etc. For example, a map matching point may include at least one of a ground control point, which is a correspondence pair of points on the ground in a specific area, a building control point, which is a correspondence pair of points on a building in a specific area, or a structure control point, which is a correspondence pair of points on a structure in a specific area.

[0080] A map matching line may indicate a corresponding pair of a line in a street view image and a line in a 3D model. Various types of map matching lines may be used depending on the type of 3D model, the position of the line, etc. For example, a map matching line may include at least one of a ground control line, which is a corresponding pair of lines on the ground in a specific area; a building control line, which is a corresponding pair of lines on a building in a specific area; a structure control line, which is a corresponding pair of lines on a structure in a specific area; or a lane control line, which is a corresponding pair of lines on a lane in a specific area.

[0081] In addition, for estimating the location information and the direction information, the processor may perform feature matching between the plurality of Street View images to extract a plurality of feature point correspondence sets. According to one embodiment, the 3D model of the specific area may include a 3D mesh model of buildings in the specific area, and the processor may perform feature matching between the plurality of Street View images using at least a portion of the 3D mesh model of the buildings.

[0082] The processor may then estimate absolute coordinate position information and orientation information for the plurality of street view images with a second accuracy based on at least one of the plurality of map matching points or the plurality of map matching lines and the plurality of feature point correspondence sets. For example, the processor may estimate absolute coordinate position information and orientation information for the plurality of street view images with a second accuracy using a bundle adjustment technique. According to one embodiment, the estimated absolute coordinate position information and orientation information with the second accuracy may be information in an absolute coordinate system representing a 3D model.

[0083] Furthermore, the processor may align the 3D model with the street view data based on the estimated absolute coordinate position information and orientation information with respect to the plurality of street view images with a second accuracy. According to one embodiment, the processor may project the 3D geometric information included in the 3D model onto the plurality of street view images.

[0084] 9 and the above description are merely examples, and the scope of the present disclosure is not limited thereto. For example, at least one step may be added / modified / deleted, or the order of the steps may be changed.

[0085] The above-described method may be provided as a computer program stored on a computer-readable recording medium for execution by a computer. The medium may be a medium that continuously stores a computer-executable program or a medium that temporarily stores the program for execution or download. The medium may be various recording or storage means in the form of a single piece of hardware or a combination of multiple pieces of hardware. The medium is not limited to a medium directly connected to a computer system but may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Other examples of media include recording or storage media managed by app stores that distribute applications or by websites or servers that provide or distribute various software.

[0086] The methods, operations, or techniques of the present disclosure may be implemented in various ways. For example, such techniques may be implemented as hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the present disclosure may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design requirements imposed on the overall system. Those of ordinary skill in the art may also implement the described functionality in various ways for each particular application, but such implementation should not be interpreted as a departure from the scope of the present disclosure.

[0087] In a hardware implementation, the processing units used in performing the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to have the functionality described in this disclosure, computers, or combinations thereof.

[0088] Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be implemented as or performed as a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate and transistor logic, discrete hardware components, or any combination designed to have the functionality described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

[0089] In a firmware and / or software implementation, the techniques may be realized as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, etc. The instructions may be executable by one or more processors and may cause the processors to perform certain aspects of the functions described in this disclosure.

[0090] While the embodiments described above utilize aspects of the presently disclosed subject matter on one or more stand-alone computer systems, the present disclosure is not limited thereto and may be implemented in connection with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the subject matter in this disclosure may be implemented on multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include PCs, network servers, and handheld devices.

[0091] Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes that can be understood by those of ordinary skill in the art to which the present disclosure pertains can be made without departing from the scope of the present disclosure, and such modifications and changes should be considered to fall within the scope of the claims appended hereto.

Claims

1. 1. A method for aligning a three-dimensional model with street view data, performed by at least one processor, comprising: receiving a 3D model of a specific area including 3D geometric information and texture information expressed in absolute coordinate positions; receiving street view data of the specific area, the street view data including a plurality of street view images taken at a plurality of nodes within the specific area and absolute coordinate position information with a first accuracy for the plurality of street view images; estimating absolute coordinate position information and direction information with a second accuracy for the plurality of street view images based on the 3D model and the street view data; Including, 10. The method of aligning a three-dimensional model with street view data, wherein the second accuracy is greater than the first accuracy.

2. and aligning the 3D model with the street view data based on absolute coordinate position information and direction information of a second accuracy for the plurality of street view images. The method of claim 1 further comprising:

3. projecting 3D geometric information included in the 3D model onto the plurality of street view images; The method of claim 2 further comprising:

4. The estimating step includes: performing feature matching between the 3D model and the street view data to extract at least one of a plurality of map matching points or a plurality of map matching lines; 2. The method of claim 1 for aligning a three-dimensional model with street view data, comprising:

5. The estimating step includes: converting at least some of the street view images into top-view images. The method of claim 4 for aligning a three-dimensional model with street view data, further comprising:

6. 5. The method of claim 4, wherein each map matching point represents a corresponding pair of one point in the Street View image and one point in the 3D model.

7. The plurality of map matching points are: Ground control points, which are corresponding pairs of points on the ground in the specific area; Building control points that are point-matching pairs of buildings in the specific area; or Structure control points, which are point-corresponding pairs of structures in the specified area The method of claim 4 for aligning a three-dimensional model with street view data, comprising at least one of:

8. 5. The method of claim 4, wherein each map matching line represents a corresponding pair of one line in the street view image and one line in the three-dimensional model.

9. The plurality of map matching lines are Ground control lines, which are corresponding pairs of lines on the ground in the specific area; building control lines, which are line-corresponding pairs for buildings in the specific area; Structure control lines that are corresponding pairs of lines at structures in the specified area; or Lane control lines that are line-corresponding pairs of lanes in the specific area The method of claim 4 for aligning a three-dimensional model with street view data, comprising at least one of:

10. The estimating step includes: performing feature matching between the plurality of street view images to extract a plurality of feature point correspondence sets; estimating absolute coordinate position information and direction information of the plurality of street view images with a second accuracy based on at least one of the plurality of map matching points or the plurality of map matching lines and the plurality of feature point correspondence sets; The method of claim 4 for aligning a three-dimensional model with street view data, further comprising:

11. the three-dimensional model of the specific area includes a three-dimensional mesh model of a building in the specific area; The method of claim 10 , further comprising: performing feature matching between the plurality of street view images using at least a portion of a 3D mesh model of the building.

12. 11. The method of claim 10, wherein the absolute coordinate position information and orientation information for the plurality of street view images with a second accuracy is estimated using a bundle adjustment technique.

13. The method of claim 1 , wherein the estimated absolute coordinate position information and direction information with a second accuracy is information in an absolute coordinate system that represents the three-dimensional model.

14. The method of claim 1 , wherein the three-dimensional model is generated based on aerial photographs.

15. The three-dimensional model is a digital elevation model and precise orthoimage of the specific area; a digital elevation model of the specific area, a plurality of aerial photographs, and absolute coordinate position information and direction information of each aerial photograph; atlas data including a three-dimensional mesh model of a building in the specific area and texture information corresponding to the three-dimensional mesh model; and A three-dimensional mesh model of buildings in the specific area, a plurality of aerial photographs, and absolute coordinate position information and direction information of each aerial photograph The method of claim 1 , wherein the three-dimensional model is generated based on any one of the following:

16. The method for aligning a three-dimensional model with street view data according to claim 1 , wherein the plurality of nodes are arranged at predefined intervals within roads in the specific area.

17. The method of claim 1 , wherein the plurality of street view images are 360° panoramic images.

18. 2. The method of claim 1, wherein texture information included in the 3D model is texture information of lower quality than the street view image included in the street view data.

19. A program for executing the method of claim 1 on a computer.

20. An information processing system, a communication module; Memory and at least one processor coupled to the memory and configured to execute at least one computer-readable program contained in the memory; Including, The at least one program receiving a 3D model of a specific area including 3D geometric information and texture information expressed in absolute coordinate positions; receiving street view data of a specific area, the street view data including a plurality of street view images taken at a plurality of nodes in the specific area and absolute coordinate position information with a first accuracy for the plurality of street view images; a command for estimating absolute coordinate position information and direction information with a second accuracy for the plurality of street view images based on the 3D model and the street view data; An information processing system, wherein the second accuracy is higher than the first accuracy.

Images