GPS-based augmented reality display method and system for lost cultural heritage

Abstract

A GPS-based augmented reality display method for lost cultural heritage according to an embodiment of the present invention is a method in which an augmented reality application, executed by at least one processor of a terminal, displays lost cultural heritage in augmented reality on the basis of GPS. The method comprises the steps of: acquiring first terminal location information on the basis of the location of a first GPS installed in the terminal; acquiring second terminal location information on the basis of the location of a second GPS installed in the terminal; calibrating the acquired first terminal location information and second terminal location information; determining the coordinates of the terminal according to the calibration result; acquiring at least one piece of historic site location information on the basis of the determined coordinates; and displaying, on the acquired historic site location information, a cultural heritage AR object matched in advance.

Description

GPS-based Augmented Reality Display Method and System for Lost Cultural Properties

[0001] The present invention relates to a GPS-based augmented reality display method and system for displaying lost cultural assets. More specifically, it relates to an augmented reality display method and system for accurately displaying lost cultural assets by reducing the range of error in a user's location and posture based on high-precision GPS.

[0002] Augmented Reality (AR) is a technology that superimposes a 3D virtual image onto a real-world image or background to create a single video.

[0003] In detail, augmented reality can be implemented by extracting feature points for a 3D object, tracking them in real time, obtaining rotation and movement information of the 3D object, and tracking and re-expressing the object.

[0004] This type of augmented reality is being used in various fields of technology, and for example, in architectural technology, a technology has been developed that provides information about architectural structures corresponding to images of the architectural structures.

[0005] However, in order to implement augmented reality, it is necessary to extract the object's characteristic points. However, the problem has arisen that it is impossible to extract the characteristic points of a building in an unfinished construction site or a site where only the foundation remains after the architectural cultural heritage has been lost.

[0006] To solve this problem, a technology was developed to augment an object that originally existed at a given location based on GPS even if there was no object to extract feature points from. However, the error range of GPS information was large, so real-world coordinate recognition was inaccurate, and a problem occurred in which the reproduced image was not accurately augmented at the desired location.

[0007] Accordingly, there is an urgent need to develop a technology that can accurately augment AR by calculating the user's location and posture.

[0008] The present invention has been devised to solve the problems of the prior art as described above, and its purpose is to provide a GPS-based augmented reality display method and system for lost cultural assets that accurately determines the location and attitude of the terminal by matching each location information obtained from a first GPS built into the terminal and a second GPS attached to the terminal.

[0009] In addition, the present invention seeks to provide a GPS-based augmented reality display method and system for lost cultural assets that recognizes reality and the environment using a camera and IMU mounted on a smartphone and a captured image.

[0010] A method for displaying lost cultural heritage augmented reality based on GPS according to an embodiment of the present invention is a method for displaying lost cultural heritage augmented reality based on GPS by an augmented reality application executed by at least one processor of a terminal, the method comprising: obtaining first terminal location information based on a location of a first GPS device built into the terminal; obtaining second terminal location information based on a location of a second GPS device mounted on the terminal; calibrating the obtained first terminal location information and the second terminal location information; determining coordinates of the terminal based on a result of the calibration; obtaining at least one historic site location information based on the determined coordinates; and displaying a cultural heritage AR object that is matched to the obtained historic site location information.

[0011] In addition, the step of acquiring the second terminal location information is a step of acquiring second terminal location information with higher accuracy than the first terminal location information based on a second GPS, which is a high-precision GPS mounted on the terminal via a bracket.

[0012] In addition, the method further includes a step of correcting the result of the calibration based on the attitude information determined by the camera and position sensor (IMU) of the terminal, and the step of correcting the result of the calibration includes a step of obtaining a first attitude based on an image of the surrounding environment obtained by the camera of the terminal, a step of obtaining a second attitude based on 6 degrees of freedom calculated by the position sensor (IMU) of the terminal, and a step of determining attitude information by synthesizing the obtained first attitude and the second attitude.

[0013] Additionally, the step of correcting the results of the above calibration is performed based on at least one of VPS (Visual Positioning Service) and VIO (Visual Inertial Odometry) technologies.

[0014] In addition, the step of obtaining at least one piece of historic site location information based on the determined coordinates includes the step of obtaining all historic site location information previously stored in the database of the augmented reality service providing server, and the step of filtering and displaying at least one piece of historic site location information located within a predetermined distance based on the determined coordinates among all of the obtained historic site location information.

[0015] In addition, the step of filtering and displaying at least one piece of historic site location information further includes the step of providing a historic site map interface, the step of determining the coordinates as a first spot, which is the user's current location, based on the historic site map interface, the step of determining the detailed information as a first view, which is the user's current field of view, based on the historic site map interface, and the step of displaying first historic site location information located within a predetermined distance from the first spot and the first view.

[0016] In addition, the step of displaying a cultural heritage AR object matched to the acquired historic site location information includes a step of creating a first cultural heritage AR object based on an AR design tool including a Geospatial-Creator, and a step of matching the created first cultural heritage AR object to the first historic site location information and storing it.

[0017] In addition, the step of displaying a cultural heritage AR object matched to the acquired historic site location information includes a step of providing an AR object display interface that displays a first cultural heritage AR object by overlapping it at a location corresponding to the first historic site location information on a real image captured by the camera of the terminal.

[0018] In addition, the step of providing the AR object display interface further includes the step of adjusting the clarity of the first cultural heritage AR object based on at least one of the distance between the coordinates of the terminal and the coordinates of the first historic site location information and weather information acquired based on the sensor system of the terminal.

[0019] In addition, the step of displaying a cultural property AR object matched to the acquired historic site location information further includes a step of tracking at least one of the distance between the first GPS and the second GPS, an image of the surrounding environment acquired based on the camera of the terminal, and detailed information acquired based on a position sensor (IMU) of the terminal, and a step of updating the coordinates and detailed information of the terminal based on the result of the tracking.

[0020] Meanwhile, a GPS-based lost cultural heritage augmented reality display system according to an embodiment of the present invention comprises at least one memory; at least one processor; and at least one application stored in the memory and executed by the processor to provide an augmented reality service, wherein the at least one application acquires first terminal location information based on a location of a first GPS built into the terminal, acquires second terminal location information based on a location of a second GPS mounted on the terminal, calibrates the acquired first terminal location information and the second terminal location information, determines coordinates of the terminal based on a result of the calibration, acquires at least one historic site location information based on the determined coordinates, and displays a cultural heritage AR object matched to the acquired historic site location information.

[0021] The GPS-based augmented reality display method and system for lost cultural properties according to an embodiment of the present invention accurately determines the location and posture of the device by matching each location information obtained from a first GPS built into a terminal and a second GPS attached to the terminal, thereby enhancing the appearance of the cultural property so that it is very similar to its actual appearance before loss, thereby increasing the sense of realism and vividness.

[0022] In addition, the GPS-based lost cultural property augmented reality display method and system according to an embodiment of the present invention has the effect of lowering the barrier to access to cultural properties by allowing the user to easily access cultural properties using only the user's smartphone by recognizing reality and the environment using the camera and IMU and captured images installed in the user's smartphone.

[0023] Figure 1 is a conceptual diagram of a GPS-based augmented reality service providing system according to an embodiment of the present invention.

[0024] Figure 2 is an internal block diagram of a terminal according to an embodiment of the present invention.

[0025] FIG. 3 is a flowchart illustrating a method for displaying augmented reality of lost cultural assets based on GPS according to an embodiment of the present invention.

[0026] FIG. 4 and FIG. 5 are examples of drawings for explaining the location information of a historic site according to an embodiment of the present invention.

[0027] FIG. 6 and FIG. 7 are examples of an AR object display interface in which a cultural property AR object is displayed according to an embodiment of the present invention.

[0028] The present invention is capable of various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms. In the following embodiments, the terms "first," "second," etc. are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, in the drawings, the sizes of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the present invention is not necessarily limited to what is shown.

[0029] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals and redundant descriptions thereof will be omitted.

[0030]

[0031] Figure 1 is a conceptual diagram of a GPS-based augmented reality service providing system according to an embodiment of the present invention.

[0032] Referring to FIG. 1, a GPS-based augmented reality service providing system (hereinafter, “service providing system”) according to an embodiment of the present invention can provide a GPS-based augmented reality service (hereinafter, “augmented reality service”) that augments lost cultural assets at an accurate location by reducing the error range for the user’s location and posture based on high-precision GPS.

[0033] In the examples, a historic site may refer to an entire area containing cultural properties that have been partially or completely lost. For convenience of explanation, the examples below describe historic sites as locations where architectural cultural properties have been lost, leaving only the foundations.

[0034] At these historical sites, cultural property AR objects (augmented reality 3D images) can be recreated and displayed as presumptive images of the cultural properties using historical and reference materials about cultural properties that previously existed at the site.

[0035] In addition, an AR object according to an embodiment may mean an object whose appearance is displayed three-dimensionally according to changes in 6 degrees of freedom (6 DoF) and / or scale as content displayed in a predetermined augmented reality environment.

[0036] Additionally, in the embodiment, the user may refer to a user of a terminal (100) that receives an augmented reality service.

[0037] In an embodiment, a service providing system implementing the above augmented reality service can be connected through a terminal (100), an augmented reality service providing server (200), and a network (10: Network).

[0038] Here, the network (10) according to the embodiment means a connection structure in which information can be exchanged between each node, such as the terminal (100) and / or the augmented reality service providing server (200), and examples of such a network (10) include, but are not limited to, a 3GPP (3rd Generation Partnership Project) network, an LTE (Long Term Evolution) network, a WIMAX (World Interoperability for Microwave Access) network, the Internet, a LAN (Local Area Network), a Wireless LAN (Wireless Local Area Network), a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) network, etc.

[0039] Hereinafter, the terminal (100) and the augmented reality service provision server (200) that implement the service provision system will be described in detail with reference to the attached drawings.

[0040]

[0041] - Terminal (100)

[0042] A terminal (100) according to an embodiment of the present invention may be a computing device having installed an augmented reality application (hereinafter, “application”) that displays and provides an augmented reality service generated based on a database of an augmented reality service providing server (200).

[0043] In detail, from a hardware perspective, the terminal (100) may include a mobile type computing device (100-1) and / or a desktop type computing device (100-2) on which an application is installed.

[0044] Here, the mobile type computing device (100-1) may be a mobile device such as a smart phone or tablet PC on which an application is installed.

[0045] For example, mobile type computing devices (100-1) may include smart phones, mobile phones, digital broadcasting devices, personal digital assistants (PDAs), portable multimedia players (PMPs), tablet PCs, etc.

[0046] In addition, the desktop type computing device (100-2) may include a device having a program installed for executing an augmented reality service based on wired / wireless communication, such as a personal computer such as a fixed desktop PC, laptop computer, or ultrabook on which an application is installed.

[0047] Additionally, depending on the embodiment, the terminal (100) may further include a predetermined server computing device that provides an augmented reality service environment.

[0048] Figure 2 is an internal block diagram of a terminal according to an embodiment of the present invention.

[0049] Referring to FIG. 2, from a functional perspective, the terminal (100) may include a memory (110), a processor assembly (120), a communication processor (130), an interface module (140), an input system (150), a sensor system (160), and a display system (170). These components may be configured to be included within the housing of the terminal (100).

[0050] In detail, in the memory (110), an application (111) is stored, and the application (111) can store one or more of various application programs, data, and commands for providing an augmented reality service environment.

[0051] That is, the memory (110) can store commands and data that can be used to create an augmented reality service environment.

[0052] Additionally, the memory (110) may include a program area and a data area.

[0053] Here, the program area according to the embodiment can be linked between the operating system (OS) that boots the terminal (100) and functional elements, and the data area can store data generated according to the use of the terminal (100).

[0054] Additionally, the memory (110) may include at least one non-transitory computer-readable storage medium and one or more temporary computer-readable storage medium.

[0055] For example, the memory (110) may be a variety of storage devices such as a ROM, EPROM, flash drive, hard drive, etc., and may include web storage that performs the storage function of the memory (110) on the Internet.

[0056] The processor assembly (120) may include at least one processor capable of executing commands of an application (111) stored in the memory (110) to perform various tasks for creating an augmented reality service environment.

[0057] In an embodiment, the processor assembly (120) can control the overall operation of the components through an application (111) of the memory (110) to provide an augmented reality service.

[0058] This processor assembly (120) may be a system on chip (SOC) suitable for a terminal (100) including a central processing unit (CPU) and / or a graphics processing unit (GPU), and may execute an operating system (OS) and / or application programs stored in a memory (110) and control each component mounted on the terminal (100).

[0059] Additionally, the processor assembly (120) can communicate with each component internally via a system bus and can include one or more predetermined bus structures including a local bus.

[0060] Additionally, the processor assembly (120) may be implemented by including at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0061] The communication processor (130) may include one or more devices for communicating with external devices. The communication processor (130) may communicate via a wireless network.

[0062] In detail, the communication processor (130) can communicate with a terminal (100) that stores a content source for implementing an augmented reality service environment, and can communicate with various user input components, such as a controller that receives user input.

[0063] In an embodiment, the communication processor (130) can transmit and receive various data related to the augmented reality service to and from other terminals and / or external servers.

[0064] This communication processor (130) can wirelessly transmit and receive data with at least one of a base station, an external terminal, and an arbitrary server on a mobile communication network constructed through a communication device capable of performing technical standards or communication methods for mobile communication (e.g., LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G NR (New Radio), WIFI) or short-range communication methods.

[0065] The interface module (140) can communicatively connect the terminal (100) to one or more other devices. In detail, the interface module (140) can include wired and / or wireless communication devices compatible with one or more different communication protocols.

[0066] Through this interface module (140), the terminal (100) can be connected to multiple input / output devices.

[0067] For example, the interface module (140) can be connected to an audio output device such as a headset port or speaker to output audio.

[0068] As an example, the audio output device is described as being connected via an interface module (140), but an embodiment in which it is installed inside the terminal (100) may also be included.

[0069] Additionally, for example, the interface module (140) may be connected to an input device such as a keyboard and / or mouse to obtain user input.

[0070] Such an interface module (140) may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0071] The input system (150) can detect user input related to the augmented reality service (e.g., gestures, voice commands, button operation, or other types of input).

[0072] In detail, the input system (150) may include a predetermined button, a touch sensor, and / or an image sensor (161) that receives user motion input.

[0073] Additionally, the input system (150) can be connected to an external controller through an interface module (140) to receive user input.

[0074] The sensor system (160) may include various sensors such as an image sensor (161), a position sensor (IMU, 163), an audio sensor (165), a distance sensor, a proximity sensor, and a contact sensor.

[0075] Here, the image sensor (161) can capture images and / or videos of the physical space around the terminal (100).

[0076] In an embodiment, the image sensor (161) can capture and acquire various images and / or videos related to the augmented reality service.

[0077] In addition, the image sensor (161) can capture an image by photographing the direction in which it is positioned on the front or / and rear of the terminal (100), and can capture a physical space through a camera positioned toward the outside of the terminal (100).

[0078] This image sensor (161) may include an image sensor device and an image processing module. In detail, the image sensor (161) may process still images or moving images obtained by an image sensor device (e.g., CMOS or CCD).

[0079] In addition, the image sensor (161) can process still images or moving images acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to the processor.

[0080] Such an image sensor (161) may be a camera assembly including at least one camera. The camera assembly may include a general camera that captures images in the visible light band, and may further include special cameras such as an infrared camera or a stereo camera.

[0081] In addition, the image sensor (161) as described above may be included in the terminal (100) and operated according to an embodiment, or may be included in an external device (e.g., an external server, etc.) and operated through linkage based on the communication processor (130) and / or interface module (140) described above.

[0082] The position sensor (IMU, 163) can detect at least one of the movement and acceleration of the terminal (100). For example, it can be formed by a combination of various position sensors such as an accelerometer, a gyroscope, and a magnetometer.

[0083] Additionally, the position sensor (IMU, 163) can recognize spatial information about the physical space around the terminal (100) by working in conjunction with a position communication processor (130), such as the GPS of the communication processor (130).

[0084] In an embodiment, the position sensor (IMU, 163) can detect and obtain six degrees of freedom and / or scale changes of the terminal (100).

[0085] The audio sensor (165) can recognize sounds around the terminal (100).

[0086] In detail, the audio sensor (165) may include a microphone capable of detecting voice input from a user using the terminal (100).

[0087] In an embodiment, the audio sensor (165) can receive voice data required for augmented reality service from the user.

[0088] The display system (170) can output various information related to the augmented reality service as graphic images.

[0089] As an example, the display system (170) can display various user interfaces for augmented reality services (as an example, a historic site map interface and / or an AR object display interface, etc.).

[0090] Such displays may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0091] The above components may be arranged within the housing of such a terminal (100), and the user interface may include a touch sensor (173) on a display (171) configured to receive user touch input.

[0092] In detail, the display system (170) may include a display (171) that outputs an image and a touch sensor (173) that detects a user's touch input.

[0093] For example, the display (171) may be implemented as a touch screen by forming a mutual layer structure with the touch sensor (173) or forming an integral structure. Such a touch screen may function as a user input unit that provides an input interface between the terminal (100) and the user, and at the same time, provide an output interface between the terminal (100) and the user.

[0094] The terminal (100) including the above-described components can store at least one terminal location information, terminal coordinates, historic site location information, cultural property AR objects, clarity by distance between coordinates, and clarity by weather type in the memory (110) according to an embodiment.

[0095] Meanwhile, in another embodiment, the terminal (100) may be implemented as a wearable device, such as smart glasses. For convenience of explanation, the embodiments of the present invention are described based on the terminal (100) being implemented as a smart phone.

[0096] Additionally, in the embodiment, the terminal (100) may include a built-in GPS.

[0097] At this time, the built-in GPS may be a general GPS that is affected by the strength of the received signal and has poor accuracy in areas where the signal is obstructed.

[0098] The above-mentioned built-in GPS can correct the result value based on a position sensor (IMU) composed of a combination of various position sensors such as an accelerometer, a gyroscope, and a magnetometer included in the terminal (100) to compensate for the above-mentioned limitations.

[0099] However, in order to obtain more accurate GPS results, an external GPS may be attached to an external side of the terminal (100) in the embodiment. Specifically, in the embodiment, the external GPS may be attached to an external side of the terminal (100) using a bracket. In another embodiment, the external GPS may be a separate electronic device separated from the terminal (100). In another embodiment, the external GPS may be attached to an external side of the terminal (100) or may be detached and carried separately by the user.

[0100] At this time, the external GPS may include one or more GPS receivers and may be a high-precision GPS that obtains relatively accurate location information by using multipath signal processing technology, advanced signal processing technology, and / or multi-satellite signal processing technology.

[0101] These external GPSs, unlike the built-in GPS which only provides location information, can provide location information and / or direction information.

[0102] Therefore, in the embodiment, the terminal (100) can obtain accurate location information and / or direction information based on a sensor system including a camera, built-in GPS, external GPS, and / or IMU.

[0103] Meanwhile, depending on the embodiment, the terminal (100) may further perform at least some of the functional operations performed by the augmented reality service providing server (200) described below.

[0104]

[0105] - Augmented Reality Service Providing Server (200)

[0106] Meanwhile, the augmented reality service providing server (200) according to an embodiment of the present invention can perform a series of processes for providing an augmented reality service.

[0107] In detail, in an embodiment, the augmented reality service providing server (200) can provide the augmented reality service by exchanging data necessary to enable the augmented reality service process to be driven in an external device such as a terminal (100) with the external device.

[0108] In more detail, in an embodiment, the augmented reality service providing server (200) can provide an environment in which an application (111) can operate on an external device (in an embodiment, a mobile type computing device (100-1) and / or a desktop type computing device (100-2) etc.).

[0109] To this end, the augmented reality service providing server (200) may include an application program, data and / or commands for the application (111) to operate, and may transmit and receive various data based thereon with the external device.

[0110] Additionally, in the embodiment, the augmented reality service providing server (200) can perform various deep learning for augmented reality services in conjunction with a deep-learning neural network.

[0111] Here, the deep learning neural network according to the embodiment may include a convolutional neural network (CNN), an R-CNN (Regions with CNN features), a Fast R-CNN, a Faster R-CNN, a Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiment described below, and the embodiment of the present invention does not limit or restrict such deep learning neural network itself.

[0112] At this time, depending on the embodiment, the deep learning neural network may be installed directly in the augmented reality service providing server (200) or may operate as a separate device from the augmented reality service providing server (200) to perform deep learning for the augmented reality service.

[0113] In the following examples, an example is described in which a deep learning neural network is directly installed on an augmented reality service providing server (200) to perform deep learning.

[0114] In addition, in the embodiment, the augmented reality service providing server (200) can read out a predetermined deep learning neural network driving program constructed to perform the above-described deep learning from memory and perform the deep learning described below according to the read-out predetermined deep learning neural network system.

[0115] Additionally, in the embodiment, the augmented reality service providing server (200) can store and manage various application programs, commands, and / or data for implementing the augmented reality service.

[0116] As an example, the augmented reality service providing server (200) can store and manage terminal location information, terminal coordinates, historic site location information, cultural property AR objects, clarity by distance between coordinates, clarity by weather type, historic site map interface, and / or AR object display interface, etc.

[0117] In an embodiment, the augmented reality service providing server (200) can design historical site content based on the BIM (Building Information Modeling: digital modeling that can create a virtual model of a building based on a three-dimensional (3D) model) method.

[0118] To this end, in the embodiment, the augmented reality service providing server (200) can store at least one digital model for displaying various information related to the historic site, such as the shape of a cultural asset included in the historic site and / or a description of the historic site, in augmented reality.

[0119] For example, a digital model may include architectural features such as floors, walls, and columns; arrows to indicate paths; and text, images, and / or videos to describe the site.

[0120] Additionally, in the embodiment, the augmented reality service providing server (200) can preset the location information of a historic site to be displayed in augmented reality (hereinafter, historic site location information).

[0121] These historical site location information may be associated with designed cultural heritage AR objects.

[0122] Additionally, in the embodiment, the augmented reality service providing server (200) can extract and display a cultural heritage AR object corresponding to preset historic site location information.

[0123] However, in the embodiment of the present invention, the functional operations that the augmented reality service providing server (200) can perform are not limited to those described above, and other functional operations can be performed.

[0124] Meanwhile, referring further to FIG. 1, in the embodiment, the augmented reality service providing server (200) as described above may be implemented as a computing device including at least one processor module (210: Processor Module) for data processing, at least one communication module (220: Communication Module) for data exchange with an external device, and at least one database (230: Database) for storing various application programs, data, and / or commands for providing augmented reality services.

[0125] Here, the database (230) can store one or more of an operating system (OS), various application programs, data, and commands for providing augmented reality services.

[0126] Additionally, the database (230) may include a program area and a data area.

[0127] Here, the program area according to the embodiment may be linked between the operating system (OS) that boots the server and functional elements, and the data area may store data generated according to the use of the server.

[0128] In an embodiment, such a database (230) may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and may also be a web storage that performs the storage function of the database (230) on the Internet.

[0129] Additionally, the database (230) may be a removable storage medium on the server.

[0130] Meanwhile, the processor module (210) can control the overall operation of each unit described above to implement an augmented reality service.

[0131] This processor module (210) may be a system on chip (SOC) suitable for a server including a central processing unit (CPU) and / or a graphics processing unit (GPU), and may execute an operating system (OS) and / or application programs stored in a database (230) and control each component mounted on the server.

[0132] In addition, the processor module (210) can communicate with each component internally via a system bus and can include one or more predetermined bus structures including a local bus.

[0133] Additionally, the processor module (210) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0134] In the above description, it has been described that the augmented reality service providing server (200) according to the embodiment of the present invention performs the functional operation as described above, but depending on the embodiment, at least a part of the functional operation performed by the augmented reality service providing server (200) may be performed by an external device (e.g., terminal (100), etc.), and at least a part of the functional operation performed by the external device may be further performed by the augmented reality service providing server (200), and various other embodiments may be possible.

[0135]

[0136] - GPS-based augmented reality display method for lost cultural assets

[0137] Hereinafter, a method of displaying augmented reality of lost cultural assets based on GPS to provide augmented reality service by an application (111) executed by at least one processor of a terminal (100) according to an embodiment of the present invention will be described in detail with reference to the attached FIGS. 3 to 7.

[0138] In an embodiment of the present invention, at least one processor of the terminal (100) can execute at least one application (111) stored in at least one memory (110) or operate in a background state.

[0139] Hereinafter, the method of displaying augmented reality of lost cultural assets based on the GPS described above by operating at least one processor to execute the command of the application (111) is briefly described as being performed by the application (111).

[0140] FIG. 3 is a flowchart illustrating a method for displaying augmented reality of lost cultural assets based on GPS according to an embodiment of the present invention.

[0141] Referring to FIG. 3, in the embodiment, the application (111) can obtain first terminal location information based on the first GPS of the terminal (100). (S101)

[0142] At this time, the first GPS may be an internal GPS included in the terminal (100).

[0143] Additionally, the first GPS may be a general GPS that is affected by the strength of the received signal and whose accuracy decreases in areas where the signal is obstructed.

[0144] That is, in the embodiment, the application (111) can obtain the first terminal location information estimated to be the location where the user is currently located.

[0145] Here, the first terminal location information is information indicating the user's current location, and its accuracy may be lower than that of the second terminal location information described below.

[0146] Additionally, in the embodiment, the application (111) can obtain second terminal location information based on the second GPS of the terminal (100). (S103)

[0147] At this time, the second GPS may be a high-precision GPS attached to one external side of the terminal (100).

[0148] In addition, the second GPS may include one or more GPS receivers and may be a high-precision GPS that obtains relatively accurate location information by using multipath signal processing technology, advanced signal processing technology, and / or multi-satellite signal processing technology.

[0149] That is, in the embodiment, the application (111) can obtain second terminal location information estimated to be where the user is currently located.

[0150] At this time, the second GPS, unlike the built-in GPS that provides only location information, can provide location information, detail information, and / or direction information.

[0151] Here, the second terminal location information is information indicating the user's current location, similar to the first terminal location information, but may have higher accuracy than the first terminal location information.

[0152] Additionally, in the embodiment, the application (111) can correct the first and second terminal location information acquired by the first and second GPS based on the camera and / or location sensor (IMU, 163), and details thereof will be described later.

[0153] Additionally, in the embodiment, the application (111) can calibrate the acquired first terminal location information and / or second terminal location information. (S105)

[0154] At this time, calibration may mean an act of checking and adjusting the precision and / or accuracy of the location information of the terminal (100).

[0155] In an embodiment, the first terminal location information may be obtained based on a first location where the first GPS is located on an internal side of the terminal (100), and the second terminal location information may be obtained based on a second location where the second GPS is located on an external side of the terminal (100) and / or an independent location.

[0156] At this time, since the external GPS is located somewhere outside the terminal (100), the second position may change depending on the shape of the bracket and / or the carrying position of the external GPS, and thus a difference is bound to occur between the first position and the second position. In addition, if the terminal (100) is a wearable device and the external GPS is implemented as a separate device from the terminal (100), the difference will be even greater.

[0157] Therefore, in the embodiment, the application (111) can calibrate the first terminal location information and the second terminal location information corresponding to the first location and the second location.

[0158] In an embodiment, the application (111) can calibrate the first terminal location information and / or the second terminal location information based on a preset calibration function.

[0159] For example, the calibration function may be a function that utilizes the least square error method, etc.

[0160] Additionally, in the embodiment, the application (111) can measure the distance between the coordinates of the first terminal location information and / or the second terminal location information.

[0161] Additionally, in the embodiment, the application (111) can adjust the constant value of the calibration function according to the ratio of the distance between the measured coordinates.

[0162] Accordingly, in the embodiment, the application (111) can determine the coordinates of the terminal (100) based on the calibration result. (S107)

[0163] Meanwhile, in the embodiment, the application (111) may determine the coordinates of the terminal (100) by correcting the calibration result based on the detailed information of the terminal (100).

[0164] To this end, in the embodiment, the application (111) can determine the position information of the terminal (100) using predetermined data acquired based on at least one of the camera, position sensor (IMU, 163), built-in GPS, and / or external GPS of the terminal (100).

[0165] At this time, in the embodiment, the application (111) can determine the position information of the terminal (100) by performing VPS (Visual Positioning Service) and / or VIO (Visual Inertial Odometry) technology on the acquired predetermined data.

[0166] In an embodiment, the VPS technology may be performed based on an image search (an image-based location determination method) that matches a query image with an image in a database. To this end, the application (111) may store a data-based image, which is a big data version of a street view photograph, in the database (230) of the augmented reality service providing server (200).

[0167] Additionally, in an embodiment, the VIO technology may be a computer vision technology that estimates the three-dimensional posture and speed of a terminal (100) moving based on a local starting position based on at least one of a position sensor (IMU, 163) of the terminal (100), an internal GPS, and / or an external GPS. To this end, in an embodiment, the application (111) may obtain and store determined coordinates and / or location information measured in real time.

[0168] In an embodiment, the application (111) can activate the camera function of the terminal (100) through automatic and / or manual input.

[0169] Additionally, in the embodiment, the application (111) can acquire an image of the surrounding environment including the surrounding environment of the terminal (100) based on the camera. At this time, the surrounding environment may include a surrounding terrain feature, such as a historic site environment according to the embodiment.

[0170] Additionally, in the embodiment, the application (111) can determine the first pose by performing tracking, environmental understanding (planar tracking) and / or light source estimation based on the acquired surrounding environment image.

[0171] In detail, in an embodiment, the application (111) can perform plane tracking and / or 3D coordinate position tracking by performing tracking on an image of the surrounding environment.

[0172] Additionally, in the embodiment, the application (111) can place an AR object or determine a play area of ​​the AR object by recognizing a plane in the surrounding environment image.

[0173] Additionally, in the embodiment, the application (111) can measure the illumination intensity of the current space where the user is located to implement the brightness of the AR object by estimating the light source for the surrounding environment image.

[0174] That is, in the embodiment, the application (111) can determine the first posture by synthesizing predetermined data acquired based on the camera of the terminal (100).

[0175] Additionally, in the embodiment, the application (111) can determine the second posture based on the six degrees of freedom calculated by the position sensor (IMU, 163) of the terminal (100).

[0176] To this end, in the embodiment, the application (111) can perform location tracking and / or map creation (SLAM) through the location sensor (IMU, 163) of the terminal (100).

[0177] Additionally, in an embodiment, the application (111) may determine the second attitude by calculating dead reckoning based on a sensor system (160) (specifically, an accelerometer and / or gyroscope).

[0178] That is, in the embodiment, the application (111) can determine the final posture information of the terminal (100) by synthesizing the first and second postures based on the VIO technology.

[0179] In other words, in the embodiment, the application (111) can perform plane tracking based on the camera of the terminal (100), perform position tracking and mapping (SLAM) through the position sensor (IMU, 163) of the terminal (100), and determine the attitude information of the terminal (100) based on the VIO technology that calculates dead reckoning based on the sensor system (160).

[0180] Accordingly, in the embodiment, the application (111) may determine the coordinates of the terminal (100) by correcting the calibration result based on the determined detailed information of the terminal (100).

[0181] Additionally, in the embodiment, the application (111) can increase the accuracy of the coordinates of the terminal (100) based on a predetermined marker (e.g., QR code, etc.).

[0182] In an embodiment, the application (111) can obtain a reference point based on a predetermined marker.

[0183] Additionally, in the embodiment, the application (111) can create a new anchor in a digital world based on an AR object design tool (e.g., Unity) based on the acquired reference point.

[0184] Additionally, in the embodiment, the application (111) can update the position and rotation of the terminal (100) and the position and rotation of the camera to match by synchronizing the coordinates of the terminal (100) based on the generated anchor.

[0185] Additionally, in the embodiment, the application (111) can convert the updated coordinates into position, rotation and size in the AR design space.

[0186] Accordingly, in the embodiment, the application (111) can increase the accuracy of the coordinates of the terminal (100).

[0187] Additionally, in the embodiment, the application (111) can obtain and display at least one piece of historical site location information based on determined coordinates. (S109)

[0188] Here, the location information of the historic site according to the embodiment is information including the location (coordinates) where the AR object of the cultural property reproducing the shape of the lost cultural property is to be displayed, and may be the location (coordinates) where the actual lost cultural property existed.

[0189] Such historical site location information may be preset by a database manager and stored in the memory (110) and / or the database (230) of the augmented reality service providing server (200). Accordingly, a cultural heritage AR object pre-matched to the historical site location information may also be pre-stored.

[0190] In addition, in the embodiment, when the coordinates of the terminal (100) are determined, the application (111) can obtain and display at least one piece of historic site location information filtered based on the determined coordinates among all previously stored historic site location information.

[0191] FIG. 4 and FIG. 5 are examples of drawings for explaining the location information of a historic site according to an embodiment of the present invention.

[0192] Referring to FIG. 4, in an embodiment, the application (111) can display at least one piece of historic site location information based on a historic site map interface (HMI).

[0193] At this time, in the embodiment, the historic site map interface (HMI) may display all previously stored historic site location information, but may only expose historic site location information located within a certain distance based on the user's current location.

[0194] For example, if the user's current location is the first spot (S1), the application (111) can display the location information (301, 302) of the first and second historic sites located within a predetermined distance from the first spot (S1).

[0195] At this time, when the user moves, the user's current location can be reflected in real time and the location information of the historic site can be displayed based on the changed location.

[0196] That is, when the user's current location moves from the first spot (S1) to the second spot (S2) as the user moves, the location information (303) of the third historic site located within a predetermined distance based on the second spot (S2) can be displayed.

[0197] At this time, if the movement from the first spot (S1) to the second spot (S2) is within a predetermined distance that can be displayed on one screen without switching screens on the historic site map interface (HMI), the first to third historic site location information (301, 302, 303) can all be displayed. On the other hand, if the movement from the first spot (S1) to the second spot (S2) is outside a predetermined distance that requires switching screens on the historic site map interface (HMI), only the first and second historic site location information (301, 302) can be displayed on the first spot (S1), and only the third historic site location information (303) can be displayed on the second spot (S2).

[0198] Additionally, in the embodiment, the application (111) can only display location information of historic sites located within a predetermined viewing distance based on the user's current posture.

[0199] For example, if the user's current field of view is the first view (V1), the application (111) can display the first historic site location information (301) located within a predetermined field of view distance from the first view (V1).

[0200] At this time, if the user changes his / her posture or turns in place, the user's current field of view can be reflected in real time and the location information of the historic site can be displayed based on the changed field of view.

[0201] That is, when the user's current position moves from the first view (V1) to the second view (V2) as the user changes his / her posture in place, the second relic site location information (302) located within a predetermined viewing distance based on the second view (V2) can be displayed.

[0202] At this time, since the first view (V1) and the second view (V2) only change the user's posture and angle in the same spot, both the first and second historic site location information (301, 302) can be displayed on the same screen on the historic site map interface (HMI). However, unlike the historic site map interface (HMI) that only displays the locations of historic sites located near the user, the AR object display interface described below can only display cultural heritage AR objects included in the user's field of view.

[0203] This site map interface (HMI) can be provided independently or as part of an AR object display interface described below. This will be described later.

[0204] Meanwhile, the above-described historic site location information may each include a matched cultural heritage AR object.

[0205] Accordingly, in the embodiment, the application (111) can display a cultural heritage AR object matched to the acquired historic site location information. (S111)

[0206] In detail, in the embodiment, the application (111) can display a cultural property AR object matched to the acquired historic site location information based on the AR object display interface.

[0207] Here, the cultural heritage AR object according to the embodiment may be AR content that recreates the shape of a lost cultural heritage by combining multiple digital models. Furthermore, the cultural heritage AR object may be designed directly within the application (111), but may also be received as a pre-designed object from an external terminal and / or server.

[0208] Additionally, in the embodiment, the tool for designing the cultural heritage AR object may be an AR platform called 'Geospatial-Creator' that can visualize, create, and publish AR content in Unity and / or Adobe Aero.

[0209] That is, on the application (111) of the terminal (100), a cultural property AR object can be designed based on the coordinates of the terminal (100) through the geospatial creator.

[0210] Additionally, multiple digital models that constitute a single cultural heritage AR object do not necessarily have to be contained in a single independent building, and can be considered as a single cultural heritage AR object even if they are located separately outside the building.

[0211] Additionally, in the embodiment, the cultural property AR object can be displayed in three dimensions according to changes in 6 degrees of freedom (6 DoF) and / or scale.

[0212] FIG. 6 and FIG. 7 are examples of an AR object display interface in which a cultural property AR object is displayed according to an embodiment of the present invention.

[0213] Referring to FIG. 6, in the embodiment, the application (111) can display a cultural property AR object matched to the location information of the historic site based on the AR object display interface (HAI).

[0214] At this time, in the embodiment, the AR object display interface (HAI) can display, among all previously stored cultural heritage AR objects, a cultural heritage AR object that matches the location information of a historic site located within a predetermined distance from the user.

[0215] That is, in the embodiment, when a user looks at a historic site where a lost cultural asset existed, the application (111) displays an AR object of the lost cultural asset overlaid on a real background based on the AR object display interface (HAI), thereby providing a sense of vividness and mystery as if the user is actually looking at a cultural asset that does not exist.

[0216] Such AR object display interface (HAI) may include a heritage site map area (410), a menu area (420), and / or a cultural property AR object (500).

[0217] The historic site map area (410) may be an area where the historic site map interface (HMI) is reduced and displayed on one side of the screen.

[0218] The menu area (420) may be an area including a menu for providing at least one function for controlling the AR object display interface (HAI).

[0219] The description area (430) may be an area that displays a detailed description of a cultural heritage AR object in text through automatic and / or manual (e.g., user touch) input.

[0220] The above-mentioned historic site map area (410), menu area (420) and / or description area (430) can be hidden according to the user's control.

[0221] Additionally, the AR Object Display Interface (HAI) can display the real-world image captured by the camera on the screen.

[0222] The cultural property AR object (500) can be displayed by overlapping coordinates corresponding to the location information of the historic site based on the real-world image captured by the camera of the terminal (100).

[0223] At this time, the cultural property AR object (500) can be exposed in a three-dimensional shape like an actual building according to the coordinates, posture, and / or angle of the terminal (100).

[0224] That is, in the embodiment, when the posture of the terminal (100) changes as the posture of the user changes, the application (111) can output an image of the cultural asset AR object (500) when viewed from a corresponding viewing angle, as shown in FIG. 7, so as to be similar to the user's actual field of view.

[0225] In detail, FIG. 6 may be an example of a screen when a cultural property AR object (500) is located on the right side based on the user's position, and FIG. 7 may be an example of a screen when a cultural property AR object (500) is located on the front side based on the user's position.

[0226] Additionally, in the embodiment, the application (111) can continuously track the posture and position of the terminal (100) and change and determine the coordinates of the terminal (100) in real time.

[0227] At this time, in the embodiment, the application (111) can change and determine the attitude, position and / or direction of the terminal (100) based on at least one of the camera, position sensor (IMU, 163), built-in GPS and / or external GPS of the terminal (100).

[0228] Here, similarly to the above-described step S107, in the embodiment, the application (111) can change and determine the attitude information of the terminal (100) by performing VPS (Visual Positioning Service) and VIO (Visual Inertial Odometry) technology based on a camera and / or a position sensor (IMU, 163).

[0229] Additionally, in the embodiment, the application (111) may determine the coordinates of the terminal (100) by using a deep learning-based VPS technology that uses a 3D point cloud to learn the 3D point cloud from a deep learning network and to backtrack the user's location.

[0230] The coordinates of the terminal (100) determined in this way can be composed of center position coordinates (x, y, z), size (length, width, height), and rotation value (X-axis rotation-Roll, Y-axis rotation-Pitch, Z-axis rotation-Yaw).

[0231] That is, in the embodiment, the application (111) can obtain location information and direction information by comparing an image captured by the camera of the terminal (100) with a VPS index to produce a positioning result (e.g., a matching rate).

[0232] In other words, in the embodiment, the application (111) can continuously track the posture and position of the terminal (100) and adjust the shape, angle, size, etc. of the AR object displayed according to the posture and position of the terminal (100).

[0233] In other words, the appearance of the cultural heritage AR object (500) displayed in correspondence with the location information of the historic site according to the changed coordinates of the terminal (100) can also be changed and determined in real time according to the changed coordinates of the terminal (100). At this time, the appearance of the cultural heritage AR object (500) does not mean that the shape of the object is changed, but may mean the appearance exposed to the user according to the angle and position.

[0234] Meanwhile, even if the user is far away from the cultural heritage AR object or part or all of the cultural heritage AR object is obscured by the surrounding environment and only part of it is visible in the user's field of view, the cultural heritage AR object according to the embodiment may be continuously exposed like a real architectural structure.

[0235] To this end, in the embodiment, the application (111) can calculate a predetermined viewing distance for each coordinate of each historic site location information and adjust the clarity of the cultural heritage AR object exposed for each viewing distance.

[0236] In an embodiment, the application (111) can measure the distance between coordinates by comparing the coordinates of the terminal (100) (hereinafter, first coordinates) and the coordinates of at least one or more relic site location information.

[0237] At this time, in the embodiment, the application (111) can set the clarity for each distance between coordinates for each relic site location information based on the measured distance between coordinates.

[0238] In detail, in the embodiment, the application (111) can extract the clarity corresponding to the determined first distance when the distance between the first coordinate and the coordinate of the first relic site location information is determined as the first distance.

[0239] Accordingly, in the embodiment, the application (111) can apply the extracted clarity to the first cultural heritage AR object matched to the first historic site location information.

[0240] For example, if the distance between coordinates is divided into five sections: less than 1 km, 1 km or more but less than 3 km, 3 km or more but less than 5 km, 5 km or more but less than 7 km, and 7 km or more, then differential clarity levels of 100%, 90%, 80%, 70%, and 60% can be set for each section.

[0241] As a more detailed example, if a user is 7 km away from the location information of a historic site, the cultural heritage AR object corresponding to the location information of the historic site can be displayed on the user's terminal (100) with a clarity of 60%.

[0242] These coordinate-based distance-based clarity levels can be preset differently depending on the location information of the site and the corresponding AR cultural asset object. This is because, depending on the size of the cultural asset, it may be visible only from very close range (e.g., within 30 meters) or from very far away (e.g., over 10 kilometers).

[0243] That is, the first cultural asset AR object with the corresponding clarity applied may appear blurry depending on the distance to the historic site, which has the effect of increasing the user's sense of immersion by making the AR object perceived as a more realistic building.

[0244] Additionally, in cases where visibility is reduced due to bad weather, AR cultural heritage objects that are continuously exposed like actual architectural structures may appear blurry beyond a certain distance to provide the same vividness as reality.

[0245] To this end, in the embodiment, the application (111) can preset the clarity applied to all historic site location information according to at least one weather type.

[0246] In an embodiment, the application (111) may preset the clarity level according to the weather type. For example, if the weather type is 'clear', the clarity level may be 100%, and if the weather type is 'cloudy', the clarity level may be 60%.

[0247] Additionally, in the embodiment, the application (111) can obtain weather information based on the sensor system (160) of the terminal (100).

[0248] Additionally, in the embodiment, the application (111) can determine the first weather type based on the acquired weather information.

[0249] Additionally, in the embodiment, the application (111) can extract a preset clarity for the determined first weather type.

[0250] Additionally, in the embodiment, the application (111) can apply the extracted clarity to the cultural heritage AR object matched to all historical site location information.

[0251] At this time, since the weather varies by hour / region, the clarity applied to the cultural heritage AR object can be continuously changed while identifying the time / region of the location of each user in real time.

[0252] Above, the GPS-based augmented reality display method and system for lost cultural properties according to an embodiment of the present invention accurately determines the location and posture of the device by matching each location information acquired from a first GPS built into a terminal and a second GPS attached to the terminal, thereby enhancing the appearance of the cultural property so that it is very similar to its actual appearance before loss, thereby increasing the sense of realism and vividness.

[0253] In addition, the GPS-based lost cultural property augmented reality display method and system according to an embodiment of the present invention has the effect of lowering the barrier to access to cultural properties by allowing the user to easily access cultural properties using only the user's smartphone by recognizing reality and the environment using the camera and IMU and captured images installed in the user's smartphone.

[0254] The embodiments of the present invention described above may be implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be changed into one or more software modules to perform processing according to the present invention, and vice versa.

[0255] The specific implementations described in the present invention are exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components illustrated in the drawings are merely representative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0256] Although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the present invention without departing from the spirit and technical scope of the present invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0257] The present invention is a method for providing augmented reality services through various augmented reality devices and a system including the same, and thus has industrial applicability.

Claims

1. A method for displaying augmented reality of lost cultural assets based on GPS by an augmented reality application executed by at least one processor of a terminal, A step of obtaining first terminal location information based on the location of the first GPS built into the terminal; A step of obtaining second terminal location information based on the location of a second GPS mounted on the terminal; A step of calibrating the acquired first terminal location information and the acquired second terminal location information; A step of determining the coordinates of the terminal according to the result of the above calibration; A step of obtaining at least one relic site location information based on the above-determined coordinates; and A step of displaying a cultural property AR object matched to the above-mentioned acquired historic site location information; including GPS-based augmented reality display method for lost cultural assets.

2. In paragraph 1, The step of obtaining the above second terminal location information is: A step of obtaining second terminal location information with higher accuracy than the first terminal location information based on the second GPS, which is a high-precision GPS mounted on the above terminal through a bracket. GPS-based augmented reality display method for lost cultural assets.

3. In paragraph 1, It further includes a step of correcting the result of the calibration based on the attitude information determined by the camera and position sensor (IMU) of the terminal. The step of correcting the results of the above calibration is: A step of acquiring a first posture based on an image of the surrounding environment acquired by the camera of the terminal, A step of acquiring a second attitude based on the 6 degrees of freedom calculated by the position sensor (IMU) of the above terminal, and A step of determining posture information by synthesizing the first posture and the second posture acquired above GPS-based augmented reality display method for lost cultural assets.

4. In paragraph 3, The step of correcting the results of the above calibration is: It is performed based on at least one of the technologies of VPS (Visual Positioning Service) and VIO (Visual Inertial Odometry). GPS-based augmented reality display method for lost cultural assets.

5. In paragraph 1, The step of obtaining at least one relic site location information based on the above-determined coordinates is: Steps for obtaining all historical site location information already stored in the database of the augmented reality service providing server, A step of filtering and displaying at least one relic site location information located within a predetermined distance based on the determined coordinates among all the relic site location information acquired above. GPS-based augmented reality display method for lost cultural assets.

6. In paragraph 3, The step of filtering and displaying at least one of the above relic site location information is as follows: Steps for providing a heritage site map interface, A step of determining the coordinates as the first spot, which is the user's current location, based on the above-mentioned historic site map interface, A step of determining the detailed information as the first view, which is the user's current field of view, based on the above-mentioned historic site map interface; and Further comprising a step of displaying location information of a first relic site located within a predetermined distance from the first spot and the first view. GPS-based augmented reality display method for lost cultural assets.

7. In paragraph 6, The step of displaying the cultural property AR object matched to the above-mentioned acquired historic site location information is as follows. A step of creating a first cultural property AR object based on an AR design tool including Geospatial-Creator, A step of storing the first cultural heritage AR object generated above by matching it to the first historic site location information. GPS-based augmented reality display method for lost cultural assets.

8. In paragraph 1, The step of displaying the cultural property AR object matched to the above-mentioned acquired historic site location information is as follows. A step of providing an AR object display interface that displays a first cultural heritage AR object by overlaying it at a location corresponding to the first historic site location information on a real image captured by the camera of the terminal GPS-based augmented reality display method for lost cultural assets.

9. In paragraph 8, The step of providing the above AR object display interface is: It further includes a step of adjusting the clarity of the first cultural heritage AR object based on at least one of the distance between the coordinates of the terminal and the coordinates of the first relic site location information and weather information acquired based on the sensor system of the terminal. GPS-based augmented reality display method for lost cultural assets.

10. In paragraph 1, The step of displaying the cultural property AR object matched to the above-mentioned acquired historic site location information is as follows. A step of tracking at least one of the distance between the first GPS and the second GPS, the surrounding environment image acquired based on the camera of the terminal, and the detailed information acquired based on the position sensor (IMU) of the terminal, Further comprising a step of updating the coordinates and detailed information of the terminal according to the results of the above tracking. GPS-based augmented reality display method for lost cultural assets.

11. At least one memory; comprising at least one processor; At least one application stored in the above memory and executed by the above processor to provide an augmented reality service, wherein the at least one application is: Acquire the first terminal location information based on the location of the first GPS built into the terminal, Acquire second terminal location information based on the location of the second GPS mounted on the above terminal, Calibrate the first terminal location information and the second terminal location information obtained above, The coordinates of the terminal are determined based on the results of the above calibration, Obtain at least one relic site location information based on the above-determined coordinates, Displaying the AR object of cultural heritage matching the above acquired location information GPS-based augmented reality display system for lost cultural assets.

Images