Method and device for providing three-dimensional digital twin content in which two-dimensional image associated with specific location is placed at corresponding location in three-dimensional digital twin content

Abstract

Provided is a method for providing three-dimensional digital twin content corresponding to a real-world space. The method for providing three-dimensional digital twin content comprises: on the basis of visual localization for three-dimensional digital twin content using a first image indicating a first location in a real-world space, determining a second location in the three-dimensional digital twin content corresponding to the first image; or identifying a predetermined second location and associating partial content of the three-dimensional digital twin content displayed at the second location with the first image. The partial content and the first image are displayed in association with each other on a user terminal.

Description

Method and device for providing three-dimensional digital twin content in which a two-dimensional image associated with a specific location is placed at a corresponding location of the three-dimensional digital twin content

[0001] The present invention relates to a method and device for providing three-dimensional digital twin content corresponding to a real-world space, and to a method and device for providing three-dimensional digital twin content in which a two-dimensional image associated with a specific location is placed at a corresponding location of the three-dimensional digital twin content.

[0002] Digital twins are a technology that enables various simulations by creating a virtual space identical (or similar) to a real-world space. For example, by modeling a virtual space in three dimensions corresponding to a real-world space, a three-dimensional digital twin of the real-world space can be created.

[0003] 3D digital twin content can be useful for visualizing information about real-world spaces, exploring information existing in them, and monitoring them, by expressing three-dimensional information about real-world spaces that are difficult to accurately represent in two dimensions. This allows 3D digital twin content to provide consumers with an immersive experience of real-world spaces.

[0004] To increase the usability of 3D digital twin content and to facilitate comparison between 3D digital twin content and real-world spaces, separate 2D images capturing the location or place represented by the 3D digital twin content are sometimes used.

[0005] However, when such a two-dimensional image is displayed together with three-dimensional digital twin content, the two-dimensional image displayed may not properly represent the part of the three-dimensional digital twin content at the corresponding location, or it may be difficult for a user consuming the three-dimensional digital twin content to know which part of the three-dimensional digital twin content the two-dimensional image is related to.

[0006] Korean Patent No. 10-1989982 (registration date: June 11, 2019) discloses a modeling system using digital twin-based indoor space analysis that can predict changes in the floating population and number of people indoors according to indoor structure and environmental conditions through digital twin-based simulation analysis, thereby planning indoor structures and managing environmental conditions in advance.

[0007] The information described above is for the purpose of understanding only and may contain matters that do not form part of the prior art and may not contain what the prior art would suggest to a person skilled in the art.

[0008] Based on visual localization, a method can be provided for determining a second location corresponding to a first image in three-dimensional digital twin content using a first image representing a first location in a real-world space, and displaying the first image and a partial content of the three-dimensional digital twin content displayed at the second location by relating them to each other.

[0009] A method can be provided to automatically place each of a large number of two-dimensional images in an appropriate location within the three-dimensional digital twin content by receiving a plurality of two-dimensional images captured at a plurality of first locations in a real-world space, determining second locations within three-dimensional digital twin content corresponding to the plurality of two-dimensional images based on visual localization, and defining each of the second locations as a location within the three-dimensional digital twin content for displaying the two-dimensional images captured at the corresponding respective first locations.

[0010] When a first location in a real-world space is displayed within an AR (augmented reality) view of a user terminal, a method may be provided to display a first image representing the first location within the AR view.

[0011] According to one aspect, a method for providing three-dimensional digital twin content corresponding to a real-world space performed by a computer system is provided, the method comprising: determining a second location within the three-dimensional digital twin content corresponding to the first image based on visual localization of the three-dimensional digital twin content using a first image representing a first location in the real-world space, or identifying the second location, which is predetermined to correspond to the first image, based on the visual localization; and rendering the three-dimensional digital twin content for display on a user terminal, wherein the user terminal displays the first image in association with a first partial content of the three-dimensional digital twin content displayed at the second location or the second location corresponding to the first image.

[0012] In the user terminal, the first image may be displayed together with the first partial content so that the first partial content and the first image can be visually compared with each other.

[0013] The displaying step may include a step of rendering the 3D digital twin content to display the 3D digital twin content based on the second location at the user terminal; and a step of displaying the first image in association with the first partial content of the 3D digital twin content rendered based on the second location at the user terminal.

[0014] The rendering step may render the 3D digital twin content so that the second location or the first partial content is displayed from the front by changing the viewpoint of the 3D digital twin content to be displayed on the user terminal.

[0015] The rendering step may render the 3D digital twin content to display the first partial content at a predetermined size by performing at least one of zooming in on the 3D digital twin content to be displayed on the user terminal and zooming out on the 3D digital twin content to be displayed on the user terminal.

[0016] The above first image may be an image with a relatively higher resolution than the image included in the first partial content.

[0017] The first location may be a POI defined on a two-dimensional or three-dimensional map of the real-world space.

[0018] The second position may be predetermined by comparing displacement information associated with the first image with displacement information defined in the three-dimensional digital twin content.

[0019] The second position may be determined in advance by a step of acquiring a plurality of two-dimensional images from the three-dimensional digital twin content by scanning the three-dimensional digital twin content or a plurality of two-dimensional images taken of the real-world space used to construct the three-dimensional digital twin content as second images; a step of determining a second image matching the first image by comparing each second image of the first image and the second images; and a step of determining the second position based on the determined second image.

[0020] The second position is determined based on displacement information associated with the determined second image, and the displaying step may display the first image on the three-dimensional digital twin content based on displacement information associated with the determined second image, thereby causing the user terminal to display the first image in association with the second position or the first partial content.

[0021] The above displacement information may include 6 degrees of freedom (6 DoF) information.

[0022] The above-described acquiring step may acquire, as the second images, images associated with a position within a predetermined radius from a position indicated by displacement information associated with the first image among the plurality of two-dimensional images.

[0023] The displacement information may include position information and rotation information of the camera that captured the determined second image as meta information of the determined second image.

[0024] The method for providing the above three-dimensional digital twin content may further include the step of displaying a user interface marking the second location on the user terminal; and the step of providing additional information about the second location to the user terminal when the user interface is selected by the user terminal.

[0025] In the area where the first image is displayed on the user terminal, text describing the first location or the second location may be displayed together with the first image.

[0026] The method for providing the above three-dimensional digital twin content further includes a step of receiving an interaction for searching or selecting the first location from the user terminal, and when the interaction is received, the determining step and the displaying step can be performed.

[0027] At least a part of the three-dimensional digital twin content is displayed within an augmented reality (AR) view including an image of the real-world space captured by a camera of the user terminal, and the method for providing the three-dimensional digital twin content may further include a step of displaying the first image in association with the second location within the AR view when the second location is included within the AR view.

[0028] The method for providing the three-dimensional digital twin content may further include: receiving a plurality of two-dimensional images representing a plurality of first locations in the real-world space; and determining second locations within the three-dimensional digital twin content corresponding to the plurality of two-dimensional images based on visual localization of the three-dimensional digital twin content using each of the two-dimensional images of the plurality of two-dimensional images; and defining each second location of the second locations as a location within the three-dimensional digital twin content for displaying each of the two-dimensional images representing the corresponding first location.

[0029] In another aspect, a computer system for providing three-dimensional digital twin content corresponding to a real-world space is provided, comprising at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor determines a second location within the three-dimensional digital twin content corresponding to the first image based on visual localization of the three-dimensional digital twin content using a first image representing a first location in the real-world space, or identifies the second location, which is predetermined to correspond to the first image, based on the visual localization, and renders the three-dimensional digital twin content for display on a user terminal, wherein the computer system displays the first image on the user terminal in association with a first partial content of the three-dimensional digital twin content displayed at the second location or the second location corresponding to the first image.

[0030] Based on visual localization, a second location corresponding to the first image in the three-dimensional digital twin content is determined using a first image representing a first location in the real-world space, and a partial content of the three-dimensional digital twin content displayed at the second location and the first image are displayed in relation to each other, thereby facilitating comparison between the first location in the real-world space and the second location of the three-dimensional digital twin content corresponding thereto.

[0031] By automatically associating a large number of input two-dimensional images captured at multiple first locations in real-world space with corresponding second locations within three-dimensional digital twin content, a method for providing three-dimensional digital twin content with high usability in location information provision services, etc. can be implemented.

[0032] In using dimensional digital twin content and the information included therein to implement an AR view of a user terminal (100) and AR content augmented therein, when a first location in a real-world space is displayed within the AR view, a first image representing the first location can be displayed within the AR view, thereby increasing the usability of the AR view.

[0033] FIG. 1 illustrates a method for providing three-dimensional digital twin content corresponding to a real-world space according to one embodiment.

[0034] FIG. 2 illustrates a computer system for generating and providing three-dimensional digital twin content according to one embodiment.

[0035] FIG. 3 is a flowchart illustrating a method for providing three-dimensional digital twin content corresponding to a real-world space according to one embodiment.

[0036] FIG. 4 is a flowchart illustrating a method for determining a second location corresponding to a first image in three-dimensional digital twin content using a first image representing a first location in real-world space, according to an example.

[0037] FIG. 5 is a flowchart illustrating a method for displaying a first image and a partial content of a three-dimensional digital twin content displayed at a second location corresponding to a first image in a real-world space in relation to each other, according to an example.

[0038] FIG. 6 is a flowchart illustrating a method for automatically associating multiple two-dimensional images with corresponding locations within three-dimensional digital twin content, according to an example.

[0039] FIG. 7 is a flowchart illustrating a method for providing three-dimensional digital twin content to be displayed through an AR view on a user terminal according to an example.

[0040] FIG. 8 illustrates a method in which a first image representing a first location in real-world space is provided in a manner comparable to partial content of a three-dimensional digital twin content displayed at a second location corresponding to the first image, according to an example.

[0041] FIG. 9 illustrates a method for determining a second position corresponding to a first image by comparing a first image(s) representing a first position in a real-world space with two-dimensional images generated from or used to generate three-dimensional digital twin content, and placing the first image(s) at the second position, according to an example.

[0042] FIG. 10 illustrates a method for displaying a first image representing a first location through an AR view including the first location, according to an example.

[0043] FIG. 11 illustrates a scenario of a method for determining a first image representing a first location in a real-world space, determining a second location corresponding to the first image, and displaying the first image and partial content of a three-dimensional digital twin content displayed at the second location by relating them to each other, according to an example.

[0044] Hereinafter, examples are described in detail with reference to the attached drawings.

[0045]

[0046] FIG. 1 illustrates a method for providing three-dimensional digital twin content corresponding to a real-world space according to one embodiment.

[0047] In Fig. 1, a method is illustrated in which three-dimensional digital twin content (20) corresponding to real-world space is provided through a screen of a user terminal (100) by a computer system (200).

[0048] The computer system (200) may be an electronic device that generates three-dimensional digital twin content (20) and provides it to a user terminal (100). The computer system (200) may be a device that renders the three-dimensional digital twin content (20) so that it can be displayed on the user terminal (100). For example, the computer system (200) may be a server that generates three-dimensional digital twin content (20) and transmits it to the user terminal (100).

[0049] In FIG. 1, the screen of a user terminal (100) is indicated as a user terminal (100) for convenience of explanation. The user terminal (100) may be an electronic device such as a smartphone, a personal computer (PC), a laptop computer, a tablet, an Internet of Things (IoT) device, or a wearable computer.

[0050] The 3D digital twin content (20) is constructed by modeling a real-world space in three dimensions, and may be generated based on images acquired by photographing real-world spaces using a camera (e.g., a camera-equipped drone, autonomous vehicle, or unmanned aerial vehicle). For example, the images acquired by photographing real-world spaces may be aligned to generate a 3D model (or 3D map) corresponding to the real-world space, and the 3D digital twin content (20) may be generated based on this 3D model. Accordingly, the 3D digital twin content (20) is not limited to simply diagramming a real-world space using shapes, etc., but may also include details of features (objects) included in the real-world space.

[0051] The 3D digital twin content (20) can simulate a real-world space, such as a 2D map or a 3D map, or a semantically distinct space. For example, in the illustrated embodiment, the digital twin content (20) can correspond to a specific apartment complex (i.e., Apartment H (complex)). Accordingly, the 3D digital twin content (20) can include multiple buildings (apartment buildings, commercial buildings, other facilities, etc.) included in Apartment H.

[0052] The illustrated 3D digital twin content (20) can be displayed on a user terminal (100) to provide a service that provides information (e.g., a real estate information service, a location service, a route guidance service, an AR (Augmented Reality) service, etc.) to the user terminal (100).

[0053] In an embodiment, to increase the usability of the three-dimensional digital twin content (20) and to facilitate comparison between the three-dimensional digital twin content (20) and real-world space, a separate two-dimensional image (32) capturing a location or place indicated by the three-dimensional digital twin content (20) may be further used. In the present disclosure, such a two-dimensional image (32) may be referred to as a first image (32).

[0054] The first image (32) may be a two-dimensional image representing a first location within real-world space. The first location may represent, for example, a specific coordinate, a coordinate range, a POI, a location, or at least a portion of a location or facility within real-world space.

[0055] The computer system (200) can determine a second location within the three-dimensional digital twin content (20) that corresponds to the first image (32) (or corresponds to the first location) based on visual localization of the three-dimensional digital twin content (20) using the first image (32). Alternatively, the computer system (200) can identify a second location that is predetermined to correspond to the first image (32) (or corresponds to the first location) based on the visual localization.

[0056] The computer system (200) can render the three-dimensional digital twin content (20) so that the user terminal (100) displays the first image (32) in association with the second location or in association with the first partial content (22) of the three-dimensional digital twin content (20) (displayed at the second location) corresponding to the first image (32). For example, the first image (32) can be displayed around the second location or the first partial content (22) as illustrated. The content display area (30), which is the area where the first image (32) is displayed, can be a part of the three-dimensional digital twin content (20). The content display area (30) can further include text (34) for describing the first location (or the second location). For example, as illustrated, the first location may represent the main gate of the H apartment complex, and in the content display area (30) displaying the first image (32) on the user terminal (100), text describing the first location or the second location, “H apartment main gate,” may be displayed together with the first image (32).

[0057] The computer system (200) can provide three-dimensional digital twin content (20) corresponding to real-world space to the user terminal (100) based on an interaction from the user terminal (100). In other words, the computer system (200) can render the three-dimensional digital twin content (20) so that it can be displayed on the user terminal (100).

[0058] Meanwhile, the rendering of the three-dimensional digital twin content (20) by the computer system (200) in the present disclosure may encompass a case where at least a part of the rendering process is performed at the user terminal (100), and thus may encompass providing the user terminal (100) with data and information required for performing such rendering process at the user terminal (100).

[0059] The first image (32) indicating the first location may be, for example, an image that is determined by searching for the first location on the user terminal (100) and displayed on the user terminal (100). That is to say, in this embodiment, when a search for the first location is requested on the user terminal (100), the first image (32) indicating the first location may be determined, and a second location corresponding to the first image (32) may be determined. In addition, the user terminal (100) may display the three-dimensional digital twin content (20), and the first image (32) may be displayed together with the second location of the three-dimensional digital twin content (20) or the first partial content (22) of the second location. Accordingly, a user who searches for a first location using a user terminal (100) can easily compare a first image (32) representing the actual first location with a first partial content (22) corresponding thereto, and can easily identify where in the extensive three-dimensional digital twin content (20) is the second location corresponding to the first image (32). Accordingly, the effort required for the user to directly search the extensive three-dimensional digital twin content (20) through the user terminal (100) to identify the second location is reduced. In addition, this first image (32) can complement a three-dimensional digital twin content (20) in which a relatively low-height area is expressed in a coarse manner (i.e., the resolution is low in a low-altitude area / part).

[0060] A specific method for determining / identifying a second location on a three-dimensional digital twin content (20) based on visual localization using a first image (32) indicating a first location, and a specific method for displaying a first image (32) in association with a first partial content (22) on a user terminal (100) are described in more detail with reference to FIGS. 2 to 11, which will be described later.

[0061]

[0062] FIG. 2 illustrates a computer system for generating and providing three-dimensional digital twin content according to one embodiment.

[0063] The computer system (200) (or server) described above with reference to FIG. 1 according to the embodiments may be implemented through the computer system (200) illustrated in FIG. 2. For example, a computer program for implementing the method of the embodiment may be installed and run on the computer system (200), and such computer system (200) may perform the method for providing 3D digital twin content (20) according to the embodiments under the control of the run computer program.

[0064] The computer system (200) may be a computer that constitutes a server as the aforementioned electronic device. Alternatively, the computer system (200) may include a server and an administrator terminal for controlling the server. Such an administrator terminal may be a smartphone, a personal computer (PC), a laptop computer, a tablet, an Internet of Things (IoT) device, or a wearable computer. Such an administrator terminal may have installed thereon a program, application, or software including a tool for generating the aforementioned three-dimensional digital twin content (20).

[0065] As illustrated, the computer system (200) may include memory (110), a processor (120), a communication interface (130), and an input / output interface (140) as components.

[0066] The memory (110) is a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the permanent mass storage device such as the disk drive may be included in the computer system (200) as a separate permanent storage device distinct from the memory (110). In addition, the memory (110) may store an operating system and at least one program code. These software components may be loaded into the memory (110) from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (110) through a communication interface (130) rather than a computer-readable recording medium. For example, software components may be loaded into the memory (110) of a computer system (200) based on a computer program that is installed by files received over a network (160).

[0067] The processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (120) via the memory (110) or the communication interface (130). For example, the processor (120) may be configured to execute instructions received according to program code stored in a storage device such as the memory (110).

[0068] That is, the processor (120) can manage components of the computer system (200) and execute programs or applications used by the computer system (200). For example, the processor (120) can execute a program for performing a method of providing three-dimensional digital twin content (20) of the embodiment. In addition, the processor (120) can process operations necessary for executing related programs or applications and processing data, and can be at least one processor of the computer system (200) or at least one core within the processor.

[0069] The communication interface (130) may provide a function for the computer system (200) to communicate with another computer system (not shown), such as a user terminal (100) or an administrator terminal, via a network (160). For example, requests, commands, data, files, etc. generated by the processor (120) of the computer system (200) according to program codes stored in a recording device such as a memory (110) may be transmitted to another computer system via the network (160) under the control of the communication interface (130). Conversely, signals, commands, data, files, etc. from another computer system may be received by the computer system (200) via the communication interface (130) of the computer system (200) via the network (160). Signals, commands, data, etc. received through the communication interface (130) may be transmitted to the processor (120) or memory (110), and files, etc. may be stored in a storage medium (the aforementioned permanent storage device) that the computer system (200) may further include. For example, the communication interface (130) may be a hardware module such as a network interface card, a network interface chip, and a networking interface port of the computer system (200), or a software module such as a network device driver or a networking program.

[0070] The input / output interface (140) may be a means for interfacing with an input / output device (150). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (140) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (150) may also be configured as a single device with the computer system (200).

[0071] Furthermore, in other embodiments, the computer system (200) may include more components than those illustrated in FIG. 2. However, most conventional components need not be explicitly illustrated. For example, the computer system (200) may be implemented to include at least some of the input / output devices connected to the aforementioned input / output interface (140), or may further include other components such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, and the like.

[0072] When the computer system (200) is a server (200) that communicates with a user terminal (100), the computer system (200) can control the user terminal (100) to display three-dimensional digital twin content (20).

[0073] In some embodiments, the computer system (200) may include a database as a device for managing (registering, changing, deleting, storing, maintaining, etc.) data and content to be provided to the user terminal (100) and two-dimensional maps and three-dimensional maps, or may be configured to communicate with a database or external service that provides the same. In addition, the server (200) may include a map server that provides digital maps (three-dimensional maps and / or two-dimensional maps) or may be configured to communicate with such a map server.

[0074] Meanwhile, the user terminal (100) or administrator terminal described above can also be implemented as a computer system including configurations similar to those illustrated in FIG. 2, and thus, redundant descriptions are omitted.

[0075] In the detailed description to be provided below, for convenience of explanation, embodiments will be described with a focus on the computer system (200), and operations on the user terminal (100) side may be briefly described or omitted. In addition, for convenience of explanation, operations (steps) performed by components (e.g., processors, etc.) of the computer system (200) may be described as being performed by the computer system (200).

[0076] The description of the technical features described above with reference to FIG. 1 can also be applied to FIG. 2, so any redundant description will be omitted.

[0077]

[0078] FIG. 3 is a flowchart illustrating a method for providing three-dimensional digital twin content corresponding to a real-world space according to one embodiment.

[0079] Referring to FIG. 3, a method for providing three-dimensional digital twin content (20) corresponding to a real-world space performed by a computer system (200) is described in more detail.

[0080] In step (310), the computer system (200) determines a second location within the 3D digital twin content (20) corresponding to the first image (32) (corresponding to the first location) based on visual localization of the 3D digital twin content using the first image (32) representing a first location in the real-world space, or identifies a second location previously determined to correspond to the first image (32) (corresponding to the first location) based on the visual localization. The first location may represent, for example, a specific coordinate, a coordinate range, a POI, a location, or at least a part of a location or facility within the real-world space. The first location may be defined on a 2D map or 3D map for the real-world space (such as a POI). The first image (32) may be an image that captures the first location. Meanwhile, the first location may be a capture location of the first image (32) or a location estimated from the capture location. Alternatively, the first position may represent position information included as meta information in the first image (32).

[0081] Visual localization in the embodiment may encompass any technique for identifying a location associated with an image through image comparison. The first image (32) may correspond to an input image for visual localization.

[0082] The computer system (200) can determine a second location within the three-dimensional digital twin content (20) corresponding to the first image (32) (or corresponding to the first location). According to an embodiment, the computer system (200) can determine the second location within the three-dimensional digital twin content (20) corresponding to the first image (32) at a time when provision of a service is requested (e.g., at a time when provision of the three-dimensional digital twin content (20) is requested from the user terminal (100) or a search and selection of the first location are made), or can determine the second location corresponding to the first image (32) in advance before the time when provision of the service is requested, and can identify this predetermined second location at the time when provision of the service is requested.

[0083] In step (320), the computer system (200) can provide the first image (32) to the user terminal (100) in association with the three-dimensional digital twin content (20) and the first partial content (22) displayed at the determined second location or the second location corresponding to the first image (32).

[0084] In other words, the computer system (200) can render the three-dimensional digital twin content (20) to be displayed on the user terminal (100), and can display the first image (32) on the user terminal (100) in association with the determined second location or the first partial content (22) of the three-dimensional digital twin content (20). Accordingly, the first partial content (22) and the first image (32) can be displayed together.

[0085] In the user terminal (100), the first image (32) may be displayed together with the first partial content (22) so that the first partial content (22) and the first image (32) can be visually compared with each other. For example, as illustrated in FIG. 1, the first partial content (22) and the first image (32) may be displayed adjacently and displayed together on a single screen of the user terminal (100). Alternatively, the content display area (30) may be displayed in a form that indicates the second location or the first partial content (22).

[0086] In step (322), the computer system (200) can render the three-dimensional digital twin content (20) to display the three-dimensional digital twin content (20) based on the second location on the user terminal (100).

[0087] Accordingly, in step (324), the computer system (200) can display the first image (32) in association with the first partial content (22) (or second location) of the three-dimensional digital twin content (100) rendered based on the second location on the user terminal (100).

[0088] Displaying the three-dimensional digital twin content (20) based on the second position may be to change the viewpoint of displaying the three-dimensional digital twin content (20) on the user terminal (100). For example, the computer system (200) may render the three-dimensional digital twin content (20) so that the second position or the first partial content (22) is displayed on a specific area of ​​the user terminal (100), for example, the front or center area, by changing the viewpoint of the three-dimensional digital twin content (20) to be displayed on the user terminal (100). Accordingly, the first partial content (22) and thus the displayed first image (32) may be more easily recognized by the user.

[0089] In addition, the computer system (200) can adjust the display size of the three-dimensional digital twin content (20) so that the content (22) and thus the first image (32) displayed can be better recognized by the user. For example, the computer system (200) can render the three-dimensional digital twin content (22) to display the first partial content (22) at a predetermined size on the screen of the user terminal (100) by performing at least one of zooming in on the three-dimensional digital twin content (20) to be displayed on the user terminal (100) and zooming out on the three-dimensional digital twin content (20) to be displayed on the user terminal (100). The predetermined size may be a predetermined size range, and may indicate a ratio or range of ratios occupied by the partial content (22) on the screen of the user terminal (100). The predetermined size may be a size of the partial content (22) that is suitable for recognition by the user.

[0090] In this way, the partial content (22) and thus the first image (32) displayed can be displayed in a desired size in a desired area of ​​the screen of the user terminal (100), and the user can better identify the partial content (22) and the first image (32).

[0091] As mentioned above, the change of viewpoint and zoom-in or zoom-out of the digital twin content (20) can be automatically performed by the computer system (200) or the user terminal (100). That is, in the embodiment, when the provision of a service is requested (for example, when the provision of the three-dimensional digital twin content (20) is requested from the user terminal (100) or when a search and selection of the first location are performed), the viewpoint and size of the digital twin content (20) can be automatically and appropriately adjusted so that the first partial content (22) and the first image (32) can be displayed in a manner suitable for the user to recognize.

[0092] Meanwhile, the first image (32) may be a relatively higher resolution image than the image included in the first partial content (22). That is, the first image (32) may be a relatively higher resolution image than the image included in the digital twin content (20). Accordingly, this high-resolution first image (32) may be used to provide a user viewing the partial content (22) with additional information (i.e., details) about the second location indicated by the first partial content (22).

[0093] In relation to this, FIG. 8 illustrates a method in which a first image representing a first location in real-world space is provided in a manner comparable to partial content of a three-dimensional digital twin content displayed at a second location corresponding to the first image (32), according to an example.

[0094] As illustrated, both the first partial content (22) and the first image (32) represent the main gate of Apartment H, and thus, the first partial content (22) may correspond to the first image (32). The second location represented by the first partial content (22) may correspond to the first location represented by the first image (32). Since the illustrated first image (32) is a high-resolution image compared to the image included in the first partial content (22), it may include additional details regarding the main gate of Apartment H, which is the second location represented by the first partial content (22).

[0095] The display of the first partial content (22) and the first image (32) on the user terminal (100) by steps (310 to 324) can be performed in response to a request for provision of a service from the user terminal (100).

[0096] In relation to this, at step (305), the computer system (200) may receive an interaction for searching or selecting a first location from the user terminal (100). This interaction may be, for example, selecting a first location to target in a 3D map or 3D digital twin content (20) already displayed on the user terminal (100), or entering the first location in a search word for providing the 3D digital twin content (20).

[0097] The computer system (200) may perform the above-described steps (310 to 324) as the above interaction is received.

[0098] In relation to this, FIG. 11 illustrates a scenario of a method for determining a first image representing a first location in a real-world space according to an example, determining a second location corresponding to the first image, and displaying the first image and partial content of a three-dimensional digital twin content displayed at the second location in association with each other.

[0099] As in the scenario illustrated in FIG. 11, when an interaction is received from the user terminal (100), the computer system (200) can search for a first location (1110), determine a first image (32) related to the first location (1120), and determine a second location corresponding to the first image (32) (1130). Next, the computer system (200) can render the three-dimensional digital twin content (20) by appropriately adjusting the time point and / or size at which the three-dimensional digital twin content (20) is displayed on the user terminal (100) (1140), and thus display the first partial content (22) of the three-dimensional digital twin content (20) and the first image associated therewith on the user terminal (100) in an area appropriate for the user to view and at an appropriate size.

[0100] The description of the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIGS. 3, 8, and 11, so redundant descriptions are omitted.

[0101]

[0102] FIG. 4 is a flowchart illustrating a method for determining a second location corresponding to a first image in three-dimensional digital twin content using a first image representing a first location in real-world space, according to an example.

[0103] Referring to FIG. 4, a method for determining a second location corresponding to the first image (32) indicated by the first image through the aforementioned visual localization is described in more detail.

[0104] Steps (410 to 430) to be described below may be steps for determining a second location corresponding to the first image (32) in advance before the time at which provision of the aforementioned service is requested. In other words, the second location corresponding to the first image (32) may be determined in advance by steps (410 to 430) to be described below.

[0105] In step (410), the computer system (200) may acquire a plurality of two-dimensional images as second images from the three-dimensional digital twin content (20) by scanning the implemented three-dimensional digital twin content (20). Here, each of the second images may be an image extracted from the three-dimensional digital twin content (20). Alternatively, the computer system (200) may acquire a plurality of two-dimensional images photographing a real-world space used to construct the three-dimensional digital twin content (20) as second images. Here, each of the second images may be an image captured by a camera (e.g., a camera mounted on a drone, an autonomous vehicle, or an unmanned aerial vehicle) during a mapping process for constructing the three-dimensional digital twin content (20). Each of the second images may include displacement information. The displacement information associated with the second image may include position information and / or rotation information associated with the second image. Such displacement information may be information included in the meta information of the second image. For example, displacement information may include position information (coordinates) and / or rotation information of the camera that captured the second image, which is degree of freedom information of the camera that captured the second image. This displacement information may include 6 degrees of freedom (6 DoF) information associated with the second image.

[0106] In step (420), the computer system (200) can determine a second image matching the first image (32) by comparing the first image (32) with each second image of the second images. For example, the computer system (200) can determine the second image matching the first image (32) by analyzing the similarity between the first image (32) and each second image. In other words, the computer system (200) can determine the second image that is most similar to the first image (32). The determined second image may include an image of the first partial content (22) described above, may be a part of an image of the first partial content (22), or may be an image that is most similar to the image of the first partial content (22) among the second images.

[0107] At step (430), the computer system (200) may determine a second location based on the determined second image. The second location may be determined based on the displacement information associated with the determined second image. For example, the second location may be a location captured based on the shooting position and angle indicated by the displacement information. Alternatively, depending on the embodiment, the second location may be a location indicated by the displacement information.

[0108] Meanwhile, in the steps (320 to 324) described above with reference to FIG. 3, the computer system (200) can display the first image (32) on the user terminal (100) by arranging the first image (32) on the three-dimensional digital twin content (20) based on the displacement information associated with the second image determined in step (430), thereby associating the first image (32) with the second location or the first partial content (22). In other words, the first image (32) can be arranged on the three-dimensional digital twin content (20) based on the 6 DoF information indicated by the displacement information of the determined second image, and thus can be arranged in association with the first partial content (22) or the second location.

[0109] As described above, the displacement information associated with the determined second image may include 6 Degree of Freedom (6 DoF) information, or may include position information and rotation information of the camera that captured the determined second image as meta information.

[0110] In this way, in the embodiment, the first image (32) may be placed at a determined second position, but may be placed on the 3D digital twin content (20) at a second position that reflects the 6 DoF information, which is displacement information of the matching second image of the 3D digital twin content (20), as described.

[0111] Additionally, in the aforementioned step (410), if the first image (32) includes displacement information, the range of the acquired second images can be reduced. The description of the displacement information of the second image described above can be similarly applied to the displacement information of the first image (32).

[0112] The computer system (200) can acquire, as second images, images associated with positions within a predetermined radius from the position indicated by the displacement information associated with the first image (32) among the plurality of two-dimensional images in step (410). Accordingly, the range of comparison with the first image (32) can be limited to these second images. The position indicated by the displacement information may be the shooting position (coordinates) of the first image (32). Thus, in the embodiment, the collection range of the second images can be limited to the position range associated with the first image (32), and thus, the determination of the second image and the determination of the second position can be made more efficient, and the reliability of the determination of the second image and the second position can also be increased.

[0113] Meanwhile, as another embodiment, the second position corresponding to the first image (32) may be determined in advance by comparing displacement information associated with the first image (32) with displacement information defined in the three-dimensional digital twin content (20). That is, the second position of the three-dimensional digital twin content (20) corresponding to the first image (32) of the first image (320) may be determined by comparing the coordinates of the first image (320) with the coordinates of the three-dimensional digital twin content (20) (or a three-dimensional map matching the three-dimensional digital twin content (20)).

[0114] The description of the technical features described above with reference to FIGS. 1 to 3, 8 and 11 can also be applied to FIG. 4, so redundant descriptions are omitted.

[0115]

[0116] FIG. 5 is a flowchart illustrating a method for displaying a first image and a partial content of a three-dimensional digital twin content displayed at a second location corresponding to a first image in a real-world space in relation to each other, according to an example.

[0117] In step (510), the computer system (200) can further display a user interface for marking the aforementioned determined second location on the user terminal (100). The first image (32) can be positioned using a separate user interface for marking the second location. The separate user interface for marking the second location can be the aforementioned content display area (30). 'Marking' can mean that the user interface (content display area (30)) is displayed by directly overlapping the second location or that the user interface (content display area (30)) is displayed to indicate the second location.

[0118] In step (520), the computer system (200) may provide additional information about the second location to the user terminal (100) as the user interface for marking the second location is selected by the user terminal (100). For example, a content display area (30) in which a first image (32) is displayed may be provided to the user terminal (100) so that the user can select it. When the content display area (30) or the first image (32) is selected by the user terminal (100), additional information about the second location may be displayed on the user terminal (100). For example, additional information (text, images, other media content, etc.) about the second location may be displayed on the user terminal (100) through the content display area (30), a user interface in which the content display area (30) is further enlarged, or a separate user interface.

[0119] Accordingly, in the embodiment, the linkage between the first partial content (22) and the first image (32) of the second location can be strengthened through the user interface in step (510), and additional information about the second location can also be effectively provided to the user terminal (100).

[0120] The description of the technical features described above with reference to FIGS. 1 to 4, 8 and 11 can also be applied to FIG. 5, so redundant descriptions are omitted.

[0121]

[0122] FIG. 6 is a flowchart illustrating a method for automatically associating multiple two-dimensional images with corresponding locations within three-dimensional digital twin content, according to an example.

[0123] Referring to FIG. 6, a method is described for determining in advance (i.e., before a service provision is requested) second locations within three-dimensional digital twin content (20) corresponding to a plurality of first images, including the first image (32), based on the aforementioned visual localization. In this case, each first image may correspond to a two-dimensional image as an input image for visual localization.

[0124] In step (610), the computer system (200) may receive a plurality of two-dimensional images representing a plurality of first locations in real-world space. That is, the computer system (200) may receive a plurality of two-dimensional images as a plurality of input images for visual localization. The plurality of input images may be registration target images to be displayed in association with the three-dimensional digital twin content (20) (such as the first image (32) described above).

[0125] In step (610), the computer system (200) can determine second positions within the three-dimensional digital twin content (20) corresponding to the plurality of two-dimensional images based on visual localization for the three-dimensional digital twin content using each two-dimensional image of the plurality of two-dimensional images. As for the method for determining the second position corresponding to the first image (32) of each two-dimensional image, the method for determining the second position described above with reference to FIGS. 1 to 5 can be similarly applied, and therefore, a duplicate description thereof will be omitted.

[0126] In step (630), the computer system (200) can define (or determine) each of the determined second locations as a location within the three-dimensional digital twin content for displaying each of the two-dimensional images representing the corresponding first locations. In other words, the computer system (200) can map each of the second locations to a corresponding two-dimensional image, so that when a specific first location is searched for and a first image of the corresponding first location is determined (retrieved), and when the three-dimensional digital twin content (20) is displayed, the computer system can identify the second location determined in advance in response to the first image (32) and display the first image determined (retrieved) in association with the second location (or the first partial content displayed there).

[0127] In this way, in the embodiment, a large number of input images (first images) can be registered to be output in association with three-dimensional digital content, and each of the registered input images can be provided to the user terminal (100) together with three-dimensional digital twin content (20), such as the first image (32) described above.

[0128] In relation to this, FIG. 9 illustrates a method for determining a second position corresponding to the first image by comparing the first image(s) representing a first position in real-world space with two-dimensional images generated from or used to generate three-dimensional digital twin content, and placing the first image(s) at the second position, according to an example.

[0129] In FIG. 9, three-dimensional digital twin content (910) and input image(s) (920) corresponding to the two-dimensional images described above are illustrated, and each input image (920) (i.e., the first image) is compared with two-dimensional images (i.e., the second images) (932, 934) associated with the three-dimensional digital twin content (910) according to visual localization, and a method of placing them on the three-dimensional digital twin content (910) is illustrated.

[0130] First, among the images (932, 934) associated with the 3D digital twin content (910), the second images (932) may be images captured by a camera during a mapping process for constructing the 3D digital twin content (910) as described above, and the second images (934) may be images extracted from the 3D digital twin content (910) through scanning as described above. Each input image (920) may be compared with each of these images (932 or 934), a matching image may be determined among the images (932 or 934), and the determined image may be utilized to place the input image (920) in the 3D digital twin content (910). The determined image may have displacement information such as 6DoF information, and the input image (920) may be placed in the 3D digital twin content (910) according to the displacement information of the determined image. Accordingly, registration (placement) of the three-dimensional digital twin content (910) of the input image (920) can be completed. The above-described process can be automatically performed upon receipt of the input image.

[0131] The description of the technical features described above with reference to FIGS. 1 to 5, 8 and 11 can also be applied to FIGS. 6 and 9, so redundant descriptions are omitted.

[0132]

[0133] FIG. 7 is a flowchart illustrating a method for providing three-dimensional digital twin content to be displayed through an AR view on a user terminal according to an example.

[0134] The three-dimensional digital twin content (20) of the embodiment and the information contained therein may also be provided through an augmented reality (AR) view of the user terminal (100). In other words, the three-dimensional digital twin content (20) of the embodiment and the information contained therein may be used to implement the AR view of the user terminal (100) and AR content augmented therein. The first image (32) as illustrated in FIGS. 1 and 8 may be displayed not only in the three-dimensional digital twin content (20) but also in the AR view.

[0135] In relation to this, FIG. 10 illustrates a method for displaying a first image representing a first location through an AR view including the first location, according to an example.

[0136] For example, in step (710), the computer system (200) can display at least a portion of the three-dimensional digital twin content (20) by augmenting the image within an AR view (1000) that includes an image of a real-world space captured by a camera of the user terminal (100). In other words, at least a portion of the three-dimensional digital twin content (20) can be displayed within the AR view (1000) of the user terminal (100).

[0137] At step (720), the computer system (200) may display a first image (1014) in association with the second location within the AR view (1000), when the second location is included within the AR view (1000). For example, as illustrated in FIG. 10 , the first image (1014) may be displayed within the content display area (1010) along with text (1012) describing the second location.

[0138] Meanwhile, the computer system (200) may display the first image (1014) in association with the second location within the AR view (1000) even if the three-dimensional digital twin content (20) is not directly displayed within the AR view (1000), if the second location is included within the AR view (1000). In other words, an image corresponding to a real-world space may be displayed instead of the three-dimensional digital twin content (20) within the AR view (1000), but the first image (1014) may be displayed in association with an area corresponding to the second location of the image.

[0139] Whether or not to display 3D digital twin content (20) within the AR view (1000) can be selectively determined according to the user's request.

[0140] In this way, the first image (32, 1014) of the embodiment may constitute at least a portion of the AR content displayed through the AR view (1000). In this respect, for example, the registration (placement) process for the three-dimensional digital twin content (910) of the input image (920) described above with reference to FIG. 6 may correspond to the authoring process of the AR content (or at least a portion of this process).

[0141] Accordingly, in the embodiment, not only can a three-dimensional digital twin content (20) in which the first image (32) is displayed in association be easily constructed, but an AR view (1000) in which the first image (32) is displayed in association can also be easily constructed.

[0142] The description of the technical features described above with reference to FIGS. 1 to 6, FIG. 8, FIG. 9 and FIG. 11 can also be applied to FIGS. 7 and 10, so redundant descriptions are omitted.

[0143] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0144] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0145] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0146] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0147] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for providing three-dimensional digital twin content corresponding to a real-world space performed by a computer system, A step of determining a second location within the three-dimensional digital twin content corresponding to the first image based on visual localization of the three-dimensional digital twin content using a first image representing a first location in the real-world space, or identifying the second location previously determined to correspond to the first image based on the visual localization; and A step of rendering the above 3D digital twin content to be displayed on a user terminal, wherein, on the user terminal, the first image is displayed in association with the first partial content of the 3D digital twin content displayed at the second location or the second location corresponding to the first image. A method for providing three-dimensional digital twin content, comprising:

2. In paragraph 1, A method for providing three-dimensional digital twin content, wherein the first image is displayed together with the first partial content so that the first image and the first partial content can be visually compared with each other in the user terminal.

3. In paragraph 1, The steps indicated above are: In the user terminal, a step of rendering the 3D digital twin content to display the 3D digital twin content based on the second location; and A step of displaying the first image in association with the first partial content of the three-dimensional digital twin content rendered based on the second location in the user terminal A method for providing three-dimensional digital twin content, comprising:

4. In paragraph 3, The above rendering steps are: A method for providing three-dimensional digital twin content, wherein the three-dimensional digital twin content is rendered so as to display the second location or the first partial content from the front by changing the viewpoint of the three-dimensional digital twin content to be displayed on the user terminal.

5. In paragraph 3, The above rendering steps are: A method for providing three-dimensional digital twin content, wherein the three-dimensional digital twin content is rendered to display the first partial content at a predetermined size by performing at least one of zooming in on the three-dimensional digital twin content to be displayed on the user terminal and zooming out on the three-dimensional digital twin content to be displayed on the user terminal.

6. In paragraph 1, A method for providing three-dimensional digital twin content, wherein the first image is an image having a relatively higher resolution than the image included in the first partial content.

7. In paragraph 1, A method for providing three-dimensional digital twin content, wherein the first location is a POI defined on a two-dimensional or three-dimensional map of the real-world space.

8. In paragraph 1, A method for providing three-dimensional digital twin content, wherein the second position is determined in advance by comparing displacement information associated with the first image and displacement information defined in the three-dimensional digital twin content.

9. In paragraph 1, The second position above is, A step of obtaining a plurality of two-dimensional images from the three-dimensional digital twin content by scanning the three-dimensional digital twin content or a plurality of two-dimensional images taken of the real-world space used to construct the three-dimensional digital twin content as second images; A step of determining a second image matching the first image by comparing the first image with each second image of the second images; and A step of determining the second location based on the second image determined above A method for providing three-dimensional digital twin content, which is predetermined by .

10. In paragraph 9, The second position is determined based on displacement information associated with the determined second image, The steps indicated above are: A method for providing three-dimensional digital twin content, wherein the first image is positioned on the three-dimensional digital twin content based on displacement information associated with the determined second image, thereby displaying the first image in association with the second location or the first partial content at the user terminal.

11. In paragraph 10, A method for providing three-dimensional digital twin content, wherein the displacement information includes six degrees of freedom (6 DoF) information.

12. In paragraph 9, The above obtaining steps are: A method for providing three-dimensional digital twin content, wherein, among the plurality of two-dimensional images, images associated with a position within a predetermined radius from a position indicated by displacement information associated with the first image are acquired as the second images.

13. In paragraph 10, A method for providing three-dimensional digital twin content, wherein the displacement information includes position information and rotation information of a camera that captured the determined second image as meta information of the determined second image.

14. In paragraph 1, A step of further displaying a user interface marking the second location on the user terminal; and A step of providing additional information about the second location to the user terminal as the user interface is selected by the user terminal. A method for providing three-dimensional digital twin content, comprising:

15. In paragraph 1, A method for providing three-dimensional digital twin content, wherein text describing the first location or the second location is displayed together with the first image in an area where the first image is displayed on the user terminal.

16. In paragraph 1, A step of receiving an interaction for searching or selecting the first location from the user terminal. Including more, A method for providing three-dimensional digital twin content, wherein the determining step and the displaying step are performed when the above interaction is received.

17. In paragraph 1, At least a portion of the 3D digital twin content is displayed within an AR (augmented reality) view including an image of the real-world space captured by a camera of the user terminal, When the second location is included in the AR view, a step of displaying the first image in association with the second location in the AR view A method for providing three-dimensional digital twin content, further comprising:

18. In paragraph 1, A step of receiving a plurality of two-dimensional images representing a plurality of first locations in the real-world space; and A step of determining second locations within the three-dimensional digital twin content corresponding to the plurality of two-dimensional images based on visual localization of the three-dimensional digital twin content using each two-dimensional image of the plurality of two-dimensional images; and A step of defining each of the second positions as a position within the three-dimensional digital twin content for displaying each of the two-dimensional images representing the corresponding first positions. A method for providing three-dimensional digital twin content, further comprising:

19. A program stored in a computer-readable recording medium that causes the method of providing three-dimensional digital twin content of paragraph 1 to be executed on the computer system.

20. In a computer system that provides three-dimensional digital twin content corresponding to real-world space, At least one processor configured to execute computer-readable instructions contained in memory Including, At least one processor of the above, Determining a second location within the three-dimensional digital twin content corresponding to the first image based on visual localization of the three-dimensional digital twin content using a first image representing a first location in the real-world space, or identifying the second location previously determined to correspond to the first image based on the visual localization, A computer system for rendering the three-dimensional digital twin content to be displayed on a user terminal, wherein the first image is displayed on the user terminal in association with a first partial content of the three-dimensional digital twin content displayed at the second location or at the second location corresponding to the first image.

Images