Method, apparatus, system and computer program for scanning spatial image and providing virtual reality service

Abstract

The present invention relates to a method, apparatus, system, and computer program for scanning a spatial image and providing a virtual reality service and, more specifically, to a method, apparatus, system, and computer program for scanning a spatial image and providing a virtual reality service, which can scan an image for a given space and further provide a virtual reality service based thereon. The method performed in an apparatus according to the present invention comprises the steps of: transmitting, to a server, first image data corresponding to a first location in a given space, receiving map data for the space from the server, the map data being generated on the basis of an image data group including the first image data, and providing the received map data to a user; when a modification request from the user is received, transferring the modification request to the server and regenerating map data by reflecting the modification request; and transmitting, to the server, second image data corresponding to a second location different from the first location, receiving, from the server, map data generated on the basis of an image data group including the first image data and the second image data, updating the received map data, and providing the updated map data to the user.

Description

Method, device, system and computer program for providing spatial image scanning and virtual reality services

[0001] The present invention relates to a method, device, system and computer program for providing spatial image scanning and virtual reality services, and more particularly, to a method, device, system and computer program for providing spatial image scanning and virtual reality services capable of scanning an image of a given space and further providing a virtual reality service based thereon.

[0002] Recently, various services that provide virtual experiences of indoor and outdoor spaces, such as street view services and cyber model house services, have been widely utilized, and various virtual reality services, such as metaverse services that provide experiences in the virtual world, are rapidly spreading.

[0003] In relation to this, a method of configuring a virtual reality service model has been used in the past by shooting images, such as 360-degree images, from multiple locations in indoor and outdoor spaces, and then assigning the shooting locations of the images to specific locations, such as given drawings, through manual work by workers.

[0004] Accordingly, in the prior art, the method was limited to simply providing images, such as 360-degree images, previously shot at multiple locations in indoor and outdoor spaces at a location selected by the user, and therefore, it was difficult to improve or expand the service, such as deriving 3D modeling information about indoor and outdoor spaces based on the shot images or providing various additional functions based on the 3D modeling information.

[0005] In addition, in the conventional technology, after taking images at multiple locations in an indoor or outdoor space, a virtual reality service model for the indoor or outdoor space is created through post-processing. However, if a problem is identified in the location where the image was taken during the process of creating the virtual reality service model, a problem may arise in which the indoor or outdoor space must be revisited and the image must be taken again.

[0006] Accordingly, it is possible to provide various additional functions by generating 3D modeling information based on captured images, such as 360-degree images, for a given space, and furthermore, there is a need for a method to prevent the problem of having to revisit the space to re-shoot images during the post-processing process after capturing images, but an appropriate solution to this has not yet been implemented.

[0007] The present invention was created to solve the problems of the prior art as described above, and the purpose of the present invention is to provide a method, device, system and computer program for providing a spatial image scan and virtual reality service, which generates 3D modeling information based on images captured using various shooting devices such as a 360-degree camera or a smartphone for a given space, and further provides various additional functions based on the 3D modeling information.

[0008] In addition, the present invention aims to provide a method, device, system and computer program for providing a spatial image scan and virtual reality service that can prevent the problem of having to revisit the space for re-shooting of an image, etc. during the post-processing process after shooting an image of a given space.

[0009] Other detailed purposes of the present invention will be readily apparent and understandable to experts or researchers in this technical field through the specific contents described below.

[0010] According to one aspect of the present invention for solving the above problem, a method performed in a device may include: transmitting first image data corresponding to a first location in a given space to a server, receiving map data for the space generated based on an image data group including the first image data from the server and providing the map data to a user; when a modification request from the user is received, transmitting the modification request to the server to regenerate the map data reflecting the modification request; and transmitting second image data corresponding to a second location different from the first location to the server, receiving the map data generated based on the image data group including the first image data and the second image data from the server and updating and providing the map data to the user.

[0011] Here, in the step of regenerating the map data, when a deletion request for the first image data from the user is received, the regenerated 1-1 image data may be transmitted to the server to regenerate the map data based on the image data group in which the first image data is replaced with the 1-1 image data.

[0012] In addition, in the step of regenerating the map data, when a request for modification of the first location of the first image data of the user is received, the map data can be regenerated to reflect the modified first location.

[0013] Additionally, the device can receive the first image data captured by the 360-degree camera at the first location and transmit it to the server.

[0014] Additionally, the panoramic image data at the first location captured by the device can be transmitted to the server as the first image data.

[0015] Additionally, the map data may be provided with a recommended location for the second location.

[0016] At this time, in the step of updating and providing to the user, the map data can be generated by considering the alignment status of the second image data and the third image data corresponding to the third location adjacent to the second location in the image data group.

[0017] In addition, the server may further include a step of generating 3D modeling information for the space based on the image data group.

[0018] Here, in the step of generating the 3D modeling information, global optimization can be performed for each location corresponding to the image data included in the image data group.

[0019] Additionally, in the step of generating the 3D modeling information, texture data and mesh data for the space can be generated by reflecting the globally optimized location.

[0020] Additionally, 2D drawing information for the space can be generated based on the 3D modeling information.

[0021] Additionally, virtual reality data for providing virtual reality services for the space can be generated based on the above 3D modeling information.

[0022] Furthermore, the distance between two points selected in the virtual reality service for the space can be calculated and provided based on mesh data for the space.

[0023] In addition, according to another aspect of the present invention, a device including a processor; and a memory, wherein the memory includes instructions configured to cause the device to implement a specific operation when executed by the processor, the specific operation may include: transmitting first image data corresponding to a first location in a given space to a server, and receiving map data for the space generated based on an image data group including the first image data from the server and providing the map data to a user; when a modification request from the user is received, transmitting the modification request to the server to regenerate the map data reflecting the modification request; and transmitting second image data corresponding to a second location different from the first location to the server, and receiving the map data generated based on the image data group including the first image data and the second image data from the server and updating and providing the map data to the user.

[0024] In addition, according to another aspect of the present invention, a computer-readable storage medium storing instructions configured to cause a device including the processor to perform a specific operation when executed by a processor, the specific operation may include: transmitting first image data corresponding to a first location in a given space to a server, and receiving map data for the space generated based on an image data group including the first image data from the server and providing the map data to a user; when a modification request from the user is received, transmitting the modification request to the server to regenerate the map data reflecting the modification request; and transmitting second image data corresponding to a second location different from the first location to the server, and receiving the map data generated based on the image data group including the first image data and the second image data from the server and updating and providing the map data to the user.

[0025] Accordingly, the method, device, system and computer program for providing a spatial image scan and virtual reality service according to one embodiment of the present invention can generate 3D modeling information, etc. based on a captured image, such as a 360-degree image, for a given space, and further provide various additional functions based on the 3D modeling information.

[0026] In addition, in the method, device, system and computer program for providing a spatial image scan and virtual reality service according to one embodiment of the present invention, it is possible to prevent the problem of having to revisit the space for re-shooting of an image, etc. during the post-processing process after shooting an image of a given space.

[0027] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the contents described in this specification.

[0028] The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide examples of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0029] FIG. 1 is a diagram illustrating a configuration of a spatial image scanning system according to one embodiment of the present invention.

[0030] FIG. 2 is a drawing illustrating a flowchart of a spatial image scanning method according to one embodiment of the present invention.

[0031] FIG. 3 is a drawing illustrating a specific configuration and operation of a spatial image scanning system according to one embodiment of the present invention.

[0032] FIGS. 4 to 8 are drawings explaining specific processes in a spatial image scanning method according to one embodiment of the present invention.

[0033] FIGS. 9 to 12 are drawings illustrating specific examples of a spatial image scanning method according to one embodiment of the present invention.

[0034] Fig. 13 is a drawing illustrating a specific configuration of a device according to one embodiment of the present invention.

[0035] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. The purpose, specific advantages, and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments, which are linked to the attached drawings.

[0036] Prior to this, the terms and words used in this specification and claims should be interpreted as meanings and concepts that are consistent with the technical idea of ​​the present invention and are appropriately defined by the inventor to explain his own invention in the best possible way, and are only for the purpose of explaining embodiments and should not be construed as limiting the present invention.

[0037] When assigning reference numerals to components, identical or similar components will be assigned the same reference numerals, regardless of the reference numerals, and any duplicate descriptions thereof will be omitted. The suffixes "module" and "part" used in the following descriptions are assigned or used interchangeably for the convenience of writing specifications, and do not have distinct meanings or roles in themselves, and may refer to either software or hardware components.

[0038] When describing components of the present invention, if a component is expressed in the singular, it should be understood that the component also includes the plural, unless otherwise specified. Furthermore, terms such as "first," "second," etc. are used to distinguish one component from another, and the components are not limited by these terms. Furthermore, when a component is connected to another component, it means that another component may be connected between the component and the other component.

[0039] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included within the spirit and technical scope of the present invention.

[0040] Hereinafter, exemplary embodiments of a method, device, system and computer program for providing a spatial image scan and virtual reality service according to the present invention will be described in detail with reference to the attached drawings.

[0041] First, FIG. 1 illustrates the configuration of a spatial image scanning system (100) according to an embodiment of the present invention. As can be seen in FIG. 1, the spatial image scanning system (100) according to an embodiment of the present invention may be configured to include a device (110a, 110b) that collects or generates and transmits image data, such as a 360-degree image captured for a given space, and a server (120) that generates map data or 3D modeling information for the space based on the image data transmitted from the device (110a, 110b), or further generates virtual reality data for providing a virtual reality service for the space and provides the same to the device (110a, 110b).

[0042] In addition, the above device (110a, 110b) may be a device that is linked to a 360-degree camera (140) capable of shooting a 360-degree image and can receive a 360-degree image of the space and transmit it to the server (120), or may be a portable terminal device such as a smart phone that is equipped with a camera and can shoot a panoramic image of the space and transmit it to the server (120), and the above device (110a, 110b) may also be equipped with a display module or be linked to an external display device to provide a user with map data generated and transmitted by the server (120).

[0043] More specifically, terminal devices such as smartphones, tablets, and laptops can be used as the above devices (110a, 110b), but they can also be implemented in various forms, such as by being implemented as dedicated devices.

[0044] In addition, the server (120) may be implemented using one or more physical server devices, but the present invention is not necessarily limited thereto, and may be implemented in various forms, such as being implemented as a dedicated device or being implemented based on a cloud system.

[0045] Furthermore, as the communication network (130) connecting the devices (110a, 110b) and the server (120) in FIG. 1, a wired network and a wireless network can be used, and specifically, various communication networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN) can be included. In addition, the communication network (130) may also include the well-known World Wide Web (WWW).

[0046] In addition, FIG. 2 illustrates a flowchart for a spatial image scanning method according to one embodiment of the present invention.

[0047] The method illustrated in FIG. 2 can be performed, for example, by the devices (110a, 110b) illustrated in FIG. 1, and further, the devices (110a, 110b) can be implemented as a device (110) having a processor (10), as can be seen with reference to FIG. 13 and the description of FIG. 13 described below, wherein the processor (10) can execute instructions configured to implement a specific operation for scanning an image of a space.

[0048] At this time, as can be seen in FIG. 2, the spatial image scanning method according to one embodiment of the present invention is a method performed in the device (110), comprising: a step (S110) of transmitting first image data (150) corresponding to a first position of a given space to a server (120), and receiving map data (160) for the space generated based on an image data group including the first image data (150a) from the server (120) and providing it to the user; a step (S120) of transmitting, when a modification request from the user is received, the modification request to the server (120) to regenerate the map data (160) in which the modification request is reflected; and a step (S120) of transmitting second image data (150b) corresponding to a second position different from the first position to the server (120), and receiving, from the server (120), the map data (160) generated based on the image data group including the first image data (150a) and the second image data (150b) to the user. It may include a step (S130) of providing renewal.

[0049] Here, in the step (S120) of regenerating the map data (160), when a deletion request for the first image data (150a) from the user is received, the regenerated 1-1 image data (150a1) may be transmitted to the server (120) to regenerate the map data (160) based on the image data group in which the first image data (150a) is replaced with the 1-1 image data (150a1).

[0050] In addition, in the step (S120) of regenerating the map data (160), when a request for modification of the first location of the first image data (150a) from the user is received, the map data (160) can be regenerated to reflect the modified first location.

[0051] Additionally, the device (110) can receive the first image data (150a) captured by the 360-degree camera (140) at the first location and transmit it to the server (120).

[0052] Additionally, the panoramic image data at the first location captured by the device (110) can be transmitted to the server (120) as the first image data (150 a).

[0053] Additionally, the map data (160) may display and provide a recommended location for the second location.

[0054] At this time, in the step (S130) of updating and providing to the user, the map data (160) can be generated by considering the alignment status of the third image data (150c) corresponding to the third location adjacent to the second location in the image data group and the second image data (150b).

[0055] In addition, a step (not shown) of generating 3D modeling information for the space from the server (120) based on the image data group may be further included.

[0056] Here, in the step of generating the 3D modeling information (not shown), global optimization can be performed for each location corresponding to the image data (150) included in the image data group.

[0057] Additionally, in the step of generating the 3D modeling information (not shown), texture data and mesh data for the space can be generated by reflecting the globally optimized location.

[0058] Additionally, 2D drawing information for the space can be generated based on the 3D modeling information.

[0059] Additionally, virtual reality data for providing virtual reality services for the space can be generated based on the above 3D modeling information.

[0060] Furthermore, the distance between two points selected in the virtual reality service for the space can be calculated and provided based on mesh data for the space.

[0061] Accordingly, the method, device, system, and computer program for providing a spatial image scan and virtual reality service according to one embodiment of the present invention can generate 3D modeling information, etc. based on a captured image, such as a 360-degree image, for a given space, and can also provide various additional functions based on the 3D modeling information, and further, can effectively solve the problem of having to revisit the space for re-capturing the image, etc. during the post-processing process after capturing the image for a given space.

[0062] Hereinafter, a spatial image scanning method according to one embodiment of the present invention will be described in more detail with reference to FIGS. 1 and 2.

[0063] First, in the step S110, the first image data (150a) corresponding to the first location of the given space is transmitted to the server (120), and map data (160) for the space generated based on the image data group including the first image data (150a) is received from the server (120) and provided to the user.

[0064] Here, the space may be an indoor space including a room inside a building, but the present invention is not necessarily limited thereto, and may include various spaces such as a space where an indoor space is mixed with an outdoor space such as a veranda or balcony.

[0065] At this time, as can be seen in FIG. 3, the image data (150) may be a 360-degree image captured by a 360-degree camera (140), but the present invention is not necessarily limited thereto, and may also be an image captured by a general camera or an image captured by various types of cameras such as an RGBD camera, and further, may include various images captured at the first position in the space, such as a panoramic image captured by a smartphone or an image captured using a device (110) equipped with a camera module or the like.

[0066] Here, if the image data (150) is a 360-degree image or a panoramic image, virtual reality data for providing a virtual reality service can be more easily generated, but the present invention is not necessarily limited thereto, and in the present invention, a spatial image scanning method according to an embodiment of the present invention can be implemented and performed based on images of various formats, such as combining multiple images based on images of various formats or providing a virtual reality service under specific conditions or areas according to the characteristics of a given image.

[0067] For a more specific example, as can be seen in FIG. 3, the device (110) such as a smartphone can transmit a shooting request to a 360-degree camera (140) installed at a first location in the space (① in FIG. 3).

[0068] Accordingly, the 360-degree camera (140) can capture a 360-degree image at a first location in the space and transmit it to a device (110) such as a smartphone (② of FIG. 3).

[0069] At this time, the device (110) such as the smartphone transmits the 360-degree video data (150) to the server (120) through a communication network (130) (③ of FIG. 3).

[0070] In addition, the server (120) can generate map data (160) for the space in real time based on a group of image data including a series of image data (150) received and transmit it to the device (110) (④ of FIG. 3).

[0071] Here, the above map data (160) may be a 2D map, but the present invention is not limited thereto, and it is possible to implement and transmit it in various forms, such as a 3D map.

[0072] At this time, the image data group may include information about the series of image data (150) and each location corresponding to the series of image data (150).

[0073] Here, information on each location corresponding to the image data (150) can be derived through a motion-based structure derivation (Structure From Motion) technique, but the present invention is not necessarily limited thereto, and can be derived through various techniques such as the SLAM technique.

[0074] Accordingly, the device (110) such as the smartphone can provide the map data (160) received from the server (120) to the user.

[0075] Next, in the step S120, when a modification request from the user is received from the device (110) such as the smartphone, the modification request may be transmitted to the server (120) to regenerate the map data (160) in which the modification request is reflected.

[0076] More specifically, the user can transmit the first image data (150a) to the server (120) through the step S110, and receive map data (160) generated by reflecting the first image data (150a) and provide it to the user. In the step S120, the user can request deletion of the first image data (150a) and re-shooting or correction of the location of the first image data (150a) based on the map data (160).

[0077] Accordingly, in the step S120, when a deletion request for the first image data (150a) from the user is received, the regenerated 1-1 image data (150a1) may be transmitted to the server (120) so as to regenerate the map data (160) based on the image data group in which the first image data (150a) is replaced with the 1-1 image data (150a1).

[0078] In addition, in the step S120, when a request for modification of the first location of the first image data (150a) of the user is received, the map data (160) can be regenerated to reflect the modified first location.

[0079] For a more specific example, as can be seen in FIG. 4, when first image data (150a) captured at a first location (201) is transmitted from the device (110) to the server (120), the server (120) generates map data (160) based on the image data group including the first image data (150a) and transmits the map data to the device (110), and then when second image data (150b) captured at a second location (202) is transmitted from the device (110) to the server (120), the server (120) regenerates map data (160) based on the image data group including the first image data (150a) and the second image data (150b) and transmits the map data to the device (110) to provide it to the user (FIG. 4 (a)).

[0080] In addition, when the third image data (150c) captured at the third location (203) is transmitted from the device (110) to the server (120), the server (120) regenerates map data (160) based on the image data group including the first image data (150a), the second image data (150b) and the third image data (150c) and transmits the regenerated map data (160) to the device (110) to provide it to the user ((b) of FIG. 4), and then, when the respective image data for the fourth location (240) and the fifth location (205) are transmitted from the device (110) to the server (120), the updated map data (160) can be provided to the user ((c), (d) of FIG. 4).

[0081] Accordingly, as can be seen in Fig. 5, by sequentially shooting image data (150) for each room, living room, bathroom, hallway, etc. of the space, map data (160) for the space can be sequentially expanded.

[0082] However, if it is determined that there is an error in the map data (160) provided to the user, the user may request deletion of specific image data (150) of the map data (160) and re-shooting or location correction.

[0083] For a more specific example, when a user scans a spatial image in a given space, such as an apartment interior, as shown in (a) of FIG. 6, the user can receive and confirm map data (160) generated by sequentially shooting image data (150), such as a 360-degree image, from the first to the seventeenth positions (217) as shown in (b) of FIG. 6.

[0084] However, if it is determined that there is an error in the map data (160) generated by the image data (150) captured at a specific 18th location (218) as in (c) of FIG. 6 (for example, an error can be determined when a separate room ((A) of FIG. 6)) is generated in the map data (150) by the image data (150) captured at the 18th location (218), or when the shape of the room is generated incorrectly), the user can perform deletion and re-capture or location correction for the specific image data (150) in real time.

[0085] Accordingly, as can be seen in FIG. 7, in step S120, when the user requests deletion of image data (150) at a specific location (e.g., 237 of FIG. 7(a)) through a pre-allocated menu selection, etc., the deletion of image data (150) at the specific location can be confirmed and performed (310 of FIG. 7(b)), and further, the re-captured image data (150) at the specific location can be transmitted to the server (120) to regenerate and provide the map data (160) based on the image data group replaced by the re-captured image data (150).

[0086] In addition, in the step S120, it is also possible to perform location correction, etc. for the specific image data (150) instead of re-shooting after deleting the specific image data (150), and accordingly, if the user clicks on a specific location in the map data (160) through a pre-assigned menu selection, etc. or provides modified location information for the image data (150) through input of coordinate values, etc., it is also possible to regenerate the map data (160) by reflecting the modified location information.

[0087] In addition, in the step S130, following the first image data (150a) at the first location processed in the step S110, the second image data (150b) corresponding to a second location different from the first location is transmitted to the server (120), and the map data (160) generated based on the image data group including the first image data (150a) and the second image data (150b) is received from the server (120) and updated and provided to the user.

[0088] More specifically, as previously discussed in (a) of FIG. 4, when the first image data (150a) captured at the first location (201) is transmitted from the device (110) to the server (120), the server (120) generates map data (160) based on the image data group including the first image data (150a) and transmits the map data to the device (110), and then when the second image data (150b) captured at the second location (202) is transmitted from the device (110) to the server (120), the server (120) regenerates map data (160) based on the image data group including the first image data (150a) and the second image data (150b) and transmits the map data to the device (110) to provide it to the user.

[0089] Next, as seen in (b) to (d) of FIG. 4, the updated map data (160) can be provided to the user by transmitting the image data for the third location (203), the fourth location (240), and the fifth location (205) from the device (110) to the server (120).

[0090] At this time, the map data (160) may display and provide a recommended location for the second location.

[0091] For a more specific example, referring to (a) of FIG. 4, a recommended location for a second location (202) to be subsequently photographed is displayed and provided based on the map data (160) generated based on the image data group including the first image data (150a) photographed at the first location (201), thereby guiding the user to photograph the second image data (150b) at an appropriate location.

[0092] More specifically, in order to select a recommended location for the second location (202), the recommended location can be selected within an area of ​​map data (160) that is sequentially expanded according to the addition of the image data (150). At this time, it is possible to select the recommended location by considering the location and order of each image data (150) used to generate the map data (160), the distance from the location of adjacent image data (150), etc.

[0093] Furthermore, with reference to (b) of FIG. 4, it is also possible to generate the map data (150) by considering the alignment state of the third image data (150c) corresponding to the third location (203) adjacent to the second location (202) in the image data group and the second image data (150b).

[0094] For example, it is possible to correct the third position (203) corresponding to the third image data (150c) by considering the alignment state of the third image data (150c) and the second image data (150b), or to correct and apply the third image data (150c) based on the second image data (150b).

[0095] Next, as can be seen in FIG. 8, when the creation of image data (150) such as shooting for the space and the creation of map data (160) are completed, a predetermined confirmation window (320 of FIG. 8) is provided to the user so that the work in the space can be completed (330 of FIG. 8).

[0096] Accordingly, the user can capture image data (150), such as a 360-degree image, using a 360-degree camera (140) for a given space and transmit it to a server (120), and receive and confirm map data (160) generated from the server (120), and determine whether there is an error in real time and perform re-capturing of the image data (150), etc.

[0097] In addition, a spatial image scanning method according to one embodiment of the present invention may further include a step (not shown) of generating 3D modeling information for the space from the server (120) based on the image data group.

[0098] Here, 3D modeling information for the space can be derived based on a mono depth estimation technique or the like based on a series of image data (150) captured in the space, but the present invention is not necessarily limited thereto, and it is possible to derive 3D modeling information for the space through various depth estimation techniques, such as by using overlapping areas of a plurality of image data (150) to derive or correct 3D modeling information for the space.

[0099] At this time, after performing global optimization for each position corresponding to the image data included in the image data group, it is possible to produce 3D modeling information including texture data and mesh data for the space by reflecting the position corrected through the global optimization.

[0100] More specifically, in the present invention, since the global optimization and 3D modeling information generation may be tasks that require significant computational resources, a method of performing them as separate post-processing tasks on the server (120) may be considered.

[0101] At this time, the accuracy for each location corresponding to the image data can be improved through the global optimization, and this makes it possible to more accurately produce 3D modeling information such as texture data and mesh data for the space.

[0102] More specifically, as can be seen in FIG. 9, in the present invention, after the on-site scanning work for the space is completed, as a post-processing work, the server (120) can first perform correction for the position of the camera, such as the 360-degree camera (140) that captured the image data (150), such as the 360-degree image (S141).

[0103] Next, the server (120) can generate a textured mesh for the space by reflecting the globally optimized location (S142).

[0104] At this time, in order to create the above textured mesh, a mesh can be created based on multi-view stereo or various 3D modeling-related techniques such as neural implicit surface reconstruction can be applied to implement the method.

[0105] Accordingly, as can be seen in Fig. 10, in step S142, it is possible to generate a 3D model of the space by generating texture data and mesh data for the space.

[0106] In addition, in the present invention, virtual reality data for providing a virtual reality service for the space can be generated based on the 3D modeling information, and accordingly, as can be seen in FIG. 11, a virtual reality service for the space is provided, and more specifically, when a user selects a specific location (420) of the space, it is possible to provide a virtual reality experience for the space by moving to that location and providing an image of the space while changing the viewpoint according to the user's operation.

[0107] Furthermore, in the present invention, 2D drawing information for the space can be generated based on the 3D modeling information, and accordingly, as can be seen in FIG. 11, it is also possible to display the user's current experience location together with the accurate 2D drawing information generated for the space in a virtual reality service for the space, etc. (410 in FIG. 11).

[0108] In addition, in the present invention, it is also possible to provide a distance between two points selected in a virtual reality service for the space by calculating the distance based on mesh data for the space.

[0109] For a more specific example, as can be seen in Fig. 12, when a user selects two points of a virtual reality service for the space using a pointer (450) or the like through a predefined menu, a connection line (430) connecting the two points and a display window (440) displaying the distance between the two points can be provided, and at this time, the distance between the two points can be calculated based on coordinate information of each mesh corresponding to the two points in the 3D modeling information.

[0110] In this way, in the present invention, it is possible to derive 3D modeling information for the space based on a group of image data including a series of image data (150) captured in the space, and further, to provide various additional functions, such as virtual reality services for the space, based on the 3D modeling information for the space.

[0111] In addition, a computer program according to another aspect of the present invention is characterized by being a computer program stored in a computer-readable medium for executing each operation of the spatial image scanning method discussed above on a computer. The computer program may be a computer program including a machine language code created by a compiler, as well as a computer program including a high-level language code that can be executed on a computer using an interpreter, etc. In this case, the computer is not limited to a personal computer (PC) or a laptop computer, and includes any information processing device equipped with a central processing unit (CPU) capable of executing a computer program, such as a server, a smartphone, a tablet PC, a PDA, or a mobile phone.

[0112] Furthermore, a computer-readable medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. Furthermore, the medium may be a variety of recording or storage means in the form of a single or multiple hardware combinations, and is not limited to media directly connected to a computer system, but may also be distributed over a network. Therefore, the above detailed description should not be construed as limiting in any respect, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

[0113] In addition, FIG. 13 illustrates a device (110) to which the proposed method of the present invention can be applied.

[0114] Referring to FIG. 13, the device (110) may be configured to implement a method for generating a scene transition image according to the proposal of the present invention. For example, the device (110) may be a device (110) that can be used to scan a spatial image of a given space or, further, to provide a virtual reality service or the like based on the scanned spatial image.

[0115] For example, the device (110) to which the proposed method of the present invention can be applied may include network devices such as repeaters, hubs, bridges, switches, routers, gateways, etc., computer devices such as desktop computers, workstations, etc., mobile terminals such as smartphones, portable devices such as laptop computers, etc., home appliances such as digital TVs, etc., and moving means such as automobiles, etc. As another example, the device (110) to which the present invention can be applied may be included as a part of an ASIC (Application Specific Integrated Circuit) implemented in the form of a SoC (System On Chip).

[0116] The memory (20) can be connected to the processor (10) when it operates, and can store programs and / or commands for processing and controlling the processor (10), and can store data and information used in the present invention, control information required for data and information processing according to the present invention, temporary data generated during data and information processing, etc. The memory (20) can be implemented as a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, an SRAM (Static RAM), an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc.

[0117] The processor (10) may be operatively connected to a memory (20) and / or a network interface (30), and controls the operation of each module within the device (110). In particular, the processor (10) may perform various control functions for performing the proposed method of the present invention. The processor (10) may also be called a controller, a microcontroller, a microprocessor, a microcomputer, etc. The proposed method of the present invention may be implemented by hardware, firmware, software, or a combination thereof. When the present invention is implemented using hardware, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. configured to perform the present invention may be provided in the processor (10). Meanwhile, when implementing the proposed method of the present invention using firmware or software, the firmware or software may include instructions related to modules, procedures, or functions that perform functions or operations necessary to implement the proposed method of the present invention, and the instructions may be stored in the memory (20) or stored in a computer-readable recording medium (not shown) separate from the memory (20) and, when executed by the processor (10), the device (110) may be configured to implement the proposed method of the present invention.

[0118] In addition, the device (110) may include a network interface device (30). The network interface device (30) is connected to the processor (10) when in operation, and the processor (10) controls the network interface device (30) to transmit or receive wireless / wired signals carrying information and / or data, signals, messages, etc. through a wireless / wired network. The network interface device (30) supports various communication standards, such as, for example, IEEE 802 series, 3GPP LTE(-A), 3GPP 5G, etc., and may transmit and receive control information and / or data signals according to the communication standards. The network interface device (30) may also be implemented outside the device (110) as needed.

[0119] Accordingly, the method, device, system and computer program for providing a spatial image scan and virtual reality service according to one embodiment of the present invention can generate 3D modeling information, etc. based on a captured image, such as a 360-degree image, for a given space, and further provide various additional functions based on the 3D modeling information.

[0120] In addition, in the method, device, system and computer program for providing a spatial image scan and virtual reality service according to one embodiment of the present invention, it is possible to prevent the problem of having to revisit the space for re-shooting of an image, etc. during the post-processing process after shooting an image of a given space.

[0121] The embodiments and drawings described herein are merely exemplary and do not limit the scope of the present invention in any way. In addition, the lines connecting or connecting members between the components depicted in the drawings are merely exemplary of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, if there is no specific mention such as "essential" or "important," the component may not be absolutely necessary for the application of the present invention.

[0122] The use of the term "above" and similar referential terms in the specification of the present invention (especially in the claims) may refer to both singular and plural. In addition, when a range is described in the present invention, it includes inventions that apply individual values ​​belonging to the range (unless stated to the contrary), and it is the same as describing each individual value constituting the range in the detailed description of the invention. In addition, the steps presented in the method inventions of the present invention are not necessarily intended to be bound by the order of their chronological order, and the order may be appropriately changed as needed, unless a certain step must come first depending on the nature of each process. The use of all examples or exemplary terms (e.g., "for example," etc.) in the present invention is merely to describe the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms, unless limited by the claims. In addition, those skilled in the art will understand that various modifications, combinations, and variations can be configured according to design conditions and elements within the scope of the appended claims or their equivalents.

Claims

1. A method performed in a device, A step of transmitting first image data corresponding to a first location in a given space to a server, receiving map data for the space generated based on an image data group including the first image data from the server, and providing the map data to a user; When a modification request from the user is received, a step of transmitting the modification request to the server to regenerate the map data reflecting the modification request; and A method comprising the step of transmitting second image data corresponding to a second location different from the first location to the server, receiving the map data generated based on the image data group including the first image data and the second image data from the server, and updating and providing the map data to the user.

2. In claim 1, In the step of regenerating the above map data, A method for regenerating the map data based on the image data group in which the first image data is replaced with the 1-1 image data by transmitting the regenerated 1-1 image data to the server when a deletion request for the first image data of the user is received.

3. In claim 1, In the step of regenerating the above map data, A method for regenerating the map data by reflecting the modified first location when a request for modification of the first location of the first image data of the user is received.

4. In claim 1, In the above device, A method for receiving the first image data captured by the 360-degree camera at the first location and transmitting it to the server.

5. In claim 1, A method for transmitting panoramic image data at the first location captured by the device to the server as the first image data.

6. In claim 1, A method wherein the above map data provides a recommended location for the second location.

7. In claim 6, In the step of providing updates to the above users, A method for generating the map data by considering the alignment status of the second image data and the third image data corresponding to the third location adjacent to the second location in the group of the image data.

8. In claim 1, A method further comprising the step of generating 3D modeling information for the space from the server based on the group of image data.

9. In claim 8, In the step of generating the above 3D modeling information, A method for performing global optimization for each location corresponding to image data included in the above image data group.

10. In claim 9, In the step of generating the above 3D modeling information, A method for generating texture data and mesh data for the space by reflecting the globally optimized location.

11. In claim 10, A method for generating 2D drawing information for the space based on the above 3D modeling information.

12. In claim 10, A method for generating virtual reality data for providing virtual reality services for the space based on the above 3D modeling information.

13. In claim 12, A method for providing a distance between two points selected from a virtual reality service for the above space by calculating the distance based on mesh data for the above space.

14. In a device including a processor and a memory, The memory comprises instructions that, when executed by the processor, cause the device to perform a specific operation, the specific operation comprising: Transmitting first image data corresponding to a first location in a given space to a server, and receiving map data for the space generated based on an image data group including the first image data from the server and providing the map data to a user; When the above user's modification request is received, transmitting the modification request to the server to regenerate the map data reflecting the modification request; and A device comprising: transmitting second image data corresponding to a second location different from the first location to the server; receiving map data generated based on the image data group including the first image data and the second image data from the server and updating and providing the map data to the user.

15. A computer-readable storage medium storing instructions configured to cause a device including the processor to perform a specific operation when executed by the processor, the specific operation comprising: Transmitting first image data corresponding to a first location in a given space to a server, and receiving map data for the space generated based on an image data group including the first image data from the server and providing the map data to a user; When the above user's modification request is received, transmitting the modification request to the server to regenerate the map data reflecting the modification request; and A computer-readable storage medium comprising: transmitting second image data corresponding to a second location different from the first location to the server; receiving the map data generated based on the image data group including the first image data and the second image data from the server and updating and providing the map data to the user.

Images