Method and device for generating three-dimensional map of indoor space

Abstract

The present invention discloses a method and device for generating a three-dimensional map of an indoor space. The method for generating a three-dimensional map of an indoor space according to an embodiment of the present invention includes: a step of acquiring at least one indoor space image as an image related to the indoor space; a step of distinguishing a background area as an area corresponding to a structure of the indoor space and a non-background area as an area corresponding to an object located in the indoor space in at least one indoor space image; a step of generating at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image; a step of generating depth-image association information based on terrain information including at least one enlarged indoor space image and information of a depth value related to the indoor space; and a step of generating a three-dimensional map related to the indoor space using at least one enlarged indoor space image, terrain information, and depth-image association information.

Description

Technical Field

[0001] The present invention relates to a method and device for generating a three-dimensional map related to an indoor space, and more particularly, to a method and device for generating a map that uses information related to an image and shape of the indoor space to three-dimensionally display the indoor space in a virtual space. Background Art

[0002] Regarding indoor space modeling, a common method is to combine traditional cameras with laser sensors to mix the terrain values ​​obtained by the laser sensor and the image values ​​obtained by the camera to form color point cloud data. In the construction industry, companies such as FARO, RieGL, and LeiCA generally use the above-mentioned indoor space modeling.

[0003] In addition, Kinect or a three-dimensional sensor with a similar working principle is used to obtain color point cloud data, and the accumulated data is accumulated in the movement trajectory by estimating the movement trajectory of the sensor to model the three-dimensional space.

[0004] However, the above-mentioned prior art has the following problems.

[0005] First, various forms of errors related to sensor measurement (detection error, sensor position estimation error, integration error between multiple sensors, etc.) affect the three-dimensional map of the indoor space, thereby reducing the user's sense of immersion. Figure 1 Due to the small position estimation error of the image sensor, the image information that needs to be mapped on the pillar will be mapped on the ground, thus making it difficult to generate a realistic three-dimensional map.

[0006] Second, in any indoor environment, it is impossible to achieve perfect modeling by matching the shape of complex indoor structures with the pixels of the image on a one-to-one basis. Figure 2 It is known that there will be inconsistencies between the image and the shape, which makes it difficult to generate a realistic 3D map. In particular, after the decoration process, when there are many objects with extremely complex shapes such as chandeliers, matching the shape with the image 1 to 1 exceeds the measurement error level of the sensor itself, so in reality, perfect modeling cannot be achieved with the current system.

[0007] Third, in order to fully present any indoor environment, it is necessary to obtain data related to all locations related to indoor spaces such as under a table or in a narrow gap. In order to perform the above operations, a sensor system with a complex form is required, and data needs to be obtained for a long time through such a sensor system. Moreover, in the process of implementing the prior art, in the presence of a moving person or a moving object, it is impossible to guarantee sufficient time for the sensor to measure the object, and as the sensor moves, the moving person or object also moves independently, so it is impossible to specify the terrain information related thereto. Therefore, when utilizing the prior art, data is obtained when there is no moving person or object, and when there is a moving person or object, it is necessary to wait until it is out of the field of view of the sensor.

[0008] Therefore, there is a need for a new three-dimensional map generation method and apparatus that can solve the conventional problems that occur when generating a three-dimensional map as described above.

[0009] The relevant prior art is Korean authorized patent No. 10-1835434 (invention name: projection image generation method and device thereof, matching method between image pixels and depth values, publication date: January 18, 2017). Summary of the invention

[0010] The present invention solves the above-mentioned problems of the prior art and provides a method and device for generating a three-dimensional map similar to the corresponding indoor space in a virtual space by using information obtained in the indoor space.

[0011] The problems to be solved by the present invention are not limited to the problems mentioned above, and a person skilled in the art in the technical field to which the present invention belongs can clearly understand other problems not mentioned from the following description.

[0012] In order to achieve the above-mentioned purpose, the method for generating a three-dimensional map of an indoor space provided in the present invention includes: a step of acquiring at least one indoor space image as an image related to the indoor space; a step of distinguishing a background area as an area corresponding to the structure of the indoor space and a non-background area as an area corresponding to the object located in the indoor space in the at least one indoor space image; a step of generating at least one enlarged indoor space image by enlarging the background area toward the non-background area in the at least one indoor space image; a step of generating depth-image association information based on terrain information including the at least one enlarged indoor space image and information on a depth value related to the indoor space; and a step of generating a three-dimensional map related to the indoor space using the at least one enlarged indoor space image, the terrain information and the depth-image association information.

[0013] Preferably, before the above-mentioned step of distinguishing the non-background area, it may also include: a step of acquiring depth value information as information of the depth value related to the above-mentioned indoor space; and a step of generating indoor space peripheral information as peripheral information related to the above-mentioned indoor space using the above-mentioned depth value information, the above-mentioned terrain information includes the above-mentioned depth value information and the above-mentioned indoor space peripheral information.

[0014] Preferably, after the above-mentioned step of generating the depth-image association information, a step of updating at least one of the above-mentioned terrain information and the above-mentioned depth-image association information using features contained in the above-mentioned at least one enlarged indoor space image and the above-mentioned terrain information may be included, and the above-mentioned step of generating a three-dimensional map may utilize at least one of the above-mentioned updated terrain information and depth-image association information.

[0015] Preferably, the step of generating the depth-image association information and the step of generating the three-dimensional map related to the indoor space may include: a step of distinguishing the background area and the non-background area in the at least one indoor space image using the terrain information and the depth-image association information; and a step of regenerating at least one enlarged indoor space image by expanding the background area toward the non-background area in the at least one indoor space image, and the step of generating the three-dimensional map may utilize the regenerated at least one enlarged indoor space image.

[0016] Preferably, in the step of re-distinguishing the above-mentioned background area and the above-mentioned non-background area, when re-distinguishing the above-mentioned background area and the above-mentioned non-background area for the object image which is one of the above-mentioned at least one indoor space image, at least one supplementary image other than the above-mentioned object image in the above-mentioned depth-image association information and the above-mentioned at least one indoor space image can also be utilized.

[0017] Preferably, the step of generating a three-dimensional map related to the indoor space may include: a step of screening an enlarged indoor space image associated with a reference posture in the at least one enlarged indoor space image based on the depth-image association information and determining it as a representative image, the reference posture corresponding to a specified reference position on the three-dimensional map; a step of determining at least one supplementary image other than the representative image in the at least one enlarged indoor space image using the depth-image association information and the reference posture; and a step of generating a three-dimensional map using the representative image, the at least one supplementary image, the terrain information and the depth-image association information.

[0018] Preferably, the present invention may also include: a step of acquiring object information containing information related to at least one object from an object database (DB) containing information related to the shapes and images of multiple objects; and a step of adding at least one virtual object corresponding to the at least one object to the three-dimensional map generated above based on the above-mentioned object information.

[0019] Preferably, the object information may be information related to an object located in the indoor space.

[0020] Preferably, the present invention may further include the step of selectively updating the at least one virtual object added to the three-dimensional map if object-related information is added to the object database or information related to previous virtual objects is updated.

[0021] Preferably, the present invention may also include the following step, namely, when a reflective virtual object which is a virtual object of a material containing reflected light is included in the at least one virtual object, a reflected image is displayed on the reflective virtual object using the at least one enlarged indoor space image and the terrain information, and the reflected image is an image of the indoor space reflected by the reflective virtual object at a reference posture corresponding to a specified reference position on the three-dimensional map.

[0022] Preferably, the present invention may also include: a step of inferring light source information from the at least one indoor space image, wherein the light source information includes information related to the position and brightness of the light source located in the indoor space; and a step of reflecting the lighting effect based on the light source information in the at least one enlarged indoor space image.

[0023] Furthermore, in order to achieve the above-mentioned purpose, the three-dimensional map generation device of the indoor space provided in the present invention includes: an acquisition unit, which is used to acquire at least one indoor space image which is an image related to the indoor space; a distinguishing unit, which is used to distinguish, in the above-mentioned at least one indoor space image, a background area which is an area corresponding to the structure of the above-mentioned indoor space and a non-background area which is an area corresponding to the object located in the above-mentioned indoor space; an enlargement unit, which generates at least one enlarged indoor space image by enlarging the above-mentioned background area toward the above-mentioned non-background area in the above-mentioned at least one indoor space image; an association unit, which generates depth-image association information based on terrain information including the above-mentioned at least one enlarged indoor space image and information on the depth value related to the above-mentioned indoor space; and a generation unit, which generates a three-dimensional map related to the above-mentioned indoor space using the above-mentioned at least one enlarged indoor space image, the above-mentioned terrain information and the above-mentioned depth-image association information.

[0024] Preferably, the above-mentioned acquisition unit can also acquire depth value information as information of the depth value related to the above-mentioned indoor space. The present invention also includes a peripheral information generation unit which generates indoor space peripheral information as peripheral information related to the above-mentioned indoor space using the above-mentioned depth value information. The above-mentioned terrain information may include the above-mentioned depth value information and the above-mentioned indoor space peripheral information.

[0025] Preferably, the associating unit may update at least one of the terrain information and the depth-image association information using features included in the at least one enlarged indoor space image and the terrain information, and the generating unit may use at least one of the updated terrain information and the depth-image association information.

[0026] Preferably, the above-mentioned distinguishing unit can use the above-mentioned terrain information and the above-mentioned depth-image association information to further distinguish the above-mentioned background area and the above-mentioned non-background area in the above-mentioned at least one indoor space image, and the above-mentioned enlargement unit can regenerate at least one enlarged indoor space image by enlarging the above-mentioned background area toward the above-mentioned non-background area in the above-mentioned at least one indoor space image, and the above-mentioned generating unit can use the regenerated at least one enlarged indoor space image.

[0027] Preferably, when further distinguishing the background area and the non-background area for an object image which is one of the at least one indoor space images, the distinguishing unit may also utilize at least one supplementary image other than the object image in the depth-image association information and the at least one indoor space image.

[0028] Preferably, the generating unit can screen out an enlarged indoor space image associated with a reference posture in the at least one enlarged indoor space image and determine it as a representative image based on the depth-image association information, the reference posture corresponds to a specified reference position on the three-dimensional map, the depth-image association information and the reference posture can be used to determine at least one supplementary image other than the representative image in the at least one enlarged indoor space image, and the representative image, the at least one supplementary image, the terrain information and the depth-image association information can be used to generate a three-dimensional map.

[0029] Preferably, the present invention may also include a database connection unit for obtaining object information containing information related to at least one object from an object database containing information related to the shapes and images of multiple objects, and the above-mentioned generation unit adds at least one virtual object corresponding to the above-mentioned at least one object to the above-mentioned generated three-dimensional map based on the above-mentioned object information.

[0030] Preferably, the object information may be information related to an object located in the indoor space.

[0031] Preferably, if information related to an object is added to the object database or information related to a previous virtual object is updated, the generating unit may selectively update the at least one virtual object added to the three-dimensional map.

[0032] Preferably, when the at least one virtual object includes a reflective virtual object which is a virtual object made of a material containing reflected light, the generating unit displays a reflected image on the reflective virtual object using the at least one enlarged indoor space image and the terrain information, and the reflected image is an image of the indoor space reflected by the reflective virtual object at a reference posture corresponding to a specified reference position on the three-dimensional map.

[0033] Preferably, the present invention may also include: a light source estimating unit, which estimates light source information from the at least one indoor space image, wherein the light source information includes information related to the position and brightness of the light source located in the indoor space; and a light source reflecting unit, which reflects the lighting effect based on the light source information in the at least one enlarged indoor space image.

[0034] The present invention has the following effect, that is, in the case of various forms of errors related to sensor measurement, by using only the background part of the space that is relatively less affected by the corresponding errors, a three-dimensional map can be robustly and stably constructed in an environment where various errors exist. In particular, in the present invention, the boundary line (boundary) part of the background is actively used to pursue robustness and stability. More specifically, the internal boundary line formed by the background part and the non-background part increases in the direction of expanding the background part. In the background, the edge of the image and the edge of the terrain can be corrected in the direction of increasing mutual integration to construct a robust and stable three-dimensional map.

[0035] Furthermore, the present invention has the following effects, namely, after stably constituting the background, the background is prepared by using the image and shape information accumulated in the object database (DB) constructed for other non-background parts that need to be configured in the corresponding background, thereby, the constructed database can also be used for objects that are sensitive to errors and whose characteristics make it almost impossible to construct precise information from the above-mentioned sensor information to accurately express. Moreover, when a new object is registered or updated in the object database with a time difference, more precise image and shape information can also be reflected for objects in the previously created map through the updated object database, and all the above processes can be performed by computer algorithms without additional labor input.

[0036] Furthermore, the present invention has the following effects, namely, when only a part of the information is obtained for the background part, the composition related to the background can also be realized, and when only a part of the information is obtained for the non-background part, the species and posture can be predicted strongly in the past, so that the entire information can be inferred and expressed by obtaining only a part of the information in a short time, rather than obtaining all data for a long time at all time points. Furthermore, the present invention has the following effects, namely, in the process of obtaining data, when there are dynamic objects such as people that should not be presented in the final map, they can be considered as non-background parts and removed. In particular, when using the single terrain-multi-image expression method implemented in the present invention, in specific image information, when it is difficult to infer and expand the background due to occlusion such as high partitions, image information obtained in other places can also be used to restore the background part more accurately and effectively.

[0037] Furthermore, the present invention has the following effect, that is, as described above, when the single terrain-multiple image expression method is used to construct map data, the indoor lighting effect changes naturally as the user moves, thereby expressing a virtual map with a realistic feel. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 An illustration of the visual error that occurs when projecting the image information of a column onto the ground.

[0039] Figure 2 An illustration showing a visual error in the appearance of an object in a three-dimensional map due to inconsistency between the image and the shape.

[0040] Figure 3 The flowchart is used to illustrate a method for generating a three-dimensional map of an indoor space according to an embodiment of the present invention.

[0041] Figure 4 The flowchart is used to illustrate a method for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0042] Figure 5 The flowchart is a method for generating a three-dimensional map of an indoor space using a representative image and a supplementary image for explaining an embodiment of the present invention.

[0043] Figure 6 The figure is a flow chart for illustrating a method for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0044] Figure 7 The figure is a flow chart for illustrating a method for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0045] Figure 8 The block diagram shows a device for generating a three-dimensional map of an indoor space according to an embodiment of the present invention.

[0046] Fig. 9 The block diagram shows a device for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0047] Fig.10 It is a block diagram showing a device for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0048] Fig.11 The following are illustrations of sensor systems used for data acquisition, including a tire-type mobile robot, a backpack-type system, and a handheld system equipped with a camera, an inertial sensor, and a distance measurement sensor.

[0049] Fig.12 A map of three-dimensional terrain information generated using a lidar sensor.

[0050] Fig.13 A graph that associates 360-degree panoramic images with depth values ​​in the form of a mesh.

[0051] Fig.14 and Fig.15 The figures respectively show an indoor space in which no object exists and an indoor space in which multiple objects exist.

[0052] Fig.16 and Fig.17 1 and 2 are diagrams showing the background area and the non-background area of ​​the indoor space respectively.

[0053] Fig.18 The figure shows the result of expanding the background area along the non-background area according to one embodiment of the present invention.

[0054] Fig.19 FIG. 1 is a diagram showing edges included in an enlarged indoor space image and indoor structure peripheral information according to an embodiment of the present invention.

[0055] Fig. 20 This is a diagram showing an example in which the regions on the image are divided so as to have the same region in the terrain information.

[0056] Fig.21 The diagrams show various embodiments of terrain information formed by point cloud data.

[0057] Fig. 22 The diagrams show various embodiments of terrain information constructed in a grid form.

[0058] Fig.23 This is a diagram of an embodiment of using an enlarged indoor space image when the indoor space is expressed in a depth-image correlation form based on a specific posture and an arbitrary field of view in a user viewer is expressed. DETAILED DESCRIPTION

[0059] The present invention may have various variations and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, the above-mentioned specific embodiments do not limit the present invention to specific implementations, but include all variations, equivalent technical solutions or alternative technical solutions included in the concept and technical scope of the present invention. In the process of describing the various drawings, similar structural elements are given similar reference numerals.

[0060] The terms first, second, A, B, etc. may be used to describe a variety of structural elements, and the above structural elements are not limited to the above terms. The above terms are only used to distinguish two structural elements. For example, without exceeding the scope of the invention claimed in the present invention, the first structural element may be named as the second structural element, and similarly, the second structural element may also be named as the first structural element. And / or the term includes a combination of multiple related recorded items or one item among multiple related recorded items.

[0061] When it is mentioned that a structural element is "connected" or "coupled" to other structural elements, it can be directly connected or coupled to other structural elements, or it can be understood that other structural elements exist in between. On the contrary, when a structural element is "directly connected" or "directly coupled" to other structural elements, it should be understood that there are no other structural elements in between.

[0062] The terms used in this application are only used to illustrate specific embodiments and are not used to limit the present invention. Unless clearly indicated in the context, the expression in the singular includes the expression in the plural. In this application, the terms such as "including" or "having" are used to specify the existence of features, numbers, steps, actions, structural elements, components or combinations thereof recorded in the specification, and do not preclude the existence or additional possibility of one or more other features or numbers, steps, actions, structural elements, components or combinations thereof.

[0063] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs. Terms defined in dictionaries and commonly used terms have the same meanings as those in the context of the relevant technology and should not be interpreted as ideal or overly formal meanings unless otherwise defined in this application.

[0064] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0065] In order to implement the present invention, a sensor system including a sensor capable of acquiring image information may be used to acquire data in an indoor space. In particular, the indoor space is wider and larger than the field of view of the sensor, and therefore the sensor is moved and acquired in a manner that can sufficiently store data of the indoor space. In this case, the sensor may use a single camera, or may use multiple cameras for depth measurement, and may be used in combination with an inertial sensor or a laser sensor capable of measuring distance.

[0066] In this case, the sensor values ​​are acquired while moving in the indoor space, so the 3D poses of the individual sensor data may be different. In order to accurately create a 3D map, it is necessary to accurately predict the pose values ​​of the individual sensor data acquired. To this end, methods such as simultaneous localization and mapping (SLAM) can be used to predict the 3D pose. However, if Fig.11 As shown, when a sensor system uses a mobile robot or the like, data is acquired at a fixed height, and the present invention can be implemented by estimating the state vector of the two-dimensional posture. That is, data can be acquired with the inertia of the sensor acquisition posture ensured.

[0067] On the other hand, the composition of the sensor value may differ depending on the type of sensor. For example, in the case where the sensor is composed of only a single camera, the sensor value is composed of only the camera image, and the corresponding image is used to extract specific points in the image. In the case of a single camera, the relative distance between corresponding feature points can be predicted, and in the case of multiple cameras, the absolute distance between corresponding feature points can be predicted. In particular, even if feature points are not extracted, in the case of a single camera, the accumulated image information can be used to predict the depth of the pixel, and in the case of multiple cameras, the multiple camera images or the accumulated information can be used to predict the depth of the pixel.

[0068] Furthermore, when information from an additional inertial sensor, depth sensor, etc. is used together, sensor information processing can be performed in a manner that conforms to the inherent characteristics of each sensor. Fig.11 As shown, when inertial sensor information is available, the inertial sensor information is used to improve the performance of simultaneous positioning and mapping, or, when image information is processed, it is used as prediction information related to the image acquisition position to make corrections related to the image acquisition position easier. In addition, the acceleration value or angular velocity value of the inertial information can be used to predict the actual movement distance, and the actual movement distance can be used to correct the measurement (scale) of the depth value extracted from a single or multiple cameras.

[0069] In the same vein, for example, in the case of LiDAR or Kinect, where sensor information capable of measuring depth in units of light beams (rays) can be obtained, the acquisition posture of the corresponding data can be predicted by simultaneous positioning and mapping, and the depth measurement data can be listed based on the predicted posture, such as Fig.12 As shown, the three-dimensional terrain information can be predicted more accurately.

[0070] In this case, if Fig.21 As shown, the terrain information may include: point cloud morphology (top), which gathers points measured by light beams; point cloud morphology (middle), which gathers only points in the point cloud that are equivalent to the background excluding the object; and point cloud morphology (bottom), which restores the background part blocked by the object based on the terrain continuity of the background part.

[0071] And, if Fig. 22 As shown, the terrain information may include a situation in which it is expressed in an enlarged grid form (upper left: outdoor viewpoint, lower left: indoor viewpoint), a situation in which only grids equivalent to the background excluding objects are gathered in the grid (upper right: outdoor viewpoint, lower right: indoor viewpoint), etc.

[0072] In this case, the expression method of the original data of the three-dimensional map can be varied according to the structure and implementation method of the sensor. Fig.13 As shown, the acquired image is expressed by a panoramic image, and the estimated depth value is associated with the panoramic image in a grid form, and the coordinate value of a specific reference point relative to the acquired image or the grid can be stored in a three-dimensional or two-dimensional posture form. Fig.13 As an example of data for specifying a posture, the original data of the three-dimensional map may be composed of a plurality of data according to the width of the indoor space.

[0073] On the other hand, refer to Fig.11 , the three-dimensional map generating device of an indoor space of an embodiment of the present invention can obtain data for generating a three-dimensional map by receiving data from a tire-type mobile robot attached with a sensor system including a camera, an inertial sensor, a lidar sensor, etc. through wired or wireless communication. Alternatively, the three-dimensional map generating device of an indoor space of an embodiment of the present invention can be directly loaded on a tire-type mobile robot attached with a sensor system to generate a three-dimensional map. Alternatively, the three-dimensional map generating device of an indoor space of an embodiment of the present invention can be mounted on a smartphone itself having a backpack-type system, a small scanner, a camera and an inertial sensor, or a system attached with a smartphone expansion device to generate a three-dimensional map.

[0074] However, the three-dimensional map generating device of the indoor space of one embodiment of the present invention is not limited to the use of Fig.11The illustrated system may also receive data for generating a three-dimensional map from devices including various sensor systems or may be mounted on various devices to generate a three-dimensional map.

[0075] Figure 3 The flowchart is used to illustrate a method for generating a three-dimensional map of an indoor space according to an embodiment of the present invention.

[0076] In step S310, the three-dimensional map generating apparatus obtains at least one indoor space image which is an image related to the indoor space.

[0077] For example, the three-dimensional map generating apparatus may acquire at least one indoor space image as an image related to the indoor space from an external device to which a camera is attached or a camera to which the three-dimensional map generating apparatus is attached.

[0078] In this case, the indoor space image may be an image of the indoor space captured at a plurality of positions inside the indoor space.

[0079] In step S320, the three-dimensional map generating apparatus distinguishes, in at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space.

[0080] In this case, the background area may be a portion of the indoor space image that corresponds only to the structure of the indoor space itself, and the non-background area may be a portion that is not represented in the final map, such as an object located in the indoor space or a moving person in the indoor space image.

[0081] For example, refer to Figures 14 to 17 , Fig.15 For an indoor space image with multiple objects (tables, bookcases, chairs, etc.) set up in the indoor space. In this case, Fig.16 As shown in , the part of the object removed (the area shown by the white part) is the background area, such as Fig.17 As shown in FIG. 1 , the portion corresponding to the object corresponds to the non-background area. In this case, refer to Fig.14 The background area may be the structure of the building itself that forms the above-mentioned indoor space, and may also include structures attached to the structure of the above-mentioned building itself (for example, doors, windows, etc.).

[0082] On the other hand, the three-dimensional map generation device cannot be continuous due to the occlusion of other elements in the structural elements of at least one indoor space image, and the elements that are continuously constituted can be distinguished as background areas in the structure of the actual environment. In addition, the three-dimensional map generation device can distinguish the area other than the area that is distinguished as the background area in at least one indoor space image as a non-background area. In this case, the three-dimensional map generation device can distinguish the background area from the non-background area based on the pixel values ​​of the images constituting at least one indoor space image.

[0083] In this case, if Fig.14 As shown in FIG. 1 , the background area is the first part formed in time in the environment where humans live. That is, generally, the background area is a combination of structures that exist at the time point after the pasting or painting process is completed. Fig.15 As shown, non-background areas (e.g., various objects) may flow into the background area. Also, the background area may be blocked by other structural elements, so that the data is incomplete, but may have similarities with the unblocked part, which may be a part reconstructed using hole filling or inpainting. Also, the background area may be a large plaque or information desk in a building that blocks other objects, and may be a part where the integration degree of the image and the terrain in the edge of the corresponding object is consistent in all data or can be consistent through an additional integration process.

[0084] In step S330, the three-dimensional map generating apparatus generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0085] In this case, the three-dimensional map generating device may use information related to the background region of each of the at least one indoor space image to expand the background region toward the non-background region, thereby generating an expanded indoor space image.

[0086] For example, refer to Fig.16 The 3D map generation device may use information related to the background area to expand it to the non-background area. That is, the 3D map generation device may use information related to the edge included in the background area to infer and enhance the portion corresponding to the non-background area to generate an enlarged indoor space image.

[0087] More specifically, when the edge included in the background area is disconnected from the boundary line 1610 of the non-background area, the three-dimensional map generating device generates an expanded indoor space image by inferring that the extension line of the above-mentioned edge exceeds the boundary line 1610 between the background area and the non-background area and extends to the non-background area.

[0088] In this case, the three-dimensional map generating device designates one or more indoor space images in addition to the specific indoor space image as background supplementary images, and can reduce the area corresponding to the non-background area of ​​the specific indoor space image by using information of the background supplementary images.

[0089] On the other hand, refer to Fig.19 , the three-dimensional map generation device can be partially expanded as needed. In this case, Fig.18 Part (a) shows the case of 10% expansion. Fig.18 Part (b) is for a 30% expansion. Fig.18 Part (c) of FIG. 1 is a case where the entire space is expanded. When the user models only the structure of the indoor space in such a way that no object exists in the indoor space and wishes to prepare any virtual object in the virtual indoor space, Fig.18 As shown in part (c), 100% enlargement is also possible.

[0090] In step S340, the three-dimensional map generating device generates depth-image association information based on at least one enlarged indoor space image and terrain information related to the indoor space.

[0091] In this case, the depth-image association information may be information matching a depth value of the indoor space corresponding to a pixel of the at least one enlarged indoor space image.

[0092] To this end, in addition to the at least one enlarged indoor space image and terrain information mentioned above, the three-dimensional map generation device can also generate depth-image association information using an image acquisition posture and a depth acquisition posture that contain information related to the respective acquisition positions and acquisition angles of the at least one enlarged indoor space image and terrain information mentioned above.

[0093] More specifically, the three-dimensional map generation device queries the coordinate relationship between the entire pixel or a part of the individual image and the depth value included in the terrain information through the image acquisition gesture associated with at least one enlarged indoor space image. Moreover, the depth-image association information can be generated by repeatedly applying the method to at least one enlarged indoor space image.

[0094] On the other hand, when the resolution of the image sensor is greater than the resolution of the distance sensor, unmatched pixels without corresponding depth values ​​may be generated in the pixels of at least one of the above-mentioned enlarged indoor space images, and the three-dimensional map generation device can use interpolation to match the estimated depth values ​​of the unmatched pixels.

[0095] In another embodiment, the three-dimensional map generating device updates at least one of the terrain information and the depth-image association information using features included in the at least one enlarged indoor space image and the terrain information.

[0096] For example, refer to Fig.13 The three-dimensional map generating device has depth-image association information in which the whole or part of individual image pixels are linked to specific reference coordinates and the relationship between the corresponding reference coordinates and terrain information is linked. Thus, when the terrain information can be matched to the image pixels, a feature-based integration process for updating at least one of the reference coordinates and terrain information can be executed in a manner based on the prescribed reference integration while expanding the features contained in the indoor space image and terrain information.

[0097] In this case, the features based on the feature integration process exist in the enlarged indoor space image and terrain information respectively, and can be related to each other. More specifically, they can be composed of edge information, point information, segment information, line information, plane block information, plane information or a combination thereof.

[0098] For example, refer to Fig.19 In the case where at least one edge 1910 of the enlarged indoor space image is inconsistent with an edge 1920 of the terrain information, the three-dimensional map generating device can modify the reference coordinates in a manner that increases the mutual consistency between the edge 1910 of the enlarged indoor space image and the edge 1920 of the terrain information.

[0099] In another embodiment, the 3D map generating apparatus may improve the integration of the edge 1910 of the enlarged indoor space image and the edge 1920 of the terrain information by correcting the terrain information in the feature-based integration process.

[0100] That is, as described above, when the three-dimensional map generating device updates at least one of the reference coordinates and the terrain information through the feature integration process, the three-dimensional map generating device updates the reference coordinates and the terrain information based on the updated information. Fig.13 Furthermore, the three-dimensional map generating device may use one of the updated depth-image correlation information and the terrain information to generate the three-dimensional map.

[0101] In another embodiment, the 3D map generating device may utilize the terrain information and the depth-image correlation information to distinguish the background area from the non-background area and regenerate at least one enlarged indoor space image.

[0102] That is, the 3D map generation device can use the depth-image association information to more accurately distinguish the background area from the non-background area. More specifically, Fig. 20As shown, the three-dimensional map generating device matches the boundary line 1620 (i.e., feature) corresponding to the same plane in the terrain information in the image through the depth-image association information, and uses the corresponding matching information to divide the interior of the boundary line 1610 into A, B, and C. The image of the background area can be expanded to the non-background area by using the divided information.

[0103] In this case, the features of the terrain information may be in mutually related forms existing in the image and terrain information respectively, and more specifically, may be composed of edge information, point information, line segment information, line information, plane block information, plane information or a combination thereof.

[0104] Furthermore, the 3D map generation device may regenerate at least one enlarged indoor space image by enlarging the reclassified background area to the non-background area in the indoor space image. Furthermore, the 3D map generation device may use the regenerated at least one enlarged indoor space image to generate the 3D map.

[0105] In another embodiment, when the three-dimensional map generating device further distinguishes the background area and the non-background area for an object image which is one of at least one indoor space image, it can also utilize depth-image association information and at least one supplementary image other than the object image in at least one indoor space image.

[0106] That is, the three-dimensional map generation device can use the depth-image association information to identify the portion of pixels in the specific object image that is far away from the reference coordinate system, and for the corresponding portion, at least one or more supplementary images other than the specific image and the depth-image information can be used simultaneously to expand the background area to the non-background area. In addition, in the specific object image, for the area that cannot be simply added with the background due to occlusion or the like, at least one or more supplementary images with better image information related to the corresponding area can be identified through the depth-image association information, and the background area can be expanded to the non-background area.

[0107] Finally, in step S350, the three-dimensional map generating device generates a three-dimensional map related to the indoor space by using at least one enlarged indoor space image, terrain information, and depth-image association information.

[0108] In this case, the 3D map generation device matches at least one enlarged indoor space image with terrain information based on the depth-image association information, thereby generating a more realistic 3D map. The generated 3D map can be linked to a user viewer operating in a smartphone, a personal computer (PC), a tablet computer, etc., and the user can observe the 3D map related to the indoor space through the user viewer.

[0109] Furthermore, the three-dimensional map generation device can improve the realism of the user's experience by executing the feature-based integration process described above.

[0110] On the other hand, in an indoor space, when the field of view expressed in the user viewer exceeds the range of the field of view expressed by a single enlarged indoor space image, the three-dimensional map generating device can use depth-image association information to use more than one enlarged indoor space image and express the user's field of view.

[0111] More specifically, if Fig.23 As shown, when the reference position 2310 of the field of view expressed in the user's vision is different from the reference positions 2320, 2330, 2340, 2350 of the enlarged indoor space images 2321, 2331, 2341, 2351 having depth-image association information, the field of view of the user's viewer cannot be expressed by a single enlarged indoor space image, and field of view loss will occur.

[0112] In this case, in order to express all fields of view of the reference position 2310 of the user's field of view, multiple enlarged indoor space images specifically required by depth-image association information are used, wherein the enlarged indoor space image associated with the posture close to the reference posture of the user viewer is designated as a representative image, and at least one enlarged indoor space image other than the above-mentioned representative image is determined as a supplementary graphic, and the representative image and at least one supplementary image can be used simultaneously to express all fields of view of the user viewer.

[0113] On the other hand, the detailed method of the three-dimensional map generating device using the representative image and at least one supplementary image to generate a three-dimensional map will be described in detail in relation to Figure 5 Detailed description in the description.

[0114] In another embodiment, the three-dimensional map generating device may generate terrain information using at least one indoor space image.

[0115] For example, the 3D map generation device uses at least one indoor space image to extract feature points in the image. In the case of a single camera, the relative distance between corresponding feature points can be predicted, and in the case of multiple cameras, the absolute distance between corresponding feature points can be predicted. In addition, when the 3D map generation device does not extract feature points, a single camera can use accumulated image information to predict the depth of pixels, and multiple cameras can use multiple camera images or the above-mentioned accumulated information to predict the depth of pixels.

[0116] Thus, the three-dimensional map generating device can generate terrain information using at least one indoor space image.

[0117] As described above, the method for generating a three-dimensional map of an indoor space according to an embodiment of the present invention can construct a robust and stable three-dimensional map even in an environment with various errors by using only the background portion of the space that is relatively less affected by errors.

[0118] Figure 4 The flowchart is used to illustrate a method for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0119] In step S410, the three-dimensional map generating apparatus obtains at least one indoor space image which is an image related to the indoor space.

[0120] In step S420, the three-dimensional map generating apparatus acquires depth value information which is information on a depth value related to the indoor space.

[0121] For example, the three-dimensional map generating apparatus may obtain depth value information related to the indoor space from an external device equipped with a depth measurement sensor or a depth measurement sensor to which the three-dimensional map generating apparatus is attached.

[0122] In step S430, the three-dimensional map generating apparatus generates indoor space peripheral information as peripheral information related to the indoor space using the depth value information.

[0123] The indoor space peripheral information may be peripheral information related to the structure of the indoor space where no object exists.

[0124] On the other hand, the 3D map generation device can generate the peripheral information of the indoor space by removing the objects existing in the indoor space using algorithms such as convex cut, etc. The convex cut algorithm is an algorithm that uses the depth value information (point cloud data) of the indoor space to extract the structure of the indoor space in real time.

[0125] In step S440, the three-dimensional map generating apparatus distinguishes a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space in at least one indoor space image.

[0126] In step S450, the three-dimensional map generating apparatus generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0127] In step S460, the three-dimensional map generating device generates depth-image association information based on at least one enlarged indoor space image and terrain information related to the indoor space.

[0128] Finally, in step S470, the three-dimensional map generating device generates a three-dimensional map related to the indoor space using at least one enlarged indoor space image, terrain information, and depth-image association information.

[0129] The terrain information may include depth value information and indoor space periphery information.

[0130] Figure 5 The flowchart is a method for generating a three-dimensional map of an indoor space using a representative image and a supplementary image for explaining an embodiment of the present invention.

[0131] In step S510, the three-dimensional map generating device selects an enlarged indoor space image associated with a reference posture from at least one enlarged indoor space image based on the depth-image association information and determines it as a representative image, wherein the reference posture corresponds to a specified reference position on the three-dimensional map.

[0132] For example, the three-dimensional map generating device uses the gesture of the person viewer as a reference gesture in at least one enlarged indoor space image, selects an image associated with a gesture closest to the reference gesture, and determines the image as a representative image.

[0133] In step S520, the three-dimensional map generating apparatus determines at least one supplementary image other than the representative image in at least one enlarged indoor space image by using the depth-image association information and the reference pose.

[0134] For example, the three-dimensional map generating device may additionally determine at least one supplementary image adjacent to the reference posture.

[0135] Finally, in step S530, the three-dimensional map generating apparatus generates a three-dimensional map using the representative image, at least one supplementary image, terrain information, and depth-image association information.

[0136] That is, the three-dimensional map generating device may also simultaneously use the representative image and at least one supplementary image to express the entire field of view equivalent to the user's viewer.

[0137] Figure 6 The figure is a flow chart for illustrating a method for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0138] In step S610, the three-dimensional map generating apparatus obtains at least one indoor space image which is an image related to the indoor space.

[0139] In step S620, the three-dimensional map generating apparatus distinguishes, in at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space.

[0140] In step S630, the three-dimensional map generating apparatus generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0141] In step S640, the three-dimensional map generating apparatus generates depth-image association information based on at least one enlarged indoor space image and terrain information related to the indoor space.

[0142] In step S650, the three-dimensional map generating device generates a three-dimensional map related to the indoor space by using at least one enlarged indoor space image, terrain information, and depth-image association information.

[0143] In step S660 , the three-dimensional map generating apparatus acquires object information including information related to at least one object from an object database including information related to shapes and images of a plurality of objects.

[0144] In this case, the object database may include information related to the type, name, shape, image, etc. of various objects. Furthermore, the three-dimensional map generating device combines the information related to the shape of an object with the information related to the image to reproduce the three-dimensional appearance of the above object in the virtual space. On the other hand, the information related to the shape of the object may include information related to the size of the object.

[0145] On the other hand, the three-dimensional map generating device can obtain object information related to at least one object to be located on the generated three-dimensional map from the object database. In this case, the at least one object located on the map can be an object actually located in the above-mentioned indoor space or any object not located in the above-mentioned indoor space.

[0146] In another embodiment, the three-dimensional map generating device may use at least one enlarged indoor space image to determine an object located in the indoor space.

[0147] To this end, the three-dimensional map generation device can use an object database containing images and shape information of various objects, and can use feature information extracted from the information contained in the object database or the images and shape information of the objects to perform object segmentation, classification, etc.

[0148] Furthermore, when it is determined that the identified object does not exist in the object database or exists in another form, the user may be requested to add or update the shape and image information related to the new object or the revised object in the object database.

[0149] Alternatively, instead of searching for an object that is exactly the same as the identified object, the 3D map generation device may search the object database for an object with the most similar size and shape. This is because the background portion of the enlarged indoor space image is larger than the background portion of the indoor space image, and even if a small object is selected in the object database for configuration, it will not produce visual unnatural phenomena compared to the object existing in the non-background area.

[0150] Finally, in step S670, the three-dimensional map generating device adds at least one virtual object corresponding to the at least one object to the generated three-dimensional map based on the object information.

[0151] That is, the three-dimensional map generating device can complete the three-dimensional map including at least one virtual object by setting at least one virtual object corresponding to at least one object included in the object information in the three-dimensional map.

[0152] In another embodiment, if information related to a new object is added to the object database or information related to a previous virtual object is updated, at least one virtual object added to the three-dimensional map may be selectively updated.

[0153] That is, when a new object is registered or updated in the object database with a time difference, the three-dimensional map generation device can also reflect the new information registered or updated to the virtual object located in the previously generated three-dimensional map. In addition, the three-dimensional map generation device repeatedly performs the above process through a computer algorithm, thereby continuously updating the information of the object located in the three-dimensional map without additional labor.

[0154] Furthermore, when the object database adds information about a new object or updates information about a previous virtual object, the three-dimensional map generation device selectively updates the virtual object information only when the information about the virtual object located in the three-dimensional map needs to be updated.

[0155] For example, the 3D map generation device wants to add object A to the 3D map, but there is no information related to object A in the object database, so a virtual object corresponding to a similar object B can be added to the 3D map. However, if information related to object A is newly added to the object database, the 3D map generation device can replace the virtual object corresponding to object B with the virtual object corresponding to object A.

[0156] Similarly, after adding object A to the three-dimensional map, if the three-dimensional map generating device updates the information related to object A in the object database, the virtual object corresponding to object A can be updated.

[0157] In another embodiment, when the at least one virtual object includes a reflective virtual object which is a virtual object of a material including reflective light, a reflected image related to the indoor space reflected by the reflective virtual object may be displayed on the reflective virtual object.

[0158] The material of the light reflected in the reflective virtual object may be a mirror, glass, metal, etc., but is not limited thereto. In addition, the information related to the shapes and images of the multiple objects included in the object database may also include the material of the light reflected by the multiple objects and the information related to the transmittance and area of ​​the above-mentioned reflective material.

[0159] In this case, if the at least one virtual object includes a reflective virtual object, the three-dimensional map generating device displays a reflected image reflected by the reflective virtual object for a reference posture corresponding to a reference position of the three-dimensional map on a surface of the reflective virtual object.

[0160] For example, when a user who appreciates a three-dimensional map observes the reflective virtual object with the reference posture as the center, the three-dimensional map generating device can also display an image related to the indoor space reflected by the reflective virtual object on the surface of the reflective virtual object. This is because the reflective virtual object included in the object database does not reflect an image related to the indoor space.

[0161] To this end, the three-dimensional map generation device may use at least one enlarged indoor space image and terrain information to form new depth-image association information corresponding to the area of ​​the reflective material contained in the above-mentioned reflective virtual object. Alternatively, the three-dimensional map generation device may extract new depth-image association information corresponding to the area of ​​the reflective material contained in the above-mentioned reflective virtual object from the above-mentioned generated depth-image association information.

[0162] Furthermore, the three-dimensional map generation device can generate a reflection image that needs to be expressed on the reflection material when the user observes the reflection virtual object in the reference posture, taking into account the transmittance of the reflection material based on the new depth-image association information. Furthermore, the three-dimensional map generation device can present the reflection image in the area of ​​the reflection material included in the reflection virtual object.

[0163] More specifically, when the reflective material is a mirror, when the mirror is observed at any position in the indoor space, the three-dimensional map generation device may generate an image projected in the mirror to be displayed on the reflective virtual object. Furthermore, when the reflective material is translucent glass, when the translucent glass is observed at any position in the indoor space, the three-dimensional map generation device overlaps the image reflected in proportion to the transmittance of the translucent glass and the transmitted image to be displayed on the reflective virtual object.

[0164] As described above, when the three-dimensional map generating device continuously changes the user's viewpoint and updates the reflected image in real time on the reflected virtual object, the image of the indoor space mapped by reflective materials such as mirrors can change in the same way as the actual changes, thereby generating a more realistic three-dimensional map.

[0165] Figure 7 The figure is a flow chart for illustrating a method for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0166] In step S710, the three-dimensional map generating apparatus obtains at least one indoor space image which is an image related to the indoor space.

[0167] In step S720, the three-dimensional map generating apparatus distinguishes a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space in at least one indoor space image.

[0168] In step S730, the three-dimensional map generating apparatus generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0169] In step S740, the three-dimensional map generating apparatus generates depth-image association information based on at least one enlarged indoor space image and terrain information related to the indoor space.

[0170] In step S750, the three-dimensional map generating apparatus generates a three-dimensional map related to the indoor space using at least one enlarged indoor space image, terrain information, and depth-image association information.

[0171] In step S760, the three-dimensional map generating apparatus estimates light source information including information on the position and brightness of a light source located in the indoor space from at least one indoor space image.

[0172] That is, the three-dimensional map generating device can estimate the position and brightness of the light source from at least one indoor space image using information on pixel values ​​included in the image.

[0173] For example, the three-dimensional map generating device may estimate a position corresponding to a pixel value within a predetermined range in at least one indoor space image as the position of the light source, and may estimate the brightness of the light source based on the pixel value of the estimated position.

[0174] Finally, in step S770, the three-dimensional map generating device reflects the lighting effect based on the light source information in at least one enlarged indoor space image.

[0175] For example, in the process of the 3D map generation device expanding the background area toward the non-background area to generate the enlarged indoor space image, the lighting effect related to the non-background area may disappear. In this case, the 3D map generation device reflects the lighting effect based on the light source effect in the enlarged indoor space image, thereby generating a more realistic 3D map.

[0176] Furthermore, when at least one virtual object is located in the three-dimensional map, the three-dimensional map generating device may add a shadow related to the at least one virtual object according to the position and brightness of the light source included in the light source information to improve the authenticity of the three-dimensional map.

[0177] In another embodiment, when the three-dimensional map generating device reflects the lighting effect on the three-dimensional map, it is reflected through preliminary work before the user appreciates the map, rather than reflecting it in the real-time environment when the user appreciates the map, thereby reducing the load based on real-time lighting effect calculations.

[0178] As a result, in the case where the three-dimensional map generating apparatus constructs a three-dimensional map using one indoor structure peripheral information and at least one enlarged indoor space image, the real-time load can be reduced in proportion to the number of at least one enlarged indoor space image.

[0179] Figure 8 The block diagram shows a device for generating a three-dimensional map of an indoor space according to an embodiment of the present invention.

[0180] Reference Figure 8 The three-dimensional map generating device 800 of an indoor space according to an embodiment of the present invention includes an acquiring unit 810, a distinguishing unit 820, an expanding unit 830, an associating unit 840, and a generating unit 850. In addition, the device may optionally include a peripheral information generating unit (not shown).

[0181] The acquisition unit 810 acquires at least one indoor space image which is an image related to the indoor space.

[0182] The distinguishing unit 820 is configured to distinguish, in at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space.

[0183] The enlarging unit 830 generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0184] The associating unit 840 generates depth-image association information based on at least one enlarged indoor space image and topographic information related to the indoor space.

[0185] The generating unit 850 generates a three-dimensional map related to the indoor space using at least one enlarged indoor space image, terrain information, and depth-image correlation information.

[0186] Finally, the peripheral information generating unit (not shown) generates indoor space peripheral information as peripheral information related to the indoor space using the depth value information.

[0187] In another embodiment, the acquisition unit 810 further acquires depth value information as information of depth values ​​related to the indoor space, and the terrain information may include the depth value information and the indoor space peripheral information generated by the peripheral information generation unit (not shown).

[0188] In another embodiment, the associating unit 840 updates at least one of the terrain information and the depth-image association information using features included in at least one expanded indoor space image and terrain information, and the generating unit 850 may use at least one of the updated terrain information and depth-image association information.

[0189] In another embodiment, the distinguishing unit 820 uses terrain information and depth-image association information to further distinguish the background area and the non-background area in at least one indoor space image, and the expanding unit 830 regenerates at least one enlarged indoor space image by expanding the background area toward the non-background area in at least one indoor space image. The generating unit 850 can use the regenerated at least one enlarged indoor space image.

[0190] In another embodiment, when the distinguishing unit 820 further distinguishes the background area and the non-background area for the object image as one of the at least one indoor space image, depth-image association information and at least one supplementary image other than the object image in the at least one indoor space image may be used.

[0191] In another embodiment, the generating unit 850 selects an enlarged indoor space image associated with a reference posture in at least one enlarged indoor space image based on the depth-image association information and determines it as a representative image, wherein the reference posture corresponds to a specified reference position on the three-dimensional map, and at least one supplementary image other than the representative image is determined in at least one enlarged indoor space image using the depth-image association information and the reference posture, and a three-dimensional map can be generated using the representative image, the at least one supplementary image, the terrain information and the depth-image association information.

[0192] Fig. 9 It is a block diagram showing a device for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0193] Reference Fig. 9The three-dimensional map generating device 800 of an indoor space according to an embodiment of the present invention includes an acquiring unit 810 , a distinguishing unit 820 , an expanding unit 830 , an associating unit 840 , a generating unit 850 and a database connecting unit 860 .

[0194] The acquisition unit 810 acquires at least one indoor space image which is an image related to the indoor space.

[0195] The distinguishing unit 820 is configured to distinguish, in at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space.

[0196] The enlarging unit 830 generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0197] The associating unit 840 generates depth-image association information based on at least one enlarged indoor space image and topographic information related to the indoor space.

[0198] The generating unit 850 generates a three-dimensional map related to the indoor space using at least one enlarged indoor space image, terrain information, and depth-image correlation information.

[0199] Finally, the database connection unit 860 acquires object information including information related to at least one object from the object database including information related to shapes and images of a plurality of objects.

[0200] In another embodiment, the generating unit 850 can add at least one virtual object corresponding to at least one object to the generated three-dimensional map based on the object information.

[0201] In another embodiment, the object information may be information related to an object located in an indoor space.

[0202] In another embodiment, if information related to a new object is added to the object database or information related to a previous virtual object is updated, the generation unit 850 may selectively update at least one virtual object added to the three-dimensional map. In another embodiment, when the at least one virtual object includes a reflective virtual object as a virtual object of a material containing reflected light, the generation unit 850 uses the at least one enlarged indoor space image and the terrain information to display a reflected image on the reflective virtual object, and the reflected image is an image of the indoor space reflected by the reflective virtual object at a reference posture corresponding to a specified reference position on the three-dimensional map.

[0203] Fig.10 It is a block diagram showing a device for generating a three-dimensional map of an indoor space according to another embodiment of the present invention.

[0204] Reference Fig.10 The three-dimensional map generating device 800 of an indoor space according to an embodiment of the present invention includes an acquiring unit 810 , a distinguishing unit 820 , an expanding unit 830 , an associating unit 840 , a generating unit 850 , a light source estimating unit 870 , and a light source reflecting unit 880 .

[0205] The acquisition unit 810 acquires at least one indoor space image which is an image related to the indoor space.

[0206] The distinguishing unit 820 is configured to distinguish, in at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space.

[0207] The enlarging unit 830 generates at least one enlarged indoor space image by enlarging the background area toward the non-background area in at least one indoor space image.

[0208] The associating unit 840 generates depth-image association information based on at least one enlarged indoor space image and topographic information related to the indoor space.

[0209] The generating unit 850 generates a three-dimensional map related to the indoor space using at least one enlarged indoor space image, terrain information, and depth-image correlation information.

[0210] The light source estimation unit 870 estimates light source information including information on the position and brightness of a light source located in the indoor space from at least one indoor space image.

[0211] Finally, the light source reflecting unit 880 reflects the lighting effect based on the light source information on at least one enlarged indoor space image.

[0212] On the other hand, the above-mentioned embodiments of the present invention can be formulated through a computer-executable program and can be embodied in a general-purpose digital computer running the above-mentioned program using a computer-readable recording medium.

[0213] The computer-readable recording medium includes magnetic storage media (eg, ROM, floppy disks, hard disks, etc.) and optical reading media (eg, CD-ROMs, DVDs, etc.).

[0214] As described above, the present invention is described with the above preferred embodiments as the center. A person skilled in the art of the present invention can realize the present invention in a modified manner within the scope of the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from the perspective of explanation rather than from the perspective of limitation. The scope of the present invention is reflected by the scope of the invention claims, rather than by the above description, and all differences within the equivalent scope belong to the scope of the present invention.

Claims

1. A method for generating a three-dimensional map of an indoor space, characterized in that: include: A step of acquiring at least one indoor space image which is an image related to the indoor space; A step of distinguishing, in the at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space; In the at least one indoor space image, based on the boundary corresponding to the same plane in the terrain information of the background area, the step of generating at least one enlarged indoor space image by enlarging the background area toward the non-background area; A step of generating depth-image association information based on terrain information including the at least one enlarged indoor space image and information about a depth value related to the indoor space; as well as The step of generating a three-dimensional map related to the indoor space by using the at least one enlarged indoor space image, the terrain information and the depth-image association information.

2. The method for generating a three-dimensional map of an indoor space according to claim 1, characterized in that: Before the above step of distinguishing the non-background area, the following steps are further included: A step of acquiring depth value information as information of a depth value related to the indoor space; and a step of generating indoor space peripheral information as peripheral information related to the indoor space using the depth value information, The terrain information includes the depth value information and the indoor space periphery information.

3. The method for generating a three-dimensional map of an indoor space according to claim 1, characterized in that: After the step of generating the depth-image association information, the method further comprises the step of updating at least one of the terrain information and the depth-image association information using features included in the at least one enlarged indoor space image and the terrain information. The step of generating the three-dimensional map utilizes at least one of the updated terrain information and the depth-image association information.

4. The method for generating a three-dimensional map of an indoor space according to claim 1, characterized in that: The step of generating the depth-image association information and the step of generating the three-dimensional map related to the indoor space include: A step of further distinguishing the background area and the non-background area in the at least one indoor space image by using the terrain information and the depth-image association information; and In the at least one indoor space image, at least one enlarged indoor space image is regenerated by enlarging the background area toward the non-background area. The step of generating a three-dimensional map uses the at least one regenerated enlarged indoor space image.

5. The method for generating a three-dimensional map of an indoor space according to claim 4, characterized in that: In the step of re-distinguishing the above-mentioned background area and the above-mentioned non-background area, when re-distinguishing the above-mentioned background area and the above-mentioned non-background area for the object image which is one of the above-mentioned at least one indoor space image, at least one supplementary image other than the above-mentioned object image in the above-mentioned depth-image association information and the above-mentioned at least one indoor space image is also used.

6. The method for generating a three-dimensional map of an indoor space according to claim 1, characterized in that: The steps of generating a three-dimensional map related to the indoor space include: A step of selecting an enlarged indoor space image associated with a reference posture from the at least one enlarged indoor space image based on the depth-image association information and determining the image as a representative image, wherein the reference posture corresponds to a predetermined reference position on the three-dimensional map; A step of determining at least one supplementary image other than the representative image in the at least one enlarged indoor space image using the depth-image association information and the reference pose; and The step of generating a three-dimensional map using the representative image, the at least one supplementary image, the terrain information and the depth-image association information.

7. The method for generating a three-dimensional map of an indoor space according to claim 1, characterized in that: Also includes: a step of acquiring object information including information related to at least one object from an object database including information related to shapes and images of a plurality of objects; as well as The step of adding at least one virtual object corresponding to the at least one object to the generated three-dimensional map based on the object information.

8. The method for generating a three-dimensional map of an indoor space according to claim 7, characterized in that: The object information is information related to an object located in the indoor space.

9. The method for generating a three-dimensional map of an indoor space according to claim 7, characterized in that: The method further includes the step of selectively updating the at least one virtual object added to the three-dimensional map if information related to an object is added to the object database or information related to a previous virtual object is updated.

10. The method for generating a three-dimensional map of an indoor space according to claim 7, characterized in that: It also includes the following step, namely, when a reflective virtual object as a virtual object of a material containing reflected light is included in the above-mentioned at least one virtual object, a reflected image is displayed on the above-mentioned reflective virtual object using the above-mentioned at least one enlarged indoor space image and the above-mentioned terrain information, and the above-mentioned reflected image is an image of the above-mentioned indoor space reflected by the above-mentioned reflective virtual object at a reference posture corresponding to a specified reference position on the above-mentioned three-dimensional map.

11. The method for generating a three-dimensional map of an indoor space according to claim 1, characterized in that: Also includes: A step of estimating light source information from the at least one indoor space image, wherein the light source information includes information related to the position and brightness of the light source located in the indoor space; as well as In the at least one step of enlarging the indoor space image to reflect the lighting effect based on the light source information.

12. A device for generating a three-dimensional map of an indoor space, characterized in that: include: An acquisition unit, configured to acquire at least one indoor space image as an image related to the indoor space; a distinguishing unit configured to distinguish, in the at least one indoor space image, a background region corresponding to a structure of the indoor space and a non-background region corresponding to an object located in the indoor space; an enlarging unit, generating at least one enlarged indoor space image by enlarging the background area toward the non-background area based on a boundary corresponding to a same plane in the terrain information of the background area in the at least one indoor space image; a correlating unit that generates depth-image correlation information based on terrain information including the at least one enlarged indoor space image and information about a depth value related to the indoor space; as well as The generating unit generates a three-dimensional map related to the indoor space by using the at least one enlarged indoor space image, the topographic information, and the depth-image correlation information.

13. The device for generating a three-dimensional map of an indoor space according to claim 12, characterized in that: The acquisition unit further acquires depth value information which is information about a depth value related to the indoor space. The invention further comprises a peripheral information generating unit which generates indoor space peripheral information as peripheral information related to the indoor space by using the depth value information. The terrain information includes the depth value information and the indoor space periphery information.

14. The device for generating a three-dimensional map of an indoor space according to claim 12, characterized in that: The associating unit updates at least one of the terrain information and the depth-image association information using features included in the at least one enlarged indoor space image and the terrain information, The generating unit uses at least one of the updated topographic information and the depth-image correlation information.

15. The device for generating a three-dimensional map of an indoor space according to claim 12, characterized in that: The segmentation unit further segments the background area and the non-background area in the at least one indoor space image using the terrain information and the depth-image correlation information. The enlarging unit regenerates at least one enlarged indoor space image by enlarging the background area toward the non-background area in the at least one indoor space image. The generating unit uses the at least one regenerated enlarged indoor space image.

16. The device for generating a three-dimensional map of an indoor space according to claim 15, characterized in that: When further distinguishing the background area and the non-background area for an object image that is one of the at least one indoor space image, the distinguishing unit further utilizes at least one supplementary image other than the object image in the depth-image association information and the at least one indoor space image.

17. The device for generating a three-dimensional map of an indoor space according to claim 12, characterized in that: The generating unit selects an enlarged indoor space image associated with a reference posture from the at least one enlarged indoor space image based on the depth-image association information and determines the image as a representative image, wherein the reference posture corresponds to a predetermined reference position on the three-dimensional map. determining at least one supplementary image other than the representative image in the at least one enlarged indoor space image by using the depth-image association information and the reference pose, A three-dimensional map is generated using the representative image, the at least one supplementary image, the terrain information and the depth-image association information.

18. The device for generating a three-dimensional map of an indoor space according to claim 12, characterized in that: The invention also includes a database connection unit for acquiring object information including information related to at least one object from an object database including information related to shapes and images of a plurality of objects, The generating unit adds at least one virtual object corresponding to the at least one object to the generated three-dimensional map based on the object information.

19. The device for generating a three-dimensional map of an indoor space according to claim 18, characterized in that: The object information is information related to an object located in the indoor space.

20. The device for generating a three-dimensional map of an indoor space according to claim 18, characterized in that: If information related to an object is added to the object database or information related to a previous virtual object is updated, the generating unit selectively updates the at least one virtual object added to the three-dimensional map.

21. The device for generating a three-dimensional map of an indoor space according to claim 18, characterized in that: When the at least one virtual object includes a reflective virtual object which is a virtual object made of a material containing reflected light, the generating unit displays a reflected image on the reflective virtual object using the at least one enlarged indoor space image and the terrain information, and the reflected image is an image of the indoor space reflected by the reflective virtual object at a reference posture corresponding to a specified reference position on the three-dimensional map.

22. The device for generating a three-dimensional map of an indoor space according to claim 12, characterized in that: Also includes: a light source estimating unit that estimates light source information from the at least one indoor space image, wherein the light source information includes information related to the position and brightness of the light source located in the indoor space; as well as The light source reflecting unit reflects a lighting effect based on the light source information on the at least one enlarged indoor space image.

Images