Method for generating indoor floor plan based on image and computing device using same

Abstract

The present invention relates to a method for generating an indoor floor plan based on an image and a computing device using same, and the method for generating an indoor floor plan based on an image, using a computing device, according to an embodiment of the present invention, may comprise the steps of: receiving an upper image obtained by capturing an upper area of a target space; performing image processing on each frame image included within the upper image, to extract a plurality of line segments included within the frame image; dividing the plurality of line segments into vertical line segments corresponding to the height direction of the target space and horizontal line segments corresponding to the breadth or width direction within the target space; extracting intersection points at which the horizontal line segments are orthogonal to one another, generating an outline shape connecting the intersection points, and detecting a unit space corresponding to the outline shape; from among a plurality of horizontal line segments included within the frame image, extracting a horizontal line segment generating intersection points with two vertical line segments, and detecting same as a passage; and generating an indoor floor plan for the target space by connecting respective unit spaces extracted from the upper image by means of the passage.

Description

Image-based indoor floor plan generation method and computing device using the same

[0001] The present invention relates to an image-based interior floor plan generation method capable of automatically generating an interior floor plan of a building from an image taken of the interior space of the building or the like, and a computing device using the same.

[0002] Typically, real estate brokerage involves a seller entrusting a property to a licensed real estate agency. The buyer then visits the agency to confirm the property's location and terms, including the selling price, and concludes a contract between the two parties. Because real estate transactions are largely conducted offline, buyers must physically visit the property to purchase it, making it difficult to obtain information on properties with better terms.

[0003] To improve this, methods are being proposed to share real estate information via the Internet, and information that can be used to understand the internal structure of a building, such as interior floor plans, is also being provided.

[0004] In this way, the interior floor plans of buildings, houses, apartments, etc. can be utilized for interior design or furniture layout plans, so they can be useful to users who wish to purchase, use, or profit from real estate.

[0005] However, since previously, floor plans were created directly based on the actual dimensions measured by the worker, there were problems such as a lot of time and cost being required to create each floor plan.

[0006] The present invention aims to provide an image-based indoor floor plan generation method capable of automatically generating an indoor floor plan for a target space from an image taken of the target space, and a computing device using the same.

[0007] The present invention aims to provide an image-based indoor floor plan generation method capable of automatically generating an indoor floor plan using line segments extracted from an image, and a computing device using the same.

[0008] The present invention provides an image-based indoor floor plan generation method capable of automatically generating an indoor floor plan using a three-dimensional mesh and line segments generated from an image, and a computing device using the same.

[0009] According to one embodiment of the present invention, a method for generating an indoor floor plan based on an image using a computing device may include the steps of: receiving an upper image in which an upper area of ​​a target space is captured; performing image processing on each frame image included in the upper image to extract a plurality of line segments included in the frame image; dividing the plurality of line segments into vertical line segments corresponding to a height direction of the target space and horizontal line segments corresponding to a width or width direction within the target space; extracting intersection points where the horizontal line segments are orthogonal, generating an outline shape connecting the intersection points, and detecting a unit space corresponding to the outline shape; extracting a horizontal line segment that creates an intersection point with two vertical line segments from among the plurality of horizontal line segments included in the frame image, and detecting the horizontal line segment as a passage; and connecting each of the unit spaces extracted from the upper image with the passage to generate an indoor floor plan for the target space.

[0010] According to one embodiment of the present invention, a method for generating an indoor floor plan based on an image using a computing device may include the steps of: receiving an image of a target space; generating a three-dimensional mesh corresponding to the target space from the image using a deep learning network; generating an initial floor plan by projecting the three-dimensional mesh onto a two-dimensional plane corresponding to a ceiling or a floor of the target space; detecting ceiling lines located on the ceiling of the target space from each frame image included in the image; generating an intermediate floor plan by dividing the initial floor plan into a plurality of unit spaces by reflecting the ceiling lines in the initial floor plan; and generating an indoor floor plan corresponding to the target space by simplifying an outline of each unit space included in the intermediate floor plan and setting a point where the unit spaces meet as a passage.

[0011] A computing device for generating an indoor floor plan based on an image according to one embodiment of the present invention includes a processor, wherein the processor may perform the following operations: receiving an upper image in which an upper region of a target space is captured; performing image processing on each frame image included in the upper image to extract a plurality of line segments included in the frame image; dividing the plurality of line segments into vertical line segments corresponding to a height direction of the target space and horizontal line segments corresponding to a width or width direction within the target space; extracting intersection points where the horizontal line segments are orthogonal, generating an outline shape connecting the intersection points, and detecting a unit space corresponding to the outline shape; extracting a horizontal line segment that generates an intersection point with two vertical line segments from among the plurality of horizontal line segments included in the frame image, and detecting the horizontal line segment as a passageway; and connecting each of the unit spaces extracted from the upper image with the passageway to generate an indoor floor plan for the target space.

[0012] A computing device for generating an indoor floor plan based on an image according to one embodiment of the present invention includes a processor, wherein the processor may perform the following operations: receiving an image capturing a target space; generating a three-dimensional mesh corresponding to the target space from the image using a deep learning network; projecting the three-dimensional mesh onto a two-dimensional plane corresponding to a ceiling or a floor of the target space to generate an initial floor plan; detecting ceiling lines located on the ceiling of the target space from each frame image included in the image; generating an intermediate floor plan by dividing the initial floor plan into a plurality of unit spaces by reflecting the ceiling lines in the initial floor plan; and simplifying an outline of each unit space included in the intermediate floor plan and setting a point where the unit spaces meet as a passageway to generate an indoor floor plan corresponding to the target space.

[0013] Additionally, the solutions to the aforementioned problems do not enumerate all features of the present invention. The various features of the present invention, along with their corresponding advantages and effects, can be understood in more detail by referring to the specific embodiments below.

[0014] According to an image-based indoor floor plan generation method and a computing device using the same according to one embodiment of the present invention, an indoor floor plan for a target space can be automatically generated from an image, thereby saving cost and time for floor plan production.

[0015] According to an image-based indoor floor plan generation method and a computing device using the same according to one embodiment of the present invention, it is possible to produce a floor plan with high accuracy by determining the location of a passage included in an image based on a depth image.

[0016] According to an embodiment of the present invention, an image-based indoor floor plan generation method and a computing device utilizing the same, an indoor floor plan can be generated by utilizing a 3D mesh or the like generated for 3D modeling of a target space. In other words, since 3D modeling and indoor floor plan generation can be performed simultaneously, it is possible to generate an indoor floor plan more efficiently.

[0017] However, the effects that can be achieved by the image-based indoor floor plan generation method according to embodiments of the present invention and the computing device using the same are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below.

[0018] Figure 1 is an exemplary diagram showing ERP image generation using a 360-degree monocular camera according to one embodiment of the present invention.

[0019] Figure 2 is a block diagram showing an indoor floor plan generation device according to one embodiment of the present invention.

[0020] Figure 3 is an exemplary diagram showing an upper image of a target space according to one embodiment of the present invention.

[0021] Figure 4 is an exemplary diagram showing line segment detection and unit space detection according to one embodiment of the present invention.

[0022] Figure 5 is an exemplary diagram showing the outline of an open geometric shape according to one embodiment of the present invention.

[0023] Figure 6 is an exemplary diagram showing passage detection according to one embodiment of the present invention.

[0024] Figure 7 is an exemplary diagram showing an indoor floor plan according to one embodiment of the present invention.

[0025] Figure 8 is a block diagram showing an indoor floor plan generation device according to another embodiment of the present invention.

[0026] Figure 9 is an exemplary diagram showing a three-dimensional mesh generated by one embodiment of the present invention and an initial plan view obtained by projecting the same.

[0027] Figure 10 is an exemplary diagram showing a ceiling line segment detection according to one embodiment of the present invention and an intermediate plane diagram generated using the same.

[0028] Figure 11 is an exemplary diagram showing an outline simplification according to one embodiment of the present invention and an indoor floor plan created using the same.

[0029] Figure 12 is a block diagram showing a computing device according to one embodiment of the present invention.

[0030] Figure 13 is a flowchart showing a method for generating an image-based indoor floor plan according to one embodiment of the present invention.

[0031] Figure 14 is a flowchart showing a method for generating an image-based indoor floor plan according to another embodiment of the present invention.

[0032] Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are given or used interchangeably only for the convenience of writing the specification, and do not have distinct meanings or roles in themselves. That is, the term "part" used in the present invention means a hardware component such as software, FPGA, or ASIC, and the "part" performs certain roles. However, the "part" is not limited to software or hardware. The "part" may be configured to be on an addressable storage medium, or may be configured to reproduce one or more processors. Thus, as an example, a 'part' may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or further separated into additional components and 'parts'.

[0033] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0034]

[0035] Figure 1 is an exemplary diagram showing the creation of an ERP (Equirectangular Projection) image using a 360-degree monocular camera (Omnidirectional Camera) according to one embodiment of the present invention.

[0036] Referring to Fig. 1, the ERP image may be a video captured using a 360-degree monocular camera (1). At this time, the 360-degree monocular camera (1) may generate and provide information such as the camera height at the time of capturing the ERP image and the location information within the target space. Here, the target space (T) captured using the 360-degree monocular camera (1) may be an indoor space such as a building, apartment, or house.

[0037] The 360-degree monocular camera (1) can be configured independently, but depending on the embodiment, it can also be implemented by being combined with various terminal devices such as a smartphone, tablet PC, PDA (Personal Digital Assistant), laptop computer, wearable device, etc. In other words, an ERP image of the target space (T) can be created using the 360-degree monocular camera (1) equipped on one's own smartphone, etc.

[0038] Thereafter, the user can transmit the captured ERP image to an indoor floor plan generation device via a wired or wireless network, and request generation of an indoor floor plan for a target space (T) based on the captured ERP image. That is, in order to generate an indoor floor plan for a target space (T), the user can transmit the ERP image to the indoor floor plan generation device and request generation of a corresponding indoor floor plan.

[0039] Meanwhile, a user can connect to a network using his / her terminal device and communicate with an indoor floor plan generation device via the network. That is, the user can use the terminal device to transmit ERP images, etc. captured by a 360-degree monocular camera (1), to the indoor floor plan generation device, and the indoor floor plan generation device can generate an indoor floor plan for the target space (T) based on the received ERP images. Thereafter, the generated indoor floor plan can be provided along with information about the building, house, or other property for real estate transactions, etc.

[0040] Here, the communication method between the terminal device and the indoor floor plan generation device is not limited, and may include not only a communication method utilizing a communication network that the network may include (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.), but also short-range wireless communication between devices. For example, the network may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. In addition, the network may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0041] Additionally, although it is illustrated here that an ERP image is created using a 360-degree monocular camera (1), depending on the embodiment, it is also possible to utilize a panoramic image created using a panorama mode of a general camera, etc.

[0042] Meanwhile, interior floor plans for buildings, houses, and apartments can be utilized for interior design and furniture layout planning, making them a valuable resource for those seeking to purchase, use, or profit from real estate. However, previously, floor plans were created manually by workers based on actual measurements, resulting in significant time and expense for each floor plan.

[0043] On the other hand, by utilizing an indoor floor plan generation device according to one embodiment of the present invention, it is possible to automatically generate an indoor floor plan based on an image of a target space (T), thereby saving time and cost required for floor plan production. Hereinafter, indoor floor plan generation devices according to one embodiment of the present invention will be described with reference to FIGS. 2 and 8.

[0044]

[0045] Figure 2 is a block diagram illustrating an indoor floor plan generation device according to one embodiment of the present invention. Referring to Figure 2, an indoor floor plan generation device (100) according to one embodiment of the present invention may include a receiving unit (110), a posture estimation unit (120), a line segment extraction unit (130), a line segment classification unit (140), a unit space detection unit (150), a passage detection unit (160), and an indoor floor plan generation unit (170).

[0046] The receiving unit (110) can receive an upper image in which the upper area of ​​the target space (T) is captured. That is, since it is possible to create an indoor floor plan from the shape of the ceiling or floor surface of the target space (T), an upper image of the target space (T) can be received and utilized through the receiving unit (110).

[0047] FIG. 3 is an exemplary diagram showing an upper image of a target space (T). As shown in FIG. 3, the ceiling surface can minimize an area covered by other objects or people compared to the floor surface. Accordingly, the receiving unit (110) can receive an upper image of the target space (T) and generate an indoor floor plan from the received upper image. However, depending on the embodiment, it is also possible to receive a lower image in which a lower area of ​​the target space (T) is captured through the receiving unit (110) and generate an indoor floor plan based on the received lower image.

[0048] Here, the upper image can be any image that captures the upper region of the target space (T), but depending on the embodiment, it may be generated by extracting a vertical image for the target space (T) from an ERP (Equirectangular Projection) image that captures the target space (T). There may be cases where an ERP image is captured to create a three-dimensional model for the target space (T), and since the ERP image captures the target space (T) in 360 degrees, the upper region of the target space (T) may also be included in the ERP image. Therefore, after receiving the ERP image, it is also possible to generate the upper image by extracting a region corresponding to the vertically upward direction from the ERP image. Depending on the embodiment, it is also possible to extract and utilize the upper image from a panoramic image or the like that captures the target space (T). Meanwhile, it is also possible for the receiving unit (110) to receive an ERP image or a panoramic image and extract an upper image therefrom.

[0049] The attitude estimation unit (120) can generate attitude information of the photographing device that captured the corresponding frame image by applying the Structure-from-Motion (SfM) technique to each frame image included in the upper image. That is, by utilizing the SfM technique, it is possible to generate attitude information such as the shooting angle or shooting direction of the photographing device that generated the corresponding upper image. Accordingly, the attitude estimation unit (120) can generate attitude information for each frame image included in the upper image, and thereafter, the attitude information can be used to create a unit space or combine a unit space and a passageway for creating an indoor floor plan. Here, the case where the attitude estimation unit (120) utilizes the SfM technique is described as an example, but any technique other than SfM that can estimate attitude information of the photographing device from a frame image can be applied.

[0050] Additionally, the posture estimation unit (120) can also generate location information about the location of the photographing device that captured each frame image within the target space (T). Depending on the embodiment, the user may separately input location information at the time the corresponding frame image was captured within the target space (T), or the photographing device may independently generate location information based on GPS coordinates, wireless LAN connection information, etc.

[0051] The line segment extraction unit (130) can perform image processing on each frame image included in the upper image to extract multiple line segments included in the frame image. Here, the line segment extraction unit (130) can extract each line segment included in the frame image by utilizing various straight line detection algorithms such as Hough transform. That is, as illustrated in Fig. 4(a), the line segment extraction unit (130) can extract each line segment included in the frame image.

[0052] The line segment classification unit (140) can classify a plurality of line segments into vertical line segments and horizontal line segments. Here, the vertical line segment is a line segment corresponding to the height direction of the target space (T), and the horizontal line segment may be a line segment corresponding to the width or depth direction of the target space (T). That is, the line segment classification unit (140) can classify a vertical line segment representing the shape of a column or side surface within the target space (T), and a horizontal line segment representing the shape of a ceiling surface within the target space (T).

[0053] At this time, the line segment classification unit (140) can distinguish each vertical line segment and horizontal line segment based on the Manhattan world assumption for the target space (T). The Manhattan world assumption is that each surface within the indoor space is orthogonal to each other. When the Manhattan world assumption is applied, the ceiling and side surfaces within the target space (T) are orthogonal to each other, the ceiling and floor surfaces are parallel to each other, and each side surface can also be seen as being orthogonal to each other.

[0054] Specifically, the line segment classification unit (140) can generate a virtual extension line for each line segment extracted from the frame image, and if there is a line segment among the line segments whose virtual extension line passes through the principal point of the corresponding frame image, the line segment can be classified as a vertical line segment. That is, if the upper image is a vertically upward shot of the ceiling, the vanishing point according to the perspective can be located at the principal point of the corresponding frame image. Accordingly, the line segment classification unit (140) can determine the line segment whose virtual extension line passes through the principal point as a vertical line segment generated long along the height direction of the target space (T), such as a pillar or side within the target space (T). In addition, the remaining line segments excluding the vertical line segments can be classified as horizontal line segments.

[0055] Here, the line segment classification unit (140) can cluster horizontal line segments by each angle, and classify each clustered horizontal line segment into a group that is orthogonal to each other. That is, since a two-dimensional indoor floor plan can be displayed on the xy plane, each horizontal line segment can be classified in advance into a group that is orthogonal to each other so that the horizontal line segments can be aligned to the x-axis and the y-axis, respectively.

[0056] Referring to Fig. 4(a), one can identify vertical segments (V) of a virtual extension line passing through the principal point of the corresponding frame image, and horizontal segments (H1, H2) of a virtual extension line not passing through the principal point. Here, the first horizontal segment (H1) and the second horizontal segment (H2) can be orthogonal to each other, and groups of the first horizontal segments (H1) and the second horizontal segments (H2) can be created and classified.

[0057] The unit space detection unit (150) can extract the intersection points where the horizontal lines intersect each other within the frame image, generate an outline shape by connecting the intersection points, and detect the unit space corresponding to the outline shape. That is, in Fig. 4(a), the intersection points where the orthogonal horizontal lines intersect can be extracted, and by connecting each intersection point, it is possible to generate an outline shape (B) as in Fig. 4(b). Here, the outline shape can correspond to the shape of the unit space included in the indoor floor plan.

[0058] However, depending on the embodiment, there may be cases where the outline shape does not form a closed shape. As illustrated in FIG. 5(a) and FIG. 5(b), outline shapes (B1, B2) corresponding to each frame image can be generated, but it can be confirmed that these outline shapes (B1, B2) correspond to open shapes. That is, if the ceiling surface of the corresponding unit space is not completely represented within a single frame image, the outline shapes (B1, B2) do not appear as closed shapes.

[0059] In this case, since the corresponding unit space cannot be specified, the unit space detection unit (150) can match corresponding intersection points among intersection points included in other frame images to generate an outline shape as a closed geometric shape. At this time, the unit space detection unit (150) can utilize each attitude information and position information generated by the attitude estimation unit (120). That is, intersection points can be extracted from other frame images with different shooting angles or shooting directions within the same unit space, and intersection points that match each other can be found based on attitude information such as shooting angles and shooting directions to generate an outline shape for the entire unit space.

[0060] In some embodiments, the intersection points extracted from each frame image may be transferred to a global coordinate system, and intersection points located at similar locations within the global coordinate system (e.g., intersection points within a set distance range) may be merged into one. Then, by connecting each intersection point, it is possible to generate a closed outline shape for each unit space. Here, when transferring the intersection points to the global coordinate system, the detailed information and the location information for each frame image may be utilized.

[0061] The passage detection unit (160) can extract a horizontal line segment that intersects two vertical line segments from among a plurality of horizontal line segments included in a frame image and detect it as a passage. The passage is formed on a side within the target space (T) and may connect different unit spaces. For example, there may be a door connecting a living room and a bedroom, and the area where the door is located can be detected as a passage.

[0062] Specifically, as illustrated in FIG. 6(a), the frame image may include a "ㄷ" shape formed by two vertical line segments (V1, V2) and one horizontal line segment (P) that creates an intersection, and the horizontal line segment included in the "ㄷ" shape may be determined as a passage (P). However, some of the "ㄷ" shapes appearing in the frame image may not be passages. In this case, the passage detection unit (160) may additionally utilize a depth image to detect an actual passage.

[0063] Specifically, the passage detection unit (160) can first extract, from among the horizontal line segments included in the frame image, a horizontal line segment that creates an intersection with two vertical line segments as a passage candidate. Here, the passage detection unit (160) can detect all of the passage candidates included in the frame image to form a passage candidate group.

[0064] Thereafter, the passage detection unit (160) can generate a depth image corresponding to the frame image, as illustrated in FIG. 6(b), and obtain the depth value of the passage corresponding to the passage candidate (P) and the depth values ​​of the two vertical line segments (V1, V2) from the depth image, respectively. Here, the passage detection unit (160) can directly generate the depth image corresponding to the frame image by utilizing a depth estimation model, or, depending on the embodiment, can receive the depth image from an external source. The depth estimation model can be implemented in various ways, such as a deep learning model or a neural network model, and any model that generates a distance value corresponding to each pixel included in the input frame image can be utilized.

[0065] Meanwhile, the depth value of the passage may be obtained from the depth value of the center line (L) that divides the passage area into two, as illustrated in Fig. 6(c). Here, the passage area is an area surrounded by the passage candidate (P) and two vertical line segments (V1, V2), and the center line (L) can be specified as passing through the principal point of the frame image while dividing the passage area into two equal-sized areas. Thereafter, the depth value corresponding to the center line (L) can be extracted from the depth image and set as the depth value of the passage. For example, the average value of the depth values ​​of the pixels corresponding to the center line (L) can be set as the depth value of the passage as a representative value. However, the present invention is not limited thereto, and depending on the embodiment, the depth value of the passage may be set by calculating the representative value in various ways, such as the maximum value, minimum value, or mode, in addition to the average value. In addition, the depth values ​​of the two vertical line segments (V1, V2) can also be calculated and set as representative values ​​in the same manner.

[0066] Here, if the passage area corresponds to an actual passage, the interior of the passage area is empty, so the depth value of the passage may be greater than the depth values ​​of the two vertical line segments (V1, V2). Accordingly, the passage detection unit (160) can detect the passage candidate (P) as a passage if the depth value of the passage is greater than the depth values ​​of the vertical line segments by a threshold value or more. The passage detection unit (160) can individually determine whether each of the entire extracted passage candidate groups corresponds to a passage.

[0067] The indoor floor plan generation unit (170) can generate an indoor floor plan for the target space (T) by connecting each unit space extracted from the upper image with a passageway. That is, as illustrated in FIG. 7, the indoor floor plan can be generated by connecting each unit space (S1, S2, S3, S4, S5, S6, S7) and passageways (P1, P2, P4, P5, P6, P7) based on the detailed information and positional information for each frame image. Here, the passageway can be distinguished by indicating it with a separate identification object such as a thick line segment. However, the present invention is not limited thereto, and the passageway can be indicated using identification objects of various colors or shapes.

[0068] Meanwhile, depending on the embodiment, the indoor floor plan generation unit (170) may also convert and display the intersections and passageways extracted from each frame image into a global coordinate system based on depth values, detail information, and location information. In this case, intersections or passageways located at similar locations on the global coordinate system may be merged into one intersection and one passageway, respectively. Thereafter, a two-dimensional indoor floor plan may be generated by removing the z-axis coordinate corresponding to the depth direction from the global coordinates of each unit space and passageway, and the indoor floor plan may also be generated by indicating the passageway as an identification object such as a thick line segment.

[0069]

[0070] Fig. 8 is a block diagram showing an indoor floor plan generation device according to another embodiment of the present invention. Referring to Fig. 8, an indoor floor plan generation device (200) according to another embodiment of the present invention may include a receiving unit (210), a posture estimation unit (220), a mesh generation unit (230), an initial floor plan generation unit (240), a ceiling line segment detection unit (250), an intermediate floor plan generation unit (260), and an indoor floor plan generation unit (270).

[0071] The receiving unit (210) can receive an image captured of the target space (T). Here, rather than an upper image captured of the upper area of ​​the target space (T), an image captured of the entire target space (T) can be received, and depending on the embodiment, the image can be received in the form of an ERP image or a panoramic image.

[0072] The posture estimation unit (220) can generate posture information of the camera device that captured the corresponding frame image by applying the Structure-from-Motion (SfM) technique to each frame image included in the video. That is, by utilizing the SfM technique, it is possible to generate posture information such as the shooting angle or shooting direction of the camera device that generated the corresponding image, such as a 360-degree monocular camera (1). Accordingly, the posture estimation unit (220) can generate posture information for each frame image included in the video. Additionally, the posture estimation unit (220) can also generate location information about the location of the camera device that captured each frame image within the target space (T). Depending on the embodiment, it is possible for a user to separately input location information at the time of capturing the corresponding frame image within the target space (T), or for the camera device to generate location information on its own based on GPS coordinates, wireless LAN connection information, etc.

[0073] The mesh generation unit (230) can generate a 3D mesh corresponding to the target space from the image using a deep learning network. Here, the deep learning network may generate a 3D mesh based on neural rendering such as the MVS (Multi View Stereo) algorithm, NeuS (Neural Surface Reconstruction), or mono-SDF (Signed Distance Function). However, the present invention is not limited thereto, and any model or algorithm can be used as long as it generates a 3D mesh for the target space (T) from each frame image and the posture information of the shooting device. The mesh generation unit (230) can generate a 3D mesh (M) for the target space (T), as illustrated in FIG. 9 (a).

[0074] The initial floor plan generation unit (240) can generate the initial floor plan by projecting the three-dimensional mesh (M) onto a two-dimensional plane corresponding to the ceiling or floor of the target space (T). That is, as illustrated in Fig. 9(b), the three-dimensional mesh (M) can be projected onto a two-dimensional plane to generate a two-dimensional initial floor plan (F1). Here, in the case of the initial floor plan (F1), each unit space is not distinguished, the passage connecting the unit spaces is unclear, and there may be an error in the outline shape of the initial floor plan (F1). Therefore, the interior floor plan can be generated by modifying it based on the initial floor plan (F1).

[0075] The ceiling line segment detection unit (250) can detect ceiling lines located on the ceiling surface of the target space from each frame image included in the video. That is, in order to distinguish each unit space included in the initial floor plan (F1) and specify a passage connecting the unit spaces, ceiling lines that serve as a reference for distinguishing each unit space can be extracted.

[0076] Specifically, the ceiling line segment detection unit (250) can first extract an upper frame image located in the vertical direction of the target space (T) from the frame image. The frame image corresponds to an ERP image or a panoramic image, and may include a floor surface or a side surface in addition to the ceiling surface of the target space (T). Therefore, the ceiling line segment detection unit (250) can separately extract an upper frame image from the frame image in order to detect a ceiling line included in the ceiling surface. In this case, as illustrated in Fig. 10(a), an upper frame image can be generated.

[0077] Thereafter, the ceiling line segment detection unit (250) can perform image processing on the upper frame image to extract a plurality of line segments. That is, the line segments included in the upper frame image can be extracted based on various straight line detection algorithms such as Hough transform. Here, the ceiling line segment detection unit (250) can extract the ceiling line segments (C) from among the plurality of line segments by filtering based on the depth value corresponding to the ceiling surface of the target space (T). That is, since it is possible to measure each depth value from the 3D mesh (M), the depth value corresponding to the ceiling surface in the target space (T) and the depth value corresponding to each line segment can be obtained. Therefore, by filtering each line segment based on the height of the ceiling surface, it is possible to extract only the ceiling line segments (C) located on the actual ceiling surface, as illustrated in Fig. 10(a).

[0078] The intermediate floor plan generation unit (260) can generate an intermediate floor plan (F2) that divides the initial floor plan (F1) into a plurality of unit spaces by reflecting ceiling line segments (C) in the initial floor plan (F1). That is, the initial floor plan (F1) of a shape in which one space is connected can be cut by the ceiling line segment (C) to generate an intermediate floor plan (F2) of a shape divided into a plurality of unit spaces. Here, the ceiling components (C) extracted from each upper frame image can be converted into a global coordinate system corresponding to the initial floor plan (F1), and the positions of each ceiling component (C) can be reflected in the initial floor plan (F1) and processed as empty space. In this case, as illustrated in Fig. 10(b), it is possible to generate an intermediate floor plan (F2) that is cut into unit space units.

[0079] The indoor floor plan generation unit (270) can simplify the outline of each unit space included in the intermediate floor plan (F2). That is, the indoor floor plan generation unit (270) can detect the outline of each unit space separated from the intermediate floor plan (F2), and can simplify the outline of each unit space vertically or horizontally based on the Manhattan world assumption for the target space (T). According to the Manhattan world assumption, since all surfaces in the indoor space are orthogonal, all outlines can appear vertically or horizontally. Therefore, the indoor floor plan generation unit (270) can simplify the outlines shown in the intermediate floor plan (F2) to generate the second intermediate floor plan (F3) shown in Fig. 11(a).

[0080] Thereafter, the indoor floor plan generation unit (270) can generate an indoor floor plan (F4) corresponding to the target space (T) by setting the points where the unit spaces meet in each of the second intermediate floor plans (F3) as passageways. At this time, the indoor floor plan generation unit (270) can specify the first point of contact between the unit spaces through a morphology operation between the unit spaces and set the specified point as the passageway. In addition, depending on the embodiment, the indoor floor plan generation unit (270) can set the passageway by extracting a horizontal line segment that creates an intersection with two vertical line segments, similar to the passage detection unit (160). Through this, the indoor floor plan generation unit (270) can generate the indoor floor plan (F4), as illustrated in Fig. 11(b).

[0081]

[0082] Figure 12 is a block diagram illustrating a computing environment (10) suitable for use in exemplary embodiments. In the illustrated embodiment, each component may have different functions and capabilities other than those described below, and may include additional components other than those described below.

[0083] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a device for generating an image-based indoor floor plan (e.g., an indoor floor plan generating device (100, 200)).

[0084] A computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may cause the computing device (12) to operate according to the exemplary embodiments mentioned above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, which, when executed by the processor (14), may be configured to cause the computing device (12) to perform operations according to the exemplary embodiments.

[0085] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by the processor (14). In one embodiment, the computer-readable storage medium (16) may be a memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, any other form of storage medium that can be accessed by the computing device (12) and store desired information, or a suitable combination thereof.

[0086] A communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and computer-readable storage media (16).

[0087] The computing device (12) may also include one or more input / output interfaces (22) that provide interfaces for one or more input / output devices (24) and one or more network communication interfaces (26). The input / output interfaces (22) and the network communication interfaces (26) are connected to the communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) via the input / output interfaces (22). Exemplary input / output devices (24) may include input devices such as pointing devices (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touchpad or a touchscreen), a voice or sound input device, various types of sensor devices and / or photographing devices, and / or output devices such as display devices, printers, speakers and / or network cards. The exemplary input / output devices (24) may be included within the computing device (12) as a component constituting the computing device (12), or may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0088]

[0089] Fig. 13 is a flowchart illustrating a method for generating an image-based indoor floor plan according to one embodiment of the present invention. Here, each step of Fig. 13 can be performed by an indoor floor plan generating device (100) according to one embodiment of the present invention.

[0090] Referring to FIG. 13, the indoor floor plan generation device can receive an upward image in which an upper area of ​​a target space is captured (S110). That is, since it is possible to generate an indoor floor plan from the shape of a ceiling or floor surface of the target space, an upward image of the target space can be received and utilized. In particular, the ceiling surface can minimize obscuration by other objects or people compared to the floor surface. Accordingly, the indoor floor plan generation device can receive an upward image of the target space and generate an indoor floor plan from the received upward image. Here, any upward image in which an upper area of ​​the target space is captured can be utilized, and depending on the embodiment, it may be generated by extracting a vertical image of the target space from an ERP image that captured the target space.

[0091] Thereafter, the indoor floor plan generation device can generate attitude information of a photographing device that captured the corresponding frame image by applying a Structure-from-Motion (SfM) technique to each frame image included in the upper image (S120). Since the SfM technique can be used to generate attitude information such as a shooting angle or shooting direction of the photographing device that generated the corresponding upper image, the indoor floor plan generation device can generate attitude information for each frame image included in the upper image. Here, in addition to SfM, any technique that can estimate attitude information of a photographing device from a frame image can be utilized.

[0092] Thereafter, the indoor floor plan generation device performs image processing on each frame image included in the upper image, thereby extracting multiple line segments included in the frame image (S130). For example, various straight line detection algorithms, such as Hough transform, can be utilized to extract the line segments included in the frame image.

[0093] When multiple line segments are extracted, the indoor floor plan generation device can divide the multiple line segments into vertical line segments and horizontal line segments (S140). The vertical line segments correspond to the height direction of the target space, and the horizontal line segments correspond to the width or depth direction of the target space. Here, the indoor floor plan generation device may divide the vertical line segments and the horizontal line segments based on the Manhattan world assumption for the target space. Specifically, the indoor floor plan generation device can generate a virtual extension line for each line segment extracted from the frame image, and then divide the line segment among the line segments whose virtual extension line passes through the principal point of the corresponding frame image into a vertical line segment. In addition, the remaining line segments that do not correspond to the vertical component can be divided into horizontal line segments. Meanwhile, the indoor floor plan generation device can cluster the horizontal line segments according to each angle, and classify the clustered horizontal line segments into groups that are orthogonal to each other.

[0094] Thereafter, the indoor floor plan generation device can extract the intersections of the horizontal lines in the corresponding frame image that are orthogonal, generate an outline shape connecting the intersections, and detect a unit space corresponding to the outline shape (S150). Here, if the ceiling surface of the corresponding unit space is not completely shown in one frame image, the outline shape may not appear as a closed shape. In other words, since the unit space cannot be specified using the outline shape, the outline shape can be made to appear as a closed shape by matching corresponding intersections among the intersections included in other frame images. At this time, the posture information or position information corresponding to each frame image can be utilized. In addition, depending on the embodiment, it is also possible to move the intersections extracted from each frame image to a global coordinate system, merge intersections located at similar positions in the global coordinate system (for example, intersections included within a set distance range) into one, and then specify each unit space from the outline shape connecting each intersection.

[0095] Thereafter, the indoor floor plan generation device can extract a horizontal line segment that creates an intersection with two vertical line segments from among a plurality of horizontal line segments included in the frame image and detect it as a passage (S160). That is, the frame image may include a "ㄷ" shape formed by two vertical line segments and one horizontal line segment, and the indoor floor plan generation device can determine the horizontal line segment included in the "ㄷ" shape as a passage. However, among the "ㄷ" shapes appearing in the frame image, there may be cases where they are not passages, and in order to distinguish them, the indoor floor plan generation device can additionally utilize a depth image.

[0096] Specifically, the indoor floor plan generation device can first extract, from among the horizontal line segments included in the frame image, a horizontal line segment that creates an intersection with two vertical line segments as a passage candidate. Thereafter, the indoor floor plan generation device can generate a depth image corresponding to the frame image, and obtain the depth value of the passage corresponding to the passage candidate and the depth values ​​of the two vertical line segments from the depth image.

[0097] Here, the depth value of the passageway can be obtained from the depth value of the center line that divides the passageway area into two. The passageway area is an area surrounded by the passage candidate and two vertical line segments, and the center line can be specified as passing through the principal point of the frame image while dividing the passageway area into two equal-sized parts. Accordingly, the indoor floor plan generation device can extract the depth value corresponding to the center line from the depth image and set it as the depth value of the passageway. For example, the average value of the depth values ​​of the pixels corresponding to the center line can be set as the representative value and set as the depth value of the passageway.

[0098] Meanwhile, if the passage area corresponds to an actual passage, the interior of the passage area is empty, so the depth value of the passage may be greater than the depth values ​​of two vertical line segments. Accordingly, the indoor floor plan generation device can detect the passage candidate as a passage if the depth value of the passage is greater than the depth value of the vertical line segment by a threshold value or more.

[0099] Thereafter, the indoor floor plan generation device can generate an indoor floor plan for the target space by connecting each unit space extracted from the upper image with a passageway (S170). That is, the indoor floor plan can be generated by connecting each unit space and passageway based on the detailed information and location information for each frame image. Here, the passageway can be distinguished by indicating it with a separate identification object, such as a thick line segment. However, this is not limited to this, and the passageway can also be indicated with various colors or shapes.

[0100]

[0101] Fig. 14 is a flowchart illustrating a method for generating an image-based indoor floor plan according to another embodiment of the present invention. Here, each step of Fig. 14 can be performed by an indoor floor plan generating device (200) according to another embodiment of the present invention.

[0102] Referring to Figure 14, the indoor floor plan generation device can receive an image capturing a target space (S210). In this case, the indoor floor plan generation device may receive an image capturing the entire target space, rather than an upper image capturing the upper area of ​​the target space. Depending on the embodiment, the image may be received in the form of an ERP image or a panoramic image.

[0103] Thereafter, the indoor floor plan generation device can generate posture information of the camera device that captured the corresponding frame image by applying the Structure-from-Motion (SfM) technique to each frame image included in the video (S220). That is, by utilizing the SfM technique, it is possible to generate posture information such as the shooting angle or shooting direction of the camera device that generated the corresponding image, such as a 360-degree monocular camera. Accordingly, the indoor floor plan generation device can generate posture information for each frame image included in the video.

[0104] Thereafter, the indoor floor plan generation device can generate a 3D mesh corresponding to the target space from the image using a deep learning network (S230). Here, the deep learning network may generate the 3D mesh based on neural rendering such as the MVS (Multi View Stereo) algorithm, NeuS (Neural Surface Reconstruction), or mono-SDF (Signed Distance Function). However, the present invention is not limited thereto, and any model or algorithm can be utilized as long as it generates a 3D mesh for the target space from each frame image and the posture information of the shooting device.

[0105] Thereafter, the indoor floor plan generation device can generate an initial floor plan by orthogonally projecting the three-dimensional mesh onto a two-dimensional plane corresponding to the ceiling or floor of the target space (S240). Here, the initial floor plan does not distinguish between individual unit spaces, the passages connecting the unit spaces are unclear, and errors may also exist in the outline shape of the initial floor plan. Therefore, procedures such as correction may be performed to generate an indoor floor plan from the initial floor plan.

[0106] To this end, the indoor floor plan generation device can detect ceiling lines located on the ceiling surface of the target space from each frame image included in the video (S250). That is, in order to distinguish each unit space included in the initial floor plan and to specify a passage connecting the unit spaces, ceiling lines that serve as a reference for distinguishing each unit space can be extracted.

[0107] Specifically, the indoor floor plan generation device can first extract an upper frame image located in the vertical direction of the target space from the frame image. Thereafter, the indoor floor plan generation device can perform image processing on the upper frame image to extract a plurality of line segments. That is, the indoor floor plan generation device can extract line segments included in the upper frame image based on various straight line detection algorithms such as Hough transform. Thereafter, the indoor floor plan generation device can extract ceiling line segments from the plurality of line segments by filtering based on a depth value corresponding to the ceiling surface of the target space.

[0108] An indoor floor plan generation device can generate an intermediate floor plan that divides the initial floor plan into multiple unit spaces by reflecting ceiling line segments in the initial floor plan (S260). In other words, an intermediate floor plan of a shape divided into multiple unit spaces can be generated by cutting an initial floor plan of a shape in which all spaces are connected by ceiling line segments.

[0109] Thereafter, the indoor floor plan generation device can generate an indoor floor plan corresponding to the target space by simplifying the outline of each unit space included in the intermediate floor plan and setting the point where the unit spaces meet as a passageway (S270). That is, the indoor floor plan generation device can detect the outline of each unit space separated from the intermediate floor plan, and can simplify the outline of each unit space vertically or horizontally based on the Manhattan world assumption for the target space. In addition, the indoor floor plan generation device can specify the point where the unit spaces first meet through a morphological operation between the unit spaces and set the specified point as a passageway. In this case, depending on the embodiment, it is also possible to set the passageway by extracting a horizontal line segment that creates an intersection with two vertical line segments. Finally, the indoor floor plan generation device can generate an indoor floor plan from the intermediate floor plan through outline simplification and passage setting.

[0110]

[0111] The present invention described above can be implemented as computer-readable code on a medium recording a program. The computer-readable medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. Furthermore, the medium may be a variety of recording or storage means, including a single or multiple hardware components, and is not limited to media directly connected to a computer system, but may also be distributed across a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and ROM, RAM, flash memory, and other media configured to store program instructions. Furthermore, other examples of media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc. Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

[0112]

[0113] The present invention is not limited to the above-described embodiments and the attached drawings. It will be apparent to those skilled in the art that components of the present invention can be substituted, modified, and altered without departing from the technical spirit of the present invention.

Claims

1. A method for generating an indoor floor plan based on an image using a computing device, A step of receiving an upper image in which an upper area of ​​a target space is photographed; A step of performing image processing on each frame image included in the upper image to extract a plurality of line segments included in the frame image; A step of dividing the above plurality of line segments into vertical line segments corresponding to the height direction of the target space and horizontal line segments corresponding to the width or width direction within the target space; A step of extracting intersection points where the horizontal lines intersect orthogonally, generating an outline shape connecting the intersection points, and detecting a unit space corresponding to the outline shape; A step of extracting a horizontal line segment that creates an intersection with two vertical line segments from among a plurality of horizontal line segments included in the above frame image and detecting it as a passage; and An image-based indoor floor plan generation method, comprising a step of generating an indoor floor plan for the target space by connecting each unit space extracted from the upper image with the passageway.

2. In the first paragraph, the upper image is An image-based indoor floor plan generation method, which generates a vertical image of a target space by extracting the vertical image from an ERP (Equirectangular Projection) image that captures the target space.

3. In paragraph 1, the step of distinguishing is An image-based indoor floor plan generation method, wherein the vertical line segments and horizontal line segments are distinguished based on the Manhattan world assumption for the above target space.

4. In paragraph 1, A method for generating an indoor floor plan based on an image, further comprising a step of generating posture information of a photographing device that captured the frame image by applying a structure-from-motion (SfM) technique to the frame image.

5. In paragraph 1, the step of distinguishing is A method for generating an indoor floor plan based on an image, wherein a line segment whose virtual extension passes through a principal point of the frame image among the above line segments is distinguished as the vertical line segment.

6. In paragraph 1, the step of distinguishing is An image-based indoor floor plan generation method, wherein the horizontal line segments are clustered by their respective angles and classified into groups that are orthogonal to each other.

7. In the first paragraph, the step of detecting through the passage is A step of extracting, as a passage candidate, a horizontal line segment that creates an intersection with two vertical line segments among the horizontal line segments included in the above frame image; A step of generating a depth image corresponding to the frame image, and obtaining a depth value of a passage corresponding to the passage candidate and depth values ​​of the two vertical line segments from the depth image; and An image-based indoor floor plan generation method, comprising a step of detecting the passage candidate as the passage if the depth value of the passage is greater than the depth value of the vertical line segment.

8. In paragraph 7, the step of obtaining the depth value is An image-based indoor floor plan generation method, wherein the passage area formed by the passage candidate and the two vertical line segments is divided into two equal-sized parts, a center line passing through a principal point of the frame image is set, and the depth value of the center line is extracted from the depth image to obtain the depth value of the passage.

9. In the first paragraph, the step of detecting the unit space is An image-based indoor floor plan generation method, wherein if the above outline shape does not form a closed shape, the outline shape is generated as a closed shape by matching corresponding intersection points among intersection points included in other frame images.

10. In the 9th paragraph, the step of detecting the unit space is An image-based indoor floor plan generation method, which matches intersections included in different frame images based on detailed information of a photographing device that captured each frame image.

11. A method for generating an indoor floor plan based on an image using a computing device, A step of receiving an image capturing a target space; A step of generating a three-dimensional mesh corresponding to the target space from the image using a deep learning network; A step of generating an initial plan view by projecting the above three-dimensional mesh onto a two-dimensional plane corresponding to the ceiling or floor surface of the target space; A step of detecting ceiling lines located on the ceiling surface of the target space from each frame image included in the above video; A step of generating an intermediate floor plan by reflecting the ceiling lines in the initial floor plan and dividing the initial floor plan into multiple unit spaces; and An image-based indoor floor plan generation method, comprising the step of generating an indoor floor plan corresponding to the target space by simplifying the outline of each unit space included in the intermediate floor plan and setting the point where the unit spaces meet as a passageway.

12. In the 11th paragraph, the step of detecting the ceiling lines is A step of extracting an upper frame image located in the vertical direction of the target space from the above frame image; A step of extracting a plurality of line segments by performing image processing on the upper frame image; and A method for generating an indoor floor plan based on an image, comprising a step of extracting ceiling line segments from among the plurality of line segments by filtering based on a depth value corresponding to a ceiling surface of the target space.

13. In the 11th paragraph, the step of generating the intermediate plane An image-based indoor floor plan generation method, wherein the initial floor plan of a connected space is cut by the ceiling line segment to separate the initial floor plan into a plurality of unit spaces.

14. In the 11th paragraph, the step of generating the indoor floor plan is An image-based indoor floor plan generation method, wherein the first point of contact is specified through a morphology operation between the above unit spaces, and the first point of contact is set as the passageway.

15. In the 11th paragraph, the step of generating the indoor floor plan is An image-based indoor floor plan generation method, wherein the outline of the unit space is simplified vertically or horizontally based on the Manhattan world assumption for the target space.

16. In paragraph 11, A method for generating an indoor floor plan based on an image, further comprising a step of generating posture information of a photographing device that captured the frame image by applying a structure-from-motion (SfM) technique to each frame image included in the image.

17. A computing device including a processor and generating an image-based indoor floor plan, The above processor Receiving an upper image in which the upper area of ​​the target space is captured; Performing image processing on each frame image included in the upper image to extract multiple line segments included in the frame image; Dividing the above plurality of line segments into vertical line segments corresponding to the height direction of the target space and horizontal line segments corresponding to the width or width direction within the target space; Extracting the intersection points where the horizontal line segments are orthogonal, generating an outline shape connecting the intersection points, and detecting a unit space corresponding to the outline shape; Extracting a horizontal line segment that creates an intersection with two vertical line segments from among a plurality of horizontal line segments included in the above frame image and detecting it as a passage; and A computing device that performs a process including creating an indoor floor plan for the target space by connecting each unit space extracted from the upper image with the passageway.

Images