Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program which can acquire a three-dimensional shape in a building even in the case that an obstacle exists.SOLUTION: An information processing system comprises: an AR space generation unit which generates an augmented reality space, based on scan data acquired by photographing the inside of a building by using a photographing terminal comprising a camera and a sensor; a plane detection unit which generates plane information which identifies arrangement of a plane in the augmented reality space; a corner estimate unit which generates corner location information on the basis of at least one piece of plane information; a height calculation unit which calculates a perpendicular distance from a first horizontal surface to a second horizontal surface in the augmented reality space, and generates height information in the building; and a model generation unit which generates a three-dimensional model in the building, based on a plurality of pieces of corner location information plotted in the augmented reality space, and the height information.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a program.

Background Art

[0002] There is known a technique of photographing an indoor space using a three-dimensional measuring device (for example, a camera, a 3D scanner, etc.) and generating a three-dimensional model or a floor plan of the indoor space. For example, Patent Document 1 discloses synthesizing a cubic panorama image using an omnidirectional photograph of an indoor space taken by a panorama photographing device and acquiring three-dimensional coordinates of the indoor space from the cubic panorama image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order to obtain the three-dimensional shape of an indoor space from a panorama image as in Patent Document 1, various geometric corrections such as distortion correction, tilt correction, and horizontal correction are required, and complex arithmetic processing is required. Further, when furniture or other fixtures are installed in the indoor space to be photographed, these fixtures may become obstacles and the shape of the indoor space may not be accurately grasped.

[0005] An object of an exemplary embodiment in the present disclosure is to provide an information processing system, an information processing method, and a program capable of obtaining the three-dimensional shape inside a building even when there are obstacles.

Means for Solving the Problems

[0006] An information processing system according to an aspect of the present disclosure is An AR space generation unit that generates an augmented reality space based on scan data obtained by photographing the interior of a building using a photographing terminal equipped with a camera and a sensor, A plane detection unit that recognizes a plane in the building based on a photographed image and sensing data included in the scan data, and generates plane information that specifies the arrangement of the plane in the augmented reality space, A corner estimation unit that estimates corners in the building based on at least one of the plane information, and generates corner position information indicating the relative positions of the corners with respect to the photographing position in the augmented reality space, Based on the relative position of the first horizontal plane with respect to the photographing position included in the plane information regarding the first horizontal plane, and the relative position of the second horizontal plane with respect to the photographing position included in the plane information regarding the second horizontal plane, a height calculation unit that calculates the perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space, and generates height information of the building, A model generation unit that generates a three-dimensional model of the building based on the plurality of corner position information plotted in the augmented reality space and the height information.

[0007] Since the information processing system has the above characteristics, even if there are obstacles in the building, the three-dimensional shape of the building can be obtained.

Brief Description of Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

DETAILED DESCRIPTION OF THE INVENTION

[0009] The information processing system, information processing method, and program of the present disclosure have the following configurations. [Item 1] An AR space generation unit that generates an augmented reality space based on scan data acquired by photographing the inside of a building using a photographing terminal equipped with a camera and sensors, A plane detection unit that recognizes the plane inside the building based on the photographed image and sensing data included in the scan data, and generates plane information that specifies the arrangement of the plane in the augmented reality space, A corner estimation unit that estimates corners inside the building based on at least one piece of the plane information, and generates corner position information indicating the relative positions of the corners with respect to the photographing position in the augmented reality space, Based on the relative position of the first horizontal plane with respect to the shooting position included in the plane information regarding the first horizontal plane, and the relative position of the second horizontal plane with respect to the shooting position included in the plane information regarding the second horizontal plane, calculate the perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space, and a height calculation unit that generates height information within the building. An information processing system comprising: a model generation unit that generates a three-dimensional model within the building based on the plurality of corner position information plotted in the augmented reality space and the height information. [Item 2] The corner estimation unit estimates an intersection of the first horizontal plane, the first vertical plane, and the second vertical plane that are adjacent to each other as the corner within the building based on the plane information regarding the first horizontal plane, the plane information regarding the first vertical plane, and the plane information regarding the second vertical plane, and generates the corner position information. The information processing system according to Item 1. [Item 3] The plane detection unit generates the plane information regarding the first horizontal plane and the plane information regarding the first vertical plane adjacent to the first horizontal plane. The corner estimation unit receives a designation of a marker for a point on the first vertical plane via the display unit of the shooting terminal, and estimates the corner within the building based on the position information of the marker and the plane information regarding the first horizontal plane, and generates the corner position information. The information processing system according to Item 1. [Item 4] The corner estimation unit receives a designation of a marker for a point presumed to be the corner via the display unit of the shooting terminal, and estimates the corner within the building based on the position information of the marker and the plane information regarding the first horizontal plane, and generates the corner position information. The information processing system according to Item 1. [Item 5] The first horizontal plane is the floor surface within the building. The corner estimation unit generates the corner position information corresponding to each corner existing on the floor surface. The information processing system according to any one of Items 1 to 4. [Item 6] further comprising a component specifying unit that generates component information for specifying the arrangement of components existing in the building The model generation unit generates the three-dimensional model in which the arrangement of the components is reflected based on the plurality of corner position information, the height information, and the component information plotted in the augmented reality space. The information processing system according to any one of Items 1 to 4 [Item 7] The component specifying unit receives, via the display unit of the photographing terminal, designation of column-beam markers corresponding to at least two corners of column-beam components existing in the building, and generates column-beam information for specifying the arrangement of the column-beam components in the augmented reality space based on the position information of the column-beam markers and the plane information regarding the plane adjacent to the column-beam components. The information processing system according to Item 6 [Item 8] The component specifying unit receives, via the display unit of the photographing terminal, designation of fixture markers corresponding to at least two corners of fixtures existing in the building, and generates fixture information for specifying the arrangement of the fixtures in the augmented reality space based on the position information of the fixture markers and the plane information regarding the plane on which the fixtures are installed. The information processing system according to Item 6 [Item 9] generating an augmented reality space based on scan data obtained by photographing the interior of a building using a photographing terminal including a camera and a sensor; recognizing a plane in the building based on a photographed image and sensing data included in the scan data, and generating plane information for specifying the arrangement of the plane in the augmented reality space; estimating corners in the building based on at least one of the plane information, and generating corner position information indicating the relative positions of the corners with respect to the photographing position in the augmented reality space; Based on the relative position of the first horizontal plane with respect to the shooting position included in the plane information regarding the first horizontal plane, and the relative position of the second horizontal plane with respect to the shooting position included in the plane information regarding the second horizontal plane, calculate the perpendicular distance from the first horizontal plane to the second horizontal plane in the extended reality space, and generate height information within the building; Based on the plurality of corner position information plotted in the extended reality space and the height information, generate a three-dimensional model within the building. An information processing method executed by a computer. [Item 10] A process of generating an extended reality space based on scan data obtained by photographing the inside of a building using a photographing terminal equipped with a camera and a sensor; Based on the photographed image and sensing data included in the scan data, recognize the plane inside the building and generate plane information for specifying the arrangement of the plane in the extended reality space; Estimate the corners inside the building based on at least one of the plane information, and generate corner position information indicating the relative position of the corners with respect to the shooting position in the extended reality space; Based on the relative position of the first horizontal plane with respect to the shooting position included in the plane information regarding the first horizontal plane, and the relative position of the second horizontal plane with respect to the shooting position included in the plane information regarding the second horizontal plane, calculate the perpendicular distance from the first horizontal plane to the second horizontal plane in the extended reality space, and generate height information within the building; Based on the plurality of corner position information plotted in the extended reality space and the height information, generate a three-dimensional model within the building. A program for causing a computer to execute the process.

[0010] Hereinafter, an information processing system according to an embodiment of the present disclosure will be described with reference to the drawings. In each of the attached drawings, the same or similar elements are given the same or similar reference numerals and names, and redundant descriptions regarding the same or similar elements may be omitted in the description of the embodiment. Note that the content shown in each drawing is merely an example for explaining the present embodiment, and is only a schematic example shown for ease of explaining the present embodiment. The content of each drawing may be modified or changed within a range where no technical problems occur.

[0011] <System Overview> The information processing system according to the present embodiment is a system for constructing a three-dimensional model of the internal space of buildings such as houses, stores, buildings, and factories. The target space of the three-dimensional model to be constructed is an area within a building partitioned by at least two horizontal planes such as a floor and a ceiling, and a three-dimensional model may be generated for each room in the building, or a three-dimensional model integrating a plurality of rooms located on the same floor may be generated.

[0012] As shown in FIG. 1, the information processing system of the present embodiment includes a management server 1 and a user terminal 2 (imaging terminal) owned by a measurer (user). The management server 1 and the user terminal 2 are communicably connected to each other via a network, and a plurality of user terminals 2 may be connected to the management server 1.

[0013] In the method for generating a three-dimensional model realized by the information processing system of the present embodiment, the target space in the building is scanned using the user terminal 2 equipped with the camera 25 and the sensor 26 to generate an augmented reality space. When scanning the target space, based on the captured image and the sensing data, the plane of the target space is specified, and the position of the corner (angle) of the target space and the height of the target space are estimated from the specified plane information. The corner estimated in this way is plotted in the augmented reality space, and a three-dimensional model is generated based on the position information of the plotted corner and the height information of the target space.

[0014] <Management Server 1> The management server 1 is an information processing device that executes processes related to the information processing system, and may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. The management server 1 may be deployed, for example, by a business operator that provides a user with a 3D model creation tool. FIG. 2 is a diagram showing the hardware configuration of the management server 1. As shown in FIG. 2, the management server 1 includes a processor 10, a memory 11, a storage 12, a transmission / reception unit 13, and an input / output unit 14, etc., which are electrically connected to each other through a bus 15. Note that the illustrated configuration is an example, and it may have other configurations.

[0015] The processor 10 is an arithmetic device that controls the operation of the entire management server 1, controls the transmission and reception of data between each element, and performs information processing necessary for the execution and authentication processing of applications. For example, the processor 10 is a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and executes programs for this system stored in the storage 12 and expanded in the memory 11 to perform each information processing.

[0016] The memory 11 includes a main memory composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory composed of a non-volatile storage device such as a flash memory or an HDD (Hard Disc Drive). The memory 11 is used as a work area of the processor 10, and stores a BIOS (Basic Input / Output System) executed at the startup of the management server 1 and various setting information, etc.

[0017] The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12. Also, a storage unit 120 described later may be provided in a part of the storage area.

[0018] The transmission / reception unit 13 is a communication interface for connecting the management server 1 to the network. Note that the transmission / reception unit 13 may be provided with a short-range communication interface for Bluetooth (registered trademark) and BLE (Bluetooth Low Energy).

[0019] The input / output unit 14 is an information input device such as a keyboard and mouse, and an output device such as a display.

[0020] The bus 15 is commonly connected to the above elements and transmits, for example, address signals, data signals, and various control signals.

[0021] <User terminal 2> The user terminal 2 shown in FIG. 3 is an information processing device (for example, a smartphone, a mobile phone terminal, a tablet terminal, a dedicated photographing terminal, or other computers) held by a user (a measurer) who uses services related to the information processing system. This user terminal 2 also includes a processor 20, a memory 21, a storage 22, a transmission / reception unit 23, an input / output unit 24, etc., similar to the management server 1, and these are electrically connected to each other through a bus 27. Since the functions of these elements can be configured in the same manner as the above-described management server 1, detailed descriptions of each element are omitted.

[0022] The user terminal 2 is also a photographing terminal for scanning inside a building, and in addition to the above elements, it includes a camera 25 capable of video shooting and a sensor 26. The camera 25 may be equipped with a vision / image sensor. The sensor 26 may include, for example, an inertial sensor (an acceleration sensor, a gyro sensor), a GPS sensor, a proximity sensor (for example, LiDAR (Light Detection And Ranging), etc.), and the sensor 26 of the user terminal 2 preferably includes LiDAR.

[0023] FIG. 4 is a block diagram illustrating the functions implemented in the management server 1 and the user terminal 2. Hereinafter, each function provided in the management server 1 and the user terminal 2 will be described.

[0024] <Functions of the Management Server 1> As shown in FIG. 4, the management server 1 includes a processing unit 110, a storage unit 120, and a server communication unit 130. The processing unit 110 may include an AR space generation unit 111, a plane detection unit 112, a corner estimation unit 113, a height calculation unit 114, a component identification unit 115, a model generation unit 116, and a floor plan generation unit 117. Each functional unit of the processing unit 110 is realized, for example, by the processor 10 of the management server 1 reading data and programs stored in the storage 12 and executing various programs in the working area of the memory 11. Note that part or all of the various functional units may be realized by the processor 20 on the user terminal 2 side.

[0025] The storage unit 120 is a storage area of the management server 1 that stores programs for executing various control processes and each function in the management server 1, input data, etc., and may store the processing results by the various functional units of the processing unit 110. For example, the storage unit 120 may have a floor plan / 3D model storage unit 121 that stores a 3D model, a floor plan, etc. inside the building generated by the model generation unit 116 and the floor plan generation unit 117. The server communication unit 130 communicates with the user terminal 2 via the network NW. The server communication unit 130 may have a function as a transmission unit that transmits the processing results executed by the processing unit 110 to the user terminal 2, and a function as a reception unit that receives various requests and data sent from the user terminal 2.

[0026] The AR space generation unit 111 generates an augmented reality space based on scan data obtained by photographing a target space in a building using the user terminal 2. In the present embodiment, when acquiring information necessary for constructing a 3D model, the target space in the building (the space for generating the 3D model) is scanned using the user terminal 2. The scan data acquired by the scan includes a photographed image (moving image, video) by the camera 25 and sensing data by the sensor 26. The sensing data is data measured by the sensor 26 at the time of photographing the target space. For example, the sensing data may include 3D point cloud data of the target space generated by sensing with LiDAR, and in addition, measurement data by an inertial sensor may be included. The 3D point cloud data acquired by LiDAR may be data represented by a 3D orthogonal coordinate system with an arbitrarily set point as the origin (reference point). These scan data acquired by the user terminal 2 are transmitted to the AR space generation unit 111 via the server communication unit 130.

[0027] The AR space generation unit 111 arranges planes, corners, other components, etc. specified by other functional units (plane detection unit 112, corner estimation unit 113, height calculation unit 114, component identification unit 115, etc.) described later on the video of the real space acquired by the camera 25 to construct an augmented reality space. At the time of executing the scan of the target space (at the time of plane recognition and corner estimation), the AR space generation unit 111 constructs an augmented reality space in real-time processing based on the video of the target space photographed by the camera 25, and on the display unit 230 of the user terminal 2 (the display screen at the time of executing the scan), objects such as planes, corners, and other components specified by the functional units described later may be superimposed and displayed on the video of the real target space being photographed by the camera 25 in real-time. Note that the method for generating the augmented reality space is not particularly limited, and a known method may be adopted.

[0028] Based on the captured image acquired by the camera 25 and the sensing data acquired by the sensor 26, the plane detection unit 112 recognizes the plane inside the building and generates plane information for specifying the arrangement of the plane in the augmented reality space. Fig. 5(a) is a schematic diagram illustrating the state of the operator's terminal operation when scanning the plane of the target space, and Figs. 5(b) and (c) are schematic diagrams illustrating the display screen when scanning a predetermined plane. As shown in Fig. 5(a), with the camera 25 of the user terminal 2 activated, the operator points the camera 25 at the plane to be identified and captures the plane. At this time, the plane detection unit 112 may transmit instruction information for designating the plane to be scanned, such as "Please point the camera at the ceiling", to the user terminal 2 and display the instruction information on the display unit 230 of the user terminal 2 (Fig. 5(b)). As shown in Fig. 5(c), when a predetermined plane is captured, the plane detection unit 112 identifies the plane shown in the captured image. The shooting angle (shooting direction) at this time is not particularly limited, and as shown in Fig. 5(a), it may be an angle inclined obliquely with respect to the plane to be captured.

[0029] For example, the plane detection unit 112 identifies the plane included in the captured image from the three-dimensional point cloud data acquired by LiDAR, and discriminates the attribute of the identified plane (whether the identified plane is the floor, ceiling, or inner wall surface). When discriminating the plane, in addition to the sensing data by LiDAR, sensing data by an inertial sensor (for example, information such as the inclination of the user terminal 2 and the shooting angle), and instruction information may also be used. The plane recognition function by the plane detection unit 112 may be realized by an AI model subjected to machine learning. In this case, the AI model receives the captured image and the sensing data as input information, and performs segmentation of the plane included in the input captured image. The learning algorithm for constructing the AI model is not particularly limited, and an AI model based on deep learning such as a neural network may be used.

[0030] After recognizing a plane in the captured image, the plane detection unit 112 generates plane information that specifies the arrangement of the recognized plane in the augmented reality space. The plane information includes the relative position of the plane with respect to the shooting position in the augmented reality space (the position of the user terminal 2 at the time of plane recognition), and the relative position may be expressed, for example, by a three-dimensional orthogonal coordinate system (x coordinate, y coordinate, z coordinate). The plane information may also include other information such as the actually measured distance from the shooting position based on the sensing data of LiDAR to the plane and the inclination of the plane. The plane detection unit 112 recognizes at least two horizontal planes that partition the target space and generates plane information regarding these horizontal planes (e.g., plane information of the floor surface and plane information of the ceiling). In the following description, the two horizontal planes that partition the target space will be referred to as the "first horizontal plane" and the "second horizontal plane". In addition to the first horizontal plane and the second horizontal plane, the plane detection unit 112 may generate plane information regarding vertical planes such as the inner wall surface of the target space, or may recognize all the planes that make up the target space and generate plane information corresponding to each plane. Also, the shooting position when recognizing the first horizontal plane and the shooting position when recognizing the second horizontal plane may be the same position or different positions.

[0031] Based on at least one piece of plane information acquired by the plane detection unit 112, the corner estimation unit 113 estimates the corners in the building and generates corner position information indicating the relative position of the corners with respect to the shooting position in the augmented reality space (e.g., relative coordinates indicated by a three-dimensional orthogonal coordinate system). Each corner in the building is an intersection where at least three planes intersect. In the following description, any one corner existing on the first horizontal plane will be referred to as the "first corner (reference sign C1)", and the two vertical planes (e.g., inner wall surfaces) that constitute the first corner C1 together with the first horizontal plane will be referred to as the "first vertical plane" and the "second vertical plane", respectively. The corner estimation unit 113 may estimate the corners using three pieces of plane information (e.g., FIG. 6), may estimate the corners using two pieces of plane information (e.g., FIG. 7), or may estimate the corners using one piece of plane information (e.g., FIG. 8).

[0032] When using three pieces of plane information, the plane detection unit 112 described above acquires plane information corresponding to each of the first horizontal plane, the first vertical plane, and the second vertical plane. Based on the plane information regarding the first horizontal plane, the plane information regarding the first vertical plane, and the plane information regarding the second vertical plane, the corner estimation unit 113 estimates the intersection of the mutually adjacent first horizontal plane, first vertical plane, and second vertical plane as the first corner C1 within the building and generates corner position information. The position of the estimated first corner C1 can be represented by the intersection coordinates of the three planes. For example, the x - coordinate and y - coordinate of the first corner C1 are specified by the intersection line coordinates of the first vertical plane and the second vertical plane, and the z - coordinate of the first corner C1 is specified by the coordinates of the first horizontal plane. Through the above - described estimation, the corner estimation unit 113 generates corner position information that specifies the relative position of the corner with respect to the shooting position (the position of the user terminal 2 at the time of corner shooting) in the augmented reality space.

[0033] Note that the plane information (plane information regarding the first horizontal plane, plane information regarding the first vertical plane, plane information regarding the second vertical plane) for estimating the corner may be acquired in advance before shooting the corner, or may be acquired during the execution of the scan for shooting the corner. FIG. 6 is a schematic diagram illustrating the terminal operation when estimating a corner from three planes. As shown in FIG. 6(a), the measurer points the activated camera 25 at the first corner C1 and shoots the area around the first corner C1 (FIG. 6(b) is an example of the display screen at the time of shooting). In FIG. 6, an obstacle 8 (for example, furniture or other fixtures) is intervening between the first corner C1 and the shooting position, and the first corner C1 does not appear in the video of the camera 25. Even when there is an obstacle 8 near the first corner C1 like this, by estimating the first corner C1 from the three pieces of plane information of the first horizontal plane, the first vertical plane, and the second vertical plane as described above, the position information of the first corner C1 can be obtained.

[0034] When estimating a corner using two or one plane information, the corner estimation unit 113 may accept a manual operation by the measurer. The "manual operation" here means accepting the designation of a predetermined point (marker) for acquiring position information on the display screen when photographing the target space. When the marker is designated by a manual operation during corner photographing, the corner estimation unit 113 acquires the position information (relative coordinates) of the designated marker based on sensing data or the like, and uses this position information for corner estimation.

[0035] When using a manual operation, even in a situation where the plane detection unit 112 cannot recognize one or two of the three planes, corners can be estimated using the manual operation. An example of a situation where a plane cannot be recognized is when the plane is composed of a glass surface like the second vertical plane in Fig. 7(a). When the plane is a glass surface, since a laser or the like for sensing passes through the glass surface, it is difficult to acquire the position (plane information) of the glass surface. When there is a plane such as a glass surface that is difficult to recognize, the position information of the marker designated by the manual operation is used to supplement the information necessary for corner estimation.

[0036] For example, when using two plane information, the plane detection unit 112 generates plane information regarding the first horizontal plane and plane information regarding the first vertical plane. Then, the corner estimation unit 113 accepts the designation of a marker for a point on the first vertical plane via the display unit 230 of the user terminal 2, and estimates the first corner C1 and generates corner position information based on the position information of the marker (relative coordinates of the marker with respect to the photographing position) and the plane information regarding the first horizontal plane. The point on the first vertical plane is not necessarily limited, but for example, in the case of Fig. 7(a), it is preferable to accept the designation of a marker on the intersection line of the first vertical plane and the second vertical plane. The position coordinates (relative coordinates with respect to the photographing position) of the point where the marker is designated may be specified based on the sensing data when the marker is designated (for example, sensing data obtained by irradiating a laser toward the marker designated position) and the plane information of the first vertical plane (relative coordinates with respect to the photographing position).

[0037] The operation method on the user terminal side when specifying the marker is not particularly limited. For example, on the screen displaying the captured image as shown in FIG. 7(b), the cursor 9 may be displayed at the center of the screen, and the measurer can specify the marker 7 by aligning the center of the cursor 9 with the desired position and tapping the button B1. In this case, the center of the cursor 9 corresponds to the shooting direction of the camera 25, and the position information (relative coordinates with respect to the shooting position) of the point where the marker 7 is specified can be obtained as the coordinates of the intersection of the virtual straight line extending in the shooting direction from the shooting position when the button B1 is tapped (when the marker 7 is specified) and the first vertical plane. After acquiring the position information of the marker 7 as described above, the corner estimation unit 113 estimates the intersection of the virtual straight line extending perpendicularly from the marker 7 to the first horizontal plane and the first horizontal plane as the first corner C1. That is, the corner estimation unit 113 estimates the x coordinate and y coordinate of the first corner C1 based on the position coordinates of the marker 7, and estimates the z coordinate of the first corner C1 based on the coordinates of the first horizontal plane, and generates the position information of the first corner C1 in the extended reality space. In the above method, even when there are obstacles 8 or glass surfaces in the target space, the first corner C1 can be estimated from the plane information of two of the three planes.

[0038] When using manual operation, corners may be estimated from a single plane information. For example, the corner estimation unit 113 receives the designation of a marker for a point presumed to be the first corner C1 via the display unit 230 of the user terminal 2, and estimates the first corner C1 based on the position information of the marker (relative coordinates of the marker with respect to the shooting position) and the plane information regarding the first horizontal plane, and generates corner position information. FIG. 8 is a schematic diagram illustrating the state of a terminal operation (manual operation) when estimating corners from a single plane information. The point presumed to be the first corner C1 means that the measurer visually determines a location in the captured image where the first corner C1 is likely to exist and designates the position of the marker. Also in this case, similar to the example of FIG. 7(b), the designation of the marker by the measurer can be received using the cursor 9. For example, when the marker 7 is designated at the position of the cursor 9 shown in FIG. 8(b), the marker 7 is placed on the surface of the obstacle 8 that overlaps the center of the cursor 9. The position information of the point where the marker 7 is designated can be specified based on sensing data (for example, sensing data by LiDAR), and the intersection of the virtual straight line extending from the shooting position toward the marker 7 (the virtual straight line extending from the shooting position in the shooting direction) and the first horizontal plane may be estimated as the first corner C1.

[0039] Figures 6 to 8 show how one corner is estimated. However, the corner estimation unit 113 estimates a plurality of corners included in the target space within the building and generates corner position information corresponding to each estimated corner. Further, the corner estimation unit 113 may grasp the relative positional relationship between the corners by arranging (plotting) the estimated corners on the augmented reality space constructed by the AR space generation unit 111, and calculate the distance between adjacent corners along the intersection line between the planes. The corner estimation unit 113 may estimate all the corners included in the target space, or may estimate each corner located on either one of the two horizontal planes. If each corner on either one of the horizontal planes is specified, the shape of the horizontal plane is determined by the respective corner position information. Therefore, the shape of the target space can be specified based on the corner position information that partitions one horizontal plane and the height information (the perpendicular distance between the horizontal planes specified by the height calculation unit 114) described later. The corner estimation unit 113 preferably estimates each corner on the first horizontal plane which is the floor surface and generates respective corner position information, but may also estimate each corner on the ceiling (second horizontal plane) side.

[0040] Figure 9 illustrates the flow of corner scanning when estimating a plurality of corners. In Figure 9, it shows a situation where the measurer moves from position A to position B and then from position B to position C to photograph each corner on the first horizontal plane (the floor surface). By sequentially photographing the corners in this way and acquiring the position information of each corner, it is also possible to construct the relative positional relationship between the corners during the construction process of the three-dimensional space (augmented reality space). Also, the distance between the photographing positions can be grasped by the self-position estimation technique that grasps the current value of the user terminal 2, and the distance between the corners in the real space may be calculated based on the distance between the photographing positions. For example, since the distance from position A to corner I and the distance from position B to corner II can be obtained from sensing data by LiDAR or the like, if the distance from position A to position B is calculated by self-position estimation, the distance from corner I to corner II can also be calculated. Note that the self-position estimation may be performed by a known method, and the method of self-position estimation is not particularly limited.

[0041] The height calculation unit 114 calculates the height of the target space based on the plane information regarding the first horizontal plane and the plane information regarding the second horizontal plane acquired by the plane detection unit 112. Specifically, the height calculation unit 114 calculates the perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space based on the relative position of the first horizontal plane with respect to the shooting position and the relative position of the second horizontal plane with respect to the shooting position, and generates height information within the building. For example, the perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space can be calculated by subtracting the z coordinate of the first horizontal plane from the z coordinate of the second horizontal plane expressed in a three-dimensional orthogonal coordinate system.

[0042] The component identification unit 115 generates component information for identifying components such as beams, columns, and fixtures (doors, shoji screens, sliding doors, windows, etc.) existing within the building. The component information is information indicating the arrangement of the components in the augmented reality space. The component identification unit 115 accepts the designation of markers for at least two corners constituting the outer edge of the component by means of a manual operation as described in the corner estimation unit 113, and identifies the component based on the position coordinates (relative coordinates) of the designated markers, etc. In the following description, beams and columns will be collectively referred to as "column-beam components", and the markers used when identifying column-beam components will be referred to as column-beam markers. Also, doors, shoji screens, sliding doors, and windows will be collectively referred to as fixtures, and the markers used when identifying fixtures will be referred to as fixture markers.

[0043] When identifying column-beam components within the building, the component identification unit 115 accepts the designation of column-beam markers corresponding to at least two corners of the column-beam components existing within the building via the display unit 230 of the user terminal 2, and based on the position information of the column-beam markers (relative coordinates of the column-beam markers with respect to the shooting position) and the plane information regarding the plane adjacent to the column-beam components, generates column-beam information for identifying the arrangement of the column-beam components in the augmented reality space.

[0044] For example, when identifying a beam 50 extending from a first vertical plane to a second vertical plane on the opposite side along a second horizontal plane as shown in Fig. 10(a), among the eight corners of the beam 50, column-beam markers may be specified for two adjacent corners on the same plane for which plane information has been acquired. Fig. 10(b) shows an example in which column-beam markers are specified for two corners (corner A and corner B) adjacent in the width direction of the beam 50 on the first vertical plane. The position information (relative coordinates with respect to the shooting position) of the specified corners A and B of the column-beam markers can be identified based on the sensing data at the time of specifying the markers and the plane information (relative coordinates) of the first vertical plane, in the same manner as when estimating the corners.

[0045] The component identification unit 115 identifies the arrangement of the beam 50 in the augmented reality space by calculating the length L, width W, and height T of the beam 50 with the specified corners A and B of the column-beam markers as the bases. First, the width W of the beam 50 is identified by calculating the distance between the specified corners A and B of the column-beam markers based on the position information of the corners A and B. Also, the height T of the beam 50 is identified by calculating the perpendicular distance from the corner where the column-beam marker is placed to the second horizontal plane (the z coordinate of the second horizontal plane - the z coordinate of corner A or corner B) based on the position information of at least one of the corners A and B and the plane information (relative coordinates) of the second horizontal plane. The length L of the beam 50 is identified by calculating the perpendicular distance from the corner where the marker is placed to the second vertical plane based on the position information of at least one of the corners A and B and the plane information (relative coordinates) of the second vertical plane.

[0046] As described above, by obtaining the position information of two corners of the column-beam structure through manual operation, the arrangement of the column-beam structure can be specified. The number of corners for which the column-beam markers are specified is not limited to two, and column-beam markers may be specified for three or more corners of the column-beam structure. The positions of the corners for which the column-beam markers are specified are not limited to two adjacent corners as described above, and column-beam markers may be specified for two diagonally opposite corners (for example, corner A and corner D of beam 50). An object such as beam 50 of the column-beam structure generally has a hexahedral shape with each face being a quadrilateral. By simply specifying column-beam markers diagonally, the position coordinates of the other corners located on the same plane as the specified corners can also be determined. That is, when column-beam markers are specified for corner A and corner D of beam 50 as shown in FIG. 10, the position coordinates of corner B and corner C can also be determined, and the width W and height T can be calculated.

[0047] When specifying a fixture provided on a plane within a building, the structure specifying unit 115 receives, via the display unit 230 of the user terminal 2, the specification of fixture markers corresponding to at least two corners of the fixture existing within the building, and generates fixture information for specifying the arrangement of the fixture in the augmented reality space based on the position information of the fixture markers (relative coordinates of the fixture markers with respect to the shooting position) and the plane information (relative coordinates) regarding the plane on which the fixture is provided.

[0048] For example, as shown in FIG. 11, when specifying the door 60 provided on the first vertical plane, fixture markers may be specified for two diagonally opposite corners A and B out of the four corners constituting the outer edge of the door 60. The position information (relative coordinates with respect to the shooting position) of the specified corners A and B of the fixture markers can be determined based on the sensing data at the time of specifying the markers and the plane information (relative coordinates) of the first vertical plane, in the same manner as when estimating the corners.

[0049] The component specifying unit 115 specifies the arrangement of the door 60 in the augmented reality space by specifying the position coordinates of the other corners and the size of the door 60 on the first vertical plane, with the corners A and B that specify the fitting markers as the bases. Fittings such as the door 60 generally have a rectangular outer edge shape, and by specifying two diagonal points such as corners A and B to obtain position information, the position coordinates of the other corners that do not specify the fitting markers can be determined from the position coordinates of corners A and B. That is, by specifying two diagonal corners as fitting markers, the position coordinates (relative coordinates with respect to the shooting position) and size of the door 60 on the first vertical plane can be calculated.

[0050] As described above, the arrangement of the fitting can be specified by obtaining the position information of two corners of the fitting through manual operation. The number of corners specifying the fitting markers is not limited to two, and the fitting markers may be specified for three or more corners of the fitting. Note that when specifying the component as shown in FIGS. 10 and 11, the component specifying unit 115 may receive the attribute information of the target component (information specifying what the component is) from the user terminal 2. For example, an input field for receiving the attribute information on the display screen at the time of scan execution may be provided, such as the button B2 shown in FIGS. 10 and 11, and the measurer may be allowed to select the attribute of the component to be photographed.

[0051] The model generation unit 116 generates a three-dimensional model of the building based on the plurality of corner position information plotted in the augmented reality space and the height information. For example, if each corner constituting either one of the two horizontal planes is specified, the shape of one horizontal plane can be specified based on the corner position information of each specified corner. Then, a three-dimensional model of the target space can be generated by constructing a wall surface having the height calculated by the height calculation unit 114 around the one horizontal plane whose shape has been specified.

[0052] The model generation unit 116 may generate 3D data (objects) for each plane such as the floor, ceiling, and walls identified by the plane detection unit 112. Further, it may generate 3D data (objects) for each component such as beams, columns, and fixtures identified by the component identification unit 115. The model generation unit 116 may combine the objects of each such part to generate an overall model of the target space. When arranging components such as beams, columns, and fixtures in a building within a 3D model, the model generation unit 116 may generate an object of the component based on the component information acquired by the component identification unit 115, and place the object at the position identified by the component information to generate a 3D model of the target space. When a plurality of 3D models are generated for each predetermined area such as for each room or for each floor of a single building, the model generation unit 116 may execute a process of integrating (merging) the plurality of 3D models.

[0053] Figure 12(a) is an example of the generated 3D model. The data structure of the 3D model output by the model generation unit 116 is not particularly limited, and the 3D model may be, for example, BIM (Building Information Modeling) data, CIM (Construction Information Modeling) data, CAD data, or GML data such as City - GML, and may be constructed in a plurality of data formats.

[0054] The processing unit 110 of the management server 1 may include a floor plan generation unit 117 that generates a floor plan of the target space within the building. The floor plan generation unit 117 may output a floor plan from the 3D model generated by the model generation unit 116, or may directly output a floor plan based on the information generated by the plane detection unit 112, the corner estimation unit 113, the height calculation unit 114, and the component identification unit 115. Figure 12(b) is an example of the generated floor plan. Although omitted in Figure 12(b), the floor plan generation unit 117 may output a floor plan having dimension information and / or scale information. Note that the data structure of the floor plan output by the floor plan generation unit 117 is not particularly limited.

[0055] The three-dimensional model and the plan view generated by the model generation unit 116 and the plan view generation unit 117 are stored in the plan view / three-dimensional model storage unit 121. Note that when storing the three-dimensional model and the plan view, the model generation unit 116 and the plan view generation unit 117 may receive information related to the target space input from the user terminal 2. For example, information such as building name, number of floors, room name, and address is received, and the received information is associated with the processing results such as the three-dimensional model and / or the plan view to be output, and then stored in the plan view / three-dimensional model storage unit 121.

[0056] <Functions of the user terminal 2> As shown in FIG. 4, the user terminal 2 includes a processing unit 210, a storage unit 220, a display unit 230, and a terminal communication unit 240. The processing unit 210 of the user terminal 2 may include an information acquisition unit 211, a self-position estimation unit 212, and an operation information reception unit 213. Similar to the processing unit 110 of the management server 1, the processing unit 210 of the user terminal 2 is realized by reading data and programs stored in the storage 22 by the processor 20 and executing various programs in the work area of the memory 21.

[0057] The storage unit 220 is a storage area of the user terminal 2 that stores programs for executing various control processes and each function within the user terminal 2, input data, etc., and may store the processing results by various functional units of the processing unit 210. For example, the storage unit 220 may have an acquisition information storage unit 221 that stores various data such as photographed images, sensing data, and self-position estimation results obtained when the target space within the building is scanned by the user terminal 2. The display unit 230 is a display on which output results such as photographed images when scanning the target space, the augmented reality space generated by the AR space generation unit 111, and the constructed 3D model are displayed, and the display may be a touch panel capable of receiving input operations of the measurer (user) by contact with the display screen. The terminal communication unit 240 communicates with the management server 1 via the network NW. The terminal communication unit 240 may have a function as a transmission unit that transmits photographed images and sensing data acquired by the user terminal 2 to the management server 1, and a function as a reception unit that receives processing results and the like sent from the management server 1.

[0058] The information acquisition unit 211 acquires photographed images and sensing data when photographing (scanning) the target space within the building from the camera 25 and the sensor 26, and stores them in the acquisition information storage unit 221 as appropriate. Further, the information acquisition unit 211 transmits the acquired photographed images and sensing data to the management server 1 via the terminal communication unit 240.

[0059] The self-position estimation unit 212 estimates the current self-position of the user terminal 2 when photographing (scanning) the target space within the building by the camera 25 and the sensor 26. Further, the self-position estimation unit 212 may generate movement history information indicating the transition of the self-position during the execution of the scan, the distance from the photographing start point to each photographing position, the distance between each photographing position, and the like. Note that the self-position may be acquired by a known method. For example, the self-position may be estimated from sensing data of an inertial sensor and / or LiDAR.

[0060] When photographing (scanning) a target space in a building, the operation information reception unit 213 receives input operations by a measurer via the display unit 230 (for example, an operation of switching between an auto mode and a manual mode during corner estimation, an operation of designating a marker, an operation of designating an attribute of a component, etc.), and transmits the operation information input by the measurer to the management server 1 via the terminal communication unit 240.

[0061] <An example of 3D model generation process> Next, an example of a 3D model generation method will be described based on the flowchart shown in FIG. 13.

[0062] First, when the camera 25 is activated and the photographing of the target space in the building is started (step S101), the process of generating an augmented reality space by the AR space generation unit 111 is also started (step S102). The AR space generation unit 111 constructs an augmented reality space based on the photographed image and sensing data sent in real time from the user terminal 2. In step S103, when the camera 25 is directed at any plane of the target space and the plane is photographed, the plane detection unit 112 recognizes the plane in the photographed image based on the photographed image and sensing data sent in real time from the user terminal 2, and generates plane information for specifying the arrangement of the plane in the augmented reality space (step S104). After any one plane is recognized, when continuing the scan for other planes, the process returns to step S103 to obtain plane information corresponding to the other planes. In the present embodiment, at least plane information regarding two horizontal planes (the first horizontal plane and the second horizontal plane) partitioning the target space is obtained. In the flowchart shown in FIG. 13, the plane scan process is shown from step S103 to step S105, but the plane information may be obtained in the corner scan process of step S106 and / or step S110, etc.

[0063] In step S106, when the camera 25 is directed toward the vicinity of any corner of the target space and the corner is photographed, the corner estimation unit 113 estimates the corners in the building based on at least one plane information, and generates corner position information indicating the relative position of the corners with respect to the photographing position in the augmented reality space. After one piece of corner position information is acquired, another corner adjacent to the one corner is photographed, and corner position information is acquired by the same processing as above. The process of scanning the corner (steps S106 to S107) is repeated until each corner on the first horizontal plane is estimated. When corner position information is acquired for each corner on the first horizontal plane, the process proceeds to step S108. The first horizontal plane on the side where the corner position information is acquired is preferably the floor surface, but is not necessarily limited thereto, and corner position information may be acquired for each corner on the ceiling side.

[0064] In step S108, the height calculation unit 114 calculates the perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space based on the relative position of the first horizontal plane with respect to the photographing position and the relative position of the second horizontal plane with respect to the photographing position, and generates height information in the building. Note that the photographing position when acquiring the plane information regarding the first horizontal plane and the photographing position when acquiring the second horizontal plane may be the same position or different positions. After the height information is acquired, when arranging components existing in the target space such as beams, columns, and fixtures in the three-dimensional model, the process proceeds to step S110. On the other hand, when there are no components in the target space, or when generating a simple three-dimensional model that does not reflect the components, the photographing of the target space in the building is terminated (the camera 25 is turned off), and the process proceeds to step S112. In this case, the model generation unit 116 generates a three-dimensional model of the target space based on the plurality of corner position information plotted in the augmented reality space and the height information.

[0065] In step S110, the camera 25 is directed at the component, and the component of the specific target is photographed. At this time, the designation of markers for at least two corners of the component is accepted. Based on the position coordinates of the corners (relative coordinates with respect to the photographing position) designated by the measurer and the plane information corresponding to the plane around the component, the component information (column and beam information, fixture information) that specifies the arrangement of the component in the augmented reality space is generated by the component specifying unit 115. After specifying the arrangement of one component, if other components are to be specified subsequently, step S110 is repeated. When the position information regarding the desired component has been acquired, the photographing of the target space within the building is terminated (the camera 25 is turned off), and the process proceeds to step S112. When reaching step S112 after going through the component scanning step S110, the model generation unit 116 may generate a three-dimensional model in which the arrangement of the components is reflected based on the plurality of corner position information, height information, and component information plotted in the augmented reality space.

[0066] As described above, in the three-dimensional modeling of the space inside the building provided by the information processing system of the present embodiment, instead of directly specifying the corners of the space inside the building from a photographed image or the like, the corner position information is acquired from the plane information around the corners. By such a corner estimation process, even when there are obstacles such as furniture (obstacles that prevent the imaging of corners) in the space inside the building, the shape of the space inside the building can be accurately and easily specified. Also, in the three-dimensional modeling of the present embodiment, the height information of the space inside the building is also specified using the plane information in the same manner as the corner position information, and the working time required for recording the shape of the space inside the building can be shortened compared to the conventional method. In particular, in the three-dimensional modeling of the present embodiment, a three-dimensional model can be constructed by specifying only the corners on one horizontal plane without specifying all the corners, and the efficiency of the processing for three-dimensional modeling can be improved.

[0067] In addition, in the 3D modeling of this embodiment, during the shooting of the interior space of a building, an augmented reality space is constructed by real-time processing, and the plane information, corner position information, and height information identified during the shooting are reflected in the augmented reality space simultaneously with the identification to construct a 3D model. Through such 3D modeling, it is possible to create a 3D model and a floor plan simultaneously with the measurement of the interior space of a building. In addition, by constructing a 3D model on the augmented reality space constructed by real-time processing, various dimensional information of the real space (such as the distance between corners and the height of a room in the real space) can be easily obtained, and a 3D model with a scale matching the actual dimensions can be easily constructed.

[0068] The information processing system of this embodiment can be applied, for example, to the creation of floor plans and indoor models used in the real estate industry, the creation of maps and 3D models for operating unmanned moving bodies, and the like.

[0069] The above-described embodiments are merely examples for facilitating the understanding of the present disclosure, and are not intended to limit the interpretation of the present disclosure. The present disclosure can be changed and improved without departing from its spirit, and it goes without saying that equivalents of the present disclosure are included therein.

[0070] For example, in the above embodiment, an example of identifying mainly the corners on the floor surface side and constructing a 3D model has been described, but it is also possible to obtain only the corner position information on the ceiling side and construct a 3D model. Further, in the description regarding the component identification unit 115, beams, columns, and fixtures have been exemplified as components, but the components that can be identified by the processing related to the component identification unit 115 are not limited to these. For example, the component identification unit 115 may execute processing for identifying various fixtures such as furniture and home appliances, and accessories such as whiteboards and screens installed on the wall surface.

[0071] In the above embodiment, various functional units such as the AR space generation unit 111, the plane detection unit 112, the corner estimation unit 113, the height calculation unit 114, the structure identification unit 115, the model generation unit 116, and the floor plan generation unit 117 were described as functions of the management server 1. However, these functions may be realized on the user terminal 2 side based on a program transmitted from the management server 1. Further, in the above embodiment, sensing inside a structure by a terminal (user terminal 2) carried by a user such as a smartphone was exemplified. However, any configuration may be used as long as the inside of the building can be sensed by a sensor. For example, scanning inside the building may be performed using a moving object such as a drone or a UGV (unmanned ground vehicle).

Description of Reference Numerals

[0072] 1 Management server 2 User terminal

Claims

1. An AR space generation unit that generates an augmented reality space based on scan data obtained by photographing the interior of a building using a photographing terminal equipped with a camera and a sensor, A plane detection unit that recognizes a plane inside the building based on a photographed image and sensing data included in the scan data, and generates plane information that specifies the arrangement of the plane in the augmented reality space; A corner estimation unit that estimates a corner inside the building based on at least one of the plane information, and generates corner position information indicating a relative position of the corner with respect to a photographing position in the augmented reality space; Based on a relative position of the first horizontal plane with respect to the photographing position included in the plane information regarding the first horizontal plane, and a relative position of the second horizontal plane with respect to the photographing position included in the plane information regarding the second horizontal plane, a height calculation unit that calculates a perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space and generates height information inside the building; An information processing system comprising: a model generation unit that generates a three-dimensional model inside the building based on the plurality of corner position information plotted in the augmented reality space and the height information.

2. The information processing system according to claim 1, wherein the corner estimation unit estimates an intersection of the first horizontal plane, the first vertical plane, and the second vertical plane adjacent to each other as the corner inside the building based on the plane information regarding the first horizontal plane, the plane information regarding the first vertical plane, and the plane information regarding the second vertical plane, and generates the corner position information.

3. The plane detection unit generates the plane information regarding the first horizontal plane and the plane information regarding a first vertical plane adjacent to the first horizontal plane, The information processing system according to claim 1, wherein the corner estimation unit receives a designation of a marker for a point presumed to be the corner via a display unit of the photographing terminal, and estimates the corner inside the building based on the position information of the marker and the plane information regarding the first horizontal plane, and generates the corner position information.

4. The information processing system according to claim 1, wherein the corner estimation unit receives a designation of a marker for a point presumed to be the corner via a display unit of the photographing terminal, and estimates the corner inside the building based on the position information of the marker and the plane information regarding the first horizontal plane, and generates the corner position information.

5. The first horizontal plane is the floor surface inside the building, The corner estimation unit generates the corner position information corresponding to each corner existing on the floor surface, respectively, for the information processing system according to any one of claims 1 to 4.

6. The information processing system further includes a component identification unit that generates component information for identifying the arrangement of components existing in the building. The model generation unit generates the three-dimensional model in which the arrangement of the components is reflected based on the plurality of corner position information, the height information, and the component information plotted in the augmented reality space, for the information processing system according to any one of claims 1 to 4.

7. The component identification unit receives, via the display unit of the photographing terminal, a designation of at least two column-beam markers corresponding to at least two corners of the column-beam components existing in the building, and generates column-beam information for specifying the arrangement of the column-beam components in the augmented reality space based on the position information of the column-beam markers and the plane information regarding the plane adjacent to the column-beam components, for the information processing system according to claim 6.

8. The component identification unit receives, via the display unit of the photographing terminal, a designation of at least two furniture markers corresponding to at least two corners of the furniture existing in the building, and generates furniture information for specifying the arrangement of the furniture in the augmented reality space based on the position information of the furniture markers and the plane information regarding the plane on which the furniture is installed, for the information processing system according to claim 6.

9. Generating an augmented reality space based on scan data obtained by photographing the inside of a building using a photographing terminal equipped with a camera and a sensor; Recognizing a plane inside the building based on the photographed image and sensing data included in the scan data, and generating plane information for specifying the arrangement of the plane in the augmented reality space; Estimating a corner inside the building based on at least one of the plane information, and generating corner position information indicating the relative position of the corner with respect to the photographing position in the augmented reality space; Calculating the perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space based on the relative position of the first horizontal plane with respect to the photographing position included in the plane information regarding the first horizontal plane and the relative position of the second horizontal plane with respect to the photographing position included in the plane information regarding the second horizontal plane, and generating height information inside the building. An information processing method executed by a computer, comprising: generating a three-dimensional model of the building based on the plurality of corner position information plotted in the augmented reality space and the height information. **Claim 10** A process of generating an augmented reality space based on scan data obtained by photographing the interior of a building using a photographing terminal equipped with a camera and a sensor; A process of recognizing a plane in the building based on a photographed image and sensing data included in the scan data, and generating plane information for specifying the arrangement of the plane in the augmented reality space; A process of estimating a corner in the building based on at least one of the plane information, and generating corner position information indicating the relative position of the corner with respect to the photographing position in the augmented reality space; A process of calculating a perpendicular distance from the first horizontal plane to the second horizontal plane in the augmented reality space based on the relative position of the first horizontal plane with respect to the photographing position included in the plane information regarding the first horizontal plane and the relative position of the second horizontal plane with respect to the photographing position included in the plane information regarding the second horizontal plane, and generating height information of the building; A program for causing a computer to execute a process of generating a three-dimensional model of the building based on the plurality of corner position information plotted in the augmented reality space and the height information.

Images