Data processing method, recording medium, program, and system

Abstract

To provide a novel technique for the practical application and operation of architectural-structure reality capture.SOLUTION: A method of processing data in an embodiment includes: preparing design data including virtual-member information about an attribute of each virtual member of a virtual architectural structure, measurement data acquired from a real architectural structure constructed based on the design data, and a virtual image of the virtual architectural structure; generating real-member data of an attribute of each real member based on the measurement data, associating the virtual member and the real member together based on the virtual-member information and the real-member data, and associating the virtual-member information and the real-member data together based on the attribute in each pair of the virtual and real members obtained by the association; and furthermore creating movement-control information for acquiring data of the architectural structure using a moving body, based on the virtual image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a data processing method, a recording medium, a program, and a system for architectural reality capture. [Background technology]

[0002] Reality capture is a technology that acquires data of real objects using a digital camera or laser scanner to construct a 3D model, and is used in various fields such as measurement, virtual reality, and augmented reality (see, for example, Patent Documents 1 and 2). In recent years, the application of reality capture in the fields of architecture and civil engineering has been attracting attention.

[0003] In the field of architecture, the application of reality capture is being promoted for construction management, maintenance management, repair management, etc., and efforts are being made to integrate it with the following technologies (see, for example, Patent Documents 3 to 12): mobile objects such as unmanned aerial vehicles (UAVs); surveying equipment such as total stations; data processing technologies such as Structure from Motion (SfM), Multi-View Stereo (MVS), and Simultaneous Localization And Mapping (SLAM); and Building Information Modeling (BIM).

[0004] To put such integrated systems into practical use and operate them in such fields, it is necessary to manage various types of data and information in an integrated, efficient, and consistent manner, but such management methods and systems have not yet been realized. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Publication No. 2016 / 0034137 [Patent Document 2] European Patent Publication No. 3522003 [Patent Document 3] Japanese Patent Application Publication No. 2018-116572 [Patent Document 4] Japanese Patent Application Publication No. 2018-119882 [Patent Document 5] Japanese Patent Application Laid-Open No. 2018-124984 [Patent Document 6] Japanese Patent Application Publication No. 2018-151964 [Patent Document 7] Japanese Patent Application Publication No. 2019-023653 [Patent Document 8] Japanese Patent Application Publication No. 2019-105789 [Patent Document 9] Japanese Patent Application Publication No. 2019-194883 [Patent Document 10] Japanese Patent Application Publication No. 2019-219206 [Patent Document 11] Japanese Patent Application Publication No. 2020-004278 [Patent Document 12] Japanese Patent Publication No. 2020-008423 Summary of the Invention [Problem to be solved by the invention]

[0006] One object of the present invention is to provide a new technology for the practical application and operation of architectural reality capture. [Means for solving the problem]

[0007] Some exemplary aspects include a data structure to be processed by an architectural reality capture system, the structure including: design data prepared in advance, including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building; and physical component data regarding the multiple attributes for each of multiple physical components, generated based on measurement data acquired from a physical building constructed based on the design data; and the structure is used for a component matching process that matches the multiple virtual components and the multiple physical components based on the virtual component information and the physical component data; and an attribute matching process that matches the virtual component information and the physical component data according to the multiple attributes for each of multiple pairs of virtual components and physical components acquired by the component matching process, the virtual component information including virtual component position information, and the physical component data including physical component position data, and the virtual component position information and the physical component position data are used in the component matching process.

[0008] Some exemplary aspects are data structures to be processed by an architectural reality capture system, including design data prepared in advance, including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building; physical component data regarding the multiple attributes for each of multiple physical components, generated based on measurement data acquired from a physical building constructed based on the design data; and a virtual image of the virtual building, generated in advance, and used for a component matching process that matches the multiple virtual components with the multiple physical components based on the virtual component information and the physical component data; an attribute matching process that matches the virtual component information with the physical component data according to the multiple attributes for each of multiple pairs of virtual components and physical components acquired by the component matching process; and a movement control information creation process that creates movement control information for acquiring building data using a moving object based on the virtual image.

[0009] A data structure according to some exemplary aspects further includes a photographed image acquired in advance, the photographed image being used in the movement control information creation process.

[0010] A data structure according to some exemplary embodiments further includes an inference model for identifying images of building components from photographed images of a building, constructed by applying machine learning using at least the virtual image to a predetermined neural network, and the inference model is used in the movement control information creation process.

[0011] Some exemplary aspects are a data structure to be processed by an architectural reality capture system, comprising: design data prepared in advance, including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; real component data regarding the plurality of attributes for each of a plurality of real components, generated based on measurement data acquired from a real building constructed based on the design data; and a pre-generated virtual image of the virtual building, and are used for a component matching process that matches the plurality of virtual components with the plurality of real components based on the virtual component information and the real component data; an attribute matching process that matches the virtual component information with the real component data according to the plurality of attributes for each of a plurality of pairs of virtual components and real components acquired by the component matching process; and a reference information creation process that creates reference information based on the virtual image for determining whether architectural component data has been acquired when acquiring building data using a mobile object.

[0012] According to some exemplary aspects, the data structure further includes a photographed image acquired in advance, the photographed image being used in the reference information creation process.

[0013] A data structure according to some exemplary embodiments further includes an inference model for identifying images of building components from photographed images of a building, constructed by applying machine learning using at least the virtual image to a predetermined neural network, and the inference model is used in the reference information creation process.

[0014] A data structure according to some exemplary embodiments, a data structure to be processed by an architectural reality capture system, comprising: design data prepared in advance, including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; physical component data regarding the plurality of attributes for each of a plurality of physical components, generated based on measurement data acquired from a physical building constructed based on the design data; a virtual image of the virtual building generated in advance; and data of the building acquired in advance, and is used for a data object detection process, which is a process of detecting a data object from data of the building based on the virtual image; a component correspondence process, which is a correspondence between the plurality of virtual components and the plurality of physical components based on the virtual component information, the physical component data, and the data objects detected by the data object detection process; and an attribute correspondence process, which is a correspondence between the virtual component information and the physical component data according to the plurality of attributes for each of a plurality of pairs of virtual components and physical components acquired by the component correspondence process.

[0015] According to some exemplary aspects, the data structure further includes a previously acquired photographed image, which is used in the data object detection process.

[0016] A data structure according to some exemplary embodiments further includes an inference model for identifying images of building components from photographed images of a building, constructed by applying machine learning using at least the virtual image to a predetermined neural network, and the inference model is used in the data object detection process.

[0017] Some exemplary aspects are data structures processed by an architectural reality capture system, including design data prepared in advance, including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building; representative component information created in advance indicating a representative component of the virtual component and measurement data acquired from a physical building constructed based on the design data; and physical component data regarding the multiple attributes for each of the multiple physical components, generated based on the measurement data, and are used for a component matching process that matches the multiple virtual components with the multiple physical components based on the virtual component information and the physical component data; an attribute matching process that matches the virtual component information with the physical component data according to the multiple attributes for each of multiple pairs of virtual components and physical components acquired by the component matching process; and a partial area identification process that identifies a partial area of ​​the measurement data corresponding to a representative component of any of the multiple virtual components based on the representative component information.

[0018] In a data structure according to some exemplary aspects, the partial area identified by the partial area identification process is used in a substantial component data generation process, which is a process for generating the substantial component data from the measurement data.

[0019] Some exemplary aspects include a data structure to be processed by an architectural reality capture system, the data structure including: design data prepared in advance, including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building; component selection information indicating some of the virtual components among the multiple virtual components, generated in advance based on the design data; measurement data acquired from a physical building constructed based on the design data; and physical component data regarding the multiple attributes for each of the multiple physical components, generated based on the measurement data; and the data structure is used for a component matching process that matches the multiple virtual components with the multiple physical components based on the virtual component information and the physical component data; an attribute matching process that matches the virtual component information with the physical component data according to the multiple attributes for each of multiple pairs of virtual components and physical components acquired by the component matching process; and a physical component data generation process that generates the physical component data from a partial area of ​​the measurement data corresponding to the some of the virtual components indicated in the component selection information.

[0020] Some exemplary aspects are a data structure to be processed by a building reality capture system, including design data prepared in advance, including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building, and real component data regarding the multiple attributes for each of multiple real components, generated based on measurement data acquired from a real building constructed based on the design data, and are used for a component matching process that matches the multiple virtual components with the multiple real components based on the virtual component information and the real component data, and an attribute matching process that matches the virtual component information with the real component data according to the multiple attributes for each of multiple pairs of virtual components and real components acquired by the component matching process, wherein the virtual component information includes construction date information indicating the construction date of each of the multiple virtual components, and the real component data includes measurement date information indicating the measurement date of the real building, and the construction date information and the measurement date information are used in the component matching process.

[0021] Some exemplary embodiments are computer-readable non-transitory recording media having data structured according to the exemplary embodiments recorded thereon.

[0022] Some exemplary embodiments are programs that cause a computer included in an architectural reality capture system to execute processing using data having a structure according to the exemplary embodiments.

[0023] Some exemplary aspects are a computer-readable non-transitory recording medium on which a program according to the exemplary aspects is recorded.

[0024] Some exemplary aspects include a system for reality capture of a building, comprising: a memory unit that stores design data including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; and real component data regarding the plurality of attributes for each of a plurality of real components, generated based on measurement data acquired from a real building constructed based on the design data; a component matching unit that associates the plurality of virtual components with the plurality of real components based on the virtual component information and the real component data to generate a plurality of pairs of virtual components and real components; and an attribute matching unit that associates the virtual component information with the real component data for each of the plurality of pairs according to the plurality of attributes, wherein the virtual component information includes virtual component position information and the real component data includes real component position data, and the component matching unit generates the plurality of pairs based on the virtual component position information and the real component position data.

[0025] Some exemplary aspects are a system for reality capture of a building, including: a memory unit that stores design data including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; a virtual image of the virtual building; and real component data regarding the plurality of attributes for each of a plurality of real components, generated based on measurement data acquired from a real building constructed based on the design data; a component matching unit that associates the plurality of virtual components with the plurality of real components based on the virtual component information and the real component data, to generate a plurality of pairs of virtual components and real components; an attribute matching unit that associates the virtual component information with the real component data for each of the plurality of pairs according to the plurality of attributes; and a movement control information creation unit that creates movement control information for acquiring building data using a moving object based on the virtual image.

[0026] In the system according to some exemplary aspects, the storage unit further stores a captured image, and the movement control information creation unit further creates the movement control information based on the captured image.

[0027] A system according to some exemplary embodiments further includes an inference model construction unit that applies machine learning using at least the virtual image to a predetermined neural network to construct an inference model for identifying images of building components from photographed images of a building, and the movement control information creation unit creates the movement control information based on the inference model.

[0028] Some exemplary aspects are a system for reality capture of a building, including: a memory unit that stores design data including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; a virtual image of the virtual building; and real component data regarding the plurality of attributes for each of a plurality of real components, generated based on measurement data acquired from a real building constructed based on the design data; a component matching unit that associates the plurality of virtual components with the plurality of real components based on the virtual component information and the real component data, to generate a plurality of pairs of virtual components and real components; an attribute matching unit that associates the virtual component information with the real component data for each of the plurality of pairs according to the plurality of attributes; and a reference information creation unit that creates reference information, based on the virtual image, for determining whether data of a building component has been acquired when data of a building is acquired using a moving object.

[0029] In the system according to some exemplary aspects, the storage unit further stores a captured image, and the reference information creation unit further creates the reference information based on the captured image.

[0030] A system according to some exemplary embodiments further includes an inference model construction unit that applies machine learning using at least the virtual image to a predetermined neural network to construct an inference model for identifying images of building components from photographed images of a building, and the reference information creation unit creates the reference information based on the inference model.

[0031] Some exemplary aspects are a system for reality capture of a building, including: design data including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; a memory unit that stores a virtual image of the virtual building; real component data regarding the plurality of attributes for each of a plurality of real components generated based on measurement data acquired from a real building constructed based on the design data; and building data; a data object detection unit that detects data objects from the building data based on the virtual image; a component matching unit that generates a plurality of pairs of virtual components and real components by performing correspondence between the plurality of virtual components and the plurality of real components based on the virtual component information, the real component data, and the data objects; and an attribute matching unit that performs correspondence between the virtual component information and the real component data for each of the plurality of pairs according to the plurality of attributes.

[0032] In a system according to some exemplary aspects, the storage unit further stores a captured image, and the data object detection unit further detects the data object based on the captured image.

[0033] A system according to some exemplary embodiments further includes an inference model construction unit that applies machine learning using at least the virtual image to a predetermined neural network to construct an inference model for identifying images of building components from photographed images of a building, and the data object detection unit further detects the data object based on the inference model.

[0034] Some exemplary aspects are a system for reality capture of a building, including: a memory unit that stores design data including virtual component information regarding a plurality of attributes for each of a plurality of virtual components of a virtual building; representative component information indicating a representative component of the virtual component; measurement data acquired from a real building constructed based on the design data; and real component data regarding the plurality of attributes for each of a plurality of real components, generated based on the measurement data; a component matching unit that matches the plurality of virtual components with the plurality of real components based on the virtual component information and the real component data, thereby generating a plurality of pairs of virtual components and real components; an attribute matching unit that matches the virtual component information with the real component data for each of the plurality of pairs according to the plurality of attributes; and a partial area identification unit that identifies a partial area of ​​the measurement data that corresponds to a representative component of any of the plurality of virtual components based on the representative component information.

[0035] According to some exemplary embodiments, the system further includes a first substantial component data generator that generates the substantial component data from the measurement data based on the partial region identified by the partial region identifier.

[0036] Some exemplary embodiments are a system for reality capture of a building, including: a memory unit that stores design data including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building; component selection information indicating some of the virtual components among the multiple virtual components generated in advance based on the design data; measurement data acquired from a real building constructed based on the design data; and real component data regarding the multiple attributes for each of the multiple real components generated based on the measurement data; a component correspondence unit that associates the multiple virtual components with the multiple real components based on the virtual component information and the real component data to generate multiple pairs of virtual components and real components; an attribute correspondence unit that associates the virtual component information with the real component data for each of the multiple pairs according to the multiple attributes; and a second real component data generation unit that generates the real component data from a partial area of ​​the measurement data corresponding to the some of the virtual components based on the component selection information.

[0037] Some exemplary aspects are a system for reality capture of a building, comprising: a memory unit that stores design data including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building; and real component data regarding the multiple attributes for each of the multiple real components, generated based on measurement data acquired from a real building constructed based on the design data; a component matching unit that associates the multiple virtual components with the multiple real components based on the virtual component information and the real component data to generate multiple pairs of virtual components and real components; and an attribute matching unit that associates the virtual component information with the real component data for each of the multiple pairs according to the multiple attributes, wherein the virtual component information includes construction date information indicating the construction date of each of the multiple virtual components, and the real component data includes measurement date information indicating the measurement date of the real building, and the component matching unit further generates the multiple pairs based on the construction date information and the measurement date information. [Effects of the Invention]

[0038] According to example aspects, new techniques for the practical application and operation of architectural reality capture are provided. [Brief explanation of the drawings]

[0039] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 2] FIG. 10 is a schematic diagram illustrating an example of a configuration of virtual member information according to an exemplary embodiment. [Figure 3] FIG. 10 is a schematic diagram illustrating an example of a data structure of virtual member information according to an exemplary embodiment. [Figure 4] FIG. 10 is a schematic diagram illustrating an example of a configuration of actual component data according to an exemplary embodiment. [Figure 5] FIG. 2 is a schematic diagram illustrating an example of a data structure of physical component data according to an exemplary embodiment. [Figure 6] FIG. 10 is a schematic diagram illustrating an example of a component association process executed by a system according to an exemplary embodiment. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of an attribute association process executed by a system according to an exemplary embodiment. [Figure 8] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 9] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 10] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 11] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 12] FIG. 2 is a schematic diagram illustrating an example of processing executed by a system according to an exemplary embodiment. [Figure 13] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 14] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment of the present invention. [Figure 15]1 is a schematic diagram illustrating an example of a data format (data configuration) used in a system according to an exemplary embodiment of the present invention. [Figure 16A] 10 is a flowchart illustrating an example of the operation of a system according to a usage pattern of an exemplary embodiment. [Figure 16B] 10 is a flowchart illustrating an example of the operation of a system according to a usage pattern of an exemplary embodiment. [Figure 16C] 10 is a flowchart illustrating an example of the operation of a system according to a usage pattern of an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0040] The following disclosure describes several exemplary aspects of a data structure, a recording medium on which structured data is recorded, a program, a recording medium on which a program is recorded, and a system. The exemplary aspects can be used to suitably implement and operate an architectural reality capture system, but can also be applied to reality capture systems in other fields. Furthermore, the matters described in the documents cited in this specification and any other known technologies can be incorporated into the exemplary aspects.

[0041] At least a portion of the functionality of the elements described in this disclosure is implemented using circuitry or processing circuitry. The circuitry or processing circuitry may be a general-purpose processor, a special-purpose processor, an integrated circuit, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA)), or a combination of these devices configured and / or programmed to perform at least a portion of the disclosed functionality. The term "circuitry," "unit," "means," or the like refers to hardware that performs at least a portion of the disclosed functions or that is programmed to perform at least a portion of the disclosed functions. The hardware may be the hardware disclosed herein or may be known hardware that is programmed and / or configured to perform at least a portion of the described functions. In the case of a processor, where the hardware can be considered a type of circuitry, the term "circuitry," "unit," "means," or the like refers to a combination of hardware and software, where the software is used to configure the hardware and / or the processor.

[0042] The exemplary aspects described below may be combined in any manner, for example, two or more exemplary aspects may be at least partially combined.

[0043] <First aspect> In the first aspect, elements and matters common to the second to sixth aspects will be described. An example configuration of a system according to this aspect is shown in FIG. 1. System 1 is included in a building reality capture system. The building reality capture system has the function of measuring an actual building and acquiring digital data. The building reality capture system (system 1) of this aspect is configured to provide a data structure (format) for facilitating comparison between measurement data of the actual building and design data. The reality capture system (system 1) of this aspect includes, for example, a data management system and a measurement system.

[0044] The system 1 according to this embodiment includes at least a storage unit 14 and a processing unit 15, and further includes a control unit 11, a user interface 12, and a data acquisition unit 13.

[0045] The control unit 11 is configured to execute various controls of the system 1. The control unit 11 is realized, for example, by cooperation between hardware including a processor and a storage device and control software. The control unit 11 is provided in one computer, or distributed across two or more computers.

[0046] The user interface 12 includes, for example, a display device, an operation device, an input device, etc. In some exemplary embodiments, the user interface 12 includes a graphical user interface (GUI) that uses a touch screen, a pointing device, computer graphics, etc. The user interface 12 may be provided on one computer or distributed across two or more computers.

[0047] The data acquisition unit 13 is configured to perform either or both of data generation and data reception. The data generation function includes, for example, a function of collecting data from entities, a function of processing data collected from entities, a function of generating data using a computer, and a function of processing pre-generated data.

[0048] The function of collecting data from an actual object includes, for example, a function of photographing the actual object using a camera (omnidirectional camera) or a video camera (omnidirectional video camera) mounted on a mobile object such as an unmanned aerial vehicle (UAV) or a worker, and a function of scanning the actual object using a laser scanner or a total station to collect data. The data acquisition unit 13 for this function may include one or more measuring devices.

[0049] The function of processing data collected from a physical object is realized, for example, by using at least a processor, and includes a function of applying predetermined processing to photographed images or scanned data of the physical object to generate other data. Examples of such a function include the data processing function using the aforementioned SfM, MVS, SLAM (V-SLAM; Visual SLAM), etc. Another example is a data processing function using a trained model constructed using machine learning. The data acquisition unit 13 for this function is provided in one computer or distributed across two or more computers.

[0050] The function of generating data using a computer includes, for example, a function of generating data using computer graphics, such as a function of generating data (referred to as BIM data) using a BIM application or a function of generating data (referred to as CAD data) using a CAD (Computer-Aided Design) application. The function of generating data using a computer also includes a function of generating data using various architectural applications, such as a construction management application, a maintenance management application, and a repair management application. The data acquisition unit 13 for this function may be provided in one computer or distributed across two or more computers.

[0051] The function of processing pre-generated data is realized, for example, using at least a processor, and includes a function of applying predetermined processing to data of an entity previously acquired and / or processed by either the system 1, another device, or another system to generate other data. The technology applicable to this function may be the same as the technology applicable to the function of processing data collected from an entity. An example of pre-generated data is BIM data. The data acquisition unit 13 for this function may be provided in one computer or distributed across two or more computers.

[0052] The data reception function is a function for receiving data from an external source. The data reception function may be realized, for example, using a communication device for communicating data with an external device (such as a device, system, or database), or may be realized using a drive device for reading data recorded on a recording medium. The data received from an external source by the data acquisition unit 13 may be, for example, data generated using a computer (such as BIM data or CAD data), or may be data previously acquired and / or processed by the system 1, another device, or another system. Furthermore, a recording medium applicable to the data reception function may be a computer-readable non-transitory recording medium, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0053] In this embodiment, the entity is a building. The building is constructed based on pre-generated design data (BIM data, design documents, construction drawings, construction plans, etc.). In this disclosure, the building data recorded in the design data (and / or building data obtained by processing the design data) may be referred to as a virtual building, and its components (building components) may be referred to as virtual components. In some exemplary embodiments, the virtual components are component models provided by a BIM model, and a building constructed using multiple component models is a virtual building.

[0054] Meanwhile, in this disclosure, a building constructed based on design data may be referred to as a real building, and its constituent elements (building components) may be referred to as real components. The real building corresponds to the real building referred to here. A real building may not only be a building completed based on design data, but also a building in the middle of construction (an unfinished building), or even a construction site before construction.

[0055] In the present disclosure, a building component may include structural components such as columns, beams, walls, slabs, roofs, and foundations, as well as non-structural components such as windows, doors, stairs, tiles, and flooring, as well as various parts, machines, equipment, and facilities. More generally, a building component in the present disclosure may be any object that can be registered as a virtual component, or any object that can be used as a physical component.

[0056] The storage unit 14 is configured to store various data (information). The storage unit 14 stores, for example, data acquired by the data acquisition unit 13. The storage unit 14 includes, for example, a relatively large-capacity storage device (memory, storage device) such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage unit 14 includes one storage device or two or more storage devices. In this embodiment, the storage device 14 stores design data 141 and actual component data 142.

[0057] The design data 141 may be any data or information related to the design of a building. The design data 141 may include, for example, BIM data, design documents, construction drawings, construction plans, etc. The design data 141 may also include data generated from any one or more of any data or information (for example, BIM data, design documents, construction drawings, construction plans).

[0058] In some exemplary embodiments, the design data 141 is data of a virtual building (multiple virtual components) designed using a BIM application (BIM tool) external to the system 1.

[0059] The design data 141 of this embodiment includes virtual component information. The virtual component information includes information about multiple virtual components that are components of the virtual building. More specifically, the virtual component information includes information about multiple pre-set attributes for each virtual component. The attributes here refer to the properties, characteristics, properties, etc., of the virtual component.

[0060] In some exemplary embodiments, the attributes of the virtual component include, for example, virtual component identification information (virtual component ID), virtual component shape information, virtual component position information, component construction date information, etc. Note that the attributes of the virtual component are not limited to these, and may be, for example, any property, characteristic, or property such as material.

[0061] An example of virtual component information is shown in Fig. 2. The virtual component information 2 according to this example includes a virtual component ID 21, virtual component shape information 22, virtual component position information 23, and component construction date information 24.

[0062] The virtual component ID 21 is information for identifying a virtual component. The virtual component ID 21 indicates the type of the virtual component (column, beam, wall, slab, roof, foundation, window, door, staircase, tile, flooring, part, machine, device, equipment, etc.). The virtual component ID 21 may be, for example, identification information (such as a component number) assigned to a real component. The virtual component ID 21 is acquired, for example, from BIM data, design documents, construction drawings, etc. The virtual component ID 21 may also be individually unique identification information. An example of such a virtual component ID 21 is identification information provided by Industry Foundation Classes (IFC), a neutral and open file format for CAD data models.

[0063] The virtual member shape information 22 is information that represents the shape of a virtual member. The virtual member shape information 22 may include information that represents the posture, orientation, etc. of the virtual member. The virtual member shape information 22 is acquired from, for example, BIM data, design documents, construction drawings, etc.

[0064] The virtual component position information 23 indicates the position of a virtual component in a virtual building. The position of a virtual component is expressed, for example, by the coordinates of the virtual component in a virtual space in which the virtual building is defined (a three-dimensional virtual space defined in a three-dimensional coordinate system). The virtual component position information 23 is acquired, for example, from BIM data, design documents, construction drawings, etc.

[0065] The member construction date information 24 indicates the date (construction date, planned construction date) when the actual member corresponding to the virtual member will be installed at the construction site. The member construction date information 24 is acquired from, for example, a construction drawing or a construction plan.

[0066] The system 1 (for example, the control unit 11 and the storage unit 14) provides, for example, a design database for managing design data 141. For example, the design database stores data on a virtual building (plurality of virtual components) designed using a BIM application. The design database is configured to manage each of the plurality of virtual components included in this virtual building. For example, the design database stores design data 141 including actual BIM data. The design database manages, for example, the design data 141 for each virtual building.

[0067] FIG. 3 shows an example of a data structure (format) of virtual component information 2 (see FIG. 2) managed by such a design database. In the data structure 3 of this example, the virtual component information 2 is managed in a table. Specifically, the table of the data structure 3 includes a virtual component ID column including multiple cells in which virtual component IDs are recorded, a virtual component shape information column including multiple cells in which virtual component shape information is recorded, a virtual component position information column including multiple cells in which virtual component position information is recorded, and a component construction date information column including multiple cells in which component construction date information is recorded. For example, virtual component ID 21 "BBB" of a certain virtual component is associated with virtual component shape information 22 "Bb", virtual component position information 23 "Cb", and component construction date information 24 "Da".

[0068] The actual component data 142 may be any data or information related to an actual component. The actual component data 142 is generated, for example, based on measurement data (photographed images, point cloud data, etc.) acquired by measuring (photographing, laser scanning, etc.) an actual building constructed based on the design data 141. The measurement of the actual building is performed by the data acquisition unit 13 or an external system. Furthermore, the generation of the actual component data 142 based on the measurement data is performed by the data acquisition unit 13 or an external system. The actual component data 142 is generated and managed, for example, as BIM data in the same format as the design data 141.

[0069] The physical component data 142 includes information about multiple physical components that are components of the physical building. More specifically, the physical component data 142 includes information about multiple pre-defined attributes for each physical component. The attributes here refer to the properties, characteristics, properties, etc., that the physical component possesses.

[0070] In some exemplary embodiments, the attributes of the physical component correspond to the attributes of the virtual component described above. For example, the attributes of the physical component include physical component identification information (physical component ID), physical component shape information, physical component position information, measurement date information, etc. Note that the attributes of the physical component are not limited to these and may be any property, characteristic, or property, such as material.

[0071] An example of the actual member data is shown in Fig. 4. The actual member data 4 according to this example includes an actual member ID 41, actual member shape data 42, actual member position data 43, and measurement date information 44.

[0072] The entity component ID 41 is information for identifying an entity component. Like the virtual component ID 21, the entity component ID 41 is information indicating the type of a virtual component, and may be, for example, identification information (such as a component number) assigned to the entity component. In some exemplary embodiments, the entity component ID 41 may be the same as the corresponding virtual component ID 21 (for example, identification information provided by IFC). Alternatively, the entity component ID 41 may be information different from the corresponding virtual component ID 21 and may be in a predetermined format that allows the system 1 (and external systems, etc.) to recognize the association with the corresponding virtual component ID 21. The entity component ID 41 is generated, for example, in a component association process described below.

[0073] The physical component shape data 42 is data representing the shape of a physical component acquired based on measurement data. The physical component shape data 42 may include data representing the posture and orientation of the physical component. The physical component location data 42 is generated, for example, in the component association process described below.

[0074] The actual component position data 43 represents the position of an actual component in an actual building. The position of the actual component is expressed, for example, by the coordinates of the actual component in a virtual space (a three-dimensional virtual space defined in a three-dimensional coordinate system) in which a BIM model of the actual building created based on measurement data is defined. The actual component position data 43 is generated, for example, in the component association process described below.

[0075] The measurement date information 44 indicates the date on which the measurement of the actual building was carried out. The measurement date information 44 is generated, for example, by a measurement system (such as a mobile object, a total station, or a computer) that measures the actual building.

[0076] The component construction date information 24 and the measurement date information 44 include at least information on the year, month, and day, and may further include information on the hour, minute, second, etc. Furthermore, if the standard time of the location where the design was made differs from the standard time of the location where the actual building exists, the system 1 (or an external system, etc.) may be configured to convert the component construction date information 24 and the measurement date information 44 so that they are expressed in the same standard time.

[0077] The system 1 (e.g., the control unit 11 and the memory unit 14) provides, for example, a physical component database for managing physical component data 142. For example, the physical component database stores data on a BIM model of a physical building (a BIM model of multiple physical components) obtained by processing measurement data of the physical building. The physical component database is configured to manage multiple physical component models included in the physical building model one by one. For example, the physical component database stores a physical building BIM model. The physical component database manages, for example, the physical component data 142 for each physical building BIM model.

[0078] FIG. 5 shows an example of the data structure (format) of the physical component data 4 (see FIG. 4) managed by such a physical component database. In the data structure 5 of this example, the physical component data 4 is managed in a table. Specifically, the table of the data structure 5 includes a physical component ID column containing multiple cells in which physical component IDs are recorded, a physical component shape data column containing multiple cells in which physical component shape data are recorded, a physical component position data column containing multiple cells in which physical component position data are recorded, and a measurement date information column containing multiple cells in which measurement date information is recorded. For example, the physical component ID 41 "222" of a certain physical component is associated with the physical component shape data 42 "B2," the physical component position data 43 "C2," and the measurement date information 44 "D2."

[0079] The processing unit 15 is configured to execute data processing. The processing unit 15 is realized, for example, by cooperation between hardware including a processor and a storage device and data processing software. The processing unit 15 is provided in one computer or distributed across two or more computers. The processing unit 15 includes a component associating unit 151 and an attribute associating unit 152.

[0080] The component association unit 151 is configured to associate a plurality of virtual components with a plurality of actual components (component association processing) based on the virtual component information and actual component data included in the design data 141. This generates a plurality of pairs of virtual components and actual components. The component association unit 151 is realized, for example, by cooperation between hardware including a processor and a storage device and component association software.

[0081] An example of processing executed by the component association unit 151 will be described. First, the component association unit 151 matches the coordinate space of the design data 141 with the coordinate space of the actual component data 142. For example, the component association unit 151 matches the origin or a predetermined reference point in the coordinate space of the design data 141 with the origin or a predetermined reference point in the coordinate space of the actual component data 142. In other words, the component association unit 151 performs registration between the design data 141 and the actual component data 142. In some exemplary embodiments, the component association unit 151 performs registration so as to match the definition coordinate system of the BIM data (virtual building model) in the design data 141 with the definition coordinate system of the BIM data (actual building model) in the actual component data 142. This enables coordinate conversion between the coordinate system representing the design data 141 and the coordinate system representing the actual component data 142.

[0082] Next, the member associating unit 151 associates an object in the design data 141 (for example, a face, vertex, or center point of a virtual member) with an object in the physical member data 142. For example, the member associating unit 151 may be configured to associate an object in the design data 141 with an object in the physical member data 142 when the distance between these objects is equal to or less than a predetermined threshold. As a result, physical member position data (for example, physical member position data 43 in FIG. 4) of this object (physical member) is generated.

[0083] Furthermore, the component association unit 151 can recognize the shape of an object in the physical component data 142. For example, data representing the shape of a pillar (surface shape, cross-sectional shape, etc.), the shape of a beam, the shape of a wall, and the shape of a ceiling can be acquired. The component association unit 151 can also acquire data representing the arrangement (orientation, posture) of an object in the physical component data 142. Furthermore, the component association unit 151 may be configured to grasp the texture, material, quality, etc. of the surface of an object. Based on the shape, arrangement, texture, material, quality, etc. of the object, physical component shape data (e.g., the physical component shape data 42 in FIG. 4) of this object (physical component) is generated. The physical component shape data obtained in this manner may be used to improve component association processing according to the distance of the object (e.g., to improve accuracy and precision).

[0084] In addition, the component associating unit 151 assigns identification information to each of the entity components identified from the entity component data 142. That is, the component associating unit 151 may be configured to assign an entity component ID to the entity component data 142.

[0085] For example, the member associating unit 151 assigns the same identification information (virtual member ID) to a physical member associated with a certain virtual member. In this case, the same identification information is assigned to a pair of a virtual member and a physical member associated with each other. In other words, the virtual member and the physical member associated with each other are linked by the same identification information. The identification information commonly assigned to the virtual member and the physical member may be, for example, unique identification information used in IFC.

[0086] In another example, the member associating unit 151 may be configured to assign identification information similar to a virtual member to a physical member associated with the virtual member. For example, a physical member ID consisting of a character string including at least a part of the character string of the virtual member ID of the virtual member is assigned to the physical member.

[0087] In yet another example, the component association unit 151 may be configured to assign unique identification information (a component ID) to a component associated with a certain virtual component, and to record the virtual component ID of this virtual component in an additional recording area in the component data.

[0088] By such a component correspondence process, correspondence relationships between a plurality of virtual components and a plurality of actual components are obtained. That is, a plurality of pairs of virtual components and actual components (component pairs) are obtained. For example, as shown in Fig. 6, correspondence relationships are obtained between identification information (virtual component IDs) of a plurality of virtual components in data structure 3 of Fig. 3 and identification information (actual component IDs) of a plurality of actual components in data structure 5 of Fig. 5.

[0089] The correspondence shown in FIG. 6 is a bijection between a set of multiple virtual components and a set of multiple real components, but is not limited to this. For example, if there are no real components in the vicinity of a certain virtual component (within a range of a predetermined distance), there is no real component associated with this virtual component. Conversely, if there are no virtual components in the vicinity of a certain real component (within a range of a predetermined distance), there is no virtual component associated with this real component. Furthermore, if there are two or more real components in the vicinity of a certain virtual component, two or more real components can be associated with one virtual component. Conversely, if there are two or more virtual components in the vicinity of a certain real component, two or more virtual components can be associated with one real component. In cases where a bijection cannot be obtained, such as in these cases, the control unit 11 can, for example, display the obtained correspondence on the user interface 12. The user can edit this correspondence using the user interface 12.

[0090] The attribute associating unit 152 is configured to associate virtual component information with actual component data according to a plurality of attributes for each component pair (a pair of a virtual component and an actual component) obtained by the component associating unit 151. The attribute associating unit 152 is realized, for example, by the cooperation of hardware including a processor and a storage device with attribute associating software.

[0091] Here, the multiple attributes of the virtual component may be the same as the multiple attributes of the actual component. Furthermore, the multiple attributes of the actual component may include all of the multiple attributes of the virtual component. For example, a metadata structure that allows attributes common to virtual component information and actual component data to be registered can be prepared in advance. The attribute associating unit 152 may be configured to associate the attributes of the virtual component with the attributes of the actual component using such a metadata structure.

[0092] For example, as shown in Fig. 7, a correspondence relationship can be obtained between a plurality of attributes of each virtual component (each virtual component ID) in data structure 3 of Fig. 3 and a plurality of attributes of each actual component (each actual component ID) in data structure 5 of Fig. 5. In this example, for example, for a pair of a virtual component with virtual component ID "AAA" and an actual component with actual component ID "111", virtual component shape information "Ba" and actual component shape data "B1" are associated, virtual component position information "Ca" and actual component position data "C1" are associated, and component construction date information "Da" and measurement date information "D1" are associated. The same applies to other component pairs.

[0093] Thus, the system 1 according to this embodiment is a system for reality capture of a building and includes a memory unit 14, a component association unit 151, and an attribute association unit 152. The memory unit 14 stores design data including virtual component information related to multiple attributes for each of multiple virtual components of a virtual building. Furthermore, the memory unit 14 stores actual component data related to multiple attributes for each of multiple actual components, generated based on measurement data acquired from a real building constructed based on the design data. The component association unit 151 is configured to associate multiple virtual components with multiple actual components based on the virtual component information and the actual component data, thereby generating multiple pairs of virtual components and actual components. The attribute association unit 152 is configured to associate virtual component information with actual component data according to multiple attributes for each of the multiple pairs generated by the component association unit 151.

[0094] This aspect also provides a data structure to be processed by the building reality capture system. The data structure includes design data and actual component data. The design data is prepared in advance and includes virtual component information related to multiple attributes for each of multiple virtual components of a virtual building. The actual component data is generated based on measurement data acquired from a real building constructed based on the design data and includes data related to multiple attributes for each of the multiple actual components. The design data and actual component data are used in a component association process and an attribute association process. The component association process determines correspondences between multiple virtual components and multiple actual components based on the virtual component information and the actual component data. The attribute association process determines correspondences between virtual component information and actual component data for each of multiple pairs of virtual components and actual components acquired by the component association process, based on multiple attributes. Furthermore, a computer-readable non-transitory recording medium having data having such a structure recorded thereon can be created. This non-transitory recording medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0095] In the system 1 of this embodiment, the virtual member information 2 may include virtual member position information 23, and the actual member data 4 may include actual member position data 43. Furthermore, the member associating unit 151 may be configured to generate a plurality of pairs of virtual members and actual members based on the virtual member position information 23 and the actual member position data 43.

[0096] This configuration provides a data structure having the following characteristics: virtual component information 2 includes virtual component position information 23; physical component data 4 includes physical component position data 43; virtual component position information 23 and physical component position data 43 are used in the component matching process.

[0097] In the system 1 of this embodiment, the virtual component information 2 may include construction date information 24 indicating the construction date of each of the multiple virtual components, and the actual component data 4 may include measurement date information 44 indicating the measurement date of the actual building. Furthermore, the component association unit 151 may be configured to generate multiple pairs of virtual components and actual components based on the construction date information 24 and the measurement date information 44.

[0098] This configuration provides a data structure having the following features: virtual component information 2 includes construction date information 24 indicating the construction date of each of multiple virtual components; actual component data 4 includes measurement date information 44 indicating the measurement date of the actual building; construction date information 24 and measurement date information 44 are used in the component matching process.

[0099] When the construction date information 24 and the measurement date information 44 are used, a correspondence between a plurality of pieces of virtual component information corresponding to a plurality of different construction dates and a plurality of pieces of actual component data corresponding to a plurality of different measurement dates can be established by comparing the construction date with the measurement date. For example, for a certain construction date, the component association unit 151 can select the measurement date closest to the construction date from one or more measurement dates after the construction date, and associate the actual component data corresponding to the selected measurement date with the virtual component information corresponding to the construction date. This allows the correspondence between virtual components (virtual component information) and actual components (actual component data) to be established based on the date, making it possible to manage time-series data according to, for example, a construction plan.

[0100] This embodiment also provides a program for causing a computer included in the building reality capture system to function as a data accepting means, a component matching means, and an attribute matching means. The data accepting means (data acquisition unit 13) accepts design data including virtual component information regarding multiple attributes for each of multiple virtual components of a virtual building, and actual component data regarding multiple attributes for each of multiple actual components, generated based on measurement data acquired from a real building constructed based on the design data. The component matching means matches the multiple virtual components with the multiple actual components based on the virtual component information and the actual component data. The attribute matching means matches the virtual component information and the actual component data according to the multiple attributes for each of multiple pairs of virtual components and actual components acquired by the component matching means. Furthermore, a computer-readable non-transitory recording medium having such a program recorded thereon can be created. This non-transitory recording medium may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0101] According to this aspect, the design data of a building and the data of the building constructed based on it can be matched on a component-by-component basis (or as a series of component-by-component basis) and also on a component attribute-by-component basis.

[0102] Furthermore, virtual component information is information used to easily measure actual buildings, and actual component data is data obtained from measurements using this virtual component information. By associating such virtual component information with actual component data on a component-by-component and attribute-by-attribute basis, it becomes possible to easily compare design data with actual measurement data. For example, it becomes easier to compare design BIM data (design BIM data) with BIM data based on measurement data (measurement BIM data).

[0103] In this way, according to this aspect, it is possible to integrate the management of various types of data and information, which in turn makes it possible to improve the efficiency of management and make it consistent.

[0104] <Second aspect> An example of the configuration of a system according to the second aspect is shown in FIG. 8. System 8 is obtained by adding an inference model construction unit 153 and a movement control information creation unit 154 to the processing unit 15 of system 1 according to the first aspect. The inference model construction unit 153 is realized, for example, by cooperation between hardware including a processor and a storage device and inference model construction software. Furthermore, the movement control information creation unit 154 is realized, for example, by cooperation between hardware including a processor and a storage device and movement control information creation software. Explanations of elements common to the first aspect will be omitted unless otherwise noted.

[0105] In this embodiment, the storage unit 14 stores, in addition to the design data 141 and the actual component data 142, a virtual image 143 and a photographed image 144. The virtual image 143 is an image representing a virtual building, and is created by, for example, rendering BIM data (CAD data). The virtual image 143 is, for example, a plurality of images (a virtual image set) of the interior and / or exterior of a virtual building along a predetermined movement path. The photographed images 144 include photographed images of any object and photographed images of people. Examples of photographed images of objects include images of building components and images of tools such as ladders and stepladders. An example of an image of a person is an image of a worker at a construction site.

[0106] The inference model construction unit 153 is configured to construct an inference model for identifying images of building components from photographed images of a building by applying machine learning to a predetermined neural network. Applications of the inference model include, for example, controlling a mobile object that measures (photographs, laser scans) an actual building, and creating BIM data based on measurement data (photographed images, scan data) of an actual building.

[0107] The neural network for constructing the inference model may include a convolutional neural network. The convolutional neural network is configured to, for example, receive an image as input, generate a feature map using a convolutional layer, compress the feature map using a pooling layer multiple times, and provide a final output (image classification, segmentation, regression, etc.) using a fully connected layer that receives the output from the final pooling layer as input. In some exemplary embodiments, the convolutional neural network may not include a fully connected layer, or may include a support vector machine.

[0108] The training data used in machine learning to build an inference model includes, for example, at least virtual images 143 and may further include photographed images 144. The training data may also include other measurement data (such as scan data) of the actual building. In this embodiment, by using virtual images 143 created from BIM data, it is possible to eliminate the need to collect a large amount of measurement data of the actual building. In particular, by creating a large number of virtual images 143 (virtual image sets) from a large number of BIM data and using them as training data, it is possible to improve the quality of machine learning and the quality of the inference model.

[0109] By using photographed images 144, such as photographed images of building components, photographed images of objects other than building components, and photographed images of people, as training data, it is possible to train an inference model to detect tools and workers. This makes it possible to control moving objects to avoid obstacles and workers, and to exclude data on obstacles and workers when creating BIM data from measurement data.

[0110] The virtual images 143 and / or measurement data (photographed images 144, scan data) included in the training data may include texture information that represents the surface condition of building components. By using texture information, it is possible to obtain a final output that reflects the texture of the object's surface. This can improve the quality of mobile object control and BIM data creation.

[0111] The training method for constructing an inference model may be any method, for example, supervised learning, unsupervised learning, or reinforcement learning, or a combination of two or more of these. Typically, supervised learning is performed using training data in which input images are labeled with the final output.

[0112] The movement control information creation unit 154 is configured to create movement control information for acquiring building data using a mobile object, based on the virtual image 143, or based on the virtual image 143 and the photographed image 144. In this embodiment, the movement control information creation unit 154 can create movement control information based on an inference model constructed using the virtual image 143 (and the photographed image 144, etc.). The created movement control information is provided to the mobile object, for example, via a wired network or a wireless network, or via a recording medium.

[0113] In some exemplary embodiments, the movement control information generator may be configured to generate the movement control information by applying rule-based processing to the virtual image 143 (and the captured image 144, etc.) without using an inference model. The movement control information generator may also be configured to selectively apply processing using an inference model and rule-based processing.

[0114] The movement control information may include, for example, a movement path that is set in advance based on the design data 141 (BIM data) and that is used to measure the interior of an actual building constructed based on the design data 141. The movement control information may also include a virtual image (143) along this movement path that is constructed by rendering the BIM data. The movement control information may also include an inference model constructed by the inference model construction unit 153 (i.e., a neural network whose parameters (weighting coefficients, etc.) have been tuned by the inference model construction unit 153).

[0115] The travel path may be set, for example, so that the distance to at least some of the multiple virtual components recorded in the BIM data falls within a predetermined tolerance range. This allows real components in the real building to be photographed from an appropriate distance. The travel path may also be a one-dimensional area, a two-dimensional area, or a three-dimensional area. The two-dimensional area or three-dimensional area represents, for example, the range within which a moving object performing measurements can move to avoid obstacles.

[0116] The action to avoid a collision with an obstacle is not limited to the avoidance action. For example, the moving object may be controlled to stop moving when an obstacle is detected. The moving object may also be controlled to emit auditory or visual information when an obstacle is detected. The auditory information may be, for example, a warning sound, and the visual information may be, for example, a warning light. Furthermore, when an obstacle is detected, the moving object may be controlled to output auditory or visual information to a device (for example, a tablet, smartphone, or laptop) for operating the moving object.

[0117] The system 8 of this embodiment provides a data structure having the following features in addition to the data structure of the first embodiment: it further includes a pre-generated virtual image 143 of the virtual building; the virtual image 143 is used to create movement control information for acquiring building data using a mobile object.

[0118] Furthermore, the system 8 of this embodiment provides a data structure further having the following features: it further includes a photographed image 144 that has been acquired in advance; the photographed image 144 is used to create the movement control information.

[0119] In addition, the system 8 of this embodiment provides a data structure further having the following features: it further includes an inference model for identifying images of building components from photographed images of a building, constructed by applying machine learning using at least virtual images 143 to a predetermined neural network; this inference model is used to create movement control information.

[0120] Furthermore, it is possible to create a computer-readable non-transitory recording medium on which data having the structure provided by the system 8 of this embodiment is recorded. The same applies to a program and a computer-readable non-transitory recording medium on which the program is recorded.

[0121] According to this aspect, similar to the first aspect, it is possible to integrate, streamline, and ensure consistency in the management of various data and information. Furthermore, since movement control information can be created based on the virtual image 143 (and the photographed image 144), it is possible to support building measurement using moving objects (UAVs, self-propelled vehicles, people, etc.). It is also possible to reduce the labor and time required for measurement work.

[0122] <Third aspect> An example of the configuration of a system according to the third aspect is shown in FIG. 9. System 9 is provided with a reference information creation unit 155 instead of the movement control information creation unit 154 of the processing unit 15 of system 8 according to the second aspect. The inference model construction unit 153 is realized, for example, by cooperation between hardware including a processor and a storage device and inference model construction software. Furthermore, the reference information creation unit 155 is realized, for example, by cooperation between hardware including a processor and a storage device and reference information creation software. Explanations of elements common to the second aspect will be omitted unless otherwise noted.

[0123] The reference information creation unit 155 creates reference information based on the virtual image 143 (and the captured image 144) to determine whether data on building components has been acquired when acquiring data on a building using a mobile body. In this embodiment, the reference information creation unit 155 can create reference information based on the inference model constructed by the inference model construction unit 153. The created reference information is provided to the mobile body, for example, via a wired network or a wireless network, or via a recording medium.

[0124] In some exemplary embodiments, the reference information creation unit may be configured to create the reference information by applying rule-based processing to the virtual image 143 (and the captured image 144, etc.) without using an inference model. Furthermore, the reference information creation unit may be configured to selectively apply processing using an inference model and rule-based processing.

[0125] As described above, the reference information is used to determine whether data on the building components (actual components) that make up a real building have been acquired when measuring the real building using a mobile object. In other words, the reference information is used to determine whether measurement data on the actual components, which are the object of measurement using a mobile object, have been reliably acquired.

[0126] The reference information may include, for example, a travel path for measuring the interior of a real building, which is set in advance based on the design data 141 (BIM data, virtual component information, etc.). The reference information may also include a virtual image 143. The reference information may also include an inference model constructed by the inference model construction unit 153.

[0127] For example, a mobile object can use V-SLAM to create a map (3D model) of the area around the mobile object from captured images, use an inference model to detect components from the 3D model, compare the detected components with design data 141 (BIM data, virtual component information, etc.), and determine whether measurement data of the actual components has been acquired based on the results of this comparison. The mobile object may be configured to output auditory or visual information indicating the result of this determination. Furthermore, a device for operating the mobile object may be configured to output auditory or visual information indicating the result of the determination.

[0128] The system 9 of this embodiment provides a data structure having the following features in addition to the data structure of the first embodiment: it further includes a pre-generated virtual image 143 of the virtual building; the virtual image 143 is used to create reference information for determining whether data of building components has been acquired when acquiring data of the building using a mobile object.

[0129] Furthermore, the system 9 of this embodiment provides a data structure that further has the following features: it further includes a photographed image 144 that has been acquired in advance; the photographed image 144 is used to create the reference information.

[0130] In addition, the system 9 of this embodiment provides a data structure further having the following features: it further includes an inference model for identifying images of building components from photographed images of a building, constructed by applying machine learning using at least virtual images 143 to a predetermined neural network; this inference model is used to create reference information.

[0131] Furthermore, it is possible to create a computer-readable non-transitory recording medium on which data having the structure provided by the system 9 of this embodiment is recorded. The same applies to a program and a computer-readable non-transitory recording medium on which the program is recorded.

[0132] According to this aspect, similar to the first aspect, it is possible to integrate, streamline, and ensure consistency in the management of various data and information. Furthermore, since reference information can be created based on the virtual image 143 (and the photographed image 144), it is possible to determine in real time whether building measurements using moving objects (UAVs, self-propelled vehicles, people, etc.) are being performed appropriately. If measurements are not being performed appropriately, at least partial re-measurement can be performed. This makes it possible to avoid, for example, a situation in which meaningless measurement data is provided to subsequent processing. Furthermore, since it is possible to automate the determination of whether re-measurement is necessary, it is possible to reduce the labor required for measurement work and shorten the time required.

[0133] <Fourth aspect> An example of the configuration of a system according to the fourth aspect is shown in FIG. 10. The system 10 is provided with a data object detection unit 156 instead of the movement control information creation unit 154 of the processing unit 15 of the system 8 according to the second aspect, and measurement data 145 is further stored in the memory unit 14. The inference model construction unit 153 is realized, for example, by cooperation between hardware including a processor and a storage device and inference model construction software. Furthermore, the data object detection unit 156 is realized, for example, by cooperation between hardware including a processor and a storage device and data object detection software. Explanations of elements common to the second aspect will be omitted unless otherwise noted.

[0134] The measurement data 145 is data obtained by actually measuring a building, such as photographed images or scanned data. The measurement data 145 is obtained, for example, by photographing or laser scanning while moving within the building along a predetermined route.

[0135] The data object detection unit 156 is configured to detect data objects from the measurement data 145 based on the virtual image 143 (and the photographed image 144). If the measurement data 145 is a photographed image, the data objects are image objects in the photographed image, typically images of building components (actual components). If the measurement data 145 is scan data, the data objects are data regions in the scan data, typically partial data corresponding to building components (actual components).

[0136] In this embodiment, the data object detector 156 can detect data objects from the measurement data 145 based on the inference model constructed by the inference model constructor 153. In some exemplary embodiments, the data object detector may be configured to detect data objects by applying rule-based processing to the measurement data 145 without using an inference model. Alternatively, the data object detector may be configured to selectively apply processing using an inference model and rule-based processing.

[0137] An example of processing executed by the data object detection unit 156 will be described. First, the data object detection unit 156 applies SfM to the measurement data 145 to obtain multiple positions and orientations of measurement devices (cameras, laser scanners), and then restores the shapes of objects in the environment that are captured in the measurement data 145.

[0138] Furthermore, the data object detection unit 156 inputs the measurement data 145 into an inference model to identify data objects corresponding to building components from the measurement data 145. This allows multiple data objects included in the measurement data 145 to be recognized, and the attributes of the components corresponding to each data object to be identified.

[0139] Next, the data object detection unit 156 applies a mask to the external region of the identified data object, detects feature points of the data object, and obtains the three-dimensional positions (three-dimensional coordinates) of the detected feature points using MVS.

[0140] The processing unit 15 can compare the 3D coordinate data of the building components obtained by the data object detection unit 156 with the design data 141 (BIM data). For example, the processing unit 15 first registers the 3D coordinate data obtained by the data object detection unit 156 and coordinate information of surface data (or point data, etc.) in the BIM data within a common 3D coordinate system. Next, the processing unit 15 selects 3D coordinate points located near specific surface data in the BIM data from the 3D coordinate data. Next, the processing unit 15 defines a surface (plane, free-form surface) in the 3D coordinate data using the selected 3D coordinate points. The defined surface is associated with the specific surface in the BIM data. Through this series of processes, a correspondence is established between multiple surfaces (typically surfaces of virtual components) represented in the BIM data and multiple surfaces (typically surfaces of actual components) of a building constructed based on this BIM data.

[0141] The processing unit 15 can create new BIM data using the three-dimensional coordinate data, surface data, and correspondence relationships obtained in this manner. For example, the processing unit 15 can change the BIM data of the design data 141 using data obtained from the measurement data 145. For example, the processing unit 15 compares the position of a column surface in the BIM data with the actual position of the column surface obtained from the measurement data 145, and replaces the virtual position in the BIM data with the actual position. This results in new BIM data that reflects the actual column position (orientation, shape, posture, etc.). In this series of processes, the component association unit 151 can associate virtual components in the BIM data of the design data 141 with actual components in the actual component data 142 using the data objects (three-dimensional coordinate data, surface data, correspondence relationships, etc.) detected by the data object detection unit 156.

[0142] The data objects detected by the data object detection unit 156 are not limited to building components. For example, the data object detection unit 156 may be configured to detect obstacles as data objects. In this case, the BIM data can be edited by excluding data objects corresponding to obstacles. This makes it possible to avoid confusing obstacles in the measurement data 145 with building components, thereby improving the BIM data editing.

[0143] The system 10 of this embodiment provides a data structure having the following features in addition to the data structure of the first embodiment: further including a pre-generated virtual image 143 of the virtual building and pre-acquired measurement data 145; the virtual image 143 is used in a data object detection process to detect data objects from the measurement data 145; the data objects detected by the data object detection process are used in a component matching process.

[0144] The system 10 of this embodiment also provides a data structure that further includes the following features: a previously acquired photographed image 144; the photographed image 144 is used in the data object detection process.

[0145] In addition, the system 10 of this embodiment provides a data structure further having the following features: it further includes an inference model for identifying images of building components from photographed images of a building, constructed by applying machine learning using at least virtual images 143 to a predetermined neural network; this inference model is used in the data object detection process.

[0146] Furthermore, it is possible to create a computer-readable non-transitory recording medium on which data having the structure provided by the system 10 of this embodiment is recorded. The same applies to a program and a computer-readable non-transitory recording medium on which the program is recorded.

[0147] According to this aspect, similar to the first aspect, it is possible to integrate, streamline, and ensure consistency in the management of various data and information. Furthermore, it is possible to detect data objects from building data based on the virtual image 143 (and the photographed image 144), and to generate multiple component pairs (pairs of virtual components and actual components) based on the detected data objects, thereby automating the creation of new BIM data. This makes it possible to easily obtain BIM data that reflects the construction status, and also to easily accumulate and / or update BIM data according to the construction status. Therefore, this aspect contributes to reducing the labor and time required for measurement work.

[0148] <Fifth aspect> A building is composed of many building components. Creating a 3D model for each of these building components is desirable from the viewpoint of data detail, but is not practical when considering processing resources and load. As mentioned somewhat in the fourth aspect, the fifth aspect provides a technology for dealing with such problems. An example configuration of a system according to this aspect is shown in FIG. 11.

[0149] System 110 is a system in which a partial area identification unit 157 and a substantial component data generation unit 158 ​​are added to the processing unit 15 of system 1 of the first embodiment, and measurement data 146 and representative portion information 147 are further stored in the storage unit 14. The partial area identification unit 157 is realized, for example, by cooperation between hardware including a processor and a storage device and partial area identification software. Furthermore, the substantial component data generation unit 158 ​​is realized, for example, by cooperation between hardware including a processor and a storage device and substantial component data generation software. Explanation of elements common to the first embodiment will be omitted unless otherwise noted.

[0150] The system 110 is configured to generate the physical component data 142 from the measurement data 146, and the physical component data 142 may not be stored in the storage unit 14 when the system 110 starts processing. On the other hand, the physical component data 142 generated in the past may be stored in the storage unit 14 when the system 110 starts processing.

[0151] The measurement data 146 is data obtained by measuring an actual building constructed based on the design data 141 (BIM data), etc., and is, for example, photographed images or scanned data. The measurement data 146 is acquired, for example, by photographing or laser scanning while moving inside the actual building along a preset route.

[0152] The representative part information 147 indicates a representative part of a virtual part recorded in the virtual part information of the design data 141. The representative part may be any part of the virtual part, for example, one or more points, one or more lines, or one or more surfaces. Note that the representative part may be the entire virtual part; however, considering one objective of this aspect, which is to reduce processing resources and processing load, it is not practical to set the entire virtual part as the representative part for all virtual parts recorded in the virtual part information. Therefore, in this aspect, the representative part for at least one of the multiple virtual parts recorded in the virtual part information is not the entire virtual part.

[0153] Some examples of the representative part information 147 will be described. When the building component is a pillar with a rectangular horizontal cross section, the representative part of this pillar may be, for example, one or more of the four side surfaces, the center point of the bottom surface, and one or more of the center point of the top surface. When the representative part of this pillar includes two or more side surfaces, information (ID) that can identify these side surfaces is used.

[0154] If the building component is a floor, the representative portion may be the top surface. Similarly, if the building component is a ceiling, the representative portion may be the bottom surface.

[0155] For other building components, any one or more points, any one or more lines, or any one or more surfaces may be used as the representative portion to facilitate measurement and data processing. When two or more representative portions are set for a single building component, identification information is assigned to each of these representative portions. This identification information is referenced to distinguish between these representative portions during data processing.

[0156] The partial area specifying unit 157 is configured to specify a partial area of ​​the measurement data 146 corresponding to one of the representative areas of the plurality of virtual members that make up the virtual building, based on the representative area information 147.

[0157] To this end, first, the partial area identification unit 157 identifies a data area of ​​the measurement data 146 that corresponds to the virtual member. That is, the partial area identification unit 157 identifies a data area representing a real member that corresponds to this virtual member from the measurement data 146. This process is performed using SfM and an inference model, for example, as with the data object detection unit 156 of the fourth aspect. By performing this process for each virtual member, multiple data areas (multiple real member areas) corresponding to multiple real members respectively are identified from the measurement data 146. Note that rule-based image processing may be applied instead of the inference model.

[0158] Next, for each virtual component, the partial region identifying unit 157 identifies a partial region corresponding to a representative portion of the virtual component from the actual component region corresponding to the virtual component. Here, MVS can be used to determine the three-dimensional coordinates of the measurement points in the measurement data 146. For example, MVS can be used to determine the three-dimensional coordinates of the actual component region in the measurement data 146 and to determine the three-dimensional coordinates of the partial region in the actual component region.

[0159] When the representative part of the virtual component is a point, the partial area identification unit 157 can, for example, determine the distance between each measurement point in the measurement data 146 and the representative part (point), and identify the partial area corresponding to the representative part (point) by threshold processing of the distance.

[0160] Furthermore, when the representative part of the virtual part is a surface, the partial area specifying unit 157 can, for example, determine a straight line (perpendicular line) that passes through each measurement point in the measurement data 146 and is perpendicular to the representative part (surface), determine the distance between this measurement point and the foot of the perpendicular line, and specify the partial area corresponding to the representative part (surface) by threshold processing of the distance. Even when the representative part of the virtual part is a line, the partial area in the measurement data 146 that corresponds to the representative part (line) can be specified in the same manner.

[0161] The substantive component data generating unit 158 ​​generates substantive component data from the measurement data 146 based on the partial region acquired by the partial region identifying unit 157. The generated substantive component data is stored in the storage unit 14 as substantive component data 142 having the data structure shown in Fig. 5, for example. At this time, the system 110 (for example, the substantive component data generating unit 158, the processing unit 15, or the control unit 11) can assign attributes (ID, shape, position, measurement date, etc.) shown in Fig. 4 to the generated substantive component data.

[0162] The processing unit 15 can compare the three-dimensional coordinates of the representative portion of the actual member obtained by the partial area identification unit 157 with the three-dimensional coordinates of the corresponding representative portion of the virtual member (three-dimensional coordinates in the design BIM data). For example, it can determine the deviation of the representative portion of the actual member from the representative portion of the virtual member. This deviation indicates the deviation of the actual position from the design position.

[0163] Furthermore, the processing unit 15 can make changes to the design BIM data based on the displacement of the representative position. For example, the processing unit 15 can replace the three-dimensional coordinates of the representative part of the virtual member with the three-dimensional coordinates of the representative part of the actual member to change the position and / or shape of the virtual member.

[0164] An example of editing design BIM data will be described with reference to Figure 12. Reference numeral 120 denotes a column (virtual column) as a virtual member. The representative part of the virtual column 120 is the center of the top surface (virtual top surface center) 121. Reference numeral 122 denotes the three-dimensional coordinates (actual top surface center) in the measurement data 146 corresponding to the virtual top surface center 121, identified by the partial area identification unit 157. Reference numeral 123 denotes an arrow indicating the deviation of the actual top surface center 122 from the virtual top surface center 121.

[0165] The processing unit 15 first obtains an upper surface area 124 centered on the actual upper surface center 122. The shape and orientation of the upper surface area may be the same as the shape and orientation of the upper surface of the virtual pillar 120. Alternatively, the upper surface area 124 may be obtained by changing at least one of the shape and orientation of the upper surface of the virtual pillar 120 in accordance with the displacement 123.

[0166] Next, the processing unit 15 obtains lines 126 connecting each vertex of the rectangular upper surface region 124 with the corresponding vertex of the bottom surface 125 of the virtual pillar 120. This obtains four lines 126 connecting the upper surface region 124 with the bottom surface 125. The lines 126 may be straight lines or curves (for example, free curves) according to the deviation 123.

[0167] In this way, the position and shape (direction, posture, etc.) of the virtual pillar 120 are changed based on the measurement data 146. The information (attributes) of the virtual pillar 120 after the change is recorded as the actual member data 142 of the corresponding actual member (pillar). This actual member data 142 has the data structure shown in FIG.

[0168] The system 110 of this embodiment provides a data structure having the following features in addition to the data structure of the first embodiment: it further includes measurement data 146 and representative part information 147 indicating a representative part of a virtual component created in advance; the representative part information 147 is used in a partial area identification process to identify a partial area of ​​the measurement data corresponding to a representative part of any of multiple virtual components of the virtual building.

[0169] Furthermore, the system 110 of this embodiment provides a data structure further having the following characteristics: the partial area identified by the partial area identification process is used in the process of generating the actual component data 142 from the measurement data 146 .

[0170] Furthermore, a computer-readable non-transitory recording medium can be created that records data having the structure provided by the system 110 of this embodiment. The same applies to a program and a computer-readable non-transitory recording medium that records the program.

[0171] According to this aspect, similar to the first aspect, it is possible to integrate, streamline and ensure consistency in the management of various types of data and information.

[0172] Furthermore, since it is possible to identify a partial area of ​​measurement data corresponding to a representative portion of a virtual component, it is possible to manage data corresponding to a representative portion of a virtual component instead of managing a 3D model of a real component. This reduces the resources required for management and processing. For example, it is possible to improve the efficiency of the process of generating real component data from measurement data.

[0173] <Sixth aspect> As mentioned above, a building is composed of many building components, but several components can be considered as a group. For example, a group of columns can be considered as a group, or a column and beam that are in contact with each other can be considered as a group. By collectively converting several components into data, it is possible to improve the efficiency of measurement and processing. In the sixth aspect, a technology conceived from this perspective will be described. An example of the configuration of a system according to this aspect is shown in Fig. 13.

[0174] System 130 is the same as system 1 of the first embodiment except that it adds a substantive component data generation unit 159 to the processing unit 15, and further stores measurement data 148 and component selection information 149 in the storage unit 14. The substantive component data generation unit 159 is realized, for example, by cooperation between hardware including a processor and a storage device and substantive component data generation software. Unless otherwise specified, explanations of elements common to the first embodiment will be omitted.

[0175] The system 130 is configured to generate the physical component data 142 from the measurement data 148, and the physical component data 142 may not be stored in the storage unit 14 when the system 130 starts processing. On the other hand, the physical component data 142 generated in the past may be stored in the storage unit 14 when the system 130 starts processing.

[0176] The measurement data 148 is data obtained by measuring an actual building constructed based on the design data 141 (BIM data), etc., and is, for example, photographed images or scanned data. The measurement data 148 is acquired, for example, by photographing or laser scanning while moving inside the actual building along a preset route.

[0177] The member selection information 149 is generated in advance based on the design data 141 and indicates some of the virtual members of the virtual building. Each virtual member indicated in the member selection information 149 is called a selected member, and they are collectively called a selected member group. The selected member group represents all the virtual members. Furthermore, the number of selected members is less than the number of all the virtual members. Therefore, a certain selected member represents multiple virtual members.

[0178] For example, if a group of columns is considered as a group as described above, one of the columns in the group can be registered as a selected member in the member selection information 149. Also, if a column and a beam that are in contact with each other are considered as a group, either the column or the beam can be registered as a selected member in the member selection information 149. The member selection information 149 is created in this way. Note that information indicating the multiple virtual members that the selected member represents can be attached to each selected member.

[0179] The actual component data generation unit 159 is configured to generate actual component data 142 from a partial region of the measurement data corresponding to the selected component group based on the component selection information 149. For example, the actual component data generation unit 159 can obtain data (position, shape, etc.) of actual components from the measurement data 148 in the same manner as in the fourth or fifth aspect, and can also associate virtual components with actual components.

[0180] However, in this embodiment, the actual component data generation unit 159 does not determine the position and shape of all actual components (i.e., all actual components corresponding to all virtual components of the virtual building), but rather determines the position and shape only of the actual components corresponding to the selected components indicated in the component selection information 149.

[0181] Furthermore, the actual member data generating unit 159 records the attributes (position, shape, etc.) found for the actual member corresponding to the selected member as actual member data 142 having the data structure shown in FIG.

[0182] The system 130 of this embodiment provides a data structure having the following features in addition to the data structure of the first embodiment: it further includes measurement data and component selection information indicating some virtual components among a plurality of virtual components pre-generated based on design data; the component selection information is used in the process of generating actual component data from a partial area of ​​the measurement data corresponding to the some virtual components indicated in the component selection information.

[0183] Furthermore, a computer-readable non-transitory recording medium can be created that stores data having the structure provided by the system 130 of this embodiment. The same applies to a program and a computer-readable non-transitory recording medium that stores the program.

[0184] According to this aspect, similar to the first aspect, it is possible to integrate, streamline and ensure consistency in the management of various types of data and information.

[0185] Furthermore, in this embodiment, instead of generating actual component data corresponding to all virtual components, actual component data corresponding to only some of the virtual components can be generated, which reduces the resources required for management and processing. For example, the efficiency of the process of generating actual component data from measurement data can be improved.

[0186] <Seventh aspect> The seventh aspect describes a usage pattern of the system that can be realized by combining several aspects. This system is used for information management in the construction field (construction management, maintenance management, repair management, etc.), and is an integrated system that utilizes various technologies, such as mobile objects (UAVs, self-propelled vehicles, people, etc.), surveying equipment (total stations, laser range finders, theodolites, range finders, etc.), data processing technologies (SfM, MVS, SLAM, etc.), and modeling technologies (computer graphics, CAD, BIM, etc.).

[0187] An example configuration of a system according to this embodiment is shown in Fig. 14. The system 140 includes a UAV 1410, a UAV controller 1420, a total station 1430, and an edge computer 1440. The cloud computer 1450 may be included in the system 140, or may be an external device capable of data communication with the system 140. Note that any of the UAV 1410, the UAV controller 1420, the total station 1430, and the edge computer 1440 may be an external device to the system 140.

[0188] It should be noted that the techniques of Patent Documents 1 to 12 can be combined with this embodiment, and the techniques described in JP-A-2018-138922, JP-A-2018-138923, and the like can be used.

[0189] The UAV 1410 is a small unmanned aerial vehicle that flies inside and / or outside a (physical) building to collect data about the building. The UAV 1410 includes a control unit 1411 that performs various controls, an imaging unit 1412 that collects data about the building, a V-SLAM unit 1413 that performs V-SLAM processing, and an obstacle detection unit 1414 that performs obstacle detection processing.

[0190] Although not shown, the UAV 1410, like a typical UAV, is equipped with elements for flight, such as multiple propellers and propeller motors for rotating each propeller. Although not shown, the UAV 1410 may also be equipped with any means that can be mounted on a standard UAV, such as an inertial measurement unit (IMU) or a device for position measurement, navigation, and timing using a global navigation satellite system (GNSS). Although not shown, the UAV 1410 may also be equipped with a member for tracking the UAV 1410 by a total station 1430. This member may be, for example, a retroreflector such as a prism or a reflective sticker.

[0191] The control unit 1411 is realized, for example, by cooperation between hardware including a processor and a storage device and control software. The UAV 1410 is capable of autonomous flight under the control of the control unit 1411. The UAV 1410 is also capable of remotely controlled flight using a UAV controller 1420 or the like. In this case, the control unit 1411 controls the flight of the UAV 1410 based on an operation instruction signal from the UAV controller 1420 or the like. The control unit 1411 includes a communication device for performing data communication with other devices (such as the UAV controller 1420 and the edge computer 1440). This data communication is typically wireless communication, but wired communication may also be possible.

[0192] The image capturing unit 1412 may include, for example, one or more of a digital camera, a laser scanner, and a spectral camera. As the digital camera, an omnidirectional camera (also called a panoramic camera or a 360-degree camera) is typically used. Although a case where the image capturing unit 1412 acquires an image (video) of the surrounding environment using an omnidirectional camera will be described in detail, similar processing can be performed in other cases as well.

[0193] The V-SLAM unit 1413 is realized, for example, by cooperation between hardware including a processor and a storage device and V-SLAM software. The V-SLAM unit 1413 analyzes the video acquired by the imaging unit 1412 in real time to generate three-dimensional information of the surrounding environment (such as buildings) of the UAV 1410, and estimates the position and orientation of the UAV 1410 (particularly the imaging unit 1412). The processing performed by the V-SLAM unit 1413 may be similar to known V-SLAM processing. Note that other technologies capable of generating output similar to V-SLAM can be used instead.

[0194] The obstacle detection unit 1414 is realized, for example, by cooperation between hardware including a processor and a storage device and obstacle detection software. The obstacle detection unit 1414 detects an image area corresponding to an obstacle (tool, worker, etc.) by, for example, inputting images (frames) constituting the video captured by the image capture unit 1412 into the above-mentioned inference model (trained model). Note that the obstacle detection unit 1414 may use rule-based processing to detect an image area corresponding to an obstacle. Alternatively, the obstacle detection unit 1414 may be configured to perform obstacle detection by combining processing using a trained model and rule-based processing.

[0195] The output from the obstacle detection unit 1414 is input to the control unit 1411. The control unit 1411 performs control to avoid collision with the detected obstacle based on the output from the obstacle detection unit 1414. This control may be, for example, any of changing the flight path, levitation, landing, switching from autonomous flight to heteronomous flight, outputting a warning sound, and instructing the UAV controller 1420 to output warning information (such as a warning sound or a warning display).

[0196] The UAV controller 1420 is used as a remote controller for remotely operating the UAV 1410. The UAV controller 1420 is also used to display information about the building to be measured (BIM model, CAD model, component information, construction plan, etc.). The UAV controller 1420 is also used to output information about the UAV 1410 (flight path, images obtained by the image capture unit 1412, warnings, etc.). The UAV controller 1420 may also be used to create or edit a flight plan (flight path) for the UAV 1410.

[0197] The UAV controller 1420 includes a control unit 1421 and a user interface 1422. The control unit 1421 controls each unit of the UAV controller 1420. This is realized by cooperation between hardware including a processor and a storage device and control software. The control unit 1421 includes a communication device for performing data communication with other devices (such as the UAV 1410 and the edge computer 1440). This data communication is typically wireless communication, but wired communication may also be possible.

[0198] The user interface 1422 includes, for example, a display device, an operation device, an input device, etc. The user interface 1422 is typically a mobile computer such as a tablet or a smartphone, and includes a touch screen, a GUI, etc.

[0199] The total station 1430 is used to track the flying UAV 1410. If the UAV 1410 is equipped with a retroreflector, the total station 1430 outputs tracking light (ranging light) and receives the tracking light reflected by the retroreflector, thereby tracking the retroreflector. While tracking the retroreflector, the total station 1430 measures three-dimensional coordinates (slope distance, horizontal angle, vertical angle, etc.) based on the installation position of the total station 1430 (or another reference position). The tracking function is realized, for example, by cooperation between hardware including a processor and a storage device and tracking software. Furthermore, the three-dimensional coordinate measurement function is realized, for example, by cooperation between hardware including a processor and a storage device and three-dimensional coordinate measurement software.

[0200] If the UAV 1410 is not provided with a retroreflector, for example, the UAV 1410 may be provided with multiple light-receiving sensors (not shown). Each light-receiving sensor is capable of receiving tracking light from the total station 1430. By determining which of the multiple light-receiving sensors has received the tracking light, it is possible to estimate the orientation of the UAV 1410 relative to the total station 1430. This estimation process is performed by, for example, any of the UAV 1410, the UAV controller 1420, the total station 1430, and the edge computer 1440.

[0201] The total station 1430 includes a communication device for performing data communication with other devices (such as the UAV 1410, the UAV controller 1420, and the edge computer 1440). This data communication is typically wireless, but wired communication may also be possible. The total station 1430 can transmit to the UAV 1410 in real time the position information (three-dimensional coordinates) of the UAV 1410 that is sequentially acquired along with the above-described tracking.

[0202] In this way, the UAV 1410 can recognize its own current position based on the information transmitted from the total station 1430. In addition, the UAV 1410 can recognize its own current position based on the information obtained by the V-SLAM unit 1413.

[0203] When the UAV 1410 is flying in a blind spot area of ​​the total station 1430, the UAV 1410 only recognizes (relatively rough) position information based on V-SLAM in real time.

[0204] On the other hand, when the UAV 1410 is flying in an area other than the blind spot area, the UAV 1410 can recognize in real time both (relatively rough) position information based on the V-SLAM and (relatively detailed) position information based on the total station 1430. When both pieces of position information can be acquired in real time, the UAV 1410 can associate both pieces of position information.

[0205] In addition, the UAV 1410 may be configured to fly autonomously by referring to (relatively detailed) position information based on the total station 1430 when both types of position information are available, and to fly autonomously by referring to (relatively rough) position information based on V-SLAM at other times.

[0206] The edge computer 1440 is a computer for realizing edge computing at the construction site, and processes data from devices such as the UAV 1410, UAV controller 1420, and total station 1430 at (or near) the construction site. By introducing such edge computing, it is possible to eliminate increased load and communication delays in the entire system 140.

[0207] The edge computer 1440 includes a communication device for performing data communication with devices used at a construction site, such as the UAV 1410, the UAV controller 1420, and the total station 1430. This data communication is typically wireless communication, but wired communication may also be possible.

[0208] The edge computer 1440 also includes a communication device for performing data communication with the cloud computer 1450. This data communication is typically wireless communication, but wired communication may also be possible.

[0209] Furthermore, the edge computer 1440 may include a BIM data processing application and a building data management application. In this embodiment, the edge computer 1440 includes an SfM·MVS unit 1441 and a construction management unit 1442.

[0210] The SfM·MVS unit 1441 is realized by, for example, cooperation between hardware including a processor and a storage device, SfM software, and MVS software. The SfM·MVS unit 1441 is configured to create position information of the UAV 1410 (the actual flight path) and a 3D model of the (real) building based on, for example, an image acquired by the imaging unit 1412 of the UAV 1410, position information of the UAV 1410 acquired by the V-SLAM unit 1413 of the UAV 1410, and position information of the UAV 1410 acquired by the total station 1430.

[0211] The SfM·MVS unit 1441 performs SfM processing to estimate the position information of the UAV 1410 from the video acquired by the UAV 1410 while it is flying. More specifically, the SfM·MVS unit 1441 applies SfM processing to the video acquired by the UAV 1410 while it is flying, thereby obtaining time-series three-dimensional coordinates representing the path actually flown by the UAV 1410 and the attitude of the UAV 1410 corresponding to each three-dimensional coordinate in the time-series three-dimensional coordinates. That is, the SfM·MVS unit 1441 obtains time-series three-dimensional coordinates representing the movement path of the camera included in the imaging unit 1412 and time-series attitude information of the camera along this movement path. In the SfM processing, the position information (three-dimensional coordinates) of the UAV 1410 acquired by the total station 1430 can be referenced. This can improve the accuracy of the acquired time-series three-dimensional coordinates and time-series attitude information. The SfM processing in this example may be similar to known SfM processing. It should be noted that other techniques capable of producing an output similar to the SfM processing of this example may alternatively be used.

[0212] Furthermore, as MSV processing, the SfM·MVS unit 1441 generates point cloud data of the (physical) building based on the position information of the UAV 1410 (time-series position information and time-series attitude information of the camera) obtained by the SfM processing and the images acquired by the UAV 1410 while flying.

[0213] In this example, the edge computer 1440 (the SfM-MVS unit 1441, the construction management unit 1442, or another data processing unit) may be configured to detect image regions (component regions) corresponding to building components by inputting images (frames) constituting the video captured by the imaging unit 1412 into the inference model described above. The edge computer 1440 (the SfM-MVS unit 1441, the construction management unit 1442, or another data processing unit) may also identify component attributes (such as type, identification information (ID), shape, position, measurement date, and measurement time) corresponding to each detected component region. The edge computer 1440 may also use rule-based processing to detect component regions. The edge computer 1440 may also be configured to perform component region detection by combining processing using a trained model and rule-based processing.

[0214] The construction management unit 1442 is realized, for example, by cooperation between hardware including a processor and a storage device and construction management software. The construction management unit 1442 is configured to manage various data handled by the system 140. The processing executed by the construction management unit 1442 will be described later.

[0215] The cloud computer 1450 is a computer for realizing cloud computing, which provides computer resources as a service from a remote location to a construction site via a computer network. The introduction of such cloud computing can improve the scalability, flexibility, efficiency, etc. of the services that can be provided to the system 140.

[0216] The cloud computer 1450 is configured to manage, for example, BIM tools, data management tools, and data used in these tools (design BIM, construction information, measurement BIM, etc.), and provide them to the edge computer 1440. In this embodiment, the cloud computer 1450 includes a BIM unit 1451 and a data management unit 1452.

[0217] The BIM unit 1451 provides various tools such as a BIM tool and various data such as data used in the BIM tool to the edge computer 1440. The BIM unit 1451 is realized, for example, by cooperation between hardware including a processor and a storage device and BIM software. The processing performed by the BIM unit 1451 will be described later.

[0218] The data management unit 1452 manages various tools and various data. The data management unit 1452 is realized, for example, by cooperation between hardware including a processor and a storage device and data management software. The processing executed by the data management unit 1452 will be described later.

[0219] 15 shows an example of the structure (data format) of data handled by system 140 of this embodiment. Data format 150 includes design data 1510, rendering data 1520, measurement data 1530, inspection information 1540, inspection knowledge base 1550, and inspection data 1560. In other words, data format 150 has a data structure that includes areas for recording at least these types of data.

[0220] In the area where the design data 1510 is recorded, the various types of design data described above, such as BIM data (design BIM) and design drawings, are recorded.

[0221] The area where the rendering data 1520 is recorded stores image data (virtual images) obtained by rendering the design BIM. Rendering data is constructed for each of multiple positions in the design BIM. For example, for each of multiple positions on a preset flight path, volume rendering is applied to the design BIM with that position as the viewpoint. This results in multiple rendering data (multiple virtual images) along the flight path in the design BIM. Corresponding position information (3D coordinates of the viewpoint in the design BIM) can be attached to each rendering data as attribute information. The attribute information of the rendering data is not limited to this. For example, the attribute information of the rendering data may include any information related to the design BIM, any information related to the rendering process, any information related to the rendering data, etc.

[0222] The area where the measurement data 1530 is recorded records various data acquired by measuring the (physical) building. Examples of such data include point cloud data of the building, three-dimensional coordinates of each measurement position, video acquired by the image capturing unit 1412 of the UAV 1410, two-dimensional images, dimensional information, measurement date, measurement time, etc. Parameter information related to any of these data can also be recorded. For example, parameter information related to the point cloud data creation process, parameter information related to the image capturing process by the image capturing unit 1412, etc. can be recorded.

[0223] In the area where the inspection information 1540 is recorded, various information (inspection information) for inspecting the actual component corresponding to the virtual component is recorded. The inspection information 1540 includes, for example, the shape, dimensions, construction date, construction time, etc. of the inspection target (virtual component) at each inspection position (position of each virtual component). In other words, the inspection information 1540 may include information on multiple attributes of each virtual component. The inspection information is not limited to these, and may include any information related to the inspection.

[0224] The area where the inspection knowledge base 1550 is recorded stores various types of knowledge used to inspect a real component corresponding to a virtual component. The inspection knowledge base 1550 includes, for example, a measurement path (multiple inspection positions along a flight path) and the aforementioned inference model (weighting coefficients of trained models, neural network models, etc.). The inspection knowledge base 1550 may also include rule-based algorithms.

[0225] The area for recording inspection data 1560 records various data (inspection data) acquired by inspecting a real member corresponding to a virtual member. The inspection data 1560 includes, for example, the presence or absence of an object at the inspection position (presence or absence of a real member corresponding to a virtual member), the deviation of the real member relative to the virtual member (presence or absence of misalignment, the direction of the misalignment, orientation of the misalignment, etc.), and a determination result as to whether the inspection position satisfies a predetermined condition. This determination may include, for example, a determination as to whether an inspection was performed based on data obtained from a preset flight path, in other words, a determination as to whether obstacle avoidance (or deviation from the flight path for other reasons) was performed at that time.

[0226] An example of the operation of the system 140 of this embodiment will be described with further reference to FIGS. 16A to 16C.

[0227] The timing for performing the series of steps (inspections) in this operational example is arbitrary. For example, inspections can be performed for each specified construction process. As an exemplary application, by performing the following series of steps on each construction day, the construction progress status on each construction day can be digitized. If the application of this operational example is only to check the construction progress status, only an inspection of the presence or absence of actual components corresponding to each virtual component may be performed. For other applications, inspections may be performed with higher accuracy.

[0228] (S1: Generate flight path) In this operation example, first, the cloud computer 1450 transmits to the edge computer 1440 (construction management unit 1442) the design BIM, construction information (construction date information for each component), the date when inspection using the UAV 1410 will be performed (measurement date information), obstacle images (virtually generated images, actually captured images), etc. to the edge computer 1440. Based on the information provided by the cloud computer 1450, the edge computer 1440 determines the flight path of the UAV 1410 on the measurement date.

[0229] The flight path may be generated by the cloud computer 1450, the UAV controller 1420, or another computer (same below). Also, instead of obtaining the measurement date information from the cloud computer 1450, the edge computer 1440 or the UAV controller 1420 may generate the measurement date information. The flight path may be generated fully automatically, semi-automatically, or manually. The construction date information may include the construction time.

[0230] The flight path is determined, for example, so that the distance between the imaging unit 1412 of the UAV 1410 and the inspection target (components, floors, ceilings, walls, equipment, pillars, etc.) is within an allowable range. For example, the allowable range (maximum distance) can be set taking into consideration that the closer this distance is, the higher the inspection accuracy. On the other hand, the allowable range (minimum distance) can be set so that the entire building can be photographed.

[0231] (S2: Send flight path to UAV) The edge computer 1440, for example, receives instructions from a user and transmits information about the flight path generated in step S1 to the UAV 1410.

[0232] (S3: Generate a virtual image) The edge computer 1440 also generates a virtual image (rendering data) by rendering the design BIM based on the flight path generated in step S1. For example, the edge computer 1440 generates a virtual image obtained when a virtual UAV (virtual camera) flies along the flight path in the three-dimensional virtual space in which the design BIM is defined. More specifically, for each of a plurality of positions on the flight path, the edge computer 1440 generates an image of the virtual BIM (virtual building) acquired by the virtual camera from that position.

[0233] (S4: Build an inference model) The edge computer 1440 applies machine learning using the virtual image generated in step S3 (and multiple other virtual images) as training data to a predetermined neural network, thereby constructing an inference model (first inference model) for identifying component data from data on the actual building.

[0234] In addition, the edge computer 1440 applies machine learning to a predetermined neural network using the obstacle image (and multiple other obstacle images) provided by the cloud computer 1450 in step S1 as training data, thereby constructing an inference model (second inference model) for identifying data on obstacles mixed in with data on actual buildings.

[0235] By using both the virtual image and the obstacle image as training data, a single inference model can be constructed that functions as both the first inference model and the second inference model. This inference model is trained to identify component data and obstacle data from data on the actual building. The following description will use such an inference model, but is not limited to this. The inference model may also have other functions.

[0236] The training data may include texture information of components and obstacles. The neural network model used to build the inference model is typically a CNN. The method used to build the inference model is not limited to this example, and any method can be applied, such as a support vector machine, a Bayesian classifier, boosting, k-means, kernel density estimation, principal component analysis, independent component analysis, self-organizing map, random forest, or generative adversarial network (GAN).

[0237] (S5: Send the inference model to the UAV) The edge computer 1440 transmits the inference model (weighting coefficients, neural network model, etc.) constructed in step S4 to the UAV 1410.

[0238] (S6: Start measuring the building) After the preparations in steps S1 to S5, the measurement work of the actual building begins.

[0239] (S7: Synchronize the UAV and the total station) In the measurement operation, first, the UAV 1410 and the total station (TS) 1430 are synchronized. That is, the clock in the UAV 1410 and the clock in the total station 1430 are synchronized. This synchronizes the time (imaging time) attached to the image acquired by the UAV 1410 and the time (measurement time) attached to the position information of the UAV 1410 acquired by the total station 1430. Here, the imaging time is attached to the position information acquired from the image using V-SLAM processing.

[0240] When both the UAV 1410 and the total station 1430 are outdoors, the clocks of both can be synchronized, for example, by using a navigation signal (including time information based on an atomic clock) from a navigation satellite. On the other hand, when one of the UAV 1410 and the total station 1430 is indoors, the clocks of both can be synchronized, for example, by using a time server on a network to which both the UAV 1410 and the total station 1430 can connect. Note that the synchronization method is not limited to these.

[0241] (S8: Start tracking the UAV using the total station) After the time synchronization in step S7, the user issues an instruction to start measurement using, for example, the UAV controller 1420. The total station 1430 starts tracking the UAV 1410 that has received the measurement start instruction from the UAV controller 1420. The total station 1430 also starts generating position information of the UAV 1410 and transmitting this position information to the UAV 1410 in real time.

[0242] (S9: Start flying, taking pictures and detecting obstacles) In addition, upon receiving the measurement start instruction in step S8, UAV 1410 begins autonomous flight by referencing the flight path received in step S2, photographing (and saving the video) by the photographing unit 1412, and obstacle detection processing by the V-SLAM unit 1413 and obstacle detection unit 1414.

[0243] (S10: Control the flight based on external information) The UAV 1410 performs autonomous flight by flight control based on the tracking information (position information of the UAV 1410 being tracked) transmitted in real time from the total station 1430 and the flight path received in step S2. Steps S10 to S14 are repeatedly executed until a "Yes" result is determined in step S14.

[0244] (S11: When tracking is interrupted, flight control is performed based on the information obtained by the aircraft itself) When the UAV 1410 enters a blind spot range of the tracking by the total station 1430, for example, the UAV 1410 will no longer be able to receive tracking information from the total station 1430. In response to the loss of reception of the tracking information, the UAV 1410 may be configured to switch the position information referenced for autonomous flight control from the tracking information from the total station 1430 to position information successively acquired by the V-SLAM unit 1413. Furthermore, in response to the resumption of reception of the tracking information, the UAV 1410 may be configured to switch the position information referenced for autonomous flight control from the position information successively acquired by the V-SLAM unit 1413 to the tracking information from the total station 1430.

[0245] Even if the UAV 1410 enters a blind spot of the total station 1430, some tracking information may reach the UAV 1410 due to reflection or transmission of radio waves. Anticipating such a case, the UAV 1410 may be configured to detect a problem from the position information indicated by the tracking information. For example, the UAV 1410 may be configured to detect a problem by comparing the position information indicated by the tracking information with the position information obtained by the V-SLAM unit 1413 (as described above, both pieces of position information are synchronized). As an example of this configuration, the UAV 1410 may be configured to calculate an error between the two pieces of position information (three-dimensional coordinates), and determine that there is a "problem" if the error is equal to or greater than a predetermined threshold, and determine that there is no "problem" if the error is less than the predetermined threshold. In response to the determination result changing from "no problem" to "problem," the UAV 1410 may be configured to switch the position information referenced for autonomous flight control from the tracking information from the total station 1430 to the position information sequentially acquired by the V-SLAM unit 1413. Furthermore, in response to the determination result changing from "problem" to "no problem," the UAV 1410 may be configured to switch the position information referenced for autonomous flight control from the position information sequentially acquired by the V-SLAM unit 1413 to the tracking information from the total station 1430.

[0246] With this configuration, while tracking of the UAV 1410 by the total station 1430 is suspended, the UAV 1410 can perform autonomous flight control based on the position and attitude successively determined by the V-SLAM unit 1413.

[0247] (S12: Did you detect an obstacle?) If the obstacle detection unit 1414 detects an obstacle (S12: Yes), the operation proceeds to step S13. If no obstacle is detected (S12: No), the operation skips step S13 and proceeds to step S14.

[0248] (S13: Perform obstacle avoidance control) If an obstacle is detected in step S12 (S12: Yes), the UAV 1410 performs control for the obstacle avoidance operation described above. For example, the UAV 1410 determines the position, direction, dimensions, etc. of the detected obstacle, determines a route to avoid collision with the obstacle, and flies along this route. Typically, the start and end points of this collision avoidance route are located on the flight route received in step S2. That is, the UAV 1410 deviates from the flight route received in step S2, detours around the obstacle, and returns to the flight route.

[0249] (S14: Have you reached the end of the flight?) The UAV 1410 can determine whether it has reached the end point (flight end point) of the flight path received in step S2 based on tracking information from the total station 1430 and position information acquired by the V-SLAM unit 1413. If it is determined that it has not reached the flight end point (S14: No), the operation returns to step S10. On the other hand, if it is determined that it has reached the flight end point (S14: Yes), the operation proceeds to step S15. This completes the measurement (photography) of the actual building.

[0250] (S15: Send the captured image to the edge computer) When the flight end point is reached (S14: Yes), the UAV 1410 sends the captured images (video) acquired during flight to the edge computer 1440. The UAV 1410 may sequentially send the captured images to the edge computer 1440 while flying, or may accumulate the captured images during flight and send them all at once to the edge computer 1440 after the flight is completed. The UAV 1410 may also repeatedly accumulate and transmit the captured images at predetermined time intervals. Alternatively, the UAV 1410 may transmit a predetermined amount of captured images each time they are accumulated.

[0251] All of the following steps are a series of processes (post-processing) based on images acquired by the UAV 1410. When the UAV 1410 transmits captured images to the edge computer 1440 while flying, post-processing may be started before the flight of the UAV 1410 is completed. In other words, image capture and post-processing may be performed sequentially in parallel. This reduces the work time. On the other hand, when post-processing is started after image capture is completed, post-processing can be performed by referring to all captured images, which improves the accuracy and precision of the processing. For example, when analyzing a captured image, it is possible to refer to images captured before and after it.

[0252] Here, the edge computer 1440 can determine whether the captured image is suitable for post-processing. For example, the edge computer 1440 may be configured to evaluate the quality (brightness, contrast, focus, color, definition, etc.) of the captured image. If the quality is determined to be insufficient, the UAV 1410 can capture the image again, or image processing can be applied to the captured image to improve image quality. This image processing may utilize, for example, a trained model constructed by machine learning. This image processing may also include interpolation processing using previous and subsequent captured images. Image quality evaluation may be started while the UAV 1410 is capturing the image. If the image quality is determined to be insufficient during capture, the UAV 1410 may return to the position where the captured image determined to be of insufficient quality was acquired (or a position upstream in the flight path) to capture the image again.

[0253] Similarly, the total station 1430 transmits the time-series three-dimensional coordinates and time-series attitude information of the UAV 1410 (the image capturing unit 1412, the retroreflector) collected while tracking the UAV 1410 to the edge computer 1440.

[0254] (S16: Calculate the camera position and orientation) The edge computer 1440 performs SfM processing based on the captured images (each frame of the video) acquired by the UAV 1410 and the time-series information (time-series three-dimensional coordinates, time-series attitude information) acquired by the total station 1430, thereby determining the position and attitude of the image capturing unit 1412 (camera) at each of multiple positions on the flight path. Here, since the UAV 1410 and the total station 1430 were synchronized in step S7, it is possible to associate the time information of the video with the time information of the time-series information. By performing this time association, the edge computer 1440 can apply SfM processing to the combination of the video and the time-series information.

[0255] (S17: Extract the component area from the captured image) The edge computer 1440 uses the inference model constructed in step S4 to extract image areas (component areas, component areas) corresponding to components (virtual components) included in the design BIM from the captured images (each frame of the video) acquired by the UAV 1410. This extraction process includes, for example, a process of identifying the component areas in the design BIM and a process of masking image areas other than the identified component areas. Here, the edge computer 1440 may perform a process of identifying predetermined attributes of the actual components corresponding to the identified component areas.

[0256] The edge computer 1440 can determine whether the information obtained in step S17 is suitable for subsequent processing. For example, if the component area of ​​the corresponding real component is not extracted for many virtual components, it is determined to be "unsuitable." If it is determined to be unsuitable, the edge computer 1440 can, for example, perform control to request re-imaging.

[0257] (S18: Find the characteristic points and their coordinates of the component area) The edge computer 1440 detects feature points of the component area extracted in step S17. The feature points may be one or more points, one or more lines, or one or more surfaces. The feature points may also be patterns or the like. Furthermore, the edge computer 1440 calculates the three-dimensional coordinates of the detected feature points.

[0258] (S19: Generate surface data of components) The edge computer 1440 generates surface data of the actual component corresponding to the component area extracted in step S17. For example, the edge computer 1440 aligns (registers) two or more captured images taken at different positions based on the feature points detected in step S18. Typically, these captured images partially overlap. The edge computer 1440 uses MVS processing to generate surface data of the component commonly depicted in these captured images based on the capture positions of these captured images.

[0259] The edge computer 1440 performs registration between the design BIM and the video. As a result, for example, each frame of the video (photographed image) is embedded in the three-dimensional space (three-dimensional coordinate system) in which the design BIM is defined.

[0260] Next, the edge computer 1440 obtains a point cloud in the captured image that is located near a specific surface of the virtual component in the design BIM. For example, the edge computer 1440 identifies the point cloud by identifying points in the captured image whose distance to the specific surface of the virtual component is within a specific threshold. By performing this process for each frame of the video, a 3D coordinate point cloud located near the specific surface of the virtual component is obtained.

[0261] Next, the edge computer 1440 determines a surface in the image corresponding to a predetermined surface of the virtual component based on the obtained three-dimensional coordinate point cloud. For example, the edge computer 1440 determines an approximate surface (plane, free-form surface, etc.) based on at least a portion of the three-dimensional coordinate point cloud. This approximate surface is the above-mentioned surface data. In other words, this approximate surface is treated as surface data (surface image) in the image corresponding to the predetermined surface of the virtual component. In other words, this approximate surface is a portion (surface) of a real component corresponding to the virtual component, and is treated as a portion (surface) corresponding to the predetermined surface of this virtual component. The edge computer 1440 associates the predetermined surface of the virtual component with a surface in the image (surface of the real component) determined based on it. This associates the virtual component with the real component, and associates the attributes (position, shape, etc.) of the virtual component with the attributes (position, shape, etc.) of the real component.

[0262] (S20: Create measurement BIM) The edge computer 1440 creates a three-dimensional model (measured BIM) based on the multiple surface data generated in step S19. In other words, the edge computer 1440 creates a measured BIM based on the data of the multiple actual components obtained in step S19. The created measured BIM is sent to and stored in the cloud computer 1450. The measured BIM is used for comparison with the design BIM, construction management, maintenance management, repair management, etc. This completes this operation example.

[0263] The configurations described above are merely examples of embodiments of the present invention, and any modifications (omissions, substitutions, additions, etc.) can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0264] 1, 8, 9, 10, 110, 130 systems 11 Control section 12 User Interface 13 Data Acquisition Section 14 Storage section 141 Design Data 142 Physical component data 143 Virtual Images 144 images 145, 146, 148 Measurement data 147 Representative part information 149 Material Selection Information 15 Processing section 151 Component matching section 152 Attribute Mapping Unit 153 Inference Model Construction Unit 154 Movement Control Information Creation Department 155 Reference Information Creation Department 156 Data Object Detection Unit 157 Partial area identification part 158, 159 Entity data generation unit

Claims

1. 1. A data processing method for architectural reality capture, comprising: preparing design data including virtual component information relating to a plurality of attributes for each of a plurality of virtual components of the virtual building; A step of preparing measurement data obtained by measuring an actual building constructed based on the design data; generating physical component data relating to the plurality of attributes for each of a plurality of physical components based on the measurement data; preparing a virtual image of the virtual building and an obstacle image; executing a member association process for associating the plurality of virtual members with the plurality of actual members based on the virtual member information and the actual member data; executing an attribute association process for associating the virtual component information with the actual component data based on the plurality of attributes in each of the plurality of pairs of virtual components and actual components acquired by the component association process; a step of executing a movement control information creation process, which is a process of creating movement control information used to control a moving object that measures the real building to acquire the measurement data, based on the virtual image and the obstacle image; Including, The step of executing the movement control information creation process includes: A step of constructing a first inference model for identifying images of building components from images obtained by photographing the real building by applying machine learning using training data including the virtual image to a neural network; a step of constructing a second inference model for identifying an image of an obstacle from the image obtained by photographing the actual building by applying machine learning using training data including the obstacle image to a neural network; Including, the mobility control information includes the first inference model and the second inference model; Data processing methods.

2. The step of executing the movement control information creation process further includes a step of setting a movement path of the moving object for performing the measurement of the actual building based on the virtual image; The movement control information further includes the movement route. The data processing method according to claim 1.

3. The step of executing the movement control information creation process includes setting the movement path so that a distance to at least a part of the plurality of virtual members falls within a predetermined tolerance range. The data processing method according to claim 2.

4. The step of executing the movement control information creation process includes setting, as the movement path, a two-dimensional area or a three-dimensional area indicating a range within which the moving body can move to avoid the obstacle.

4. The data processing method according to claim 2 or 3.

5. The step of executing the movement control information creation process further includes a step of creating a virtual image along the movement path based on the virtual image; the movement control information further includes the virtual image along the movement path; A data processing method according to any one of claims 2 to 4.

6. A program that causes a computer included in an architectural reality capture system to execute each step of the data processing method according to any one of claims 1 to 5.

7. A computer-readable non-transitory recording medium on which the program of claim 6 is recorded.

8. 1. A system for architectural reality capture, comprising: a storage unit that stores design data including virtual component information relating to a plurality of attributes for each of a plurality of virtual components of a virtual building, a virtual image of the virtual building, an obstacle image, and real component data relating to the plurality of attributes for each of a plurality of real components, the real component data being generated based on measurement data acquired by measuring a real building constructed based on the design data; a member associating unit that associates the plurality of virtual members with the plurality of actual members based on the virtual member information and the actual member data to generate a plurality of pairs of virtual members and actual members; an attribute associating unit that associates the virtual component information with the actual component data according to the plurality of attributes for each of the plurality of pairs; a movement control information creation unit that creates movement control information to be used for controlling a moving object that measures the real building to acquire the measurement data, based on the virtual image and the obstacle image; Including, The movement control information creation unit Applying machine learning using training data including the virtual image to a neural network to construct a first inference model for identifying images of building components from images obtained by photographing the real building; and Applying machine learning using training data including the obstacle image to a neural network to construct a second inference model for identifying an obstacle image from the image obtained by photographing the actual building; the mobility control information includes the first inference model and the second inference model; system.

Images