Information processing device, inference model construction method, program, and recording medium

Abstract

To make construction management more efficiently.SOLUTION: An information processing apparatus 1 according to an exemplary embodiment of the present invention has a data set receiving unit 13 which receives a data set including at least BIM data. A route setting unit 151 sets, based on the received data set, routes inside and / or outside a virtual building represented by the BIM data. A virtual image set generating unit 152, based on the received data set and the set route, generates a virtual image set of the virtual building along the route. An inference model constructing unit 153 applies machine learning using training data including at least the generated virtual image set to a neural network to thereby construct an inference model for use in specifying data of a building member from measured data of the building.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and an inference model construction method. , P This invention relates to a program and a recording medium. [Background technology]

[0002] Known technologies for managing the construction, maintenance, and repair of buildings include those that use mobile objects such as unmanned aerial vehicles (UAVs), those that use surveying equipment such as total stations, those that use image processing technologies such as Structure from Motion (SfM), Multi-View Stereo (MVS), and Simultaneous Localization And Mapping (SLAM), and those that use Building Information Modeling (BIM) (see, for example, Patent Documents 1 to 10).

[0003] In this type of construction management, the condition of each and every building component (e.g., pillars, beams, walls, floors, ceilings, etc.) is checked. For example, in construction management, tasks include checking whether each component included in the BIM data has actually been constructed, and whether the location where it was constructed is appropriate. This type of condition checking is often done manually, which requires a great deal of work from workers. In addition, it is difficult to assign workers with sufficient experience to many construction sites. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Publication No. 2016 / 0034137 [Patent Document 2] European Patent Publication No. 3522003 [Patent Document 3] JP 2018-116572 A [Patent Document 4] JP 2018-119882 A [Patent Document 5] JP 2018-124984 A [Patent Document 6] JP 2018-151964 A [Patent Document 7] JP 2019-023653 A [Patent Document 8] JP 2019-105789 A [Patent Document 9] JP 2019-194883 A [Patent Document 10] JP 2019-219206 A [Patent Document 11] JP 2020-004278 A [Patent Document 12] JP 2020-008423 A Summary of the Invention [Problem to be solved by the invention]

[0005] One object of the present invention is to improve the efficiency of construction management. [Means for solving the problem]

[0006] An information processing device according to some exemplary embodiments includes a dataset receiving unit that receives a dataset including at least BIM (Building Information Modeling) data, a route setting unit that sets a route inside and / or outside a virtual building represented by the BIM data based on the dataset, a virtual image set generation unit that generates a virtual image set of the virtual building along the route based on the dataset and the route, and an inference model construction unit that applies machine learning using training data including at least the virtual image set to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of a building.

[0007] In some exemplary information processing devices, the dataset further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data and measurement date information indicating the measurement date of the building, and the inference model construction unit constructs an inference model by applying machine learning corresponding to the construction date and the measurement date to the neural network.

[0008] The information processing device in some exemplary embodiments further includes a BIM data construction unit that constructs first BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed based on the dataset, the route setting unit sets a first route based on the first BIM data, the virtual image set generation unit generates a first virtual image set based on the first BIM data and the first route, and the inference model construction unit applies machine learning using first training data including at least the first virtual image set to the neural network to construct an inference model corresponding to the measurement date.

[0009] The information processing device in some exemplary embodiments further includes a BIM data construction unit that constructs second BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the dataset, the virtual image set generation unit generates a second virtual image set based on the second BIM data and the route, and the inference model construction unit applies machine learning using second training data including at least the second virtual image set to the neural network to construct an inference model corresponding to the measurement date.

[0010] In some exemplary information processing devices, the route setting unit sets a route so that a distance to a virtual building component recorded in the BIM data falls within a predetermined tolerance range.

[0011] In the information processing device according to some exemplary aspects, the route setting unit sets one of a one-dimensional area, a two-dimensional area, and a three-dimensional area as the route.

[0012] An information processing device according to some exemplary embodiments includes a dataset receiving unit that receives a dataset including at least BIM (Building Information Modeling) data and route information representing a route inside and / or outside a virtual building represented by the BIM data, a virtual image set generation unit that generates a virtual image set of the inside and / or outside of the virtual building along the route based on the dataset, and an inference model construction unit that applies machine learning using training data including at least the virtual image set to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of a building.

[0013] An information processing device according to some exemplary embodiments includes a virtual image set receiving unit that receives a set of virtual images along a predetermined route inside and / or outside a virtual building represented by BIM (Building Information Modeling) data, and an inference model construction unit that applies machine learning using training data including at least the virtual image set to a predetermined neural network to construct an inference model for identifying data on building components from measurement data of the building.

[0014] In some exemplary embodiments of the information processing device, the dataset includes texture information representing the surface condition of a virtual building component, and further includes a first information synthesis unit that synthesizes the texture information with the virtual image set to generate first synthesized information, and the inference model construction unit applies machine learning to the neural network using training data that includes at least the first synthesized information.

[0015] In some exemplary embodiments, the information processing device further includes a second information synthesis unit that synthesizes the obstacle information with the virtual image set to generate second synthesized information, and the inference model construction unit applies machine learning to the neural network using training data that includes at least the second synthesized information.

[0016] In some exemplary embodiments, the information processing device further includes a label assignment unit that assigns labels related to virtual building components recorded in the BIM data to the virtual image set, and the inference model construction unit applies supervised learning to the neural network using training data including at least the virtual image set and the labels.

[0017] In some exemplary embodiments, the neural network includes a convolutional neural network.

[0018] The information processing device in some exemplary embodiments further includes a transmitting unit that transmits the inference model constructed by the inference model construction unit to a mobile body configured to perform measurements while autonomously moving inside and / or outside a building.

[0019] An information processing device according to some exemplary embodiments includes a memory unit that stores an inference model constructed by applying machine learning using training data including at least a set of virtual images along a predetermined path inside and / or outside a virtual building represented by BIM (Building Information Modeling) data to a predetermined neural network, and a processing unit that performs data processing using the inference model, and the processing unit includes a feature extractor that extracts features from measurement data of the building, and an identifier that identifies data of building components based on the features.

[0020] An inference model construction method according to some exemplary embodiments includes preparing a dataset including at least BIM (Building Information Modeling) data, setting a route inside and / or outside a virtual building represented by the BIM data based on the dataset, generating a set of virtual images of the virtual building along the route based on the dataset and the route, and applying machine learning using training data including at least the set of virtual images to a predetermined neural network to construct an inference model for identifying data on building components from measurement data of a building.

[0021] In some exemplary embodiments of the inference model construction method, the dataset further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data and measurement date information indicating the measurement date of the building, and an inference model is constructed by applying machine learning corresponding to the construction dates and the measurement dates to the neural network.

[0022] The inference model construction method of some exemplary embodiments further includes constructing first BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed based on the dataset, setting a first route based on the first BIM data, generating a first virtual image set based on the first BIM data and the first route, and applying machine learning using first training data including at least the first virtual image set to the neural network to construct an inference model corresponding to the measurement date.

[0023] In some exemplary embodiments, the inference model construction method further constructs second BIM data based on the dataset, the second BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed, generates a second virtual image set based on the second BIM data and the route, and applies machine learning using second training data including at least the second virtual image set to the neural network to construct an inference model corresponding to the measurement date.

[0024] In an inference model construction method according to some exemplary embodiments, a route is set so that the distance to the virtual building components recorded in the BIM data falls within a predetermined tolerance range.

[0025] In an inference model construction method according to some exemplary embodiments, the route is set to be one-dimensional, two-dimensional, or three-dimensional area.

[0026] An inference model construction method according to some exemplary embodiments prepares a dataset including at least BIM (Building Information Modeling) data and route information representing a route inside and / or outside a virtual building represented by the BIM data, generates a set of virtual images of the inside and / or outside of the virtual building along the route based on the dataset, and applies machine learning using training data including at least the set of virtual images to a specified neural network to construct an inference model for identifying data on building components from measurement data of a building.

[0027] An inference model construction method according to some exemplary embodiments prepares a set of virtual images along a specified route inside and / or outside a virtual building represented by BIM (Building Information Modeling) data, and applies machine learning using training data including at least the set of virtual images to a specified neural network to construct an inference model for identifying data on building components from measurement data of the building.

[0028] In an inference model construction method according to some exemplary embodiments, the dataset includes texture information representing the surface condition of a virtual building component, and further, the texture information is synthesized with the virtual image set to generate first synthesized information, and machine learning using training data including at least the first synthesized information is applied to the neural network.

[0029] In some exemplary embodiments, the inference model building method further includes a method in which the dataset further includes obstacle information, the method further includes synthesizing the obstacle information with the virtual image set to generate second synthetic information, and applying machine learning to the neural network using training data including at least the second synthetic information.

[0030] In some exemplary embodiments, the inference model building method further includes assigning labels to the virtual image set regarding virtual building components recorded in the BIM data, and applying supervised learning to the neural network using training data including at least the virtual image set and the labels.

[0031] In an inference model building method according to some exemplary embodiments, the neural network includes a convolutional neural network.

[0032] The inference model construction method according to some exemplary embodiments further includes transmitting the inference model to a mobile object configured to take measurements while autonomously moving inside and / or outside a building.

[0033] An inference model construction method according to some exemplary embodiments accepts an inference model constructed by applying machine learning using training data including at least a set of virtual images along a specified route inside and / or outside a virtual building represented by BIM (Building Information Modeling) data to a specified neural network, extracts features from measurement data of the building by a processor that processes data using the inference model, and identifies data of building components based on the features by the processor.

[0034] Some exemplary aspects are programs that cause a computer to execute the methods of the exemplary aspects.

[0035] Some exemplary embodiments are inference models constructed by the inference model construction method of the exemplary embodiments.

[0036] Some exemplary aspects are programs that cause a computer to execute a process of identifying data on building components from measurement data of a building using an inference model of an exemplary aspect.

[0037] Some exemplary embodiments are a computer-readable non-transitory recording medium having a program of the exemplary embodiments recorded thereon. Effect of the Invention

[0038] According to an exemplary embodiment, it is possible to improve the efficiency of construction management. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Diagram 2] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Diagram 3] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 6] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 7] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 8] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 9] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 10] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 11] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 12] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 13] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 14] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 15] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 16] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 17] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 18] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 19] FIG. 1 is a schematic diagram illustrating an example of a configuration of an information processing device according to an exemplary embodiment. [Figure 20] 11 is a flowchart illustrating an example of an operation of an information processing device according to an exemplary embodiment. [Figure 21] FIG. 1 is a schematic diagram illustrating an example of a configuration of a system according to an exemplary embodiment. [Figure 22] FIG. 2 is a schematic diagram illustrating an example of a data format (data configuration) used in the system according to the exemplary embodiment. [Figure 23A] 1 is a flowchart illustrating an example of the operation of a system according to an exemplary embodiment. [Figure 23B] 1 is a flowchart illustrating an example of the operation of a system according to an exemplary embodiment. [Figure 23C] 1 is a flowchart illustrating an example of the operation of a system according to an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] In the present disclosure, several exemplary aspects of an information processing device, an inference model construction method, an information processing method, an inference model, a program, and a recording medium will be described. Also, data structures that can be used in the several aspects will be described.

[0041] The exemplary aspects can be used, for example, to favorably put into practical use and operate an architectural reality capture system. Reality capture is a technology for acquiring data of an actual object using a digital camera or a laser scanner to construct a three-dimensional model, and is used in various fields such as measurement, virtual reality, and augmented reality. In this disclosure, the application of reality capture in the architectural field is described, but the exemplary aspects can also be applied to reality capture in other fields (e.g., the civil engineering field) or technologies other than reality capture. In addition, matters described in the documents cited in this specification and any other publicly known technologies can be incorporated into the exemplary aspects.

[0042] At least a portion of the functionality of the elements described in this disclosure is implemented using circuitry or processing circuitry. The circuitry or processing circuitry may be a general purpose processor, a special purpose processor, an integrated circuit, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), or a combination of a programmable logic device (PLC) and a programmable logic device (PLC)) configured and / or programmed to perform at least a portion of the disclosed functionality. The term "circuitry", "unit", "means" or the like includes any of the above-mentioned circuitry, including a transistor and / or other circuitry, a processor, a processor array, a conventional circuitry, and any combination thereof. A processor is considered to be a processing circuitry or circuitry, including transistors and / or other circuitry. In this disclosure, a circuitry, a unit, a means, or the like is hardware that performs at least some of the disclosed functions or is hardware that is programmed to perform at least some of the disclosed functions. The hardware may be hardware disclosed herein or may be known hardware that is programmed and / or configured to perform at least some of the described functions. In the case where the hardware is a processor that can be considered to be a type of circuitry, a circuitry, a unit, a means, or the like is a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

[0043] The exemplary aspects described below may be combined in any manner, for example, two or more exemplary aspects may be at least partially combined.

[0044] <First aspect> 1 and 2 show configuration examples of an information processing device according to this embodiment. The information processing device 1 may be included in, for example, a building reality capture system. The building reality capture system has a function of measuring an actual building and acquiring digital data. This embodiment may be configured to provide a data structure (format) for facilitating comparison between measurement data of an actual building and design data.

[0045] The information processing device 1 according to this embodiment includes at least a data set receiving unit 13 and a processing unit 15, and further includes a control unit 11, a user interface 12, and a storage unit .

[0046] The control unit 11 is configured to execute various controls of the information processing device 1. The control unit 11 is realized, for example, by cooperation between hardware including a processor and a storage device and control software. The control unit 11 is provided in one computer, or distributed across two or more computers.

[0047] The user interface 12 includes, for example, a display device, an operation device, an input device, etc. In some exemplary embodiments, the user interface 12 includes a graphical user interface (GUI) that utilizes a touch screen, a pointing device, computer graphics, etc. The user interface 12 may be provided in one computer or distributed across two or more computers.

[0048] The data set reception unit 13 has a function of receiving data to be processed by the information processing device 1. This data reception function is a function of receiving data from the outside. The data reception function may be realized, for example, by using a communication device for performing data communication with the outside (such as a device, a system, or a database), or may be realized by using a drive device for reading data recorded in a recording medium. The data received from the outside by the data set reception unit 13 may be, for example, data generated using a computer (such as BIM data, CAD data, etc.), or may be data previously acquired and / or processed by any of the information processing device 1, other devices, and other systems. In this embodiment, the data set reception unit 13 receives a data set including at least BIM data created in advance. In addition, a recording medium applicable to the data reception function is a computer-readable non-transitory recording medium, and may be, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like.

[0049] An information processing device (or a system including the same) according to some exemplary embodiments may have a data generation function in addition to or instead of the data reception function. The data generation function may include, for example, any of a function of collecting data from an entity, a function of processing data collected from an entity, a function of generating data using a computer, and a function of processing data generated in advance.

[0050] The function of collecting data from an actual object may include, for example, one or both of a function of photographing an actual object with a camera (all-around camera) or a video camera (all-around video camera) mounted on a moving object such as an unmanned aerial vehicle (UAV) or a worker, and a function of scanning an actual object with a laser scanner or a total station to collect data. An information processing device (or a system including the same) having such a function may include one or more measuring devices.

[0051] The function of processing data collected from an actual object may include, for example, a function that is realized by using at least a processor and applies a predetermined process to a photographed image or scanned data of the actual object to generate other data. Examples of such a function include a data processing function using the above-mentioned SfM, MVS, SLAM (V-SLAM; Visual SLAM), etc. Another example is a data processing function using a trained model constructed using machine learning. An information processing device (or a system including the same) having such a function is provided in one computer, or distributed across two or more computers.

[0052] The function of generating data using a computer may include a data generation function using computer graphics, such as a function of generating BIM data using a BIM application or a function of generating CAD (Computer-Aided Design) data using a CAD application. The function of generating data using a computer may also include a function of generating data using various architectural applications such as a construction management application, a maintenance management application, and a repair management application. An information processing device (or a system including the same) having such a function may be provided in one computer, or distributed across two or more computers.

[0053] The function of processing pre-generated data may include, for example, a function that is realized using at least a processor, and applies a predetermined process to data of an entity previously acquired and / or processed by any of the information processing device 1, other devices, and other systems to generate other data. The technology applicable to the function may be similar to the technology applicable to the function of processing data collected from an entity. An example of pre-generated data is BIM data. An information processing device (or a system including the same) having the function may be provided in one computer, or distributed across two or more computers.

[0054] In this embodiment, the entity is a building. The building is constructed based on design data (BIM data, design drawings, construction drawings, construction plans, etc.) generated in advance. The data set receiving unit 13 receives a data set including at least a part of the BIM data included in the design data.

[0055] In this disclosure, the building data recorded in the design data (and / or the building data obtained by processing the design data) may be referred to as a virtual building, and its components (building components) may be referred to as virtual components. In some exemplary embodiments, the virtual components are component models provided by the BIM model, and a building constructed using multiple component models is a virtual building.

[0056] Meanwhile, in this disclosure, a building constructed based on design data may be called an actual building, and its constituent elements (building components) may be called actual components. The above-mentioned actual building corresponds to the actual building referred to here. The actual building may be not only a building completed based on design data, but also a building in the middle of construction (an unfinished building), or even a construction site before construction.

[0057] In the present disclosure, a building component may include structural components such as columns, beams, walls, slabs, roofs, foundations, etc., as well as non-structural components such as windows, doors, stairs, tiles, flooring, etc., various parts, various machines, various equipment, various facilities, etc. More generally, a building component in the present disclosure may be any object that can be registered as a virtual component, or any object that can be used as a real component.

[0058] The storage unit 14 is configured to store various data (information). The storage unit 14 stores, for example, data acquired by the data set receiving unit 13. The storage unit 14 includes, for example, a relatively large-capacity storage device (memory, storage device) such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage unit 14 includes one storage device or two or more storage devices. In this embodiment, the storage device 14 stores a data set 141, route information 142, training data 143, and an inference model 144.

[0059] The dataset 141 is received by the dataset receiving unit 13. The dataset 141 may be any data or information related to the design of a building. The dataset 141 of this embodiment includes at least BIM data of a virtual building (multiple virtual components) designed using a BIM application (BIM tool) outside the information processing device 1, and may further include design drawings, construction drawings, construction plans, etc. In addition, the dataset 141 may include data generated from any one or more of any data or information (for example, BIM data, design drawings, construction drawings, construction plans).

[0060] The data set 141 of this embodiment may include virtual member information. The virtual member information includes information on a plurality of virtual members that are components of a virtual building. More specifically, the virtual member information includes information on a plurality of attributes that are set in advance for each virtual member. The attributes referred to here refer to properties, characteristics, properties, etc. that a virtual member has.

[0061] In some exemplary embodiments, the attributes of the virtual parts include, for example, virtual part identification information (virtual part ID), virtual part shape information, virtual part position information, part construction date information, etc. Note that the attributes of the virtual parts are not limited to these, and may be any property, characteristic, or property, such as material.

[0062] The virtual member ID is information for identifying a virtual member. The virtual member ID indicates the type of the virtual member (column, beam, wall, slab, roof, foundation, window, door, staircase, tile, flooring, parts, machine, equipment, facility, etc.). The virtual member ID may be, for example, identification information (such as a member number) given to a real member. The virtual member ID is obtained, for example, from BIM data, design documents, construction drawings, etc. Also, the virtual member ID may be individually unique identification information. As such a virtual member ID, there is identification information provided by Industry Foundation Classes (IFC), which is a file format of a neutral and open CAD data model. The virtual member shape information is information representing the shape of a virtual member. The virtual member shape information may include information representing the posture, orientation, etc. of a virtual member. The virtual member shape information is obtained, for example, from BIM data, design documents, construction drawings, etc. The virtual member position information represents the position of a virtual member in a virtual building. The position of a virtual component is expressed, for example, by the coordinates of the virtual component in a virtual space in which a virtual building is defined (a three-dimensional virtual space defined in a three-dimensional coordinate system). Virtual component position information is obtained, for example, from BIM data, design drawings, construction drawings, etc. Component construction date information indicates the date (construction date, planned construction date) when a real component corresponding to a virtual component will be installed at a construction site. Component construction date information is obtained, for example, from construction drawings and construction plans.

[0063] The information processing device 1 (for example, the control unit 11 and the storage unit 14) provides, for example, a design database for managing a data set 141. For example, the design database stores data of a virtual building (multiple virtual components) designed using a BIM application. The design database is configured to manage each of the multiple virtual components included in this virtual building. For example, the design database stores a data set 141 including actual BIM data. The design database manages, for example, the data set 141 for each virtual building. The design database is configured to manage virtual component information using, for example, a table-type data structure (format).

[0064] The data and information stored in the memory unit 14 are not limited to these, and one example is actual component data. The actual component data may be any data or information related to an actual component. The actual component data is generated based on measurement data (photographed images, scan data, point cloud data, etc.) acquired by measuring (photographing, laser scanning, etc.) an actual building constructed based on BIM data included in the dataset 141, for example. The measurement of the actual building is performed by an external system (UAV, total station, etc.). In addition, the generation of the actual component data based on the measurement data is performed by the information processing device 1 or an external system. The actual component data is generated and managed, for example, as BIM data in the same format as the dataset 141.

[0065] The real component data includes information about a plurality of real components that are components of the real building. More specifically, the real component data includes information about a plurality of predefined attributes for each real component. The attributes here refer to properties, characteristics, properties, etc., that the real component possesses.

[0066] In some exemplary embodiments, the attributes of the real member correspond to the attributes of the virtual member described above. For example, the attributes of the real member include real member identification information (real member ID), real member shape information, real member position information, measurement date information, etc. Note that the attributes of the real member are not limited to these, and may be any property, characteristic, or property, such as material. The attribute information of the real member is generated, for example, by the information processing device 1 or an external system.

[0067] The entity member ID is information for identifying an entity member. As with the virtual member ID, the entity member ID is information indicating the type of a virtual member, and may be, for example, identification information (such as an entity number) given to an entity member. In some exemplary embodiments, the entity member ID may be the same as the corresponding virtual member ID (for example, identification information provided by IFC). Alternatively, the entity member ID may be information different from the corresponding virtual member ID, and may be in a predetermined format that allows the information processing device 1 (and an external system, etc.) to recognize the association with the corresponding virtual member ID. The entity member shape data is data representing the shape of an entity member acquired based on measurement data. The entity member shape data may include data representing the posture, orientation, etc. of the entity member. The entity member position data represents the position of an entity member in an entity building. The position of the entity member is expressed, for example, by the coordinates of the entity member in a virtual space (a three-dimensional virtual space defined in a three-dimensional coordinate system) in which a BIM model of an entity building created based on measurement data is defined. The measurement date information indicates the date on which the measurement of the entity building was performed. The measurement date information is generated, for example, by a measurement system (such as a mobile object, a total station, or a computer) that measures the actual building.

[0068] Note that the component construction date information, which is an attribute of a virtual component, and the measurement date information, which is an attribute of a real component, include at least information of the year, month, and day, and may further include information such as hour, minute, second, etc. Furthermore, when the standard time of the location where the design was made differs from the standard time of the location where the real building exists, the information processing device 1 or an external system, etc. may be configured to perform conversion so that the component construction date information and the measurement date information are expressed in the same standard time.

[0069] The information processing device 1 (for example, the control unit 11 and the storage unit 14) provides, for example, an entity component database for managing entity component data. For example, the entity component database stores data of a BIM model (a BIM model of multiple entity components) of an entity building obtained by processing measurement data of the entity building. The entity component database is configured to manage multiple entity component models included in the entity building model one by one. For example, the entity component database stores an entity building BIM model. The entity component database manages entity component data for each entity building BIM model. The entity component database is configured to manage entity component data using a table-type data structure (format) similar to the design database, for example.

[0070] The route information 142 represents a route set by the processing unit 15 (a route setting unit 151 described later). The training data 143 includes a virtual image set generated by the processing unit 15 (a virtual image set generating unit 152 described later). The inference model 144 is constructed by the processing unit 15 (an inference model constructing unit 153 described later). The route information 142, the training data 143, and the inference model 144 will be described later.

[0071] The processing unit 15 is configured to execute data processing. The processing unit 15 is realized, for example, by cooperation between hardware including a processor and a storage device and data processing software. The processing unit 15 is provided in one computer, or distributed among two or more computers. The processing unit 15 includes a path setting unit 151, a virtual image set generating unit 152, and an inference model building unit 153.

[0072] The route setting unit 151 is configured to set routes inside and / or outside a virtual building represented by the BIM data, based on a data set 141 including at least the BIM data. The control unit 11 stores information representing the route set by the route setting unit 151 in the storage unit 14 (route information 142). The route setting unit 151 is realized, for example, by cooperation between hardware including a processor and a storage device and route setting software.

[0073] Examples of algorithms that can be used for routing include dynamic programming, Heldkarp algorithm, linear programming, logic trees, branch and bound, linear programming, group theoretic algorithms, cutting plane methods, branch and cut, Lin-Kernighan algorithm, local search algorithms, simulated annealing, Hopfield networks, Boltzmann machines, genetic algorithms, tabu search, heuristic algorithms, Christofide algorithm, and polynomial time approximation algorithms.

[0074] The route setting unit 151 may set a route using a trained model constructed by machine learning. For example, training data generated from a large amount of BIM data is used for this machine learning. For example, this machine learning is performed in the same manner as a method for solving the traveling salesman problem, and may include at least one method of supervised learning, unsupervised learning, and reinforcement learning.

[0075] The path setting unit 151 is set, for example, so as to pass near all building components (virtual building components, virtual components) that constitute the virtual building. Typically, the path setting unit 151 may be configured to set the path so that the distance to the virtual components is within a predetermined allowable range. This allows the real components of the real building constructed based on this virtual building to be photographed from a suitable distance. For example, the allowable range (maximum distance) can be set in consideration of the fact that the closer the distance to the virtual components, the higher the accuracy of measuring the corresponding real components tends to be. In addition, the allowable range (minimum distance) can be set in consideration of the fact that the farther the distance to the virtual components, the higher the efficiency of measurement tends to be.

[0076] The route setting unit 151 may set a one-dimensional area as the route. In other words, the route setting unit 151 may set a linear route inside and outside the virtual building. The route set by the route setting unit 151 may be a two-dimensional area or a three-dimensional area. The two-dimensional area or the three-dimensional area represents, for example, a range within which a moving object performing measurement while moving can move in order to avoid obstacles.

[0077] The virtual image set generator 152 is configured to generate a plurality of images of a virtual building along a route represented by the route information 142, based on the dataset 141 and the route information 142. Such an image of a virtual building is called a virtual image, and a plurality of virtual images is called a virtual image set. The virtual image set generator 152 is realized, for example, by cooperation between hardware including a processor and a storage device and virtual image set generation software.

[0078] The control unit 11 stores the virtual image set generated by the virtual image set generation unit 152 in the storage unit 14 (training data 143). The training data 143 includes at least the virtual image set, and may further include other data (described later).

[0079] The virtual image is an image representing a virtual building, and is, for example, rendering data obtained by rendering BIM data (also called design BIM) included in the data set 141. Such rendering data (virtual image) is constructed for each of a plurality of positions in the design BIM. In this embodiment, for each of a plurality of positions on the route represented in the route information 142, the virtual image set generator 152 applies volume rendering with the position as the viewpoint to the design BIM. This allows a plurality of rendering data (virtual image sets) of the design BIM along the route represented in the route information 142 to be obtained. Corresponding position information (three-dimensional coordinates of the viewpoint in the design BIM) can be attached to each rendering data as attribute information. The attribute information of the rendering data is not limited to this. For example, the attribute information of the rendering data may include any information related to the design BIM, any information related to the rendering process, any information related to the rendering data, etc.

[0080] The inference model construction unit 153 is configured to apply machine learning using training data 143 including at least the virtual image set generated by the virtual image set generation unit 152 to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of a building. The control unit 11 stores the inference model constructed by the inference model construction unit 153 in the memory unit 14 (inference model 144). The inference model construction unit 153 is realized, for example, by cooperation between hardware including a processor and a storage device and inference model construction software.

[0081] Applications of inference models include, for example, controlling moving objects that measure (photograph, laser scan) real buildings, and creating BIM data based on measurement data of real buildings (photographed images, scanned data, point cloud data, etc.).

[0082] 2 shows an example of the configuration of the inference model construction unit 153. The inference model construction unit 153 of this example includes a learning processing unit 153A and a neural network 153B.

[0083] The neural network 153B typically includes a convolutional neural network (CNN). An example of the structure of a convolutional neural network is shown in Figure 2. An image is input to the input layer.

[0084] Behind the input layer, multiple pairs of convolutional layers and pooling layers are arranged. In the example shown in Figure 2, two pairs of convolutional layers and pooling layers are provided, but the number of pairs is arbitrary.

[0085] In the convolution layer, a convolution operation is performed to grasp features (such as contours) from an image. The convolution operation is a multiplication and accumulation operation of a filter function (weighting factor, filter kernel) of the same dimension as the input image for the input image. In the convolution layer, the convolution operation is applied to each of multiple parts of the input image. More specifically, in the convolution layer, the value of each pixel of the partial image to which the filter function is applied is multiplied by the value (weight) of the filter function corresponding to that pixel to calculate the product, and the sum of the products is obtained over multiple pixels of this partial image. The product-sum value thus obtained is substituted for the corresponding pixel in the output image. The convolution operation result for the entire input image is obtained by performing the product-sum operation while moving the part (partial image) to which the filter function is applied. According to such a convolution operation, a large number of images in which various features are extracted using a large number of weighting factors are obtained. In other words, a large number of filtered images such as smoothed images and edge images are obtained. The large number of images generated by the convolution layer are called feature maps.

[0086] In the pooling layer, the feature map generated by the immediately preceding convolution layer is compressed (data thinning, etc.). More specifically, in the pooling layer, statistics of a predetermined neighborhood pixel of a pixel of interest in the feature map are calculated at a predetermined pixel interval, and an image with a smaller size than the input feature map is output. The statistics applied to the pooling operation are, for example, maximum value (max pooling) or average value (average pooling). The pixel interval applied to the pooling operation is called a stride.

[0087] In a convolutional neural network, multiple pairs of convolutional layers and pooling layers are provided for processing, making it possible to extract many features from an input image.

[0088] A fully connected layer is provided after the last pair of convolutional and pooling layers. In the example shown in FIG. 2, two fully connected layers are provided, but the number of fully connected layers is arbitrary. In the fully connected layer, image classification, image segmentation, regression, and other processes are performed using features compressed by convolution and pooling. An output layer that provides output results is provided after the last fully connected layer.

[0089] In some exemplary embodiments, the convolutional neural network may not include a fully connected layer, and may include a support vector machine or a recurrent neural network (RNN). Machine learning for the neural network 153B may be transfer learning. That is, the neural network 153B may include a neural network that has already been trained using other training data (training images) and has had its parameters adjusted. The model construction unit 153 (the learning processing unit 153A) may perform fine tuning on the trained neural network (153B). The neural network 153B may include, for example, a known open source neural network architecture.

[0090] The learning processing unit 153A applies machine learning using the training data 143 to the neural network 153B. When the neural network 153B is a convolutional neural network, the parameters adjusted by the learning processing unit 153A are, for example, the filter coefficients of the convolutional layer and the connection weights and offsets of the fully connected layer.

[0091] The training data 143 includes at least a virtual image set as described above. The training data 143 may further include data other than virtual images. For example, the training data 143 may include photographed images of any object, photographed images of a person, etc. Examples of photographed images of objects include images of building components, and images of tools such as ladders and stepladders. Examples of images of people include images of workers at a construction site. The training data 143 may also include other measurement data (scan data, point cloud data, etc.) of real buildings. The number of training data may also be increased using techniques such as data expansion and data augmentation.

[0092] In this embodiment, by using virtual images created from BIM data (design BIM), it is possible to eliminate the need to collect a large amount of measurement data of actual buildings. In particular, by creating a large number of virtual images (virtual image sets) from a large number of BIM data and using them as training data, it is possible to improve the quality of machine learning and the quality of inference models.

[0093] In addition, by using images of building components, objects other than building components, and people as training data, it is possible to train an inference model to detect tools and workers. This makes it possible to control moving objects to avoid obstacles and workers, and to exclude data on obstacles and workers when creating BIM data from measurement data.

[0094] The virtual images and / or measurement data (photographed images, scanned data, point cloud data, etc.) included in the training data 143 may include texture information that represents the surface condition of building components. By using the texture information, a final output that reflects the texture of the object's surface can be obtained. This can improve the quality of mobile object control and BIM data creation.

[0095] Any training method for constructing an inference model may be used, and may be, for example, any one of supervised learning, unsupervised learning, and reinforcement learning, or a combination of two or more of them. Typically, supervised learning is performed using training data in which input images are labeled with a final output label. In addition, in order to prevent processing from concentrating on a specific unit of the neural network 153B, the learning processing unit 153A may randomly select and disable some units and perform learning using the remaining units (dropout).

[0096] In addition, the methods used to construct the inference model are not limited to the examples shown here, and any method can be used, such as support vector machines, Bayesian classifiers, boosting, k-means, kernel density estimation, principal component analysis, independent component analysis, self-organizing maps, random forests, and generative adversarial networks (GANs).

[0097] The operation of the information processing device 1 according to this embodiment will be described below. An example of the operation of the information processing device 1 is shown in FIG.

[0098] First, a dataset including at least BIM data is prepared (S1). The preparation of the dataset may be performed automatically and / or manually, for example using a computer such as a cloud computer.

[0099] Next, the information processing device 1 receives the data set prepared in step S1 through the data set receiving unit 13 (S2). The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0100] Next, the route setting unit 151 sets a route inside and / or outside the virtual building represented by the BIM data in the data set 141 based on the data set 141 received in step S2 (S3). Information representing the set route is stored in the memory unit 14 by the control unit 11 (route information 142).

[0101] Next, the virtual image set generator 152 generates (S4) a virtual image set of the virtual building represented by the BIM data in the dataset 141 along the route set in step S3 based on the dataset 141 and the route information 142. The generated virtual image set is stored (together with other data) in the memory unit 14 by the control unit 11 (training data 143).

[0102] Next, the learning processing unit 153A of the inference model construction unit 153 applies machine learning using the training data 143 including at least the virtual image set generated in step S4 to the neural network 153B to construct an inference model for identifying data on building components from the measurement data of the building (S5). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0103] The inference model 144 constructed in this embodiment is provided to, for example, a mobile body configured to collect measurement data while moving (autonomously or heteronomously) inside and / or outside a building. The inference model 144 is also provided to a computer configured to identify data of building components from the measurement data of the building collected by such a mobile body. Note that the uses of the inference model 144 are not limited to these and may be arbitrary. Also, at least a part of the inference model 144 can be used for, for example, transfer learning or fine tuning.

[0104] Thus, the information processing device 1 according to this embodiment can be used for reality capture of buildings, and includes a data set receiving unit 13, a route setting unit 151, a virtual image set generating unit 152, and an inference model construction unit 153. The data set receiving unit 13 is configured to receive a data set including at least BIM data. The route setting unit 151 is configured to set a route inside and / or outside a virtual building represented by the BIM data in the data set based on the data set received by the data set receiving unit 13. The virtual image set generating unit 152 is configured to generate a virtual image set of the virtual building along the route based on the data set received by the data set receiving unit 13 and the route set by the route setting unit 151. The inference model construction unit 153 is configured to apply machine learning using training data including at least the virtual image set generated by the virtual image set generating unit 152 to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of the building.

[0105] In the information processing device 1 according to this embodiment, the route setting unit 151 may be configured to set a route so that the distance to the building component data (virtual building component) recorded in the BIM data falls within a predetermined allowable range.

[0106] In the information processing device 1 according to this embodiment, the route setting unit 151 may be configured to set any one of a one-dimensional area, a two-dimensional area, and a three-dimensional area as a route.

[0107] In addition, in the information processing device 1 according to this embodiment, the neural network used to construct the inference model may include a convolutional neural network.

[0108] This aspect also provides a method for constructing an inference model that can be used for reality capture of a building. This inference model construction method includes at least the following steps: preparing a dataset including at least BIM data (dataset preparation step); setting a route inside and / or outside a virtual building represented by the BIM data in the dataset based on the prepared dataset (route setting step); generating a virtual image set of the virtual building along the route based on the prepared dataset and the set route (virtual image set generation step); applying machine learning using training data including at least the generated virtual image set to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of the building (inference model construction step). Note that any of these steps may include two or more sub-steps (similar below).

[0109] In the inference model construction method of this embodiment, the route setting process may be performed to set a route so that the distance to the virtual building components recorded in the BIM data falls within a predetermined tolerance range.

[0110] In addition, in the inference model construction method according to this embodiment, the route setting process may be performed to set any one of a one-dimensional area, a two-dimensional area, and a three-dimensional area as a route.

[0111] In addition, in the inference model construction method of this embodiment, the neural network used in the inference model construction process may include a convolutional neural network.

[0112] This aspect provides a program for causing a computer to execute each of the steps of preparing a data set, planning a route, generating a virtual image set, and constructing an inference model, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0113] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0114] This aspect provides an inference model constructed through a dataset preparation process, a route setting process, a virtual image set generation process, and an inference model construction process.

[0115] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0116] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0117] According to this aspect, an inference model trained using a virtual image set can be used to check the condition of each building component (e.g., pillars, beams, walls, floors, ceilings, etc.) in building management. This can reduce the labor required for checking whether each component included in BIM data has actually been constructed, and whether the construction position is appropriate. In addition, since these tasks can be performed using a trained model, there is no need to assign workers with sufficient experience to many building sites. Furthermore, since the system is configured to perform machine learning using a virtual image set, there is no need to collect a large number of images of actual buildings. Therefore, the labor required for collecting training data can be reduced, and a large amount of training data can be easily collected. In this way, according to this aspect, it is possible to improve the efficiency of building management from various perspectives.

[0118] <Second aspect> In the second and third aspects, several examples are described for constructing an inference model by applying machine learning corresponding to the construction date (date constructed, planned construction date) of each building component and the measurement date (planned measurement date) of the building to a neural network.

[0119] An example of the configuration of an information processing device according to the second aspect is shown in Fig. 4. The information processing device 4 is configured to add a BIM data construction unit 154 to the processing unit 15 of the information processing device 1 of the first aspect, and to provide the output of the BIM data construction unit 154 to both the route setting unit 151 and the virtual image set generation unit 152. The BIM data construction unit 154 is realized, for example, by cooperation between hardware including a processor and a storage device and BIM data construction software. Explanations of elements common to the first aspect will be omitted unless otherwise noted.

[0120] The dataset 141 prepared in this embodiment includes, in addition to the BIM data similar to that in the first embodiment, construction date information (component construction date information) indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building.

[0121] The member construction date information indicates the date (construction date, planned construction date) when the actual member corresponding to the virtual member is installed at the construction site. The member construction date information is acquired from, for example, a construction drawing or a construction plan.

[0122] The measurement date information indicates the measurement date of the actual building. In this embodiment, the measurement date information indicates the date of a future measurement (scheduled measurement date), for example, the next measurement date. Note that the measurement of the actual building is carried out by a building measurement system (including a mobile body, a total station, a computer, etc.). The measurement date information is obtained, for example, from a construction plan, a management plan, etc.

[0123] The component construction date information and measurement date information include at least the year, month, and day information, and may further include information such as hour, minute, second, etc. Furthermore, if the standard time of the location where the design was made differs from the standard time of the location where the actual building exists, the information processing device 4 (or an external system, etc.) can perform conversion so that the component construction date information and the measurement date information are expressed in the same standard time.

[0124] The BIM data construction unit 154 is configured to construct BIM data (first BIM data) corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed, based on a dataset 141 that includes at least BIM data, component construction date information, and measurement date information.

[0125] For example, the BIM data construction unit 154 first compares the construction date of each virtual component recorded in the BIM data in the data set 141 with the measurement date, and selects a virtual component whose construction date is earlier than the measurement date from among the multiple virtual components recorded in the BIM data. Here, the virtual component whose construction date is earlier than the measurement date may be a virtual component whose construction date is the same as the measurement date or a virtual component whose construction date is earlier than the measurement date, or a virtual component whose construction date is earlier than the measurement date.

[0126] Next, the BIM data construction unit 154 constructs the first BIM data including only the selected virtual component group. For example, the BIM data construction unit 154 may be configured to construct the first BIM data including only the selected virtual component group by deleting each virtual component that was not selected from the BIM data in the data set 141.

[0127] The first BIM data thus constructed is BIM data corresponding to the layout of building components on the measurement date. The first BIM data is effective for measuring an actual building during construction, for example.

[0128] The first BIM data constructed by the BIM data construction unit 154 is sent to both the route setting unit 151 and the virtual image set generation unit 152.

[0129] The route setting unit 151 of this embodiment sets a route inside and / or outside the virtual building represented by the first BIM data based on the first BIM data constructed by the BIM data construction unit 154. The route set based on the first BIM data is called a first route.

[0130] The first route set in this way takes into consideration only the building components installed on the measurement date. In other words, the first route can be said to be a route optimized for the measurement work on the measurement date. Therefore, it is possible to efficiently perform the measurement work on the measurement date.

[0131] The virtual image set generation unit 152 of this embodiment generates a virtual image set of a virtual building along a first route based on the first BIM data constructed by the BIM data construction unit 154 and the first route set based on the first BIM data by the route setting unit 151. The virtual image set generated based on the first BIM data and the first route is called a first virtual image set.

[0132] The inference model construction unit 153 in this embodiment applies machine learning using training data (first training data) that includes at least such a first virtual image set to a neural network to construct an inference model corresponding to the measurement date.

[0133] The first virtual image set is a plurality of virtual images along a first route optimized for the measurement work on the measurement date. By performing machine learning using such a first virtual image set, it is possible to improve the performance of an inference model (used for controlling a moving object on the measurement date and for analyzing measurement data acquired on the measurement date).

[0134] It is possible to create in advance an inference model corresponding to each of the multiple measurement dates. In other words, it is possible to prepare multiple inference models corresponding to the multiple measurement dates. In addition, the large number of virtual images obtained in the process (the large number of virtual images on different measurement dates) can be used to build each inference model (machine learning). This makes it possible to increase the amount of training data. In addition, it is possible to train a component identification process according to changes in the number and position of building components.

[0135] Thus, the information processing device 4 according to this embodiment includes a BIM data construction unit 154 in addition to the elements of the first embodiment. Moreover, the dataset received by the dataset receiving unit 13 further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data and measurement date information indicating the measurement date of the building. The BIM data construction unit 154 is configured to construct first BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed based on this dataset. Moreover, the route setting unit 151 of this embodiment is configured to set a first route based on the first BIM data. Furthermore, the virtual image set generation unit 152 of this embodiment is configured to generate a first virtual image set based on the first BIM data and the first route. In addition, the inference model construction unit 153 of this embodiment is configured to apply machine learning using first training data including at least the first virtual image set to a neural network to construct an inference model corresponding to the measurement date.

[0136] This aspect also provides a method for constructing an inference model that can be used for reality capture of a building, etc. In this inference model construction method, the dataset prepared in the dataset preparation step further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building. Furthermore, the inference model construction step is executed to apply machine learning corresponding to the construction date and the measurement date to a neural network to construct an inference model.

[0137] More specifically, the inference model construction method of this embodiment further includes a first BIM data construction step. The first BIM data construction step is executed to construct first BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the data set prepared in the data set preparation step. The route setting step of this embodiment is executed to set a first route based on the first BIM data. Furthermore, the virtual image set generation step of this embodiment is executed to generate a first virtual image set based on the first BIM data and the first route. In addition, the inference model construction step of this embodiment is executed to apply machine learning using first training data including at least the first virtual image set to a neural network to construct an inference model corresponding to the measurement date.

[0138] This aspect provides a program for causing a computer to execute each of a data set preparation step, a first BIM data construction step, a route planning step, a virtual image set generation step, and an inference model construction step. The computer is, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0139] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0140] This aspect provides an inference model constructed through a dataset preparation process, a first BIM data construction process, a route setting process, a virtual image set generation process, and an inference model construction process.

[0141] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0142] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0143] According to this aspect, as described above, it is possible to set an optimal measurement route for the measurement work on the measurement date based on the BIM data corresponding to the arrangement of building components on the measurement date, generate an optimal virtual image set along this optimal route, and construct an optimized inference model for the measurement date by performing machine learning using this optimal virtual image set. Therefore, it is possible to optimally perform the measurement work on the measurement date and process the measurement data obtained on the measurement date.

[0144] <Third aspect> In the second aspect, the BIM data (first BIM data) constructed by the BIM data construction unit 154 is used by both the route setting unit 151 and the virtual image set generation unit 152. In contrast, in the third aspect described below, the BIM data (called second BIM data in this aspect) constructed by the BIM data construction unit 154 is used only by the virtual image set generation unit 152.

[0145] That is, in this embodiment, the virtual image set is generated from BIM data corresponding to the layout of the building components on the measurement date, but the route is set regardless of the measurement date. For example, the route in this embodiment may be a route set based on the BIM data in the data set 141 accepted by the data set accepting unit 13, as in the first embodiment. Alternatively, the route in this embodiment may be a route set for a certain measurement date.

[0146] An example of the configuration of the information processing device according to this embodiment is shown in Fig. 5. The information processing device 5 is configured to add a BIM data construction unit 154 to the processing unit 15 of the information processing device 1 of the first embodiment, and to provide the output of the BIM data construction unit 154 to a virtual image set generation unit 152.

[0147] As in the second embodiment, the dataset 141 prepared in this embodiment includes, in addition to the BIM data similar to that of the first embodiment, construction date information (component construction date information) indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building.

[0148] The BIM data construction unit 154 has the same function and configuration as the second embodiment, and is configured to construct BIM data (second BIM data) corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed, based on the data set 141 including at least the BIM data, component construction date information, and measurement date information. The second BIM data constructed in this manner is BIM data corresponding to the arrangement of building components on the measurement date. The second BIM data constructed by the BIM data construction unit 154 is sent to the virtual image set generation unit 152.

[0149] As described above, the route setting unit 151 of this embodiment sets a route based on the BIM data in the data set 141, for example, in the same manner as in the first embodiment. Alternatively, a route already set for a certain measurement date may be used.

[0150] The virtual image set generation unit 152 of this embodiment generates a virtual image set of a virtual building along this route based on the second BIM data constructed by the BIM data construction unit 154 and the route set by the route setting unit 151. The virtual image set generated based on the second BIM data is called the second virtual image set.

[0151] The inference model construction unit 153 in this embodiment applies machine learning using training data (second training data) that includes at least such a second virtual image set to a neural network to construct an inference model corresponding to the measurement date.

[0152] Thus, the information processing device 5 according to this embodiment includes a BIM data construction unit 154 in addition to the elements of the first embodiment. Moreover, the dataset received by the dataset receiving unit 13 further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building. The BIM data construction unit 154 is configured to construct second BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed based on this dataset. Moreover, the virtual image set generation unit 152 of this embodiment is configured to generate a second virtual image set based on the second BIM data and a route set from other BIM data. Furthermore, the inference model construction unit 153 of this embodiment is configured to apply machine learning using second training data including at least the second virtual image set to a neural network to construct an inference model corresponding to the measurement date.

[0153] This aspect also provides a method for constructing an inference model that can be used for reality capture of a building, etc. In this inference model construction method, the dataset prepared in the dataset preparation step further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building. Furthermore, the inference model construction step is executed to apply machine learning corresponding to the construction date and the measurement date to a neural network to construct an inference model.

[0154] More specifically, the inference model construction method of this embodiment further includes a second BIM data construction step. The second BIM data construction step is executed to construct second BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the data set prepared in the data set preparation step. The virtual image set generation step of this embodiment is executed to generate a second virtual image set based on the second BIM data and a route set based on the other BIM data. Furthermore, the inference model construction step of this embodiment is executed to apply machine learning using second training data including at least the second virtual image set to a neural network to construct an inference model corresponding to the measurement date.

[0155] This aspect provides a program for causing a computer to execute each of a data set preparation step, a second BIM data construction step, a route planning step, a virtual image set generation step, and an inference model construction step. The computer is, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0156] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0157] This aspect provides an inference model constructed through a dataset preparation process, a second BIM data construction process, a route setting process, a virtual image set generation process, and an inference model construction process.

[0158] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0159] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0160] According to this embodiment, the measurement route is not optimized as in the second embodiment, but machine learning can be performed using a virtual image set based on BIM data corresponding to the arrangement of building components on the measurement date, making it possible to construct an inference model that is suitable for that measurement date.

[0161] In the second mode, the BIM data corresponding to the measurement date is used for both route setting and virtual image set generation, and in the third mode, the BIM data corresponding to the measurement date is used only for the virtual image set. In contrast to these, it is also possible to adopt a configuration in which the BIM data corresponding to the measurement date is used only for route setting. Examples of these operations are shown in Figure 6.

[0162] First, a data set including at least BIM data, construction date information, and measurement date information is prepared (S11). The data set is prepared automatically and / or manually, for example, using a computer such as a cloud computer.

[0163] Next, the information processing device 4 or 5 receives the data set prepared in step S11 through the data set receiving unit 13 (S12). The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0164] Next, the BIM data construction unit 154 constructs BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date have been constructed based on the data set 141 received in step S12 (S13). The BIM data constructed in step S13 is called measurement date corresponding BIM data. The measurement date corresponding BIM data is stored in the memory unit 14 by the control unit 11 (data set 141).

[0165] Next, the route setting unit 151 sets a route inside and / or outside the virtual building based on, for example, the BIM data accepted in step S12 or the measurement date-compatible BIM data constructed in step S13 (S14). Information representing the set route is stored in the storage unit 14 by the control unit 11 (route information 142).

[0166] Next, the virtual image set generator 152 generates a virtual image set based on the BIM data received in step S12 or the measurement date-related BIM data constructed in step S13 and the route set in step S14 (S15). The generated virtual image set is stored in the memory unit 14 (training data 143) by the control unit 11 (together with other data).

[0167] Next, the inference model construction unit 153 applies machine learning using the training data 143 including at least the virtual image set generated in step S15 to the neural network to construct an inference model (S16). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0168] <Fourth aspect> An example of the configuration of an information processing device according to the fourth aspect is shown in Fig. 7. The information processing device 7 is obtained by adding a first information synthesis unit 155 to the processing unit 15 of the information processing device 1 of the first aspect. The first information synthesis unit 155 is realized, for example, by cooperation between hardware including a processor and a storage device and first information synthesis software. Explanations of elements common to the first aspect will be omitted unless otherwise noted.

[0169] In this embodiment, the texture of the building component is referenced. The texture is a parameter that represents the surface state of an object, and may be, for example, a parameter that represents the uniformity of the surface color, brightness, and minute irregularities. The texture in this embodiment represents the surface state of the building component according to the raw materials, materials, raw materials, quality, surface finish, etc. The texture may be specified by the type of raw materials, materials, raw materials, quality, surface finish, etc.

[0170] The data set prepared in this embodiment includes texture information representing the surface condition of the components (virtual building components) that constitute the virtual building in addition to the BIM data similar to that in the first embodiment. The data set receiving unit 13 receives a data set including at least the BIM data and the texture information. The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0171] The first information synthesis unit 155 is configured to synthesize the texture information in the data set 141 with the virtual image set generated by the virtual image set generation unit 152 based on the data set 141 and the route information 142. The information generated in this manner is called first synthesis information. The generated first synthesis information is stored in the storage unit 14 by the control unit 11 (training data 143).

[0172] For example, the first information synthesis unit 155 first collates the position (three-dimensional coordinates) of the virtual member in the BIM data with the three-dimensional coordinates in the virtual image generated from the BIM data to identify the position in the virtual image corresponding to the virtual member in the BIM data. In other words, the first information synthesis unit 155 associates the virtual member in the BIM data with the virtual member in the virtual image. Furthermore, the first information synthesis unit 155 assigns texture information of the virtual member in the BIM data to the virtual member in the virtual image corresponding to this virtual member. Note that, when one virtual member in the BIM data is depicted in two or more virtual images, the texture information of the virtual image can be added to the corresponding virtual image in each of these virtual images. In this way, the virtual image set and the texture information are synthesized to generate the first synthesis information.

[0173] The inference model construction unit 153 of this embodiment is configured to apply machine learning to a neural network using training data 143 including at least the first synthesis information generated from the virtual image set and texture information, thereby constructing an inference model for identifying data of building components from measurement data of a building.

[0174] The operation of the information processing device 7 according to this embodiment will be described below. An example of the operation of the information processing device 7 is shown in FIG.

[0175] First, a dataset including at least BIM data and texture information is prepared (S21). The dataset is prepared automatically and / or manually, for example, using a computer such as a cloud computer.

[0176] Next, the information processing device 7 receives the data set prepared in step S21 through the data set receiving unit 13 (S22). The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0177] Next, the route setting unit 151 sets a route inside and / or outside the virtual building represented by the BIM data in the data set 141 based on the data set 141 received in step S22 (S23). Information representing the set route is stored in the memory unit 14 by the control unit 11 (route information 142).

[0178] Next, the virtual image set generator 152 generates a virtual image set of the virtual building represented by the BIM data in the data set 141 along the route set in step S23 based on the data set 141 and the route information 142 (S24). The generated virtual image set is sent to the first information synthesis unit 155.

[0179] Next, the first information synthesis unit 155 synthesizes the virtual image set generated in step S24 with the texture information in the data set 141 received in step S22 to generate first synthesis information (S25). The generated first synthesis information is stored (together with other data) in the memory unit 14 by the control unit 11 (training data 143).

[0180] Next, the inference model construction unit 153 applies machine learning using the training data 143 including at least the first synthesis information (synthetic information of the virtual image set and the texture information) generated in step S25 to the neural network to construct an inference model for identifying data on building components from the measurement data of the building (S26). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0181] Thus, the information processing device 7 according to this embodiment includes a first information synthesis unit 155 in addition to the elements of the first embodiment. Furthermore, the data set 141 includes texture information representing the surface state of the virtual building components in addition to the BIM data. The first information synthesis unit 155 is configured to generate first synthesis information by synthesizing the texture information included in the data set 141 with the virtual image set generated by the virtual image set generation unit 152. Furthermore, the inference model construction unit 153 is configured to construct an inference model by applying machine learning using the training data 143 including at least the first synthesis information generated by the first information synthesis unit 155 to a neural network.

[0182] This aspect also provides a method for constructing an inference model that can be used for reality capture of buildings, etc. In this inference model construction method, the dataset preparation step is executed to prepare a dataset including BIM data and texture information representing the surface state of virtual building components. The inference model construction method of this aspect further includes a step of synthesizing the texture information with the virtual image set to generate first synthesized information (first synthesis step). The inference model construction step is executed to construct an inference model by applying machine learning using training data including at least the first synthesized information generated in the first synthesis step to a neural network.

[0183] This aspect provides a program for causing a computer to execute each of the steps of preparing a data set, setting a route, generating a virtual image set, a first synthesis, and constructing an inference model, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0184] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0185] This aspect provides an inference model constructed through a dataset preparation process, a route setting process, a virtual image set generation process, a first synthesis process, and an inference model construction process.

[0186] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0187] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0188] According to this aspect, machine learning can be performed using texture information of building components. The inference model constructed in this way is configured to extract features of the surface state (texture) of the object represented in the measurement data of the building, and to identify the data of the building components by taking into account the extracted texture features. By using such an inference model, it is possible to improve the accuracy and precision of the building component data identification process compared to when texture is not taken into account.

[0189] <Fifth aspect> An example of the configuration of an information processing device according to the fifth aspect is shown in Fig. 9. The information processing device 9 is obtained by adding a second information synthesis unit 156 to the processing unit 15 of the information processing device 1 of the first aspect. The second information synthesis unit 156 is realized, for example, by cooperation between hardware including a processor and a storage device and second information synthesis software. Explanations of elements common to the first aspect will be omitted unless otherwise noted.

[0190] This aspect takes into consideration objects (obstacles) that may be impeding the measurement of a building. In other words, this aspect refers to information about obstacles that may be present at the site when measuring a building. Examples of obstacles include tools such as ladders and stepladders, workers, and measurement devices (mobiles, total stations, computers).

[0191] The data set prepared in this embodiment includes obstacle information representing obstacles in addition to the BIM data similar to that in the first embodiment. The data set receiving unit 13 receives a data set including at least the BIM data and the obstacle information. The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0192] The obstacle information includes, for example, photographed images of obstacles and scan data of obstacles. Specifically, the obstacle information may be photographed images and / or scan data of tools such as ladders and stepladders, or photographed images and / or scan data of workers at a construction site.

[0193] The second information synthesis unit 156 is configured to synthesize the obstacle information in the data set 141 with the virtual image set generated by the virtual image set generation unit 152 based on the data set 141 and the route information 142. The information generated in this way is called second synthesis information. The generated second synthesis information is stored in the storage unit 14 by the control unit 11 (training data 143).

[0194] For example, the second information synthesis unit 155 generates the second synthesis information by embedding obstacle information (such as a photographed image of an obstacle) in a virtual image. This results in a set of virtual images in which the obstacle information is embedded at various positions (second synthesis information). Note that the embedding positions of the obstacle information are determined based on, for example, photographed images of an actual construction site.

[0195] The inference model construction unit 153 of this embodiment is configured to apply machine learning to a neural network using training data 143 including at least the second composite information generated from the virtual image set and the obstacle information, thereby constructing an inference model for identifying data of building components from measurement data of a building.

[0196] The operation of the information processing device 9 according to this embodiment will be described below. An example of the operation of the information processing device 9 is shown in FIG.

[0197] First, a data set including at least BIM data and obstacle information is prepared (S31). The data set is prepared automatically and / or manually, for example, using a computer such as a cloud computer.

[0198] Next, the information processing device 9 receives the data set prepared in step S31 through the data set receiving unit 13 (S32). The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0199] Next, the route setting unit 151 sets a route inside and / or outside the virtual building represented by the BIM data in the data set 141 based on the data set 141 received in step S32 (S33). Information representing the set route is stored in the memory unit 14 by the control unit 11 (route information 142).

[0200] Next, the virtual image set generator 152 generates a virtual image set of the virtual building represented by the BIM data in the data set 141 along the route set in step S33 based on the data set 141 and the route information 142 (S34). The generated virtual image set is sent to the second information synthesis unit 156.

[0201] Next, the second information synthesis unit 156 synthesizes the virtual image set generated in step S34 with the obstacle information in the data set 141 received in step S32 to generate second synthesis information (S35). The generated second synthesis information is stored (together with other data) in the memory unit 14 by the control unit 11 (training data 143).

[0202] Next, the inference model construction unit 153 applies machine learning using the training data 143 including at least the second synthesis information (synthetic information of the virtual image set and the obstacle information) generated in step S35 to the neural network to construct an inference model for identifying data on building components from the measurement data of the building (S36). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0203] Thus, the information processing device 9 according to this embodiment includes the second information synthesis unit 156 in addition to the elements of the first embodiment. Furthermore, the data set 141 includes obstacle information in addition to the BIM data. The second information synthesis unit 156 is configured to generate second synthesis information by synthesizing the obstacle information included in the data set 141 with the virtual image set generated by the virtual image set generation unit 152. Furthermore, the inference model construction unit 153 is configured to construct an inference model by applying machine learning using the training data 143 including at least the second synthesis information generated by the second information synthesis unit 156 to the neural network.

[0204] This aspect also provides a method for constructing an inference model that can be used for reality capture of buildings, etc. In this inference model construction method, the dataset preparation step is executed to prepare a dataset including BIM data and obstacle information. The inference model construction method of this aspect further includes a step of synthesizing the obstacle information with a virtual image set to generate second synthesized information (second synthesis step). The inference model construction step is executed to construct an inference model by applying machine learning using training data including at least the second synthesized information generated in the second synthesis step to a neural network.

[0205] This aspect provides a program for causing a computer to execute each of the steps of preparing a data set, setting a route, generating a virtual image set, a second synthesis, and constructing an inference model, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0206] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0207] This aspect provides an inference model constructed through a dataset preparation process, a route setting process, a virtual image set generation process, a second synthesis process, and an inference model construction process.

[0208] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0209] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0210] According to this aspect, machine learning can be performed using obstacle information. The inference model constructed in this way is configured to extract features of obstacles mixed in the measurement data of the building and to identify the data of the building components by taking into account the extracted obstacle features. By using such an inference model, it is possible to improve the accuracy and precision of the building component data identification process compared to when obstacles are not taken into account.

[0211] For example, according to this embodiment, by using a virtual image set in which photographed images of objects other than building components and photographed images of people are combined as training data, when a data area corresponding to an obstacle is mixed in measurement data of a building (photographed images, scan data, point cloud data, etc.), it is possible to train an inference model to detect this obstacle data area. This makes it possible to control a moving object to avoid the obstacle, and to exclude the obstacle data when creating BIM data from the measurement data.

[0212] <Sixth aspect> An example of the configuration of an information processing device according to the sixth aspect is shown in Fig. 11. The information processing device 1100 is obtained by adding a label assignment unit 157 to the processing unit 15 of the information processing device 1 of the first aspect. The label assignment unit 157 is realized, for example, by cooperation between hardware including a processor and a storage device and label assignment software. Explanation of elements common to the first aspect will be omitted unless otherwise noted.

[0213] This aspect is adopted when supervised learning is used in constructing an inference model. In supervised learning, pairs of data and labels (correct labels) are prepared, and parameters are adjusted by comparing the output of a neural network corresponding to input data with the labels. This aspect has a function of assigning labels to data. In other words, this aspect creates training data for supervised learning.

[0214] The label assignment unit 157 is configured to assign labels related to the virtual building components recorded in the BIM data in the dataset 141 to the virtual image set generated by the virtual image set generation unit 152.

[0215] An example of a label is the type of a building component, which may be, for example, a pillar, a beam, a wall, a slab, a roof, a foundation, a window, a door, a staircase, a tile, flooring, various parts, various machines, various devices, various facilities, etc. The label may also be information representing the texture of the building component.

[0216] Another example of the label type is the type of obstacle, which may be, for example, a tool such as a ladder or a stepladder, a worker, a measuring device (a mobile object, a total station, a computer), and the like.

[0217] The labeling (annotation) of this embodiment may be either manual or automatic, and manual and automatic annotation may be able to cooperate and / or switch between them.

[0218] In the case of manual annotation, the label assignment unit 157, for example, cooperates with the control unit 11 to cause the user interface 12 to display a GUI for annotation work. This annotation GUI displays, for example, a virtual image and a software key. The label assignment unit 157 is configured to assign a label input using the software key to the corresponding virtual image.

[0219] For example, the user observes the displayed virtual image and specifies an image area corresponding to a building component. Furthermore, the user specifies the type of this building component. These specification operations are performed using the user interface 12. The labeling unit 157 assigns the specified type of building component to the specified image area. As a result, a label representing the type of building component is assigned to the virtual image. Note that multiple labels may be assigned to a single virtual image. For example, labels representing the types of building components can be assigned to different image areas in a single virtual image. Also, a label representing the type of building component and a label representing the texture can be assigned to a single image area in a single virtual image.

[0220] In the case of automatic annotation, the labeling unit 157 may be configured to, for example, assign labels to the virtual images through semi-supervised learning. The training data used in the semi-supervised learning is only partially labeled. The labeling unit 157 includes, for example, a classifier constructed by machine learning using labeled data, and is configured to use the classifier to classify (estimate) unlabeled data and assign the result as a label to the unlabeled data. An inference model can be constructed by performing supervised learning using the data thus labeled.

[0221] Here, the semi-supervised learning method may be any method. For example, the following methods may be used: bootstrap method (self-training, co-training, semi-supervised support vector machine, active learning, etc.); graph-based semi-supervised learning (semi-supervised k-nearest neighbor graph, semi-supervised mixture distribution model, semi-supervised support vector machine, active learning, etc.); generative model (conditional GAN, conditional cycle GAN, semi-supervised mixture distribution model, variational autoencoding method, ladder network, adversarial autoencoder, etc.).

[0222] Furthermore, the automatic annotation is not limited to a method using semi-supervised learning, and any known automatic annotation method can be adopted.

[0223] The control unit 11 stores the virtual image set generated by the virtual image set generation unit 152 and the label assigned to the virtual image set by the label assignment unit 157 in the storage unit 14 (training data 143). In other words, the training data 143 of this embodiment includes at least the virtual image set generated by the virtual image set generation unit 152 and the label assigned to the virtual image set by the label assignment unit 157.

[0224] The inference model construction unit 153 of this embodiment is configured to apply supervised learning using such training data 143 to a neural network, thereby constructing an inference model for identifying data on building components from measurement data on a building.

[0225] In addition, in a modified example of this embodiment, the inference model construction unit 153 may be configured to construct an inference model by applying semi-supervised learning using the training data 143 to a neural network.

[0226] The following describes the operation of the information processing device 1100 according to this embodiment. An example of the operation of the information processing device 1100 is shown in FIG.

[0227] First, a dataset including at least BIM data is prepared (S41). The preparation of the dataset may be performed automatically and / or manually, for example, using a computer such as a cloud computer.

[0228] Next, the information processing device 1100 receives the data set prepared in step S41 through the data set receiving unit 13 (S42). The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0229] Next, the route setting unit 151 sets a route inside and / or outside the virtual building represented by the BIM data in the data set 141 based on the data set 141 received in step S42 (S43). Information representing the set route is stored in the memory unit 14 by the control unit 11 (route information 142).

[0230] Next, the virtual image set generation unit 152 generates a virtual image set of the virtual building represented by the BIM data in the data set 141 along the route set in step S43 based on the data set 141 and the route information 142 (S44). The generated virtual image set is sent to the label assignment unit 157.

[0231] Next, the labeling unit 157 assigns labels related to the virtual building components recorded in the BIM data in the data set 141 to the virtual image set generated in step S44 (S45). The labeled virtual image set is stored (together with other data) by the control unit 11 in the memory unit 14 (training data 143).

[0232] Next, the inference model construction unit 153 applies supervised learning using the training data 143 including at least the labeled virtual image set generated in step S45 to the neural network to construct an inference model for identifying data on building components from the measurement data of the building (S46). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0233] Thus, the information processing device 1100 according to this embodiment includes, in addition to the elements of the first embodiment, a labeling unit 157. The labeling unit 157 is configured to assign labels related to virtual building components recorded in the BIM data to the virtual image set. Furthermore, the inference model construction unit 153 is configured to construct an inference model by applying supervised learning using training data 143 including at least the virtual image set and the labels to the neural network.

[0234] This aspect also provides a method for constructing an inference model that can be used for reality capture of buildings, etc. This inference model construction method further includes a step of assigning labels related to virtual building members recorded in the BIM data to the virtual image set (labeling step). The inference model construction step is executed to construct an inference model by applying supervised learning using training data including at least the virtual image set and the labels to a neural network.

[0235] This aspect provides a program for causing a computer to execute each of the steps of preparing a data set, planning a route, generating a virtual image set, labeling, and building an inference model, for example, in a single computer or a system including two or more computers that can be used for reality capture of a building.

[0236] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0237] This aspect provides an inference model constructed through a dataset preparation process, a route setting process, a virtual image set generation process, a labeling process, and an inference model construction process.

[0238] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0239] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0240] According to this aspect, labels for virtual building components recorded in the BIM data can be manually and / or automatically assigned to a set of virtual images, making it possible to perform supervised learning (or semi-supervised learning) for constructing an inference model.

[0241] <Seventh aspect> An example of the configuration of an information processing device according to the seventh aspect is shown in Fig. 13. The information processing device 1300 is obtained by adding a transmission unit 16 to the information processing device 1 of the first aspect. In Fig. 13, the arrow connecting the control unit 11 and the user interface 12 is omitted, and the connection form between them is the same as in the first aspect. Explanation of elements common to the first aspect will be omitted unless otherwise specified.

[0242] The transmission unit 16 is configured to transmit the inference model constructed by the inference model construction unit 153 to a predetermined mobile object. The transmission unit 16 may include at least one of a wired communication device and a wireless communication device. The mobile object is configured to perform measurements while autonomously moving inside and / or outside a building. The mobile object is, for example, a UAV or a self-driving car. The mobile object may also be a person, and the inference model can be transmitted to a mobile terminal.

[0243] The processing unit 15 may be configured to create information for controlling the moving body (movement control information) based on the inference model constructed by the inference model construction unit 153. The created movement control information is provided to the moving body by the transmission unit 16. The movement control information may be created by applying rule-based processing to the information stored in the storage unit 14.

[0244] The following describes the operation of the information processing device 1300 according to this embodiment. An example of the operation of the information processing device 1300 is shown in FIG.

[0245] First, a dataset including at least BIM data is prepared (S51). The preparation of the dataset may be performed automatically and / or manually, for example, using a computer such as a cloud computer.

[0246] Next, the information processing device 1300 receives the data set prepared in step S51 through the data set receiving unit 13 (S52). The received data set is stored in the storage unit 14 by the control unit 11 (data set 141).

[0247] Next, the route setting unit 151 sets a route inside and / or outside the virtual building represented by the BIM data in the data set 141 based on the data set 141 received in step S52 (S53). Information representing the set route is stored in the memory unit 14 by the control unit 11 (route information 142).

[0248] Next, the virtual image set generator 152 generates a virtual image set of the virtual building represented by the BIM data in the dataset 141 along the route set in step S53 based on the dataset 141 and the route information 142 (S54). The generated virtual image set is stored (together with other data) in the memory unit 14 by the control unit 11 (training data 143).

[0249] Next, the inference model construction unit 153 applies machine learning using the training data 143 including at least the virtual image set generated in step S54 to a predetermined neural network to construct an inference model for identifying data on building components from the measurement data of the building (S55). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144).

[0250] Next, the control unit 11 reads out the inference model 144 from the storage unit 14 and sends it to the transmission unit 16. The processing unit 15 can also create the above-mentioned movement control information. The created movement control information is sent to the transmission unit 16. The transmission unit 16 transmits the inference model 144 and / or the movement control information to a predetermined moving body (S56). The moving body receives the inference model 144 and / or the movement control information, and performs measurements while autonomously moving inside and / or outside the building (end).

[0251] Thus, the information processing device 1300 according to this embodiment includes, in addition to the elements of the first embodiment, a transmission unit 16. The transmission unit 16 is configured to transmit the inference model 144 constructed by the inference model construction unit 153 to a mobile object configured to perform measurements while autonomously moving inside and / or outside a building.

[0252] This aspect also provides a method for constructing an inference model that can be used for reality capture of a building, etc. This inference model construction method further includes a step of transmitting the inference model to a mobile object configured to perform measurements while autonomously moving inside and / or outside the building (transmission step).

[0253] This aspect provides a program for causing a computer to execute each of the steps of preparing a data set, setting a route, generating a virtual image set, constructing an inference model, and transmitting. The computer may be a single computer or a system including two or more computers that can be used for reality capture of a building, for example.

[0254] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0255] This aspect provides an inference model obtained through a dataset preparation process, a route setting process, a virtual image set generation process, an inference model construction process, and a transmission process.

[0256] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0257] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0258] According to this aspect, the constructed inference model can be easily provided to the moving body, thereby smoothing the transition from constructing the inference model to measurements by the moving body.

[0259] <Eighth aspect> A configuration example of a system according to the eighth aspect is shown in Fig. 15. An information processing device 1500 is obtained by removing the path setting unit 151 from the processing unit 15 of the information processing device 1 of the first aspect. Explanations of elements common to the first aspect will be omitted unless otherwise noted.

[0260] In this embodiment, a data set including at least BIM data and route information is prepared. The route information is information representing a route inside and / or outside the virtual building represented by the BIM data. The route setting method may be the same as that of the first embodiment, but the route is set using a device or system other than the information processing device 1500.

[0261] The virtual image set generation unit 152 in this embodiment is configured to generate a virtual image set of the interior and / or exterior of the virtual building represented by the BIM data along the route represented by the route information, based on the dataset accepted by the dataset accepting unit 13.

[0262] The inference model construction unit 153 in this embodiment is configured to apply machine learning using training data including at least the virtual image set generated by the virtual image set generation unit 152 to a neural network to construct an inference model for identifying data on building components from measurement data of a building.

[0263] The following describes the operation of the information processing device 1500 according to this embodiment. An example of the operation of the information processing device 1500 is shown in FIG.

[0264] First, a data set including at least BIM data and route information is prepared (S61). The data set is prepared automatically and / or manually, for example, using a computer such as a cloud computer.

[0265] Next, the information processing device 1500 receives the dataset prepared in step S61 through the dataset receiving unit 13 (S62). The received dataset is stored in the storage unit 14 by the control unit 11 (dataset 141). Note that, in the example shown in FIG. 15, route information is also included in the dataset 141, but the route information included in the dataset received by the dataset receiving unit 13 may be stored in the storage unit 14 as route information 142 similar to the first aspect.

[0266] Next, the virtual image set generator 152 generates a virtual image set of the virtual building represented by the BIM data along the route represented by the route information based on the data set 141 received in step S62 (S63). The generated virtual image set is stored (together with other data) in the memory unit 14 by the control unit 11 (training data 143).

[0267] Next, the inference model construction unit 153 applies machine learning using the training data 143 including at least the virtual image set generated in step S63 to a predetermined neural network to construct an inference model for identifying data on building components from the measurement data of the building (S64). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0268] Thus, the information processing device 1500 according to this embodiment has a configuration in which the route setting unit 151 is excluded from the elements of the first embodiment. The data set receiving unit 13 is configured to receive a data set including at least BIM data and route information. The virtual image set generating unit 152 is configured to generate a virtual image set of the inside and / or outside of the virtual building represented by the BIM data along the route represented by the route information based on the data set received by the data set receiving unit 13. The inference model construction unit 153 is configured to apply machine learning using training data including at least the virtual image set generated by the virtual image set generating unit 152 to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of the building.

[0269] This aspect also provides a method for constructing an inference model that can be used for reality capture of buildings. This inference model construction method includes at least the following steps: preparing a data set including at least BIM data and route information (data set preparation step); generating a virtual image set of a virtual building represented by the BIM data along a route represented by the route information based on the prepared data set (virtual image set generation step); and applying machine learning using training data including at least the generated virtual image set to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of the building (inference model construction step). That is, the inference model construction method of this aspect is configured by excluding the route setting step from the method of the first aspect and preparing route information in the data set preparation step.

[0270] This aspect provides a program for causing a computer to execute each of the steps of preparing a data set, generating a virtual image set, and constructing an inference model, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0271] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0272] This aspect provides an inference model constructed through a dataset preparation process, a virtual image set generation process, and an inference model construction process.

[0273] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0274] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0275] According to this aspect, it is possible to improve the efficiency of building management from various viewpoints, as in the first aspect. Furthermore, since this aspect is configured to use route information created in advance, it has the advantage of being less processing load and requiring less processing resources than the first aspect.

[0276] <Ninth aspect> An example of the configuration of a system according to the ninth aspect is shown in Fig. 17. The information processing device 1700 is obtained by excluding the path setting unit 151 and the virtual image set generation unit 152 from the processing unit 15 of the information processing device 1 of the first aspect, and by providing a virtual image set reception unit 17 instead of the data set reception unit 13. In other words, the information processing device 1700 is obtained by excluding the virtual image set generation unit 152 from the processing unit 15 of the information processing device 1500 of the eighth aspect, and by providing a virtual image set reception unit 17 instead of the data set reception unit 13. Unless otherwise noted, a description of elements common to the first aspect will be omitted.

[0277] In this embodiment, a virtual image set is prepared along a predetermined route inside and / or outside the virtual building represented by the BIM data. The route setting method and the virtual image set generation method may be similar to those of the first embodiment, but the route setting and the virtual image set generation are performed using a device or system other than the information processing device 1500.

[0278] The virtual image set receiving unit 17 has a configuration similar to that of the data set receiving unit 13 of the first embodiment, and has a function of receiving data (virtual image set) to be processed by the information processing device 1700. The virtual image set receiving unit 17 may include, for example, a communication device for performing data communication with the outside (such as a device, a system, or a database). The virtual image set receiving unit 17 may also include a drive device for reading out data recorded on a recording medium.

[0279] The inference model construction unit 153 in this embodiment is configured to apply machine learning using training data including at least the virtual image set accepted by the virtual image set accepting unit 17 to a neural network to construct an inference model for identifying data on building components from measurement data of a building.

[0280] The following describes the operation of the information processing device 1700 according to this embodiment. An example of the operation of the information processing device 1700 is shown in FIG.

[0281] First, a route inside and / or outside the virtual building is set based on the BIM data, and a virtual image set of the virtual building along the route is prepared (S71). The preparation of the virtual image set is performed automatically and / or manually, for example, by using a computer such as a cloud computer.

[0282] Next, the information processing device 1700 receives the virtual image set prepared in step S71 through the virtual image set receiving unit 17 (S72). The received data set is stored in the storage unit 14 by the control unit 11 (training data 143).

[0283] Next, the inference model construction unit 153 applies machine learning using the training data 143 including at least the virtual image set received in step S72 to a predetermined neural network to construct an inference model for identifying data on building components from measurement data on the building (S73). The constructed inference model is stored in the memory unit 14 by the control unit 11 (inference model 144) (end).

[0284] In this manner, the information processing device 1700 according to this embodiment has a configuration in which the route setting unit 151 and the virtual image set generation unit 152 are excluded from the elements of the first embodiment, and a virtual image set reception unit 17 is provided instead of the data set reception unit 13. The virtual image set reception unit 17 is configured to receive a virtual image set along a predetermined route inside and / or outside the virtual building represented by the BIM data. The inference model construction unit 153 is configured to apply machine learning using training data including at least the virtual image set received by the virtual image set reception unit 17 to a predetermined neural network, and to construct an inference model for identifying data of building components from measurement data of the building.

[0285] This aspect also provides a method for constructing an inference model that can be used for reality capture of a building, etc. This inference model construction method includes at least the following steps: a step of preparing a virtual image set along a predetermined route of the interior and / or exterior of a virtual building represented by BIM data (virtual image set preparation step); a step of applying machine learning using training data including at least the prepared virtual image set to a predetermined neural network to construct an inference model for identifying data of building components from measurement data of the building (inference model construction step). That is, the inference model construction method of this aspect is obtained by excluding the route setting step and the virtual image set generation step from the method of the first aspect, and by providing a virtual image set preparation step instead of the data set preparation step.

[0286] This aspect provides a program for causing a computer to execute each of the steps of preparing a virtual image set and constructing an inference model, the computer being, for example, a single computer usable for reality capture of a building, or a system including two or more computers.

[0287] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0288] This aspect provides an inference model constructed through a virtual image set preparation process and an inference model construction process.

[0289] This aspect also provides a program that causes a computer to execute a process of identifying data on building components from measurement data of a building using such an inference model.

[0290] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0291] According to this aspect, like the first aspect, it is possible to improve the efficiency of construction management from various viewpoints. Furthermore, since this aspect is configured to use a virtual image set generated in advance, it has the advantage of lower processing load and smaller processing resources than the first aspect. The same is true in comparison with the eighth aspect.

[0292] <Tenth aspect> The exemplary aspects (first to ninth aspects) described above relate to the construction of an inference model, whereas the tenth aspect relates to the use of an inference model.

[0293] The inference model used in this embodiment may be, for example, an inference model relating to any of the first to ninth embodiments, an inference model equivalent to an inference model relating to any of the first to ninth embodiments, or an inference model constructed based on an inference model relating to any of the first to ninth embodiments.

[0294] Examples of inference models equivalent to an inference model relating to any of the first to ninth aspects include an inference model that falls within the equivalent scope of an inference model relating to any of the first to ninth aspects, an inference model obtained by substantially the same construction method as an inference model relating to any of the first to ninth aspects, an inference model that has substantially the same configuration as an inference model relating to any of the first to ninth aspects, and an inference model that has substantially the same function as an inference model relating to any of the first to ninth aspects.

[0295] Further, examples of inference models constructed based on the inference model according to any one of the first to ninth aspects include an inference model obtained by applying further machine learning to the inference model according to any one of the first to ninth aspects, an inference model obtained by applying transfer learning to the inference model according to any one of the first to ninth aspects, and an inference model including at least a portion of the inference model according to any one of the first to ninth aspects.

[0296] As described below, the present embodiment is configured to identify data of building components from data (measurement data) obtained by measuring a building, but the use of the inference model is not limited to this. For example, some exemplary embodiments are configured to identify other data from measurement data of a building, or to identify predetermined data from measurement data of an object other than a building.

[0297] A configuration example of an information processing device according to this embodiment is shown in Fig. 19. The information processing device 1900 may be included in, for example, a building reality capture system. The building reality capture system has a function of measuring an actual building and acquiring digital data (measurement data such as photographed images, scan data, and point cloud data). The acquired measurement data is input to the information processing device 1900.

[0298] For example, the information processing device 1900 may be mounted on a moving body such as a UAV or a self-propelled vehicle, may be a mobile computer carried by a measurer, may be a computer (such as an edge computer) used at the measurement site (construction site), or may be a computer (such as a cloud computer) used at a location other than the measurement site.

[0299] The information processing device 1900 according to this embodiment includes at least a storage unit 24 and a processing unit 25, and further includes a control unit 21, a user interface 22, and a data receiving unit .

[0300] The control unit 21 is configured to execute various controls of the information processing device 1900. The control unit 21 is realized, for example, by cooperation between hardware including a processor and a storage device and control software. The control unit 21 is provided in one computer, or distributed across two or more computers.

[0301] The user interface 22 includes, for example, a display device, an operation device, an input device, etc. In some exemplary embodiments, the user interface 22 includes a graphical user interface (GUI) that uses a touch screen, a pointing device, computer graphics, etc. The user interface 22 may be provided on one computer or may be distributed across two or more computers.

[0302] The data receiving unit 23 has a function of receiving data to be processed by the information processing device 1900. The data receiving unit 23 may include a communication device for performing data communication with the outside (such as a device, a system, or a database). The data receiving unit 23 may also include a drive device for reading out data recorded on a recording medium. In addition, when the information processing device 1900 is incorporated in a device (such as a measuring device or an information processing device) that generates the data to be received, the data receiving unit 23 may include a bus within the device.

[0303] The data receiving unit 23 receives, for example, measurement data of a building. The data receiving unit 23 also receives an inference model. If the information processing device 1900 has an inference model construction function, the data receiving unit 23 does not need to receive an inference model from outside.

[0304] The measurement data accepted by the data accepting unit 23 is obtained, for example, by photographing the building using a camera (panoramic camera) or a video camera (panoramic video camera) mounted on a moving body such as a UAV, a self-propelled vehicle, or a worker, or by scanning the building using a laser scanner or a total station.

[0305] Furthermore, the measurement data accepted by the data accepting unit 23 may be generated by applying a predetermined process to data (raw data) acquired by photographing or scanning. Examples of the process applied to the raw data include SfM, MVS, SLAM (V-SLAM), etc.

[0306] The storage unit 24 is configured to store various data (information). For example, the storage unit 24 stores data received by the data receiving unit 23. The storage unit 24 includes a relatively large capacity storage device such as an HDD or SSD. In this embodiment, an inference model 241 and measurement data 242 are stored in the storage device 24.

[0307] The inference model 241 is a trained model constructed by applying machine learning using training data including at least a set of virtual images along a predetermined route inside and / or outside the virtual building represented by the BIM data to a predetermined neural network. The inference model 241 is received by the data receiving unit 23 and stored in the memory unit 24 by the control unit 21.

[0308] The inference model 241 may be, for example, any of the inference models 144 of the first to ninth aspects. In addition, the inference model 241 may be an inference model constructed based on the inference model 144 of any of the first to ninth aspects, an inference model equivalent to the inference model 144, or an inference model constructed based on an inference model equivalent to the inference model 144.

[0309] The measurement data 242 is data acquired by measuring an actual building. The measurement data 242 may have additional information. The additional information may be, for example, information related to the attributes described above, or any other information.

[0310] The measurement data may be, for example, photographed images (still images, moving images (video)) obtained by photographing a building with a camera (omnidirectional camera) or a video camera (omnidirectional video camera) mounted on a moving body, or scan data obtained by scanning a building with a laser scanner or a total station. Moreover, the measurement data 242 may be data obtained by processing raw data such as photographed images and scan data.

[0311] The measurement data 242 is accepted by the data accepting unit 23 and stored in the memory unit 24 by the control unit 21. The measurement data accepted by the data accepting unit 23 may be sent to the processing unit 25, where it may be subjected to a predetermined process and then stored in the memory unit 24 as the measurement data 242. The process executed by the processing unit 25 may be considered as pre-processing of data processing using the inference model 241. Examples of this pre-processing include SfM, MVS, and SLAM (V-SLAM).

[0312] The processing unit 25 is configured to execute data processing. The processing unit 25 is realized, for example, by cooperation between hardware including a processor and a storage device and data processing software. The processing unit 25 is provided in one computer, or distributed among two or more computers. The processing unit 25 of this embodiment includes a feature extractor 251 and a specifier 252.

[0313] The feature extractor 251 is realized by using the inference model 241. The feature extractor 251 extracts features from the measurement data 242 input to the inference model 241.

[0314] When the inference model 241 includes a convolutional neural network, the feature extractor 251 includes, for example, multiple convolutional layers and multiple pooling layers (see FIG. 2).

[0315] The type of feature extracted from the measurement data 242 is determined by the configuration of the inference model 241. For example, when the inference model 241 includes a convolutional neural network, the type of feature (feature map) extracted by the feature extractor 251 is determined by the configuration of the convolutional layer (kernel (filter), activation function, etc.), etc.

[0316] The identifier 252 is realized using the inference model 241. The identifier 252 identifies data of the building components included in the measurement data 242 based on the features extracted by the feature extractor 251.

[0317] When the inference model 241 includes a convolutional neural network, the feature extractor 251 includes, for example, one or more fully connected layers and an output layer (see FIG. 2).

[0318] The type of data of a building component identified from the feature amount is determined by the configuration of the inference model 241. For example, when the inference model 241 includes a convolutional neural network, the type of data of a building component identified by the identifier 252 is determined by the configuration of the fully connected layer (weights, biases, activation functions, etc.) and the configuration of the output layer (number of nodes, etc.).

[0319] The number of nodes in the output layer is equal to the number of types of building components to be identified (pillars, beams, walls, slabs, roofs, foundations, windows, doors, stairs, tiles, flooring, various parts, various machines, various equipment, various facilities, etc.). In this case, for example, in the output layer, the probability that a certain data region in the measurement data 242 (a data region corresponding to a certain building component) is each type is calculated, and the type with the highest probability is selected. This identifies the type of building component corresponding to a certain data region in the measurement data 242. In this example, the inference model 241 is configured to perform image segmentation (e.g., semantic segmentation).

[0320] The following describes the operation of the information processing device 1900 according to this embodiment. An example of the operation of the information processing device 1900 is shown in FIG.

[0321] First, the information processing device 1900 receives a pre-constructed inference model through the data receiving unit 23 (S81). The received inference model is stored in the memory unit 24 by the control unit 21 (inference model 241).

[0322] Next, the information processing device 1900 receives the measurement data of the building acquired in advance through the data receiving unit 23 (S82). The received measurement data is stored in the storage unit 24 by the control unit 21 (measurement data 242).

[0323] Next, the control unit 21 loads the inference model 241 into the processing unit 25 (processor). This enables the processing unit 25 to operate as a feature extractor 251 and a specifier 252. The control unit 21 inputs the measurement data 242 into the processing unit 25 (inference model 241) (S83).

[0324] Next, the feature extractor 251 extracts features from the input measurement data 242 (S84).

[0325] Next, the identifier 252 identifies the data of the building components based on the feature amount extracted in step S84 (S85).

[0326] Next, the control unit 21 causes the display device of the user interface 22 to display the data of the building components identified in step S85 (S86) (END).

[0327] As described above, the information processing device 1900 according to this embodiment includes a storage unit 24 and a processing unit 25. The storage unit 24 stores an inference model 241 constructed by applying machine learning using training data including at least a set of virtual images along a predetermined route inside and / or outside the virtual building represented by the BIM data to a predetermined neural network. The processing unit 25 performs data processing using the inference model 241. More specifically, the processing unit 25 functions as a feature extractor 251 and a specifier 252 by using the inference model 241. The feature extractor 251 extracts features from the measurement data 242. The specifier 252 specifies data of the building components based on the features extracted by the feature extractor 251.

[0328] This aspect also provides an information processing method that can be used in reality capture of buildings, etc. This information processing method includes at least the following steps: a step of accepting an inference model constructed by applying machine learning using training data including at least a set of virtual images along a predetermined route inside and / or outside a virtual building represented by BIM data to a predetermined neural network (inference model accepting step); a step of extracting features from measurement data of the building by a processor that performs data processing using the accepted inference model (feature extraction step); and a step of identifying data of building components based on the features extracted in the feature extraction step by a processor that performs data processing using the accepted inference model (building component data identifying step).

[0329] This aspect provides a program for causing a computer to execute each of the inference model receiving step, the feature extraction step, and the building component data identifying step. The computer is, for example, a single computer that can be used for reality capture of a building, or a system including two or more computers.

[0330] Furthermore, the present aspect provides a computer-readable non-transitory recording medium having such a program recorded thereon. This non-transitory recording medium may be, for example, any one of a magnetic disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

[0331] According to this aspect, data on building components can be automatically identified from measurement data of a building (photographed images, scanned data, point cloud data, etc.), thereby making it possible to improve the efficiency of building management.

[0332] <Eleventh aspect> In the eleventh aspect, a system that can be realized by applying any one of the first to tenth aspects and its usage form will be described. This system is used for information management in the construction field (construction management, maintenance management, repair management, etc.), and is an integrated system that utilizes various technologies such as mobile objects (UAVs, self-driving cars, people, etc.), surveying equipment (total stations, laser range finders, theodolites, range finders, etc.), data processing technologies (SfM, MVS, SLAM, etc.), and modeling technologies (computer graphics, CAD, BIM, etc.).

[0333] A configuration example of the system according to this embodiment is shown in Fig. 21. The system 2100 includes a UAV 2110, a UAV controller 2120, a total station 2130, and an edge computer 2140. The cloud computer 2150 may be included in the system 2100, or may be an external device capable of data communication with the system 2100. Any of the UAV 2110, the UAV controller 2120, the total station 2130, and the edge computer 2140 may be an external device of the system 2100.

[0334] In addition, the techniques of Patent Documents 1 to 12 can be combined with this embodiment, and the techniques described in JP-A-2018-138922, JP-A-2018-138923, and the like can be used.

[0335] The UAV2110 is a small unmanned flying object that flies inside and / or outside a (physical) building to collect data on the building. The UAV2110 includes a control unit 2111 that performs various controls, a photographing unit 2112 for collecting data on the building, a V-SLAM unit 2113 that performs V-SLAM processing, and an obstacle detection unit 2114 that performs obstacle detection processing.

[0336] Although not shown in the figure, the UAV2110 is equipped with elements for flying, such as multiple propellers and a propeller motor for rotating each propeller, as in a general UAV. Although not shown in the figure, the UAV2110 may be equipped with any means that can be mounted on a standard UAV, such as an inertial measurement unit (IMU) or a device for position measurement, navigation, and timing using a global navigation satellite system (GNSS). Although not shown in the figure, the UAV2110 may be equipped with a member for tracking the UAV2110 by the total station 2130. This member may be, for example, a retroreflector such as a prism or a reflective seal.

[0337] The control unit 2111 is realized by, for example, cooperation between hardware including a processor and a storage device and control software. The UAV 2110 is capable of autonomous flight under the control of the control unit 2111. The UAV 2110 is also capable of remote-controlled flight using the UAV controller 2120 or the like. In this case, the control unit 2111 performs flight control of the UAV 2110 based on an operation instruction signal from the UAV controller 2120 or the like. The control unit 2111 includes a communication device for performing data communication with other devices (such as the UAV controller 2120 and the edge computer 2140). This data communication is typically wireless communication, but wired communication may also be possible.

[0338] The photographing unit 2112 may include, for example, one or more of a digital camera, a laser scanner, and a spectral camera. As the digital camera, an omnidirectional camera (also called a panoramic camera or a 360-degree camera) is typically used. Although a detailed description will be given of a case where the photographing unit 2112 acquires an image (video) of the surrounding environment using an omnidirectional camera, similar processing can be performed in other cases as well.

[0339] The V-SLAM unit 2113 is realized, for example, by cooperation between hardware including a processor and a storage device and V-SLAM software. The V-SLAM unit 2113 analyzes the image acquired by the image capturing unit 2112 in real time to generate three-dimensional information of the surrounding environment (such as buildings) of the UAV 2110, and estimates the position and attitude of the UAV 2110 (particularly the image capturing unit 2112). The processing executed by the V-SLAM unit 2113 may be similar to known V-SLAM processing. Note that other technologies capable of generating an output similar to V-SLAM can be used instead.

[0340] The obstacle detection unit 2114 is realized, for example, by cooperation between hardware including a processor and a storage device and obstacle detection software. The obstacle detection unit 2114 detects an image area corresponding to an obstacle (tool, worker, etc.) by inputting images (frames) constituting the video acquired by the shooting unit 2112 to an inference model (trained model). This inference model may be, for example, the inference model in any one of the first to tenth aspects. The obstacle detection unit 2114 may use rule-based processing to detect an image area corresponding to an obstacle. The obstacle detection unit 2114 may be configured to perform obstacle detection by combining processing using the inference model and rule-based processing.

[0341] The output from the obstacle detection unit 2114 is input to the control unit 2111. The control unit 2111 performs control to avoid collision with the detected obstacle based on the output from the obstacle detection unit 2114. This control may be, for example, any of changing the flight path, levitation, landing, switching from autonomous flight to heteronomous flight, outputting a warning sound, and an instruction to the UAV controller 2120 to output warning information (such as a warning sound or a warning display).

[0342] The UAV controller 2120 is used as a remote controller for remotely operating the UAV 2110. The UAV controller 2120 is also used to display information about the building to be measured (BIM model, CAD model, component information, construction plan, etc.). The UAV controller 2120 is also used to output information about the UAV 2110 (flight path, video obtained by the image capture unit 2112, warnings, etc.). The UAV controller 2120 may also be used to create or edit a flight plan (flight path) for the UAV 2110.

[0343] The UAV controller 2120 includes a control unit 2121 and a user interface 2122. The control unit 2121 controls each unit of the UAV controller 2120. This is realized by cooperation between hardware including a processor and a storage device and control software. The control unit 2121 includes a communication device for performing data communication with other devices (such as the UAV 2110 and the edge computer 2140). This data communication is typically wireless communication, but wired communication may also be possible.

[0344] The user interface 2122 includes, for example, a display device, an operation device, an input device, etc. The user interface 2122 is typically a mobile computer such as a tablet or a smartphone, and includes a touch screen, a GUI, etc.

[0345] The total station 2130 is used to track the flying UAV 2110. When a retroreflector is provided on the UAV 2110, the total station 2130 outputs a tracking light (distance measurement light) and receives the reflected light of the tracking light by the retroreflector, thereby tracking the retroreflector. The total station 2130 measures three-dimensional coordinates (slope distance, horizontal angle, vertical angle, etc.) based on the installation position of the total station 2130 (or another reference position) while tracking the retroreflector. The tracking function is realized, for example, by cooperation between hardware including a processor and a storage device and tracking software. Also, the three-dimensional coordinate measurement function is realized, for example, by cooperation between hardware including a processor and a storage device and three-dimensional coordinate measurement software.

[0346] When the UAV 2110 is not provided with a retroreflector, for example, a plurality of light receiving sensors can be provided on the UAV 2110 (not shown). Each light receiving sensor can receive tracking light from the total station 2130. By determining which of the plurality of light receiving sensors has received the tracking light, it is possible to estimate the orientation of the UAV 2110 with respect to the total station 2130. This estimation process is executed by, for example, any of the UAV 2110, the UAV controller 2120, the total station 2130, and the edge computer 2140.

[0347] The total station 2130 includes a communication device for performing data communication with other devices (such as the UAV 2110, the UAV controller 2120, and the edge computer 2140). This data communication is typically wireless communication, but wired communication may also be possible. The total station 2130 can transmit the position information (three-dimensional coordinates) of the UAV 2110, which is sequentially acquired along with the above-mentioned tracking, to the UAV 2110 in real time.

[0348] In this way, the UAV 2110 can recognize its own current position based on the information transmitted from the total station 2130. In addition, the UAV 2110 can recognize its own current position based on the information obtained by the V-SLAM unit 2113.

[0349] When the UAV 2110 is flying in a blind spot area of ​​tracking by the total station 2130, the UAV 2110 only recognizes (relatively rough) position information based on V-SLAM in real time.

[0350] On the other hand, when the UAV 2110 flies in an area other than the blind spot area, the UAV 2110 can recognize in real time both (relatively rough) position information based on the V-SLAM and (relatively detailed) position information based on the total station 2130. When both pieces of position information can be acquired in real time, the UAV 2110 can associate both pieces of position information.

[0351] In addition, the UAV 2110 may be configured to fly autonomously by referring to (relatively detailed) position information based on the total station 2130 when both types of position information are available, and to fly autonomously by referring to (relatively rough) position information based on V-SLAM at other times.

[0352] The edge computer 2140 is a computer for implementing edge computing at a construction site, and processes data from devices such as the UAV 2110, the UAV controller 2120, and the total station 2130 at (or near) the construction site. By introducing such edge computing, it is possible to eliminate the increase in load and communication delays in the entire system 2100.

[0353] The edge computer 2140 includes a communication device for performing data communication with devices used at a construction site, such as the UAV 2110, the UAV controller 2120, and the total station 2130. This data communication is typically wireless communication, but wired communication may also be possible.

[0354] The edge computer 2140 also includes a communication device for performing data communication with the cloud computer 2150. This data communication is typically wireless communication, but wired communication may also be possible.

[0355] Further, the edge computer 2140 may include a BIM data processing application or a building data management application. In this embodiment, the edge computer 2140 includes an SfM·MVS unit 2141 and a construction management unit 2142.

[0356] The SfM·MVS unit 2141 is realized by, for example, cooperation between hardware including a processor and a storage device, SfM software, and MVS software. The SfM·MVS unit 2141 is configured to create position information of the UAV2110 (actual flight path) and a 3D model of a (real) building based on, for example, an image acquired by the imaging unit 2112 of the UAV2110, position information of the UAV2110 acquired by the V-SLAM unit 2113 of the UAV2110, and position information of the UAV2110 acquired by the total station 2130.

[0357] As the SfM process, the SfM·MVS unit 2141 estimates the position information of the UAV 2110 from the image acquired while the UAV 2110 is flying. More specifically, the SfM·MVS unit 2141 applies the SfM process to the image acquired while the UAV 2110 is flying, thereby obtaining time-series three-dimensional coordinates representing the route that the UAV 2110 actually flew and the attitude of the UAV 2110 corresponding to each three-dimensional coordinate in the time-series three-dimensional coordinates. That is, the SfM·MVS unit 2141 obtains time-series three-dimensional coordinates representing the moving route of the camera included in the shooting unit 2112 and time-series attitude information of the camera along this moving route. In the SfM process, the position information (three-dimensional coordinates) of the UAV 2110 acquired by the total station 2130 can be referred to. This makes it possible to improve the accuracy of the acquired time-series three-dimensional coordinates and time-series attitude information. The SfM process of this example may be similar to a known SfM process. It should be noted that other techniques capable of producing an output similar to the SfM processing of this example may alternatively be used.

[0358] Furthermore, as MSV processing, the SfM·MVS unit 2141 generates point cloud data of the (physical) building based on the position information of the UAV2110 obtained by the SfM processing (time-series position information and time-series attitude information of the camera) and the images acquired while the UAV2110 is flying.

[0359] In this example, the edge computer 2140 (the SfM·MVS unit 2141, the construction management unit 2142, or another data processing unit) may be configured to detect an image region (component region) corresponding to a building component by inputting an image (frame) constituting the video acquired by the shooting unit 2112 into an inference model (image segmentation). This inference model may be, for example, an inference model in any of the first to tenth aspects. Furthermore, the edge computer 2140 (the SfM·MVS unit 2141, the construction management unit 2142, or another data processing unit) can identify component attributes (type, identification information (ID), shape, position, measurement date, measurement time, etc.) corresponding to each detected component region. The inference model can also be used in this process. The edge computer 2140 may use rule-based processing for component region detection and component attribute identification. Furthermore, the edge computer 2140 may be configured to perform component region detection and component attribute identification by combining processing using the inference model and rule-based processing.

[0360] The construction management unit 2142 is realized, for example, by cooperation between hardware including a processor and a storage device and construction management software. The construction management unit 2142 is configured to manage various data handled by the system 2100. The processing executed by the construction management unit 2142 will be described later.

[0361] The cloud computer 2150 is a computer for realizing cloud computing, which provides computer resources as a service from a remote location to a construction site via a computer network. By introducing such cloud computing, it is possible to improve the scalability, flexibility, efficiency, etc. of the services that can be provided to the system 2100.

[0362] The cloud computer 2150 is configured to manage, for example, a BIM tool, a data management tool, and data used in these tools (design BIM, construction information, measurement BIM, etc.), and provide them to the edge computer 2140. In this embodiment, the cloud computer 2150 includes a BIM unit 2151 and a data management unit 2152.

[0363] The BIM unit 2151 provides various tools such as a BIM tool and various data such as data used in the BIM tool to the edge computer 2140. The BIM unit 2151 is realized, for example, by the cooperation of hardware including a processor and a storage device and BIM software. The processing executed by the BIM unit 2151 will be described later.

[0364] The data management unit 2152 manages various tools and various data. The data management unit 2152 is realized, for example, by cooperation between hardware including a processor and a storage device and data management software. The processing executed by the data management unit 2152 will be described later.

[0365] 22 shows an example of a data structure (data format) handled by the system 2100 of this embodiment. Data format 2250 includes design data 2210, rendering data 2220, measurement data 2230, inspection information 2240, inspection knowledge base 2250, and inspection data 2260. In other words, data format 2250 has a data structure including at least areas in which these types of data are recorded.

[0366] In the area in which the design data 2210 is recorded, the various types of design data described above, such as BIM data (design BIM) and design drawings, are recorded.

[0367] In the area where the rendering data 2220 is recorded, image data (virtual images) obtained by rendering the design BIM are recorded. The rendering data is constructed for each of a plurality of positions in the design BIM. For example, for each of a plurality of positions on a preset flight path, volume rendering with the position as the viewpoint is applied to the design BIM. This results in a plurality of rendering data (a plurality of virtual images) along the flight path in the design BIM. Corresponding position information (three-dimensional coordinates of the viewpoint in the design BIM) can be attached to each rendering data as attribute information. The attribute information of the rendering data is not limited to this. For example, the attribute information of the rendering data may include any information related to the design BIM, any information related to the rendering process, any information related to the rendering data, etc.

[0368] In the area where the measurement data 2230 is recorded, various data acquired by measuring the (actual) building is recorded. Examples of such data include point cloud data (scan data) of the building, three-dimensional coordinates of each measurement position, video acquired by the imaging unit 2112 of the UAV 2110, two-dimensional images, dimensional information, measurement date, measurement time, and the like. In addition, parameter information related to any of these data can also be recorded. For example, parameter information related to the point cloud data creation process, parameter information related to the imaging process by the imaging unit 2112, and the like can be recorded.

[0369] In the area where the inspection information 2240 is recorded, various information (inspection information) for inspecting a real component corresponding to a virtual component is recorded. The inspection information 2240 includes, for example, the shape, dimensions, construction date, construction time, etc. of the inspection target (virtual component) at each inspection position (position of each virtual component). In other words, the inspection information 2240 may include information on multiple attributes of each virtual component. The inspection information is not limited to these, and may include any information related to the inspection.

[0370] In the area where the inspection knowledge base 2250 is recorded, various knowledge used to inspect a real part corresponding to a virtual part is recorded. The inspection knowledge base 2250 includes, for example, a measurement path (a plurality of inspection positions along a flight path), the inference model described above, and the like. The inspection knowledge base 2250 may also include a rule-based algorithm.

[0371] In the area where the inspection data 2260 is recorded, various data (inspection data) acquired by inspecting a real member corresponding to a virtual member is recorded. The inspection data 2260 includes, for example, the presence or absence of an object at the inspection position (presence or absence of a real member corresponding to a virtual member), the deviation of the real member with respect to the virtual member (presence or absence of positional deviation, the direction of positional deviation, etc.), and a determination result as to whether the inspection position satisfies a predetermined condition. This determination may include, for example, a determination as to whether an inspection has been performed based on data obtained from a preset flight path, in other words, a determination as to whether obstacle avoidance (or deviation from the flight path for other reasons) has been performed at the relevant time point.

[0372] An example of the operation of the system 2100 according to this embodiment will be described with further reference to FIGS. 23A, 23B, and 23C.

[0373] The series of steps (inspections) in this operational example may be performed at any timing. For example, inspections may be performed for each specified construction process. As an exemplary application, the construction progress status on each construction day may be digitized by performing the series of steps below on each construction day. If the application of this operational example is only to check the construction progress status, only an inspection may be performed to check whether or not there is an actual member corresponding to each virtual member. For other applications, inspections may be performed with higher accuracy.

[0374] (S91: Generate flight path) In this operation example, first, the cloud computer 2150 transmits to the edge computer 2140 (construction management unit 2142) the design BIM, construction information (construction date information for each component), the date when inspection using the UAV 2110 will be performed (measurement date information), obstacle images (virtually generated images, actual captured images), and the like to the edge computer 2140. Based on the information provided by the cloud computer 2150, the edge computer 2140 determines the flight route of the UAV 2110 on the measurement date.

[0375] The flight path may be generated by the cloud computer 2150, the UAV controller 2120, or another computer (same below). Also, instead of acquiring the measurement date information from the cloud computer 2150, the edge computer 2140 or the UAV controller 2120 may generate the measurement date information. The flight path is generated fully automatically, semi-automatically, or manually. The construction date information may include the construction time.

[0376] The flight path is determined, for example, so that the distance between the imaging unit 2112 of the UAV 2110 and the inspection target (components, floors, ceilings, walls, equipment, pillars, etc.) is within an allowable range. For example, the allowable range (maximum distance) can be set taking into consideration that the closer the distance, the higher the inspection accuracy. On the other hand, the allowable range (minimum distance) can be set so that the entire building can be photographed.

[0377] (S92: Send flight route to UAV) The edge computer 2140, for example, upon receiving instructions from a user, transmits information about the flight path generated in step S91 to the UAV 2110.

[0378] (S93: Generate a virtual image) In addition, the edge computer 2140 generates a virtual image (rendering data) by rendering the design BIM based on the flight path generated in step S91. For example, the edge computer 2140 generates a virtual image obtained when a virtual UAV (virtual camera) flies along the flight path in a three-dimensional virtual space in which the design BIM is defined. More specifically, the edge computer 2140 generates an image of a virtual BIM (virtual building) acquired by a virtual camera from each of a plurality of positions on the flight path.

[0379] (S94: Building an inference model) The edge computer 2140 applies machine learning using the virtual image generated in step S93 (and other virtual images) as training data to a predetermined neural network to construct an inference model (first inference model) for identifying component data from data of the actual building. This process may be performed, for example, according to any of the exemplary aspects described above.

[0380] Furthermore, the edge computer 2140 applies machine learning to a predetermined neural network using the obstacle image (and other obstacle images) provided by the cloud computer 2150 in step S91 as training data, thereby constructing an inference model (second inference model) for identifying data of an obstacle mixed in data of the actual building. This process may be executed, for example, according to any of the exemplary aspects described above.

[0381] In addition, by using both the virtual image and the obstacle image as training data, a single inference model that functions as both the first inference model and the second inference model can be constructed. This process may be performed, for example, according to any of the exemplary aspects described above. The inference model in this example is trained to identify component data and obstacle data from data of a real building. The following description employs such an inference model, but is not limited to this. The inference model may also have other functions.

[0382] The training data may include texture information of components and texture information of obstacles. The neural network model used to construct the inference model is typically a convolutional neural network. The method used to construct the inference model is not limited to this example, and any method such as a support vector machine, a Bayesian classifier, boosting, k-means, kernel density estimation, principal component analysis, independent component analysis, self-organizing map, random forest, and generative adversarial network (GAN) can be applied.

[0383] (S95: Send the inference model to the UAV) The edge computer 2140 transmits the inference model constructed in step S94 to the UAV 2110.

[0384] (S96: Start measuring the building) After the preparations in steps S91 to S95, the measurement work of the actual building is started.

[0385] (S97: Synchronize UAV and total station) In the measurement operation, first, synchronization is performed between the UAV 2110 and the total station (TS) 1430. That is, the clock in the UAV 2110 and the clock in the total station 2130 are synchronized. This synchronizes the time (imaging time) attached to the image obtained by the UAV 2110 and the time (measurement time) attached to the position information of the UAV 2110 acquired by the total station 2130. Here, the imaging time is attached to the position information acquired from the image using V-SLAM processing.

[0386] When both the UAV 2110 and the total station 2130 are outdoors, the clocks of both can be synchronized, for example, by using a navigation signal (including time information based on an atomic clock) from a navigation satellite. On the other hand, when one of the UAV 2110 and the total station 2130 is indoors, the clocks of both can be synchronized, for example, by using a time server on a network to which both the UAV 2110 and the total station 2130 can be connected. Note that the synchronization method is not limited to these.

[0387] (S98: Start tracking the UAV using the total station) After the time synchronization in step S97, the user issues an instruction to start measurement, for example, using the UAV controller 2120. The total station 2130 starts tracking the UAV 2110 that has received the measurement start instruction from the UAV controller 2120. The total station 2130 also starts generating position information of the UAV 2110 and transmitting this position information to the UAV 2110 in real time.

[0388] (S99: Start flying, taking pictures and detecting obstacles) In addition, upon receiving the measurement start instruction in step S98, the UAV 2110 starts autonomous flight by referencing the flight route received in step S92, shooting by the shooting unit 2112 (and storing the video), and obstacle detection processing by the V-SLAM unit 2113 and the obstacle detection unit 2114.

[0389] (S100: Controls flight based on external information) The UAV 2110 performs autonomous flight by flight control based on the tracking information (position information of the UAV 2110 that is the tracking target) transmitted in real time from the total station 2130 and the flight path received in step S92. Note that steps S100 to S104 are repeatedly executed until "Yes" is determined in step S104.

[0390] (S101: When tracking is interrupted, flight control is performed based on the information obtained by the vehicle itself) When the UAV 2110 enters a blind spot range of the tracking by the total station 2130, for example, the UAV 2110 cannot receive tracking information from the total station 2130. The UAV 2110 may be configured to switch the position information referred to for autonomous flight control from the tracking information from the total station 2130 to the position information sequentially acquired by the V-SLAM unit 2113 in response to the interruption of reception of the tracking information. Furthermore, the UAV 2110 may be configured to switch the position information referred to for autonomous flight control from the position information sequentially acquired by the V-SLAM unit 2113 to the tracking information from the total station 2130 in response to the restart of reception of the tracking information.

[0391] Even if the UAV 2110 enters the blind spot range of the tracking by the total station 2130, some tracking information may reach the UAV 2110 due to reflection or transmission of radio waves. In anticipation of such a case, the UAV 2110 may be configured to detect a problem from the position information indicated by the tracking information. For example, the UAV 2110 may be configured to detect a problem by collating the position information indicated by the tracking information with the position information obtained by the V-SLAM unit 2113 (as described above, both pieces of position information are synchronized). As an example of this configuration, the UAV 2110 may be configured to calculate an error between both pieces of position information (three-dimensional coordinates), and determine that there is a "problem" when this error is equal to or greater than a predetermined threshold, and determine that there is no "problem" when this error is less than the predetermined threshold. The UAV 2110 may be configured to switch the position information referred to for autonomous flight control from the tracking information from the total station 2130 to the position information sequentially acquired by the V-SLAM unit 2113 in response to the determination result transitioning from "no problem" to "problem". Furthermore, the UAV 2110 may be configured to switch the position information referred to for autonomous flight control from the position information sequentially acquired by the V-SLAM unit 2113 to the tracking information from the total station 2130 in response to the determination result transitioning from "problem" to "no problem".

[0392] With this configuration, while tracking of the UAV 2110 by the total station 2130 is suspended, the UAV 2110 can perform autonomous flight control based on the position and attitude successively determined by the V-SLAM unit 2113.

[0393] (S102: Was an obstacle detected?) When the obstacle detection unit 2114 detects an obstacle (S102: Yes), the operation proceeds to step S103. If no obstacle is detected (S102: No), the operation skips step S103 and proceeds to step S104.

[0394] (S103: Perform obstacle avoidance control) When an obstacle is detected in step S102 (S102: Yes), the UAV2110 performs control for the obstacle avoidance operation described above. For example, the UAV2110 obtains the position, direction, dimensions, etc. of the detected obstacle, determines a route to avoid collision with the obstacle, and flies along this route. Typically, the start point and end point of this collision avoidance route are located on the flight route received in step S92. That is, the UAV2110 deviates from the flight route received in step S92, detours around the obstacle, and returns to the flight route.

[0395] (S104: Have you reached the end of the flight?) The UAV 2110 can determine whether it has reached the end point (flight end point) of the flight path received in step S92 based on tracking information from the total station 2130 and position information acquired by the V-SLAM unit 2113. If it is determined that the flight end point has not been reached (S104: No), the operation returns to step S100. On the other hand, if it is determined that the flight end point has been reached (S104: Yes), the operation proceeds to step S105. This completes the measurement (photographing) of the actual building.

[0396] (S105: Send the captured image to the edge computer) When the flight end point is reached (S104: Yes), the UAV2110 sends the captured images (video) acquired during flight to the edge computer 2140. The UAV2110 may send the captured images to the edge computer 2140 one by one while flying, or may accumulate the captured images during flight and send them to the edge computer 2140 collectively after the flight is completed. The UAV2110 may repeat accumulation and transmission of the captured images at predetermined time intervals. Alternatively, the UAV2110 may transmit a predetermined amount of captured images each time they are accumulated.

[0397] The following steps are all a series of processes based on images acquired by the UAV2110 (post-processing). When the UAV2110 transmits captured images to the edge computer 2140 while flying, post-processing may be started before the flight of the UAV2110 is completed. In other words, shooting and post-processing may be performed in parallel and sequentially. This makes it possible to shorten the work time. On the other hand, when post-processing is started after shooting is completed, post-processing can be performed by referring to all captured images, so that it is possible to improve the accuracy and precision of the processing. For example, when analyzing a certain captured image, it is possible to refer to the captured images before and after it.

[0398] Here, the edge computer 2140 can determine whether the captured image is suitable for post-processing. For example, the edge computer 2140 may be configured to evaluate the quality (brightness, contrast, focus, color, definition, etc.) of the captured image. If the quality is determined to be insufficient, the image may be captured again by the UAV 2110, or image processing for improving image quality may be applied to the captured image. For example, an inference model constructed by machine learning is used for this image processing. In addition, this image processing may include an interpolation process using previous and subsequent captured images. Image quality evaluation may be started during capture by the UAV 2110. When the image quality is determined to be insufficient during capture, the image may be recaptured by returning to the position where the captured image determined to be insufficient in quality was acquired (or a position upstream of that in the flight path).

[0399] Similarly, the total station 2130 transmits to the edge computer 2140 time-series three-dimensional coordinates and time-series attitude information of the UAV 2110 (the image capturing unit 2112, the retroreflector) collected while tracking the UAV 2110.

[0400] (S106: Calculate the camera position and orientation) The edge computer 2140 executes SfM processing based on the captured images (each frame of the video) acquired by the UAV 2110 and the time-series information (time-series three-dimensional coordinates, time-series attitude information) acquired by the total station 2130, thereby obtaining the position and attitude of the shooting unit 2112 (camera) at each of a plurality of positions on the flight path. Here, since the UAV 2110 and the total station 2130 are synchronized in step S97, it is possible to associate the time information of the video with the time information of the time-series information. By performing this time association, the edge computer 2140 can apply SfM processing to the combination of the video and the time-series information.

[0401] (S107: Extract the component area from the captured image) The edge computer 2140 uses the inference model constructed in step S94 to extract image areas (component areas, component areas) corresponding to components (virtual components) included in the design BIM from the captured images (each frame of the video) acquired by the UAV 2110. This extraction process includes, for example, a process of identifying the component areas in the design BIM and a process of masking image areas other than the identified component areas. Here, the edge computer 2140 may perform a process of identifying a predetermined attribute of the actual component corresponding to the identified component area.

[0402] The edge computer 2140 can determine whether the information obtained in step S107 is suitable for the subsequent processing. For example, when the component area of ​​the corresponding real component is not extracted for many virtual components, it is determined to be "unsuitable". When it is determined to be unsuitable, the edge computer 2140 can, for example, perform control to request re-imaging.

[0403] (S108: Find the characteristic points and their coordinates of the component area) The edge computer 2140 detects the feature points of the component area extracted in step S107. The feature points may be one or more points, one or more lines, or one or more surfaces. The feature points may also be patterns, etc. Furthermore, the edge computer 2140 obtains the three-dimensional coordinates of the detected feature points.

[0404] (S109: Generate surface data for components) The edge computer 2140 generates surface data of the actual component corresponding to the component area extracted in step S107. For example, the edge computer 2140 performs registration of two or more captured images taken at different positions based on the feature points detected in step S108. Typically, these captured images partially overlap. The edge computer 2140 uses MVS processing to generate surface data of the component commonly depicted in these captured images based on the capture positions of these captured images.

[0405] The edge computer 2140 performs registration between the design BIM and the video. As a result, for example, each frame of the video (photographed image) is embedded in the three-dimensional space (three-dimensional coordinate system) in which the design BIM is defined.

[0406] Next, the edge computer 2140 obtains a point cloud in the captured image located near a predetermined surface of a virtual component in the design BIM. For example, the edge computer 2140 specifies the point cloud by specifying points in the captured image whose distance to the predetermined surface of the virtual component is within a predetermined threshold or less. By performing such processing for each frame of the video, a three-dimensional coordinate point cloud located near the predetermined surface of the virtual component is obtained.

[0407] Next, the edge computer 2140 determines a surface in the image corresponding to a predetermined surface of the virtual member based on the obtained three-dimensional coordinate point group. For example, the edge computer 2140 obtains an approximate surface (plane, free-form surface, etc.) based on at least a part of the three-dimensional coordinate point group. This approximate surface is the above-mentioned surface data. That is, this approximate surface is treated as surface data (surface image) in the image corresponding to the predetermined surface of the virtual member. In other words, this approximate surface is a part (surface) of a real member corresponding to the virtual member, and is treated as a part (surface) corresponding to the predetermined surface of this virtual member. The edge computer 2140 associates the predetermined surface of the virtual member with a surface in the image (surface of the real member) determined based on it. As a result, the virtual member and the real member are associated with each other, and the attributes (position, shape, etc.) of the virtual member and the attributes (position, shape, etc.) of the real member are associated with each other.

[0408] (S110: Create measurement BIM) The edge computer 2140 creates a three-dimensional model (measured BIM) based on the multiple surface data generated in step S109. That is, the edge computer 2140 creates a measured BIM based on the data of the multiple actual components obtained in step S109. The created measured BIM is transmitted to and stored in the cloud computer 2150. The measured BIM is used for comparison with the design BIM, construction management, maintenance management, repair management, etc. This is the end of this operation example (end).

[0409] The above disclosure is merely an example of the embodiment of the present invention, and therefore any modifications (omissions, substitutions, additions, etc.) can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0410] 1. Information processing device 11 Control section 12 User Interface 13 Data Set Reception Section 14 Storage section 141 Datasets 142 Route Information 143 Training Data 144 Inference Model 15 Processing section 151 Route setting section 152 Virtual image set generator 153 Inference Model Construction Department 153A Learning processing unit 153B Neural Networks 1900 Information processing equipment 21 Control section 22 User Interface 23 Data Set Reception Department 24 Memory section 241 Inference Model 242 Measurement data 25 Processing section 251 Feature Extractor 252 Specifier

Claims

1. A data set receiving unit that includes a communication device that performs data communication with the outside or a drive device that reads data recorded on a recording medium and receives a data set that includes at least BIM (Building Information Modeling) data; A computer that constructs an inference model for identifying data on building components from measurement data acquired by photographing or laser scanning a building; Including, The computer includes: A route setting unit that sets a route inside and / or outside the virtual building represented by the BIM data based on the data set so that a distance to a virtual building component recorded in the BIM data falls within a predetermined tolerance range; a virtual image set generator for generating a virtual image set of the virtual building along the route based on the data set and the route; an inference model construction unit that applies machine learning using training data including at least the virtual image set and photographed images of building components to a predetermined neural network to construct an inference model in which photographed images of a building and measurement data including either scan data or point cloud data obtained by laser scanning the building are input and data of the building components is output; Including, the route setting unit sets, as the route, either a two-dimensional area or a three-dimensional area indicating a range within which a moving body configured to perform measurements while autonomously moving inside and / or outside a building can move in order to avoid obstacles; Information processing device.

2. The data set further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building, Further comprising a BIM data construction unit that constructs first BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the data set, The route setting unit sets a first route so that a distance to a virtual building component recorded in the first BIM data is within a predetermined tolerance range, The virtual image set generator generates a first virtual image set based on the first BIM data and the first path; The inference model construction unit applies machine learning using first training data including at least the first virtual image set and the photographed images of the building components to the neural network to construct an inference model corresponding to the measurement date.

2. The information processing device of claim 1.

3. The data set further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building, Further comprising a BIM data construction unit that constructs second BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the data set, The virtual image set generator generates a second virtual image set based on the second BIM data and the path; The inference model construction unit applies machine learning using second training data including at least the second virtual image set and the photographed images of the building components to the neural network to construct an inference model corresponding to the measurement date.

2. The information processing device of claim 1.

4. a data set receiving unit that receives a data set including at least a communication device that performs data communication with the outside or a drive device that reads data recorded on a recording medium, the data set receiving unit receiving at least a data set including BIM (Building Information Modeling) data and route information that represents a route inside and / or outside a virtual building represented by the BIM data, the route being set so that a distance to a virtual building component recorded in the BIM data falls within a predetermined tolerance range; A computer that constructs an inference model for identifying data on building components from measurement data acquired by photographing or laser scanning a building; Including, The computer includes: a virtual image set generator for generating a set of virtual images of the interior and / or exterior of the virtual building along the path based on the data set; an inference model construction unit that applies machine learning using training data including at least the virtual image set and photographed images of building components to a predetermined neural network to construct an inference model in which photographed images of a building and measurement data including either scan data or point cloud data obtained by laser scanning the building are input and data of the building components is output; Including, The route is either a two-dimensional area or a three-dimensional area that indicates a range within which a mobile body configured to perform measurements while autonomously moving inside and / or outside a building can move in order to avoid obstacles. Information processing device.

5. A virtual image set receiving unit that includes a communication device that performs data communication with the outside or a drive device that reads data recorded on a recording medium, and receives a virtual image set along a predetermined route that is set so that the distance of the inside and / or outside of a virtual building represented by BIM (Building Information Modeling) data to a virtual building component recorded in the BIM data is within a predetermined tolerance range; A computer that constructs an inference model for identifying data on building components from measurement data acquired by photographing or laser scanning a building; Including, The computer includes an inference model construction unit that applies machine learning using training data including at least the virtual image set and photographed images of building components to a predetermined neural network to construct an inference model that uses photographed images obtained by photographing a building and measurement data including either scan data or point cloud data obtained by laser scanning the building as input and outputs data of the building components; The route is either a two-dimensional area or a three-dimensional area that indicates a range within which a mobile body configured to perform measurements while autonomously moving inside and / or outside a building can move in order to avoid obstacles. Information processing device.

6. the data set includes texture information representing a surface condition of a virtual building component; The computer further includes a first information synthesis unit that synthesizes the texture information with the virtual image set to generate first synthesis information; The inference model construction unit applies machine learning to the neural network using training data including at least the first synthetic information.

5. An information processing device according to claim 1.

7. The data set further includes obstacle information; The computer further includes a second information synthesis unit that synthesizes the obstacle information with the virtual image set to generate second synthesis information, The inference model construction unit applies machine learning to the neural network using training data including at least the second synthetic information. An information processing device according to any one of claims 1 to 4 and 6.

8. The computer further includes a labeling unit that assigns labels to the virtual image set related to virtual building components recorded in the BIM data, The inference model construction unit applies supervised learning to the neural network using training data including at least the virtual image set and the label.

8. An information processing device according to claim 1.

9. The neural network includes a convolutional neural network.

9. An information processing device according to claim 1.

10. The computer further includes a transmission unit that transmits the inference model constructed by the inference model construction unit to the mobile body.

10. An information processing device according to claim 1.

11. An inference model construction method executed by a communication device that performs data communication with the outside or a drive device that reads data recorded on a recording medium, and a computer that constructs an inference model for identifying data on building components from measurement data acquired by photographing or laser scanning a building, comprising: The communication device or the drive device receives a data set including at least Building Information Modeling (BIM) data, The computer sets a route inside and / or outside the virtual building represented by the BIM data based on the data set so that a distance to a virtual building component recorded in the BIM data falls within a predetermined tolerance range; the computer generates a set of virtual images of the virtual building along the route based on the dataset and the route; The computer applies machine learning using training data including at least the virtual image set and photographed images of building components to a predetermined neural network to construct an inference model in which the photographed images obtained by photographing the building and the measurement data including either the scan data or the point cloud data obtained by laser scanning the building are input and data of the building components is output; The computer sets, as the route, either a two-dimensional area or a three-dimensional area indicating a range within which a mobile object configured to perform measurements while autonomously moving inside and / or outside a building can move in order to avoid obstacles. How to build an inference model.

12. The data set further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building, The computer constructs first BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the data set, The computer sets a first route so that a distance to a virtual building component recorded in the first BIM data is within a predetermined tolerance range; the computer generates a first set of virtual images based on the first BIM data and the first path; The computer applies machine learning to the neural network using first training data including at least the first virtual image set and the photographed images of the building member to construct an inference model corresponding to the measurement date.

12. The method of claim 11.

13. The data set further includes construction date information indicating the construction date of each virtual building component recorded in the BIM data, and measurement date information indicating the measurement date of the building, The computer constructs second BIM data corresponding to a state in which only virtual building components whose construction date is before the measurement date are constructed based on the data set, the computer generates a second set of virtual images based on the second BIM data and the path; The computer applies machine learning to the neural network using second training data including at least the second virtual image set and the photographed images of the building member to construct an inference model corresponding to the measurement date.

12. The method of claim 11.

14. An inference model construction method executed by a communication device that performs data communication with the outside or a drive device that reads data recorded on a recording medium, and a computer that constructs an inference model for identifying data on building components from measurement data acquired by photographing or laser scanning a building, comprising: The communication device or the drive device receives a data set including at least BIM (Building Information Modeling) data and route information representing a route inside and / or outside a virtual building represented by the BIM data, the route being set so that a distance to a virtual building component recorded in the BIM data falls within a predetermined tolerance range; generating, based on the data set, a set of virtual images of the interior and / or exterior of the virtual building along the path; The computer applies machine learning using training data including at least the virtual image set and photographed images of building components to a predetermined neural network to construct an inference model in which the photographed images obtained by photographing the building and the measurement data including either the scan data or the point cloud data obtained by laser scanning the building are input and data of the building components is output; The route is either a two-dimensional area or a three-dimensional area that indicates a range within which a mobile body configured to perform measurements while autonomously moving inside and / or outside a building can move in order to avoid obstacles. How to build an inference model.

15. An inference model construction method executed by a communication device that performs data communication with the outside or a drive device that reads data recorded on a recording medium, and a computer that constructs an inference model for identifying data on building components from measurement data acquired by photographing or laser scanning a building, comprising: The communication device or the drive device receives a set of virtual images along a predetermined route set so that a distance of the interior and / or exterior of a virtual building represented by BIM (Building Information Modeling) data to a virtual building component recorded in the BIM data falls within a predetermined tolerance range; The computer applies machine learning using training data including at least the virtual image set and photographed images of building components to a predetermined neural network to construct an inference model in which the photographed images obtained by photographing the building and the measurement data including either the scan data or the point cloud data obtained by laser scanning the building are input and data of the building components is output; The route is either a two-dimensional area or a three-dimensional area that indicates a range within which a mobile body configured to perform measurements while autonomously moving inside and / or outside a building can move in order to avoid obstacles. How to build an inference model.

16. the data set includes texture information representing a surface condition of a virtual building component; The computer synthesizes the texture information with the virtual image set to generate first synthesis information, and applies machine learning to the neural network using training data including at least the first synthesis information. The method of any one of claims 11 to 14.

17. The data set further includes obstacle information; The computer generates second synthetic information by synthesizing the obstacle information with the virtual image set, and applies machine learning to the neural network using training data including at least the second synthetic information. The method according to any one of claims 11 to 14 and 16.

18. The computer assigns labels related to virtual building components recorded in the BIM data to the virtual image set, and applies supervised learning to the neural network using training data including at least the virtual image set and the labels. The method of any one of claims 11 to 17.

19. The neural network includes a convolutional neural network. The method of any one of claims 11 to 18.

20. The computer transmits the inference model toward the mobile object. The method of any one of claims 11 to 19.

21. A program for causing a computer to execute the method according to any one of claims 11 to 20.

22. A non-transitory computer-readable recording medium having the program of claim 21 recorded thereon.

23. A program that causes a computer to execute a process of identifying data on building components from measurement data of a building using an inference model constructed by any of the methods of claims 11 to 20.

24. A non-transitory computer-readable recording medium having the program of claim 23 recorded thereon.

Images