Information processing device, information processing method, and program

Abstract

To support establishing remote construction management.SOLUTION: An information processing device according to an embodiment of the present disclosure comprises: means of acquiring a site image obtained by photographing a construction site associated with a building; means of aligning the site image with a three-dimensional model of the building; and means of generating an output image obtained by merging the three-dimensional model with the site image in reference to an alignment result.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Various construction management tasks are required at construction sites. Construction managers typically visit the site to check progress, safety, quality of deliverables, worker performance, and other aspects, and issue instructions for improvements.

[0003] Patent Document 1 describes a method for managing the finished form of reinforcement by recognizing and analyzing images of the reinforcement arrangement state. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-027058 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 describes a technology for measuring the completed state based on images. Utilizing these technologies may enable the construction manager to manage the completed state even when he or she is not on-site. Remote management in this way reduces the movement of the construction manager and improves the productivity of construction management. However, the technology in Patent Document 1 does not target management other than the completed state.

[0006] The purpose of the present disclosure is to support remote construction management. [Means for solving the problem]

[0007] An information processing device according to one embodiment of the present disclosure includes a means for acquiring a site image of a construction site relating to a building, a means for aligning the site image with a three-dimensional model relating to the building, and a means for generating an output image that combines the three-dimensional model and the site image by referring to the alignment result. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to a first embodiment. [Figure 2] 1 is a block diagram illustrating an example of the configuration of an information processing apparatus according to a first embodiment. [Figure 3] FIG. 1 is an explanatory diagram of an overview of a first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an overall flow of information processing according to the first embodiment. [Figure 5] FIG. 10 is an explanatory diagram of an overview of a second embodiment. [Figure 6] FIG. 10 is a diagram illustrating an overall flow of information processing according to the second embodiment. [Figure 7] FIG. 10 is an explanatory diagram of an overview of a third embodiment. [Figure 8] FIG. 11 is an explanatory diagram of association in the third embodiment. [Figure 9] FIG. 11 is a diagram illustrating an overall flow of information processing according to the third embodiment. [Figure 10] FIG. 10 is an explanatory diagram of an overview of a fourth embodiment. [Figure 11] FIG. 13 is a diagram illustrating an overall flow of information processing according to a fourth embodiment. [Figure 12] FIG. 10 is an explanatory diagram of an overview of a fifth embodiment. [Figure 13] FIG. 13 is a diagram illustrating an overall flow of information processing according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiments, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0010] (1) First embodiment A first embodiment will be described.

[0011] (1-1) Information Processing System Configuration A description will be given of the configuration of an information processing system according to the first embodiment. Fig. 1 is a block diagram showing the configuration of the information processing system according to the first embodiment.

[0012] 1, the information processing system 1 includes an information processing device 10 and a sensor 30. The number of sensors 30 is arbitrary. The information processing device 10 and the sensor 30 are connected via a network (for example, the Internet or an intranet) NW.

[0013] The information processing device 10 is, for example, a smartphone, a tablet terminal, a personal computer, a server (for example, a web server, an application server, a database server, or a combination thereof), or a combination thereof.

[0014] The information processing device 10 generates an output image (described later) by referring to an image (site image) transmitted from the sensor 30 or an image server (not shown) and a three-dimensional model (described later). The information processing device 10 presents the output image to a construction manager (an example of a "user").

[0015] Here, the term "image" refers to the result of optical sensing. In the following description, the image is a point cloud (which may include a color point cloud) generated by a depth camera. However, the image may also be a two-dimensional image generated by an image sensor.

[0016] The sensor 30 is installed at a construction site of a target structure (referred to as a "target structure"). The work performed at the construction site may be the construction, repair, or demolition of the structure, or a combination thereof. The sensor 30 performs sensing, generates an image, and transmits the image to the information processing device 10. The sensor 30 is an optical sensor device of any type. The sensor 30 may be any of the following: ·camera Depth camera LIDAR ToF camera

[0017] (1-1-1) Configuration of information processing device The configuration of the information processing apparatus according to the first embodiment will be described below with reference to Fig. 2, which is a block diagram showing an example of the configuration of the information processing apparatus according to the first embodiment.

[0018] As shown in FIG. 2, the information processing device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface .

[0019] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0020] The programs include, for example, the following programs: OS (Operating System) programs Application programs that perform information processing (e.g., the information processing in Figure 4)

[0021] The data includes, for example, the following data: Data referenced in information processing (e.g., 3D models described below) and databases Data obtained by performing information processing (i.e., the results of performing information processing)

[0022] The processor 12 is configured to realize the functions of the information processing device 10 by running a program stored in the storage device 11 and processing data. The processor 12 is an example of a computer. The programs and data stored in the storage device 11 may be provided via a network, or may be provided by being recorded on a computer-readable recording medium. Note that at least some of the functions of the information processing device 10 may be realized by one or more dedicated hardware components.

[0023] The input / output interface 13 is configured to acquire signals (for example, user instructions) from an input device connected to the information processing device 10, and to output signals (for example, image signals) to an output device connected to the information processing device 10. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0024] The communication interface 14 is configured to control communication between the information processing device 10 and the sensor 30 .

[0025] (1-2) Overview of the embodiment An outline of the first embodiment will be described below with reference to Fig. 3, which is an explanatory diagram of the outline of the first embodiment.

[0026] As shown in Fig. 3, the sensor 30 captures an image of a construction site CS(t1) at time t1 and generates a site image SI(t1). The information processing device 10 acquires the site image SI(t1) transmitted from the sensor 30 or an image server (not shown). At the construction site CS(t1), a part P1 of a target building has already been constructed. The part P1 corresponds to a partial image PIM1(t1), which is a subset of the site image SI(t1).

[0027] A three-dimensional model 3M of the target building is stored in the storage device 11. The three-dimensional model 3M may be, for example, BIM (Building Information Modeling) data. The three-dimensional model 3M includes a model part MP1 corresponding to the part P1 and a model part MP2.

[0028] The information processing device 10 aligns the on-site image SI(t1) with the three-dimensional model 3M. As an example, the information processing device 10 converts the position of each information element (point or pixel) included in the on-site image SI(t1) into a position in the coordinate system (referred to as a "reference coordinate system") of the three-dimensional model 3M.

[0029] The information processing device 10 refers to the alignment result, generates an output image OI(t1) by combining the 3D model 3M and the on-site image SI(t1), and displays it on the display. Specifically, in the reference coordinate system, the partial image PIM1(t1) is located at the same position as the model part MP1. Therefore, in the output image OI(t1), the partial image PIM1(t1) is pasted onto the model part MP1. On the other hand, in the reference coordinate system, all information elements of the on-site image SI(t1) are located at different positions from the model part MP2. Therefore, in the output image OI(t1), none of the information elements of the on-site image SI(t1) are pasted onto the model part MP2.

[0030] According to the output image OI(t1), the user can intuitively grasp the progress of construction by viewing the 3D rendering of the target building (i.e., the 3D model 3M) and the completed state at time t1 (i.e., the partial image PIM1(t1)) on one screen.

[0031] (1-3) Information Processing The information processing of the first embodiment will be described below with reference to Fig. 4, which is a diagram showing the overall flow of the information processing of the first embodiment.

[0032] The information processing starts when a user inputs an instruction to start the information processing into the information processing device 10 or when other start conditions are met.

[0033] As shown in FIG. 4, the information processing device 10 acquires a site image (S101). Specifically, the processor 12 acquires site images from the sensor 30 or an image server (not shown) via the communication interface 14 .

[0034] After step S101, the information processing device 10 performs alignment between the site image and the 3D model (S102). Specifically, the processor 12 aligns the on-site image acquired in step S101 with the three-dimensional model. As an example, the processor 12 converts the position of each information element included in the on-site image into a position in the reference coordinate system.

[0035] In a first example of the alignment of the site image with the 3D model (S102), the processor 12 aligns the site image with the 3D model by reproducing the shooting direction of the site image in the reference coordinate system with reference to the shooting conditions and the reference coordinate system of the site image. Here, the shooting conditions are the position, orientation, focal length, or a combination thereof of the sensor 30. The shooting conditions may be transmitted by the sensor 30 together with the site image, or may be stored in the storage device 11 as known information. For example, the processor 12 identifies the position and orientation of the reference coordinate system corresponding to the real-space position and orientation of the sensor 30 that captured the site image. The processor 12 sets a virtual camera in the reference coordinate system with reference to the identified position and orientation of the reference coordinate system and the focal length of the sensor 30. The processor 12 compares the site image with a model image captured of a 3D model using the set camera. As an example, the processor 12 determines the position of the reference coordinate system of an arbitrary information element of the site image by comparing a feature (e.g., an edge) extracted from the model image with a feature extracted from the site image.

[0036] In a second example of aligning the site image with the 3D model (S102), the processor 12 compares an image of a 3D model taken from a specific viewpoint with the site image, and aligns the site image with the 3D model. For example, the processor 12 repeatedly selects a viewpoint and compares the site image with a model image taken of the 3D model with a virtual camera from the selected viewpoint, while changing the viewpoint. In this way, the processor 12 searches for a model image that best matches the site image (e.g., whose features best match). The processor 12 determines the position of the reference coordinate system of an arbitrary information element ("target element") in the model image as the position of the reference coordinate system of the information element in the site image that corresponds to the target element.

[0037] After step S102, the information processing device 10 generates an output image (S103). Specifically, the processor 12 generates an output image by combining the on-site image and the 3D model with reference to the result of the alignment in step S102. As an example, the processor 12 generates the output image by pasting each information element included in the on-site image at a corresponding position on the 3D model.

[0038] After step S103, the information processing device 10 displays the output image (S104). Specifically, the processor 12 causes the output image generated in step S103 to be displayed on the display via the input / output interface 13.

[0039] After step S104, the information processing device 10 ends the information processing of FIG.

[0040] (1-4) Summary As described above, the information processing device of the first embodiment acquires site images of a construction site related to a building, aligns the site images with a 3D model, and generates an output image that combines the 3D model and the site image. Therefore, with this information processing device, a user can intuitively grasp the progress of construction by viewing a 3D rendering of the target building and the completed state at the time the site image was captured on a single screen. In other words, with this information processing device, a user can manage the construction process without directly observing the target building during construction.

[0041] (2) Second embodiment A second embodiment will be described below, but the same description as in the above embodiment will be omitted.

[0042] (2-1) Overview of the embodiment An outline of the second embodiment will be described below with reference to Fig. 5, which is an explanatory diagram of the outline of the second embodiment.

[0043] As shown in FIG. 5, the sensor 30 captures an image of a construction site CS(t2) at time t2 and generates a site image SI(t2). The information processing device 10 acquires the site image SI(t2) transmitted from the sensor 30 or an image server (not shown). At the construction site CS(t2), part P1 and a portion of part P2 of the target building have been constructed. The part P1 corresponds to a partial image PIM1(t2), which is a subset of the site image SI(t2). The portion of part P2 that has been constructed corresponds to a partial image PIM2(t2), which is a subset of the site image SI(t2). Furthermore, a partial image PIM3(t2), which is a subset of the site image SI(t2), corresponds to a worker W1 who is not defined in the three-dimensional model 3M.

[0044] A 3D model 3M of the target building is stored in the storage device 11. The 3D model 3M may be, for example, BIM data. The 3D model 3M includes a model part MP1 corresponding to the part P1 and a model part MP2 corresponding to the part P2.

[0045] As in the first embodiment, the information processing device 10 aligns the on-site image SI(t2) with the three-dimensional model 3M. As an example, the information processing device 10 converts the position of each information element included in the on-site image SI(t2) into a position in a reference coordinate system.

[0046] The information processing device 10 refers to the alignment result and associates a subset of the on-site image with each part included in the 3D model 3M. Specifically, the information processing device 10 associates the partial image PIM1(t2) with the model part MP1, and associates the partial image PIM2(t2) with the model part MP2.

[0047] The information processing device 10 refers to the association result, generates an output image OI(t2) by combining the 3D model 3M and the on-site image SI(t2), and displays it on the display. Specifically, the partial image PIM1(t2) associated with the model part MP1 is pasted onto the model part MP1, and the partial image PIM2(t2) associated with the model part MP2 is pasted onto the model part MP2. On the other hand, the partial image PIM3(t2) not associated with any model part MP is not included in the output image OI(t2).

[0048] In this way, even if an entity unrelated to the target building, such as worker W1, appears in the on-site image SI(t2), the subset of the on-site image SI(t2) related to that entity is not reflected in the output image OI(t2). Therefore, the user can intuitively grasp the progress of construction by viewing the 3D rendering of the target building (i.e., the 3D model 3M) and the completed state at time t2 (i.e., partial image PIM1(t2) and partial image PIM2(t2)) on one screen, without being distracted by entities unrelated to the target building.

[0049] (2-2) Information Processing The information processing of the second embodiment will be described below with reference to Fig. 6, which is a diagram showing the overall flow of the information processing of the second embodiment.

[0050] The information processing starts when a user inputs an instruction to start the information processing into the information processing device 10 or when other start conditions are met.

[0051] As shown in FIG. 6, the information processing device 10 acquires a site image (S101) and aligns the site image with a 3D model (S102), similarly to FIG.

[0052] After step S102, the information processing device 10 associates the partial image with the model part (S203). Specifically, processor 12 refers to the alignment result in step S102 and associates a partial image that is at the same position as each model part in the reference coordinate system with that model part. As an example, processor 12 assigns a label that identifies that model part to a partial image that is at the same position as that model part. Furthermore, processor 12 may assign some kind of label (for example, a label indicating that there is no corresponding model part) to a partial image that is at a different position from any model part.

[0053] After step S203, the information processing apparatus 10 generates an output image (S204). Specifically, the processor 12 generates the output image by combining the site image and the 3D model with reference to the association result in step S203. As an example, the processor 12 generates the output image by attaching each subset included in the site image to the model part associated with that subset.

[0054] After step S204, the information processing device 10 displays the output image (S205). Specifically, the processor 12 displays the output image generated in step S204 on the display via the input / output interface 13.

[0055] After step S205, the information processing device 10 ends the information processing of FIG.

[0056] (2-3) Summary As described above, the information processing device of the second embodiment acquires site images of a construction site related to a building, aligns the site images with a 3D model, associates a subset of the site images with model parts of the 3D model, and generates an output image that combines the 3D model and the site image. Therefore, even if an entity unrelated to the target building is captured in the site image, this information processing device can generate an output image without reflecting the subset related to the entity. Therefore, a user can intuitively grasp the progress of construction by viewing a 3D rendering of the target building and the completed state at the time the site image was captured on a single screen, without being distracted by entities unrelated to the target building. In other words, this information processing device allows a user to manage the construction process without directly observing the target building during construction.

[0057] In the second embodiment, an example was described in which partial images that are not associated with any model parts are not reflected in the output image. This example is not limiting; the information processing device can refer to the results of associating subsets of the on-site image with model parts (i.e., labels), and individually control whether or not to reflect each subset in the output image, and the display mode when the subset is reflected in the output image. The display mode is, for example, an element related to visual effects such as color, transparency, blinking pattern, or a combination thereof.

[0058] In a first example, the information processing device determines that subsets associated with a specific model part are reflected in the output image, and that subsets not associated with the specific model part are not reflected in the output image, thereby excluding subsets not associated with the specific model part from the output image and helping to grasp the progress of construction on the specific model part.

[0059] In a second example, the information processing device determines not to reflect in the output image a subset associated with a specific model part, and to reflect in the output image a subset not associated with the specific model part, thereby excluding from the output image a subset associated with the specific model part, and helping to understand the progress of construction on model parts other than the specific model part, or the status of entities unrelated to the target building (e.g., a material storage area or construction machinery).

[0060] In a third example, the information processing device determines the display form of a subset associated with a specific model part among the subsets reflected in the output image to be different from the display form of a subset not associated with the specific model part among the subsets reflected in the output image, thereby making the subset associated with the specific model part stand out in the output image and helping to grasp the progress of construction on the specific model part.

[0061] In a fourth example, the information processing device determines the display form of subsets not associated with specific model parts among the subsets reflected in the output image to be different from the display form of subsets associated with the specific model parts among the subsets reflected in the output image, thereby making the subsets not associated with the specific model parts stand out in the output image and helping to understand the progress of construction on model parts other than the specific model parts or the status of entities unrelated to the target building (for example, material storage areas and construction machinery).

[0062] In the first to fourth examples above, the specific model parts are at least some of the model parts included in the target building. The specific model parts may be determined according to a user's selection, or may be determined automatically by an algorithm.

[0063] (3) Third embodiment A third embodiment will be described below, but the same description as in the above embodiments will be omitted.

[0064] (3-1) Overview of the embodiment An overview of the third embodiment will be described below. Fig. 7 is an explanatory diagram of the overview of the third embodiment. Fig. 8 is an explanatory diagram of association in the third embodiment.

[0065] As shown in FIG. 7, the sensor 30 captures an image of a construction site CS(t3) at time t3 and generates a site image SI(t3). The information processing device 10 acquires the site image SI(t3) transmitted from the sensor 30 or an image server (not shown). At the construction site CS(t3), part P1 and part P2 of the target building have been constructed. The part P1 corresponds to a partial image PIM1(t3), which is a subset of the site image SI(t3). The constructed part of the part P2 corresponds to a partial image PIM2(t3), which is a subset of the site image SI(t3).

[0066] A 3D model 3M of the target building is stored in the storage device 11. The 3D model 3M may be, for example, BIM data. The 3D model 3M includes a model part MP1 corresponding to the part P1 and a model part MP2 corresponding to the part P2.

[0067] As in the first and second embodiments, the information processing device 10 aligns the on-site image SI(t3) with the three-dimensional model 3M. As an example, the information processing device 10 converts the position of each information element included in the on-site image SI(t3) into a position in a reference coordinate system.

[0068] The information processing device 10 refers to the alignment result and associates a subset of the on-site image with each part unit obtained by subdividing a part included in the 3D model 3M. For example, the information processing device 10 associates the partial image PIM2(t3) with some of the part units included in the model part MP2. As shown in FIG. 8, the information processing device 10 associates each subset of the partial image PIM2(t3) with the part units U0, U1, U2, U3, U4, and U5 included in the model part MP2, respectively. On the other hand, the information processing device does not associate any information element included in the on-site image SI(t3) with the part units U6, U7, U8, . . . , U21, U22, and U23 included in the model part MP2.

[0069] The information processing device 10 refers to the association result and calculates a progress index related to the progress of construction for each model part. As an example, the information processing device 10 calculates the proportion of part units included in the model part that are associated with the subset of the site image as the progress index. In the example of FIG. 8, the information processing device 10 calculates 25% (=6 / 24), which is the proportion of part units included in model part MP2 that are associated with the subset of the site image, as the progress index.

[0070] The information processing device 10 refers to the association result, generates an output image OI(t3) by combining the 3D model 3M and the site image SI(t3), and displays the output image OI(t3) on the display. Specifically, the partial image PIM1(t3) associated with (a part unit included in) the model part MP1 is pasted onto the model part MP1, and the partial image PIM2(t3) associated with (a part unit included in) the model part MP2 is pasted onto the model part MP2. Furthermore, the information processing device 10 generates the output image OI2 so as to include progress indicators related to specific model parts. Here, the specific model parts are at least some of the model parts included in the target building. The specific model parts may be determined according to a user's selection, or may be determined automatically by an algorithm.

[0071] According to the output image OI(t3), the user can intuitively grasp the progress of construction by viewing the 3D completed rendering of the target building (i.e., the 3D model 3M) and the completed form at time t3 (i.e., the partial image PIM1(t3) and the partial image PIM2(t3)) on one screen, and can quantitatively grasp the progress of construction for each part by visually checking the progress indicators.

[0072] (3-2) Information Processing The information processing of the third embodiment will be described below with reference to Fig. 9, which is a diagram showing the overall flow of the information processing of the third embodiment.

[0073] The information processing starts when a user inputs an instruction to start the information processing into the information processing device 10 or when other start conditions are met.

[0074] As shown in FIG. 9, the information processing device 10 acquires a site image (S101) and aligns the site image with a 3D model (S102), similarly to FIG.

[0075] After step S102, the information processing apparatus 10 executes association between the partial image and the part unit (S303). Specifically, processor 12 refers to the alignment result in step S102 and associates a partial image that is at the same position as a part unit included in each model part in the reference coordinate system with that part unit. As an example, processor 12 assigns a label that identifies each part unit to a partial image that is at the same position as that part unit. Furthermore, processor 12 may assign some kind of label (for example, a label indicating that there is no corresponding model part) to a partial image that is at a position different from any part unit.

[0076] After step S303, the information processing device 10 calculates the index (S304). Specifically, the processor 12 refers to the result of the association in step S303 and calculates a progress index relating to the progress of construction for each model part.

[0077] After step S304, the information processing apparatus 10 generates an output image (S305). Specifically, processor 12 generates an output image by referring to the association result in step S303 and the progress index calculated in step S304. As an example, processor 12 combines the site image and the 3D model by attaching each subset included in the site image to the part unit associated with that subset. Processor 12 generates an output image by combining the image obtained by combining the site image and the 3D model with an image representing the progress index calculated in step S304.

[0078] After step S305, the information processing device 10 displays the output image (S306). Specifically, the processor 12 displays the output image generated in step S305 on the display via the input / output interface 13.

[0079] After step S306, the information processing device 10 ends the information processing of FIG.

[0080] (3-3) Summary As described above, the information processing device of the third embodiment acquires site images of a construction site related to a building, aligns the site images with a 3D model, and associates a subset of the site images with part units included in the model parts of the 3D model. The information processing device calculates progress indicators related to the progress of construction for the model parts and generates an output image that combines the 3D model, the site images, and an image representing the progress indicators. Therefore, a user can intuitively grasp the progress of construction by viewing a 3D rendering of the target building and the completed state at the time the site images were captured on a single screen, and can quantitatively grasp the progress of construction for each model part by visually checking the progress indicators. In other words, this information processing device allows a user to manage the construction process without directly observing the target building during construction.

[0081] The information processing device of the third embodiment can refer to the result of associating the subset of the site image with the part unit (that is, the label) and individually control the display mode of each part unit in the output image.

[0082] For example, the information processing device determines the display form of a first part unit of the model parts associated with a subset of site images to be different from the display form of a second part unit of the model parts that is not associated with the subset of site images. This makes the part units of the model parts associated with the subset of site images, i.e., the constructed parts, stand out in the output image, making it easier to understand the progress of construction on the model parts. In other words, this information processing device allows the user to manage the construction process without directly observing the target building under construction.

[0083] (4) Fourth embodiment A fourth embodiment will be described below, but the same description as in the above embodiments will be omitted.

[0084] (4-1) Overview of the embodiment An outline of the fourth embodiment will be described below with reference to Fig. 10, which is an explanatory diagram of the outline of the fourth embodiment.

[0085] As shown in FIG. 10, the sensor 30 captures an image of a construction site CS(t12) at time t12 and generates a site image SI(t12). The information processing device 10 acquires the site image SI(t12) transmitted from the sensor 30 or an image server (not shown). At the construction site CS(t12), part P1 and part P2 of the target building have been constructed. The part P1 corresponds to a partial image PIM1(t12), which is a subset of the site image SI2. The constructed part of the part P2 corresponds to a partial image PIM2(t12), which is a subset of the site image SI2.

[0086] A site image SI(t11) generated by photographing a construction site CS(t11) at time t11 is stored in the storage device 11. At the construction site CS(t11), part P1 of the target building has been constructed, but part P2 has not yet been constructed.

[0087] The information processing device 10 aligns the on-site image SI(t11) and the on-site image SI(t12). As an example, the information processing device 10 converts the position of each information element (point or pixel) included in at least one of the on-site image SI(t11) or the on-site image SI(t12) into a position in a reference coordinate system. In the fourth embodiment, the reference coordinate system may be any of the following: - Coordinate system of the 3D model of the target building Coordinate system of the on-site image SI(t11) Coordinate system of the on-site image SI(t12)

[0088] The information processing device 10 refers to the alignment result and identifies the difference between the on-site image SI(t11) and the on-site image SI(t12). The information processing device 10 generates an output image OI(t12) by referring to the on-site image SI(t12) so that a portion of the on-site image SI(t12) corresponding to the difference and a portion not corresponding to the difference are displayed in different forms, and displays the output image OI(t12) on the display. Specifically, the difference between the on-site image SI(t11) and the on-site image SI(t12) is the partial image PIM2(t12). Therefore, for example, the brightness of the partial image PIM2(t12) is set high so that the partial image PIM2(t12) stands out in the output image OI(t12).

[0089] The output image OI(t12) allows the user to intuitively understand what changes have occurred at the construction site between time t11 and time t12. For example, the user can understand the progress of construction work and the status (e.g., installation, removal, inventory increase / decrease, or operation) of objects (e.g., hazardous materials, materials, construction machinery).

[0090] (4-2) Information Processing The information processing of the fourth embodiment will be described below with reference to Fig. 11, which is a diagram showing the overall flow of the information processing of the fourth embodiment.

[0091] The information processing starts when a user inputs an instruction to start the information processing into the information processing device 10 or when other start conditions are met.

[0092] As shown in FIG. 11, the information processing device 10 acquires a site image (S401). Specifically, the processor 12 acquires site images from the sensor 30 or an image server (not shown) via the communication interface 14 .

[0093] After step S401, the information processing device 10 performs registration between the site images (S402). Specifically, the processor 12 aligns the first site image (e.g., the site image stored in the storage device 11) and the second site image (e.g., the site image acquired in step S401) that were captured at different times. As an example, the processor 12 converts the position of each information element included in at least one of the first site image or the second site image into a position in a reference coordinate system.

[0094] After step S402, the information processing device 10 executes the step of identifying the difference (S403). Specifically, the processor 12 refers to the result of the registration in step S402 and identifies the difference between the second scene image and the first scene image.

[0095] After step S403, the information processing apparatus 10 generates an output image (S404). Specifically, the processor 12 generates the output image by referring to the second site image so that a portion of the second site image that corresponds to the difference identified in step S403 and a portion that does not correspond to the difference are displayed in different forms. As an example, the processor 12 generates the output image by emphasizing the portion of the second site image that corresponds to the difference.

[0096] After step S404, the information processing device 10 displays the output image (S405). Specifically, the processor 12 displays the output image generated in step S404 on the display via the input / output interface 13.

[0097] After step S405, the information processing device 10 ends the information processing of FIG.

[0098] (4-3) Summary As described above, the information processing device of the fourth embodiment acquires a first site image and a second site image taken at different times of a construction site for a building, and aligns the first site image with the second site image. The information processing device identifies differences between the first site image and the second site image, and generates an output image in which portions of the second site image corresponding to the differences are displayed in different formats from portions that do not correspond to the differences. Therefore, with this information processing device, a user can intuitively grasp what changes occurred at the construction site between the time the first site image was taken and the time the second site image was taken. In other words, with this information processing device, a user can perform progress management without directly observing the target building during construction, and safety management without directly observing the site.

[0099] (5) Fifth embodiment A fifth embodiment will be described below, but the same description as in the above embodiments will be omitted.

[0100] (5-1) Overview of the embodiment An outline of the fifth embodiment will be described below. Fig. 12 is an explanatory diagram of the outline of the fifth embodiment.

[0101] A four-dimensional model of the target building is stored in the storage device 11. As an example, the four-dimensional model may be a model in which time information is added to the three-dimensional model described in any of the above embodiments. The four-dimensional model may be, for example, a four-dimensional BIM.

[0102] As shown in FIG. 12, the information processing device 10 refers to time information included in the four-dimensional model and identifies an intermediate goal IG(t21) that should be achieved at the construction site by a time t21 (hereinafter referred to as the "target time") at which an index is to be calculated. The sensor 30 captures an image of the construction site at the target time t21 and generates a site image SI(t21). The information processing device 10 acquires the site image SI(t21) transmitted from the sensor 30 or an image server (not shown). The information processing device 10 refers to the site image SI(t21) and determines whether the intermediate goal IG(t21) has been achieved. The information processing device 10 calculates an index IN(t21) related to the degree of achievement of the intermediate goal IG(t21) based on the determination result. The information processing device 10 presents the index IN(t21) to the user. As an example, the information processing device 10 generates an output image OI(t21) including the index IN(t21) and displays it on a display.

[0103] The output image OI(t21) allows the user to grasp the degree of achievement of the intermediate goal that should be achieved by the target time t(t21) at the construction site.

[0104] (5-2) Information Processing The information processing of the fifth embodiment will be described below with reference to Fig. 13, which is a diagram showing the overall flow of the information processing of the fifth embodiment.

[0105] The information processing starts when a user inputs an instruction to start the information processing into the information processing device 10 or when other start conditions are met.

[0106] As shown in FIG. 13, the information processing device 10 executes image acquisition (S501). Specifically, the processor 12 acquires site images from the sensor 30 or an image server (not shown) via the communication interface 14 .

[0107] After step S501, the information processing device 10 executes identification of an intermediate goal (S502). Specifically, the processor 12 refers to the 4D model and identifies intermediate goals to be achieved at the construction site by a target time. The number of intermediate goals is not limited to one, and may be multiple. The intermediate goals may include, for example, at least one of the following: - The specified parts or the building materials (building materials include surface materials or paints; the same applies below) that make up the parts have a specified appearance (for example, shape, color, pattern). - The specified parts or the building materials that make up the parts are installed in the specified positions. - The specified parts or the building materials that make up the parts are installed in the specified position. - The specified part or the building materials that make up the part are in a specified state (for example, the state in which concrete, paint, or adhesive is dry). - The specified parts pass the specified inspection.

[0108] The predetermined appearance, position, condition, or inspection may be identifiable from, for example, a four-dimensional model or inspection reference data, or may be specified by a user.

[0109] After step S502, the information processing device 10 executes a determination of whether the intermediate goal has been achieved (S503). Specifically, the processor 12 refers to the site image acquired in step S501 and determines whether the intermediate goal identified in step S502 has been achieved. If multiple intermediate goals have been identified in step S502, the processor 12 determines whether each of the intermediate goals has been achieved individually.

[0110] In a first example of determining whether an intermediate goal has been achieved (S503), the information processing device 10 determines whether a part is complete. Specifically, the processor 12 extracts a model part corresponding to the part from the 4D model. The processor 12 aligns the site image with the model part. The processor 12 refers to the alignment result and associates a subset of the site image with each part unit obtained by dividing the model part. The processor 12 refers to the association result and determines the degree of completion of the model part. As an example, the processor 12 calculates the proportion of part units included in the model part that are associated with the subset of the site image as the degree of completion.

[0111] In a second example of determining whether or not a building material (e.g., a steel frame) has been installed in a correct position, the information processing device 10 determines whether or not the building material (e.g., a steel frame) has been installed in a correct position. Specifically, the processor 12 extracts, from the four-dimensional model, model parts corresponding to the building material that should have been installed by the target time. The processor 12 aligns the site image with the model parts. The processor 12 searches for a partial image corresponding to the model part by referring to the alignment result. If the processor 12 cannot find a partial image corresponding to the model part (i.e., the building material has not yet been installed or is installed in an incorrect position), the processor 12 determines that the intermediate goal has not been achieved. If the processor 12 finds a partial image corresponding to the model part (i.e., the building material has been installed in a correct position), the processor 12 determines that the intermediate goal has been achieved.

[0112] In a third example of the intermediate goal achievement determination (S503), the information processing device 10 determines whether the object (e.g., paint or adhesive) applied to the surface of the part is dry. Specifically, the processor 12, for example, refers to a site image captured before the target time to identify the time when the object was applied. The processor 12 may refer to some kind of sensing data or a user instruction instead of the past site image. The processor 12 determines the degree of drying of the object by comparing the elapsed time from the time when the object was applied to the target time with a standard time. Here, the standard time is a standard time required for the object to dry after being applied (i.e., exposed to the outside air). The standard time is determined based on, for example, the physical properties of the object, the amount of application of the object, the environment surrounding the object (e.g., temperature, humidity, wind speed, or wind direction), or a combination thereof. The processor 12 determines that the intermediate goal has not been achieved if the elapsed time is less than the standard time (i.e., the object is considered not to be dry yet). The processor 12 determines that the intermediate goal has been achieved if the elapsed time is equal to or greater than the standard time (i.e., the object is considered to be dry). The processor 12 may quantify the degree of dryness of the object by dividing the elapsed time by the required time.

[0113] In a fourth example of determining whether the intermediate goal has been achieved (S503), the information processing device 10 determines whether the parts have passed the inspection. The parts to be inspected may be some of the parts constituting the target building (i.e., a sampling inspection) or all of them (i.e., a full inspection). In the case of a sampling inspection, the parts to be inspected may be randomly selected or manually selected. Specifically, the processor 12 refers to the inspection standard data and identifies the specifications required for the parts. The specifications are, for example, required levels regarding the appearance (e.g., dimensions) of the parts, the position of the parts, the posture of the parts, the condition of the parts, or a combination thereof. The processor 12, for example, refers to an on-site image taken at a target time and the specifications and inspects whether the parts satisfy the specifications. If the parts satisfy the specifications (i.e., the parts have passed the inspection), the processor 12 determines that the intermediate goal has been achieved. If the parts do not satisfy the specifications (i.e., the parts have not passed the inspection), the processor 12 determines that the intermediate goal has not been achieved.

[0114] After step S503, the information processing device 10 calculates the index (S504). Specifically, the processor 12 refers to the determination result in step S503 and calculates an index relating to the degree of achievement of the intermediate goal identified in step S502. The index may be a binary index (e.g., achieved / not achieved) or a multi-value index of three or more values (e.g., percentage). When multiple intermediate goals are identified in step S502, the processor 12 may calculate an index for each intermediate goal, or may calculate an index (overall degree of achievement) across the multiple intermediate goals.

[0115] After step S504, the information processing apparatus 10 generates an output image (S505). Specifically, the processor 12 generates an output image including the indices calculated in step S504.

[0116] After step S505, the information processing device 10 displays the output image (S506). Specifically, the processor 12 displays the output image generated in step S505 on the display via the input / output interface 13.

[0117] After step S506, the information processing device 10 ends the information processing of FIG.

[0118] (5-3) Summary As described above, the information processing device of the fifth embodiment refers to a 4D model in which time information is added to a 3D model of a target building, and identifies intermediate goals that should be achieved by a target time at the construction site of the target building. The information processing device refers to site images taken of the construction site and determines whether the intermediate goals have been achieved. Depending on the determination result, the information processing device calculates an index related to the degree of achievement of the intermediate goals and presents the index to the user. Therefore, the user can grasp the degree of achievement of the intermediate goals that should be achieved by a target time at the construction site. In other words, this information processing device allows the user to manage the construction process without directly observing the target building during construction.

[0119] (6) Other variations The storage device 11 may be connected to the information processing device 10 via a network NW.

[0120] The third embodiment illustrates an example in which a progress index related to the progress of construction work on a model part is calculated. The information processing device 10 can also calculate an index other than the progress index. For example, the information processing device 10 may refer to multiple site images taken at different times of a construction site, calculate an operation index related to the work of workers at the construction site, and generate an output image that includes the operation index. Specifically, the information processing device 10 performs image recognition on each site image to analyze the worker's posture type (e.g., posture during work, posture during rest). A score is assigned to each posture type in advance. The information processing device 10 can calculate the operation index by adding up the scores corresponding to the posture types analyzed from each site image. This allows a construction manager to quantitatively grasp the work of workers. In other words, this information processing device allows a user to manage costs without directly observing workers.

[0121] The information processing device 10 may refer to the progress indicators and operation indicators to calculate an efficiency indicator (e.g., a unit rate) relating to the efficiency of construction work on the model parts, and generate an output image that includes the efficiency indicator. As an example, the unit rate can be calculated based on the percentage of work that has been completed out of the total work, or the percentage of work that has not been completed out of the total work. This allows the construction manager to quantitatively grasp the efficiency of construction work on the model parts. In other words, this information processing device allows the user to perform cost management without directly observing the target building and workers.

[0122] In the third embodiment, an example was shown in which the part units included in the model parts have a rectangular shape, but the shape of the part units is not limited to a rectangular shape.

[0123] In the fourth embodiment, an example has been described in which the output image OI12 is generated with reference to the on-site image SI12 so that the portions of the on-site image SI12 that correspond to the differences and the portions that do not correspond to the differences have different display forms. However, the information processing device 10 may generate the output image with reference to the on-site image SI11 so that the portions of the on-site image SI11 that correspond to the differences and the portions that do not correspond to the differences have different display forms.

[0124] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments. Furthermore, the above-described embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined.

[0125] (7) Supplementary Notes The matters explained in the embodiment and the modified examples are additionally noted below.

[0126] (Appendix 1) A means for acquiring site images of a construction site relating to a building (S101, S201, S301); A means for aligning the on-site image with a 3D model of the building (S102, S202, S302); A means for generating an output image by synthesizing the 3D model and the on-site image with reference to the alignment result (S103, S204, S305); An information processing device (10) comprising:

[0127] (Appendix 2) The 3D model includes multiple model parts, The information processing device further includes means for associating a subset of on-site images with each model part included in the 3D model by referring to the result of the alignment (S203, S303); The generating means (S204, S305) refers to the result of the association and generates an output image. 2. The information processing device according to claim 1.

[0128] (Appendix 3) The generating means generates the output image by referring to the result of the association and individually controlling whether or not each subset of the on-site image is reflected in the output image and the display mode when the subset is reflected in the output image. 3. The information processing device according to claim 2.

[0129] (Appendix 4) The associating means associates each part unit included in the model part. 4. The information processing device according to claim 2 or 3.

[0130] (Appendix 5) the generating means generates the output image such that a first part unit, which is associated with the subset of the site image among the part units constituting the model part, and a second part unit, which is not associated with the subset of the site image among the part units constituting the model part, have different display forms; 5. The information processing device according to claim 4.

[0131] (Appendix 6) The system further includes a means (S304) for referring to the result of association for each part unit included in the model part and calculating a progress index relating to the progress of construction for the model part. 5. The information processing device according to claim 4.

[0132] (Appendix 7) The progress indicator is the percentage of part units included in the model part that are associated with a subset of site images. 7. The information processing device according to claim 6.

[0133] (Appendix 8) The construction site management system further includes a means for calculating an operation index related to the operation of workers at the construction site by referring to a plurality of site images taken at different times of the construction site. 8. The information processing device according to claim 6 or 7.

[0134] (Appendix 9) The system further includes a means for calculating an efficiency index relating to the efficiency of construction work on the model part by referring to the progress index and the operation index. 9. The information processing device according to claim 8.

[0135] (Appendix 10) A means for aligning a first site image and a second site image taken at different times of a construction site relating to a building (S402); a means for referencing the aligned first and second site images and determining a difference between the second site image and the first site image (S403); a means (S404) for generating an output image by referring to the second on-site image so that a portion of the second on-site image corresponding to the difference and a portion of the second on-site image not corresponding to the difference are displayed in different formats; An information processing device (10) comprising:

[0136] (Appendix 11) A means for acquiring site images of a construction site relating to a building (S501); A means for identifying an intermediate goal to be achieved by a target time at the construction site by referring to a four-dimensional model in which time information is added to a three-dimensional model of the building (S502); a means for determining whether or not an intermediate goal has been achieved by referring to the site image (S503); a means for calculating an index relating to the degree of achievement of the intermediate goal according to the determination result of whether the intermediate goal has been achieved (S504); An information processing device (10) comprising:

[0137] (Appendix 12) A computer (10) Acquiring site images of a construction site relating to a building (S101, S201, S301); Aligning the site image with a 3D model of the building (S102, S202, S302); By referring to the alignment result, an output image is generated by combining the 3D model and the on-site image (S103, S204, S305). An information processing method comprising:

[0138] (Appendix 13) A computer (10) Aligning a first site image and a second site image taken at different times of a construction site relating to a building (S402); Referring to the aligned first and second site images, determining a difference between the second site image and the first site image (S403); generating an output image by referring to the second on-site image so that a portion of the second on-site image corresponding to the difference and a portion of the second on-site image not corresponding to the difference are displayed in different formats (S404); An information processing method comprising:

[0139] (Appendix 14) A program for causing a computer (10) to realize each of the means described in any one of Supplementary Note 1 to Supplementary Note 11. [Explanation of symbols]

[0140] 1: Information processing system 10: Information processing device 11:Storage device 12: Processor 13: Input / output interface 14: Communication interface 30: Sensor

Claims

1. A means for acquiring site images of a construction site relating to a building; means for registering the site image with a three-dimensional model of the building; a means for generating an output image by combining the three-dimensional model and the on-site image with reference to the result of the registration; An information processing device comprising:

2. the three-dimensional model includes a plurality of model parts; the information processing device further comprises means for associating the subset of on-site images with each model part included in the three-dimensional model by referring to the result of the registration; the generating means generates the output image by referring to the result of the association. The information processing device according to claim 1 .

3. the generating means generates the output image by referring to the result of the association and individually controlling whether or not each subset of the on-site image is reflected in the output image and a display mode when the subset is reflected in the output image. The information processing device according to claim 2 .

4. the associating means performs the association for each part unit included in the model part.

4. The information processing device according to claim 2.

5. the generating means generates the output image such that a first part unit, which is associated with the subset of the site image among the part units constituting the model part, and a second part unit, which is not associated with the subset of the site image among the part units constituting the model part, have different display forms. The information processing device according to claim 4 .

6. The method further includes a means for calculating a progress index relating to the progress of construction of the model part by referring to the result of association for each part unit included in the model part. The information processing device according to claim 4 .

7. the progress indicator is a percentage of part units included in the model part that are associated with the subset of site images; The information processing device according to claim 6 .

8. The construction site management system further includes a means for calculating an operation index related to the operation of workers at the construction site by referring to a plurality of site images taken at different times of the construction site.

8. The information processing device according to claim 6 or 7.

9. The method further includes a means for calculating an efficiency index relating to efficiency of construction work on the model part by referring to the progress index and the operation index. The information processing device according to claim 8 .

10. a means for aligning a first site image and a second site image taken at different times of a construction site relating to a building; means for referencing the registered first and second site images and determining differences between the second site image and the first site image; means for generating an output image by referring to the second site image so that a portion of the second site image corresponding to the difference and a portion not corresponding to the difference are displayed in different formats; An information processing device comprising:

11. A means for acquiring site images of a construction site relating to a building; a means for identifying an intermediate goal to be achieved by a target time at the construction site by referring to a four-dimensional model obtained by adding time information to the three-dimensional model of the building; a means for determining whether the intermediate goal has been achieved by referring to the site image; a means for calculating an index relating to the degree of achievement of the intermediate goal according to a determination result as to whether the intermediate goal has been achieved; An information processing device comprising:

12. The computer Acquiring site images of a construction site relating to a building; aligning the site image with a three-dimensional model of the building; generating an output image by combining the three-dimensional model and the on-site image with reference to the result of the alignment; An information processing method comprising:

13. The computer Aligning a first site image and a second site image taken at different times of a construction site relating to a building; referencing the registered first and second site images and determining differences between the second site image and the first site image; generating an output image by referring to the second site image so that a portion of the second site image corresponding to the difference and a portion of the second site image not corresponding to the difference are displayed in different formats; An information processing method comprising:

14. A program for causing a computer to realize each of the means set forth in any one of claims 1 to 11.

Images