Program, information processing method, and information processing apparatus

Abstract

To provide a technique capable of reducing a processing load in inspection of a building.SOLUTION: A program causing an information processing apparatus to execute: acquiring design data relating to a predetermined space; acquiring a distance image relating to the space; comparing the design data and the distance image; and outputting a comparison result.SELECTED DRAWING: Figure 15

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing method, and an information processing device. [Background technology]

[0002] Conventionally, at a construction site, inspections are carried out to determine whether the completed area conforms to the shape and dimensions of the design drawings. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Shinya Yamaoka, Osamu Kanai, Hiroaki Date, "Efficient Matching and Difference Point Extraction of Indoor Environment Laser Measurement Point Clouds to BIM Components Using RANSAC," Proceedings of the JSPE Annual Conference, vol. 2013 JSPE Spring Meeting, 2013, p. 641. [Non-patent document 2] M. Bueno, F. Bosche, H. Gonzalez-Jorge, J. Martinez-Sanchez, and P. Arias, "4-plane congruent sets for automatic registration of AS-IS 3D point clouds with 3d bim models," Automation in Construction, vol. 89, 2018, pp.120-134. Summary of the Invention [Problem to be solved by the invention]

[0004] Inspecting buildings by visually comparing design drawings with completed areas requires a great deal of time and effort. Therefore, in recent years, an inspection method has been adopted in which point cloud data of completed areas is acquired using a laser scanner and the point cloud data is compared with design data. For example, the methods shown in Non-Patent Documents 1 and 2 are known as inspection methods for comparing point cloud data of completed areas with design data.

[0005] However, inspection methods that compare point cloud data of completed areas with design data impose a heavy processing load because a huge number of points in the point cloud data must be compared with the design data.

[0006] Therefore, the disclosed technology aims to provide a technology that can reduce the processing burden in building inspections. [Means for solving the problem]

[0007] A program that is one aspect of the disclosed technology causes an information processing device to acquire design data related to a specified space, acquire a distance image related to the space, compare the design data with the distance image, and output the comparison results. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the processing load in building inspections. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a screen showing the results of the finished product inspection. [Figure 3] FIG. 3 is a block diagram showing an example of an information processing device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of three-dimensional CAD data. [Figure 5] FIG. 5 is a diagram showing an example of a distance image. [Figure 6] FIG. 6 is a diagram for explaining distance information. [Figure 7] FIG. 7 is a diagram illustrating an example of a process performed by the first generation unit. [Figure 8] FIG. 8 is a diagram showing an example of a distance image. [Figure 9] FIG. 9 is a diagram illustrating an example of a process performed by the first generation unit. [Figure 10] FIG. 10 is a diagram illustrating an example of processing by the comparison unit. [Figure 11] FIG. 11 is a diagram illustrating an example of a process for moving the second plane data. [Figure 12] FIG. 12 is a diagram illustrating an example of pre-processing for calculating the difference between the first planar data and the second planar data. [Figure 13] FIG. 13 is a diagram illustrating an example of a process for calculating the difference between the first planar data and the second planar data. [Figure 14] FIG. 14 is a diagram illustrating an example of processing by the second generation unit. [Figure 15] FIG. 15 is a flowchart showing an example of processing by the information processing device according to this embodiment. [Figure 16] FIG. 16 is a diagram showing an example of how the results of the finished product inspection according to this embodiment are reflected on the screen. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The following embodiments are examples for explaining the present invention, and are not intended to limit the present invention to only these embodiments. Furthermore, the present invention can be modified in various ways without departing from the gist of the invention. Furthermore, the same components in each drawing will be designated by the same reference numerals whenever possible, and redundant explanations will be omitted whenever possible.

[0011] <System Overview> Fig. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment of the present invention. In the example shown in Fig. 1, the information processing system 1 illustratively includes an imaging device 10, an information processing device 20, and a network N. Furthermore, the number of imaging devices 10 and information processing devices 20 may be one or more.

[0012] In Fig. 1, a worker U uses an imaging device 10 and an information processing device 20 to perform an inspection (hereinafter also referred to as "finished product inspection") of a building B to check whether the completed sections are suitable for the shape and dimensions as per the design drawings. The building B is composed of, for example, walls (e.g., walls W1 and W2), a floor, and a ceiling (not shown). There may be multiple workers U.

[0013] The imaging device 10 is, for example, an electronic camera capable of depth sensing that is used by a worker U when inspecting the finished product. The imaging device 10 generates, for example, a distance image relating to the distance between an object to be imaged and the viewpoint of the camera. A detailed description of distance images will be given later. The imaging device 10 measures the distance between an object to be imaged and the viewpoint of the camera by, for example, irradiating the object with infrared rays. More specifically, the imaging device 10 measures the distance between the object to be imaged and the viewpoint of the camera based on the time it takes for the irradiated infrared rays to reflect off the object and return to the camera.

[0014] The information processing device 20 is, for example, a personal computer, a smartphone, a tablet terminal, or the like used by a worker U when performing an as-built inspection. The information processing device 20 acquires a distance image from the imaging device 10, for example, via a network N. The information processing device 20 outputs the results of the as-built inspection through processing by an information processing unit, which will be described later. Furthermore, the information processing device 20 does not necessarily have to be used by the worker U. The information processing device 20 may be used by a user other than the worker U, who is, for example, outside the building B. Furthermore, the functions of the imaging device 10 and the information processing device 20 may be integrated into a single information processing device.

[0015] In conventional inspection methods such as those described in Non-Patent Documents 1 and 2, each point in point cloud data of a completed building acquired using a laser scanner was matched to a Building Information Modeling (BIM) model, and points that could not be matched to the BIM model were considered differences from the BIM model. BIM is a computer-generated 3D model of a building or other structure that is identical to the real world. However, conventional inspection methods required a large processing load because they had to compare the vast number of points in the point cloud data with the BIM model. Furthermore, conventional inspection methods required installing laser scanners at multiple locations on the construction site and acquiring point cloud data, which took approximately 10 minutes. Furthermore, laser scanner-based methods required the installation of black-and-white patterned boards called markers at the construction site prior to inspection. Therefore, conventional inspection methods required a significant amount of time to install the markers, acquire point cloud data, and match the point cloud data with the BIM model, making real-time inspection impossible.

[0016] In contrast, in the present invention, the information processing device 20 acquires 3D CAD data related to a predetermined space (e.g., a space within building B). The information processing device 20 acquires a distance image related to the predetermined space (e.g., a space within building B). The information processing device 20 detects first plane data related to the space (e.g., a space within building B) from the 3D CAD data. The information processing device 20 generates second plane data related to a predetermined plane of the space (e.g., a space within building B) based on the distance image. The information processing device 20 compares the first plane data with the second plane data. The information processing device 20 outputs the comparison result. That is, in the present invention, since data related to the planes (the first plane data and the second plane data) are compared, the processing load can be reduced compared to conventional inspection methods that compare a huge number of points in point cloud data with a BIM model. Furthermore, since the imaging device 10 can instantly capture distance images, the information processing device 20 can generate data related to the plane (the second plane data) based on the distance image and compare it with the first plane data in real time. Furthermore, in the present invention, since there is no need to use markers, it is possible to reduce the labor required for placing markers.

[0017] FIG. 2 is an example of a screen showing the results of an as-built inspection output by the information processing device 20. The information processing device 20 displays, for example, a screen A10 of the as-built inspection shown in FIG. 2 on a screen provided in the information processing device 20. The information processing device 20 visibly displays, for example, on the screen A10, a comparison result between design data for a building B and a distance image. The design data is, for example, 3D CAD data for the building B. Wall CW1 is a wall in the 3D CAD data and corresponds to wall W1 shown in FIG. 1. Wall CW2 is a wall in the 3D CAD data and corresponds to wall W2 shown in FIG. 1. In the example shown in FIG. 2, the information processing device 20 visibly displays the comparison result on wall CW1. As a result, the comparison result of the walls in the screen region B10 (for example, the difference between the distance image for wall CW1 and wall W1) is displayed, for example, in warm colors if the wall in the screen region B10 protrudes in front of the walls other than the screen region B10, in cool colors if the wall is recessed, or in gray if the positions of the two walls are approximately the same. Therefore, when the wall in screen area B10 is displayed in a warm color or a cool color, worker U can understand that the portion of wall W1 corresponding to screen area B10 has not been constructed as designed (for example, the portion of wall W1 corresponding to screen area B10 protrudes or is recessed compared to the portion of wall W1 corresponding to an area other than screen area B10). Note that information processing device 20 may output the results of the finished product inspection on a screen provided in an external information processing device.

[0018] The network N is realized by, but is not limited to, a network such as the Internet or a mobile phone network, a LAN (Local Area Network), or a network that combines these.

[0019] <Configuration of information processing device 20> 3 is a block diagram showing an example of an information processing device 20 according to an embodiment of the present disclosure. The information processing device 20 includes one or more processors (e.g., CPUs) 210, one or more network communication interfaces 220, a storage device (storage unit) 230, and one or more communication buses 250 for interconnecting these components.

[0020] The information processing device 20 may optionally include a user interface 240. The user interface 240 includes a display and / or an input device (such as a keyboard and / or a mouse, or some other pointing device).

[0021] The storage device 230 may be, for example, a high-speed random access memory (main storage device) such as a DRAM, an SRAM, or other random access solid-state storage device. Alternatively, the storage device 230 may be a non-volatile memory (auxiliary storage device) such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Alternatively, the storage device 230 may be a non-transitory computer-readable recording medium that stores programs and the like. Alternatively, the storage device 230 may be either a main storage device (memory) or an auxiliary storage device (storage), or may include both.

[0022] The storage device 230 is a storage unit that stores data, programs, etc. used by the information processing system 1. The storage device 230 may also store, for example, design data, distance images, and results of the as-built inspection related to the building B on which the worker U is to perform an as-built inspection.

[0023] Another example of storage device 230 may be one or more storage devices located remotely from processor 210. In some embodiments, storage device 230 stores programs, modules, and data structures, or a subset thereof, that are executed by processor 210.

[0024] The processor 210 executes a program stored in the storage device 230, thereby controlling, for example, the processing executed by each unit of the information processing unit 211.

[0025] The information processing unit 211 includes, for example, an acquisition unit 212, a detection unit 213, a first generation unit 214, a comparison unit 215, an output unit 216, a second generation unit 217, an attachment unit 218, and a third generation unit 219.

[0026] The acquisition unit 212 acquires three-dimensional CAD data relating to a predetermined space. For example, the acquisition unit 212 acquires three-dimensional CAD data relating to a space within a building.

[0027] Fig. 4 is a diagram showing an example of three-dimensional CAD data acquired by the acquisition unit 212. In the example shown in Fig. 4, the acquisition unit 212 acquires three-dimensional CAD data related to building B shown in Fig. 1. The three-dimensional CAD data shown in Fig. 4 includes a wall CW1 (corresponding to wall W1 shown in Fig. 1), a wall CW2 (corresponding to wall W2 shown in Fig. 1), and a floor (corresponding to the floor of building B shown in Fig. 1). As a preferred example, the angle between wall CW1 and wall CW2 is a right angle. As a preferred example, the angle between wall CW1 and the floor is a right angle. As a preferred example, the angle between wall CW2 and the floor is a right angle.

[0028] The acquiring unit 212 may acquire the three-dimensional CAD data by reading out the three-dimensional CAD data stored in advance in the storage device 230, for example.

[0029] The acquiring unit 212 may acquire, for example, information indicating that each of the wall CW1, the wall CW2, and the floor includes a flat, horizontal surface (hereinafter also referred to as "horizontal information"). Note that the horizontal information may be acquired in association with three-dimensional CAD data.

[0030] The acquisition unit 212 acquires a distance image relating to a space. For example, the acquisition unit 212 acquires a distance image relating to a space inside a building. The distance image is an image including information relating to the distance between an object to be imaged (for example, building B shown in FIG. 1) and the viewpoint of an imaging device (for example, imaging device 10 shown in FIG. 1) that images the object to be imaged. More specifically, the distance image includes information relating to the distance between the viewpoint of the imaging device and the object to be imaged included in each pixel (hereinafter also referred to as "distance information") for each pixel (hereinafter also referred to as "pixel"). The distance information includes, for example, information such as "The distance between the object to be imaged and the viewpoint of the imaging device is XX mm."

[0031] Fig. 5 is a diagram showing an example of a distance image acquired by the acquisition unit 212. In the example shown in Fig. 5, the acquisition unit 212 acquires a distance image C10 of a wall W1 captured by the worker U shown in Fig. 1 using the imaging device 10. In the distance image C10, pixels in an image region D10 are shown in a color different from that of other pixels. That is, the distance value indicated by the distance information included in the pixels in the image region D10 is different from the distance value indicated by the distance information included in the pixels outside the image region D10.

[0032] Fig. 6 is a diagram illustrating distance information. In the example shown in Fig. 6, distance image C10 includes, as distance information, information about the distance d between the pixel at point PI and a virtual viewpoint PE of image capture device 10. Each pixel in distance image C10 may also include distance information. That is, distance image C10 may include, as distance information, information about the distance between the viewpoint PE and a pixel other than the pixel at point PI.

[0033] The detection unit 213 detects first plane data relating to the space from the three-dimensional CAD data acquired by the acquisition unit 212. The detection unit 213 detects, for example, a wall CW1 (hereinafter also referred to as "first plane data CW1") from the three-dimensional CAD data shown in FIG. 4. As a non-limiting example, the detection unit 213 may detect the first plane data from the three-dimensional CAD data based on horizontal information associated with the three-dimensional CAD data. Note that the detection method is not particularly limited as long as it can detect a plane from the three-dimensional CAD data.

[0034] The first generator 214 generates second plane data relating to a predetermined plane in space based on the distance image. The first generator 214 generates second plane data relating to the wall W1 based on, for example, the distance image C10.

[0035] The first generation unit 214 may generate the second planar data based on, for example, the distance between the viewpoint of an imaging device that captures the distance image and each of a plurality of pixels included in the distance image. The first generation unit 214 may generate the second planar data based on, for example, distance information included in each pixel in the distance image C10.

[0036] Fig. 7 is a diagram showing an example of the processing of first generation unit 214. In the example shown in Fig. 7, first generation unit 214 groups adjacent pixels having similar distance information included in distance image C10 acquired by acquisition unit 212 into one group (hereinafter also referred to as "superpixels"). Note that "adjacent pixels having similar distance information" refers to adjacent pixels whose distance information indicates a difference of less than a few mm (for example, but not limited to, 5 mm), for example.

[0037] Image C20 shown in FIG. 7 is an image obtained by performing grouping processing on distance image C10 by first generation unit 214. For example, the area surrounded by a black frame in image C20 is one superpixel. For example, superpixel SP1 in image C20 includes one or more adjacent pixels with similar distance information. For example, first generation unit 214 may create superpixels using the SLIC algorithm, a well-known technique, without limitation.

[0038] If the 3D CAD data specifies that a wall corresponding to wall W1 is flat and free of irregularities, and wall W1 is not constructed as designed (e.g., if irregularities exist on the wall surface), image C20 will contain superpixels related to the irregularities. Therefore, the first generation unit 214, for example, determines the planarity of each superpixel included in image C20. "Determining the planarity of a superpixel" means, for example, using a predetermined statistical analysis on image C20 to determine that the superpixel is planar if a predetermined mathematical condition is satisfied, and determining that the superpixel is non-planar if the condition is not satisfied. For example, and without limitation, the first generation unit 214 may perform principal component analysis, a known technique, on image C20 to determine the planarity of each superpixel.

[0039] For example, the first generation unit 214 extracts superpixels determined to be planar from the image C20 after the planarity determination process for each superpixel has been performed. For example, the first generation unit 214 may extract superpixels determined to be planar by removing superpixels determined to be non-planar from the image C20 after the planarity determination process for each superpixel has been performed.

[0040] Image C30 is an image from which superpixels determined to be planes are extracted by first generation unit 214. In image C30, superpixels other than those corresponding to pixels in image region D10 of distance image C10 shown in FIG.

[0041] The first generation unit 214 generates the second plane data based on, for example, the extracted superpixels. For example, the first generation unit 214 generates the second plane data C40 by grouping the superpixels included in the image C30 determined to be a plane into data related to a single plane. This enables the information processing device 20 to generate the second plane data based on grouping the pixels included in the distance image. Therefore, in the plane data comparison process described below, the first plane data and the second plane data can be compared, reducing the processing load compared to conventional techniques that compare a huge number of points in point cloud data with design data. In other words, by using superpixels in the processing by the first generation unit 214, the comparison result between the first plane data and the second plane data can be obtained more quickly than conventional techniques.

[0042] The first generating section 214 may generate the second plane data based on the distance between the viewpoint of the imaging device that captures the distance image and each of the plurality of pixels included in the distance image, and the normal vectors of the plurality of pixels.

[0043] 1, when image capture device 10 captures an image of wall W1 and wall W2 so that they fit into one frame, two planes (wall W1 and wall W2) fit into the frame. Therefore, first generation unit 214 calculates, for example, a normal vector of a pixel related to wall W1 in the distance image and a normal vector of a pixel related to wall W2 in the distance image.

[0044] FIG. 8 shows an example of a distance image, image D20, captured by imaging device 10 so that walls W1 and W2 fit into a single frame. In image D20, pixels to the right of line L are pixels related to wall W1. In image D20, pixels to the left of line L are pixels related to wall W2. In the example shown in FIG. 8, the distance information of pixels on the right of line L that are adjacent to line L may indicate the same value as the distance information of pixels on the left of line L that are adjacent to line L. In this case, if first generation unit 214 attempts to generate second planar data based solely on the distance information included in each pixel in distance image C10, superpixels related to walls W1 and W2 will be included in the single second planar data. In other words, although planar data based solely on pixels related to wall W1 and planar data based solely on pixels related to wall W2 should be generated separately, such planar data generation is not possible. Therefore, the first generation unit 214, for example, calculates the normal vector of each pixel included in the image D20 and generates the second plane data by further referring to the calculated normal vector in addition to the distance information. That is, as shown in FIG. 9, the normal vector N1 for the wall W1 and the normal vector N2 for the wall W2 have significantly different normal vector orientations (for example, as shown in FIG. 9, the angle α formed by the dashed line extending from the normal vector N1 and the dashed line extending from the normal vector N2 is 90 degrees). Therefore, the first generation unit 214 generates the second plane data based on the orientation of the normal vector of each pixel included in the image D20. With this configuration, it is possible to distinguish between pixels related to the wall W1 and pixels related to the wall W2, and the first generation unit 214 can separately generate plane data based only on pixels related to the wall W1 and plane data based only on pixels related to the wall W2. The angle α is set and changeable so as to be able to distinguish between pixels related to the wall W1 and pixels related to the wall W2.

[0045] The normal vector of each pixel may be calculated by a known mathematical process, for example, by calculating a cross product based on the vectors from a predetermined pixel to a plurality of pixels adjacent to the pixel, and then calculating the normal vector of the predetermined pixel based on the cross product. Note that the calculation method is not particularly limited as long as the normal vector of a pixel included in image D20 can be calculated.

[0046] For example, the first generation unit 214 extracts pixels corresponding to normal vectors of a predetermined direction from among the calculated normal vectors. As a non-limiting example, the first generation unit 214 applies cluster analysis, a known technique, to the calculated normal vectors to group the pixels based on the direction of the normal vectors. As a result, the first generation unit 214 can, for example, divide the normal vectors of the pixels included in the image D20 into a group of normal vectors of pixels related to wall W1 (hereinafter also referred to as "group A") and a group of normal vectors of pixels related to wall W2 (hereinafter also referred to as "group B"). The first generation unit 214 extracts pixels corresponding to normal vectors belonging to group A, for example. Alternatively, the first generation unit 214 may extract pixels corresponding to normal vectors belonging to group B. Note that cluster analysis is not necessarily required as long as pixels corresponding to normal vectors of a predetermined direction can be extracted from among the calculated normal vectors, and any extraction method is acceptable.

[0047] The first generation unit 214 generates the second plane data based on, for example, distance information of pixels including the extracted normal vector. This allows the first generation unit 214 to generate the second plane data based only on, for example, pixels on the right side of the line L. Therefore, even if the imaging device 10 captures images of the walls W1 and W2 so that they fit in one frame, it is possible to generate the second plane data for the wall W1.

[0048] The first generating unit 214 generates second plane data with priority for a plane having a larger area from among a plurality of planes included in the space, based on the distance image.

[0049] The first generation unit 214 may, for example, preferentially generate second plane data for a plane with a larger area from among multiple planes included in the space, based on the number of pixels included in the distance image. For example, if the pixels included in the distance image are grouped based on the direction of their normal vectors as described above, the first generation unit 214 may generate second plane data based on pixels corresponding to normal vectors belonging to the group with the largest number of normal vectors (e.g., group A). ​​Note that the generation method is not particularly limited as long as it allows preferential generation of second plane data for planes with a larger area.

[0050] This allows the information processing device 20 to preferentially generate second plane data relating to a plane with a larger area, thereby enabling a user (e.g., worker U) to easily perform as-built inspection of a wide range of planes (e.g., walls, floors, and ceilings).

[0051] The comparison section 215 compares the first plane data with the second plane data. For example, the comparison section 215 may use the first plane data as a reference and make the second plane data correspond to the first plane data.

[0052] More specifically, the comparison unit 215 may rotate the second plane data in a predetermined direction based on the normal vector of the first plane data until the direction of the normal vector of the second plane data corresponds to the direction of the normal vector of the first plane data. Furthermore, the comparison unit 215 may, for example, move the rotated second plane data to make it correspond to the first plane data.

[0053] 10 is a diagram showing an example of processing by the comparison unit 215. In the example shown in Fig. 10, the comparison unit 215 corresponds the second plane data C40 generated by the first generation unit to the wall CW1 detected by the detection unit 213, based on the wall CW1 (first plane data). For example, the comparison unit 215 rotates the second plane data C40 in a predetermined direction based on the normal vector N10 of the wall CW1 until the orientation of the normal vector N20 of the second plane data C40 corresponds to the orientation of the normal vector N10 of the wall CW1.

[0054] The wall CW1 exists in a three-dimensional orthogonal coordinate system including, for example, the X-axis, the Y-axis, and the Z-axis. Because the wall CW1 is part of the three-dimensional CAD data, the comparison unit 215 can recognize the position of the wall CW1 in the three-dimensional orthogonal coordinate system. In the example shown in FIG. 10, the wall CW1 exists at a position on the XZ plane that is tangent to the X-axis, the Z-axis, and the origin O. The second plane data C40 generated by the first generation unit 214 is set at an arbitrary position parallel to, for example, one of the XY plane, the XZ plane, and the YZ plane of the three-dimensional orthogonal coordinate system. In the example shown in FIG. 10, the generated second plane data C40 is set at an arbitrary position parallel to the XY plane. The comparison unit 215 rotates the second plane data C40 around a line passing through any point on the second plane data C40 (for example, the intersection of the diagonals of the second plane data C40) and parallel to one of the X-axis, Y-axis, and Z-axis as the rotation axis until the orientation of the normal vector N20 of the second plane data C40 corresponds to the orientation of the normal vector N10 of the wall CW1.

[0055] FIG. 11 is a diagram showing an example of a process for shifting the rotated second plane data to correspond to the first plane data. For the purpose of explanation, the X-axis shown in FIG. 11 is marked with a scale at 1 cm intervals, for example. Sides SC1 and SC2 are sides of wall CW1 shown in FIG. 10 that are parallel to the Z-axis. In the example shown in FIG. 11, wall CW1 has a width of 500 cm, so side SC1 is located at position 0 on the X-axis and side SC2 is located at position 500 on the X-axis. Sides SD10 and SD20 are sides of second plane data C40 after rotation shown in FIG. 10 that are parallel to the Z-axis.

[0056] The comparison unit 215 sets the rotated second plane data C40 to a position where, for example, the values ​​of the sides SD10 and SC2 on the X axis are the same (initial state shown in FIG. 11). The comparison unit 215 moves the second plane data C40 in 1 cm increments in the negative direction of the X axis, for example. Sides SD11 and SD21 are sides parallel to the Z axis of the second plane data C40 when the second plane data C40 is moved 499 cm from the initial state in the negative direction of the X axis. Sides SD12 and SD22 are sides parallel to the Z axis of the second plane data C40 when the second plane data C40 is moved 500 cm from the initial state in the negative direction of the X axis. The comparison unit 215 moves the second plane data C40 in the negative direction of the X axis to a position where, for example, the values ​​of the sides SD20 and SC1 on the X axis in the initial state are the same (final state shown in FIG. 11). Sides SD13 and SD23 are sides parallel to the Z axis of the second plane data C40 when the second plane data C40 is moved 1000 cm in the negative direction of the X axis from the initial state (final state).

[0057] For example, the comparison unit 215 selects the amount of movement that minimizes the difference between the X-axis values ​​for the two sides of wall CW1 and the X-axis values ​​for the two sides of second plane data C40 from the initial state to the final state. In the example shown in Fig. 11, the comparison unit 215 selects the amount of movement (500 cm in the negative direction of the X-axis from the initial state) that minimizes the difference between the X-axis value for side SC1 and the X-axis value for side SD12 and the difference between the X-axis value for side SC2 and the X-axis value for side SD22.

[0058] The comparison unit 215 performs similar processing for, for example, the Y-axis and Z-axis to select the amount of movement. The comparison unit 215 moves the second plane data C40 based on, for example, the selected amount of movement for each of the X-axis, Y-axis, and Z-axis. This allows the wall CW1 and the second plane data to correspond to each other, as shown in the right diagram in FIG.

[0059] Note that the method of moving the second plane data is not particularly limited as long as the rotated second plane data can be moved to correspond to the first plane data. For example, the comparison unit 215 may move the second plane data so that the intersection of the diagonals of the first plane data (hereinafter also referred to as "intersection A") and the intersection of the diagonals of the rotated second plane data (hereinafter also referred to as "intersection B") coincide with each other.

[0060] This allows the information processing device 20 to match the first plane data with the second plane data. Therefore, in the processing described below, it becomes possible to output a comparison result based on the matched first plane data and second plane data. Furthermore, the information processing device 20 rotates the second plane data in a predetermined direction based on the normal vector of the first plane data until the direction of the normal vector of the second plane data corresponds to the direction of the normal vector of the first plane data, and moves the rotated second plane data to match the first plane data, thereby making it possible to match the first plane data with the second plane data with high accuracy.

[0061] The comparison unit 215 may calculate a difference between the first planar data and the second planar data. The difference may include a distance between at least one feature point included in the second planar data and at least one feature point included in the first planar data.

[0062] Fig. 12 is a diagram showing an example of pre-processing for calculating the difference between the first plane data and the second plane data. In Fig. 12, the wall CW1 (first plane data) and the second plane data C40 correspond to each other through processing by the comparison unit 215. Note that, for ease of viewing the diagram, in Fig. 12, the wall CW1 (first plane data) and the second plane data C40 do not contact each other, but in reality, the wall CW1 (first plane data) and the second plane data C40 do contact each other.

[0063] In the example shown in FIG. 12, the comparison unit 215 selects any multiple points (for example, four points shown in the area E20 in FIG. 12, hereinafter also referred to as "point A") in the area E20 of the second planar data C40 generated by the first generation unit 214. Also, in the example shown in FIG. 12, the comparison unit 215 calculates nearest points (for example, four points shown in the area E10 in FIG. 12, hereinafter also referred to as "point B") that are included in the wall CW1 (first planar data) detected by the detection unit 213 and correspond to each of the multiple points in the selected area E20. Also, as a preferred example, the shapes of the areas E10 and E20 are squares. Note that as a preferred example, the squares are congruent with one surface of a voxel V, which will be described later.

[0064] 12 may include superpixels determined to be non-planar by first generation unit 214. That is, comparison unit 215 may return the superpixels removed by first generation unit 214 to second planar data associated with the first planar data.

[0065] FIG. 13 is a diagram illustrating an example of a process for calculating the difference between the first planar data and the second planar data. The comparison unit 215 calculates a feature point P20 based on a plurality of points A selected in the preprocessing illustrated in FIG. 12. The comparison unit 215 calculates, for example, a feature point P20 corresponding to the average value of the coordinate values ​​of the plurality of points A. The comparison unit 215 calculates a feature point P10 based on a plurality of points B calculated in the preprocessing illustrated in FIG. 12. The comparison unit 215 calculates, for example, the distance between the feature point P20 and the point included in the first planar data CW1 that is closest to the feature point P20 (the distance h illustrated in FIG. 13). This makes it possible to calculate the difference between the first planar data and the second planar data. Therefore, a user (for example, worker U) can understand the comparison result based on the difference. Furthermore, by calculating the difference between the first planar data and the second planar data based on the feature points (e.g., feature points P10 and P20), the processing load on the information processing device 20 can be reduced compared to calculating the difference between each point of the first planar data (e.g., multiple points A) and each point of the second planar data (e.g., multiple points B). Note that the comparison unit 215 may also calculate the difference between the first planar data and the second planar data in areas other than area E10 and area E20.

[0066] The output unit 216 outputs the comparison result between the first plane data and the second plane data. For example, the output unit 216 outputs the difference between the first plane data and the second plane data calculated by the comparison unit 215. Note that the output destination of the processing by the output unit 216 may be, for example, a screen provided in the information processing device 20 or a screen provided in an external information processing device.

[0067] The second generating unit 217 generates a data structure including a plurality of three-dimensional spaces based on, for example, three-dimensional CAD data.

[0068] Fig. 14 is a diagram showing an example of processing by the second generating unit 217. In the example shown in Fig. 12, the second generating unit 217 generates a three-dimensional space (hereinafter also referred to as "voxel V") that penetrates the wall CW1 (first plane data). The second generating unit 217 generates a plurality of voxels V without gaps so as to fill the plane related to the wall CW1 (first plane data), for example. As a suitable example, the second generating unit 217 generates a cubic voxel V. As a suitable example, the second generating unit 217 generates a voxel V that perpendicularly penetrates the plane related to the wall CW1 (first plane data).

[0069] As a preferred example, the processing of second generating unit 217 is executed before the difference between the first planar data and the second planar data is calculated by comparing unit 215. The processing of second generating unit 217 may be executed, for example, at the timing when detecting unit 213 detects the first planar data.

[0070] When the processing of the second generating unit 217 is executed before the comparing unit 215 calculates the difference between the first planar data and the second planar data, the region E10 and the region E20 shown in Fig. 12 are congruent to the plane of the cubic voxel V. In this case, the region E10 shown in Fig. 12 corresponds to the region where the voxel V and the wall CW1 (first planar data) overlap.

[0071] The assigning unit 218 assigns, for example, information relating to the difference calculated by the comparing unit 215 to the multiple three-dimensional spaces generated by the second generating unit 217. In the example shown in Fig. 14, the assigning unit 218 assigns information relating to the distance h shown in Fig. 12 to the voxel V.

[0072] The third generation unit 219 generates a composite image based on, for example, a data structure including multiple three-dimensional spaces to which information is assigned by the assignment unit 218 and the three-dimensional CAD data. For example, as shown in FIG. 2, the third generation unit 219 generates a composite image in which the difference-related information assigned to the voxel V is visibly projected onto the wall CW1 in the three-dimensional CAD data. As a result, the comparison result of the wall in the screen region B10 is displayed, for example, in warm colors if the wall in the screen region B10 protrudes in front of the wall outside the screen region B10, in cool colors if the wall is recessed, or in gray if the positions of the two walls are approximately the same. Therefore, when the wall in the screen region B10 is displayed in warm or cool colors, a user (e.g., worker U) can understand that the portion of the wall W1 corresponding to the screen region B10 has not been constructed as designed. Furthermore, the third generation unit 219 can generate a data structure in which each difference calculated by the comparison unit 215 is associated with the wall CW1 in the three-dimensional CAD data by using the voxel V. This enables output section 216 to output the comparison result between the first plane data and the second plane data.

[0073] The above processing makes it possible to reduce the processing load in building inspections.

[0074] <Operation> Next, the operation according to this embodiment will be described. Fig. 15 is a flowchart showing an example of the processing of the information processing device 20 according to this embodiment.

[0075] In step S11, the acquisition unit 212 acquires three-dimensional CAD data relating to a predetermined space (for example, a space within a construction site).

[0076] In step S12, the acquisition unit 212 acquires a distance image relating to the space.

[0077] In step S13, the detection unit 213 detects first plane data relating to space from the three-dimensional CAD data.

[0078] In step S14, the first generator 214 generates second plane data relating to a predetermined plane in space based on the distance image.

[0079] In step S15, the comparison unit 215 compares the first plane data with the second plane data.

[0080] In step S16, the output unit 216 outputs the comparison result.

[0081] <Screen example> FIG. 16 is a diagram illustrating an example of how the results of the as-built inspection according to this embodiment are reflected on a screen. In the example illustrated in FIG. 16, first, a worker U points the imaging device 10 at a wall W1 inside a building B. The information processing device 20 executes the above-described process and displays an image F10 related to the inspection result of the wall W1 on a screen that outputs the inspection result. Next, the worker U points the imaging device 10 at a wall W2 inside the building B. The information processing device 20 executes the above-described process and displays an image F20 related to the inspection result of the walls W1 and W2 on a screen that outputs the inspection result. Next, the worker U points the imaging device 10 at a floor inside the building B. The information processing device 20 executes the above-described process and displays an image F30 related to the inspection result of the walls W1, W2, and the floor on a screen that outputs the inspection result. As illustrated in FIG. 16, during the worker U's inspection, inspection results are successively accumulated and displayed in the 3D CAD data related to the building B, allowing the worker U to grasp the results of the as-built inspection in real time.

[0082] <Modification> The above-described embodiments and examples are intended to facilitate understanding of the present invention and are not to be construed as limiting the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention. Furthermore, the present invention can be formed into various disclosures by appropriately combining multiple components disclosed in the above-described embodiments or examples. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components may be appropriately combined in different embodiments.

[0083] In the above embodiment, it has been described that the worker U performs the inspection inside the building B, but this is not limited to this. For example, a robot that can move around the building B may perform the inspection using the imaging device 10 and the information processing device 20. Furthermore, a robot equipped with the functions of the imaging device 10 and the information processing device 20 may perform the inspection inside the building B.

[0084] In the above embodiment, a comparison between one first plane data and one second plane data has been described as an example, but this is not limiting. For example, a comparison between a plurality of first plane data and a plurality of second plane data may be performed. As a non-limiting example, a comparison may be performed between first plane data related to wall W1 and first plane data related to wall W2 and second plane data related to wall W1 and second plane data related to wall W2. Furthermore, a comparison may be performed using plane data of three surfaces, such as a wall, floor, and ceiling. Furthermore, in the process of associating second plane data with first plane data based on the first plane data in the comparison unit 215, a plurality of first plane data may be used as a reference, and a plurality of second plane data may be simultaneously associated with the first plane data. [Explanation of symbols]

[0085] 1...information processing system, 10...imaging device, 20...information processing device, 210...processor, 211...information processing unit, 212...acquisition unit, 213...detection unit, 214...first generation unit, 215...comparison unit, 216...output unit, 217...second generation unit, 218...assignment unit, 219...third generation unit, 220...network communication interface, 230...storage device, 240...user interface, 250...communication bus, A10...screen, B...building, B10...screen area, C10...range image, C20 ...image, C30...image, C40...second plane data, CW1...wall, CW2...wall, D10...image region, D20...image, E10...region, E20...region, F10~F30...image, L...line, N...network, N10...normal vector, N20...normal vector, P10...feature point, P20...feature point, PE...viewpoint, PI...point, SC1...edge, SC2...edge, SD10~SD13...edge, SD20~23...edge, SP1...superpixel, U...worker, V...voxel, W1...wall, W2...wall

Claims

1. In the information processing device, Obtaining design data for a given space; acquiring a range image relating to the space; comparing the design data with the distance image; outputting the comparison result; A program that executes the following.

2. The information processing device includes: detecting first plane data relating to the space from three-dimensional CAD data relating to the space included in the design data; generating second plane data relating to a predetermined plane in the space based on the distance image; The computer-readable medium according to claim 1 , wherein the comparing step includes comparing the first plane data with the second plane data.

3. The generating step comprises: The program according to claim 2 , further comprising generating the second plane data based on a distance between a viewpoint of an imaging device that captures the distance image and each of a plurality of pixels included in the distance image.

4. The generating step comprises: The program according to claim 2, further comprising generating the second plane data based on the distance between the viewpoint of an imaging device capturing the distance image and each of a plurality of pixels included in the distance image, and the normal vectors of the plurality of pixels.

5. The generating step comprises: The program according to claim 2 , further comprising generating second plane data for a plane having a larger area from among a plurality of planes included in the space, based on the distance image.

6. The comparing step comprises: The program according to claim 2 , further comprising: making the second plane data correspond to the first plane data based on the first plane data.

7. The corresponding Rotating the second plane data in a predetermined direction based on the normal vector of the first plane data until the direction of the normal vector of the second plane data corresponds to the direction of the normal vector of the first plane data; and moving the rotated second plane data to correspond to the first plane data.

8. The comparing step comprises: calculating a difference between the first plane data and the second plane data; The program according to claim 2 , wherein the difference includes a distance between at least one feature point included in the second planar data and at least one feature point included in the first planar data.

9. The information processing device includes: generating a data structure including a plurality of three-dimensional spaces based on the three-dimensional CAD data; assigning information about the differences to the plurality of three-dimensional spaces; 9. The program according to claim 8, further causing the program to execute generating a composite image based on the data structure including the plurality of three-dimensional spaces to which the information is added and the three-dimensional CAD data.

10. The information processing device Obtaining design data for a given space; acquiring a range image relating to the space; comparing the design data with the distance image; outputting the comparison result; An information processing method that performs the above.

11. Obtaining design data for a given space; acquiring a range image relating to the space; comparing the design data with the distance image; outputting the comparison result; An information processing device that executes the above.

Images