Reinforcement inspection result display system

Abstract

To display a bar arrangement inspection result that facilitates the correction work for a bar arrangement error of a worker in a site.SOLUTION: A bar arrangement inspection result display system S comprises: a measuring instrument 2 which has a communication unit 44 and a three-dimensional coordinate measurement unit 262; an eyewear display device 4 which has a display 41; and a system control unit 60 which manages a coordinate space of information about a position and orientation of the eyewear display device and a coordinate space of the measuring instrument in a space with a common reference point as an original point, and displays an image created by the system control unit on the display 41 to display the image so as to be superimposed on an actual stuff observed by wearing the eyewear display device 4. The system control unit generates a present three-dimensional model within a bar arrangement inspection range on the basis of three-dimensional point group data within the bar arrangement inspection range acquired in such a state that the position and orientation are known, generates three-dimensional inspection result display data by associating the bar arrangement inspection result for the bar arrangement inspection range with a three-dimensional model, displays the data on the display, and displays a bar arrangement error within the inspection range in a recognizable manner.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a bar arrangement inspection result display system, and more particularly to a bar arrangement inspection result display system using an eyewear display device. [Background technology]

[0002] Reinforcement inspections in rebar construction check whether the type, number, position, spacing, joint method, etc. of rebars are arranged correctly according to the design. Conventionally, inspectors check for errors by comparing the rebar arrangement drawings with the on-site rebar arrangement status, and record it by taking pictures with a digital camera.

[0003] In recent years, a technique has been proposed for acquiring reinforcing bar arrangement information, such as the number, diameter, and pitch of reinforcing bars, by using captured images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-27058 A [Patent Document 2] Patent Publication No. 2021-21622 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the output of the reinforcing bar inspection systems in Patent Documents 1 and 2 was related to the display of data for creating a report for an inspection report, or display on a tablet or the like for inspectors to check the inspection results. To correct the reinforcing bar errors, workers had to check the display on the report or tablet, which was inconvenient for workers. For this reason, there was a demand for the development of technology that could display the reinforcing bar inspection results for use by on-site workers who correct the reinforcing bar errors.

[0006] The present invention has been made in consideration of the above circumstances, and has an object to make it possible to display reinforcement inspection results in a manner that makes it easier for on-site workers to correct reinforcement errors. [Means for solving the problem]

[0007] In order to achieve the above object, a bar arrangement inspection result display system according to a first aspect of the present invention has the following configuration. 1. A reinforcement inspection result display system comprising: a measuring instrument having a communication unit and a three-dimensional coordinate measuring unit; an eyewear display device having a display, a relative position sensor that detects its own position, and a relative direction sensor that detects its own direction; and a system control unit that manages a coordinate space of information regarding the position and direction of the eyewear display device and the coordinate space of the measuring instrument in a space with a common reference point as the origin; and displays an image created by the system control unit on the display, superimposing it on an actual object observed by wearing the eyewear display device, wherein the system control unit generates a current three-dimensional model of the reinforcement inspection range based on three-dimensional point cloud data of the reinforcement inspection range acquired with the position and direction known, generates three-dimensional inspection result display data by associating the reinforcement inspection results for the reinforcement inspection range with the three-dimensional model, displays it on the display, and displays reinforcement errors in the inspection range in a recognizable manner.

[0008] 2. In the configuration 1 above, it is also preferable that the system control unit generates correction support data for correcting the reinforcement error based on 3D reinforcement design data in the inspection range, and displays on the display a correction support image that supports the task of correcting the reinforcement error based on the correction support data.

[0009] 3. In the above configuration 1 or 2, it is also preferable that the system control unit is capable of recognizing the worker's hands within the field of view of the eyewear display device, and that the rebar touched by the worker is displayed recognizably on the display. Effect of the Invention

[0010] According to the above aspect, it is possible to display the reinforcement inspection results in a manner that makes it easier for on-site workers to correct reinforcement errors. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic external view of a bar arrangement inspection system including a bar arrangement inspection result display system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram of the system. [Diagram 3] FIG. 2 is a block diagram of a scanner constituting the system. [Figure 4A] FIG. 2 is an external perspective view of an eyewear display device that constitutes the system. [Figure 4B] FIG. 2 is a block diagram of the eyewear display device. [Diagram 5] FIG. 2 is a block diagram of a data processing device constituting the above system. [Figure 6] 1A to 1C are diagrams illustrating the differences between image data, point cloud data, and a point cloud composite image. [Figure 7] FIG. 2 is a diagram illustrating a reinforcement bar state inspection unit of the data processing device. [Figure 8] FIG. 13 is a diagram illustrating a method for generating a reinforcement state identification model in the data processing device. [Figure 9] FIG. 1 is a schematic flow diagram of a reinforcing bar inspection method using the above system. [Figure 10] FIG. 4 is a flow chart of the process of the data processing device in the method. [Figure 11] FIG. 4 is a detailed flow diagram of the process of inspecting the state of reinforcement in the method. [Figure 12] FIG. 13 is a flow diagram of a process for generating display data of a reinforcement inspection result using the system. [Figure 13] FIG. 13 is a diagram showing an example of a reinforcing bar inspection result display by the system. [Figure 14] FIG. 13 is a diagram showing an example of a reinforcing bar inspection result display by the system. [Figure 15]1 is a block diagram of a reinforcement bar inspection result display system according to an embodiment of the present invention. [Figure 16] FIG. 13 is a block diagram of a reinforcement inspection result display system according to a modified example of the system. [Figure 17] FIG. 2 is a diagram illustrating a motion capture device according to the system. [Figure 18] FIG. 13 is a diagram showing an example of a reinforcing bar inspection result display by the system. [Figure 19] FIG. 13 is a schematic external view of another example of an inspection system for acquiring three-dimensional point cloud data and three-dimensional reinforcement inspection result data. [Figure 20] FIG. 2 is a configuration block diagram of the inspection system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited thereto. In addition, the same components common to each embodiment and modification are denoted by the same reference numerals, and duplicated descriptions will be omitted as appropriate.

[0013] I. Embodiment 1. Configuration of the Reinforcement Inspection System 100 (Reinforcement Inspection Result Display System S) 1 is a block diagram showing an overview of a state of use of a bar arrangement inspection system (hereinafter simply referred to as an inspection system) 100 incorporating a bar arrangement inspection result display system (hereinafter simply referred to as a display system) S according to an embodiment of the present invention. The inspection system 100 comprises at least one scanner 2, an eyewear display device (hereinafter simply referred to as an eyewear device) 4 equipped with a camera 50, and a data processing device 6. The scanner 2, eyewear device 4, and data processing device 6 are wirelessly connected and can transmit and receive information to and from each other.

[0014] The scanner 2 is a ground-mounted 3D laser scanner. The scanner 2 is installed at an arbitrary point within the site of foundation work, which is the inspection range. The instrument installation point is known by the resection method or the like. The scanner 2 is installed via a leveling stand attached to a tripod, and has a base part 2α provided on the leveling stand, a support part 2β that rotates horizontally around an axis H on the base part 2α, and a light projecting part 2γ that rotates vertically in the center of the support part 2β.

[0015] The eyewear device 4 is a so-called head-mounted display that is worn on the worker's head. The camera 50 is used to capture images of the reinforcing bar arrangement state in the inspection range. The camera 50 captures an image of the range within the field of view of the camera 50 (referred to as the inspection target range) in one shot. The display 41 can also display the inspection results superimposed on the actual reinforcing bars.

[0016] In the illustrated example, the data processing device 6 is a laptop computer. The data processing device performs a reinforcement inspection using three-dimensional point cloud data (hereinafter simply referred to as point cloud data) acquired by the scanner 2 and images acquired by the camera 50. Each component will be described in detail below.

[0017] 2. Scanner 3 is a configuration block diagram of the scanner 2. The scanner 2 includes a distance measuring unit 21, a vertical rotation drive unit 22, a vertical angle detector 23, a horizontal rotation drive unit 24, a horizontal angle detector 25, a scanner control unit 26, a display unit 27, an operation unit 28, a storage unit 29, an external storage device 30, and a communication unit 31.

[0018] The distance measuring unit 21 includes a light sending unit, a light receiving unit, a light sending optical system, a light receiving optical system that shares optical elements with the light sending optical system, and a rotating mirror 21α. The light sending unit includes a light emitting element such as a semiconductor laser, and emits pulsed light that is distance measuring light as scanning light. The emitted distance measuring light is incident on the rotating mirror 21α via the light sending optical system, and is deflected by the rotating mirror 21α to be irradiated onto the measurement object. The rotating mirror 21α is driven by the vertical rotation drive unit 22 to rotate around the axis V.

[0019] The reflected light reflected by the object to be measured passes through the rotating mirror 21α and the light receiving optical system and enters the light receiving section. The light receiving section includes a light receiving element such as a photodiode. A part of the distance measuring light enters the light receiving section as an internal reference light, and the scanner control section determines the distance to the irradiation point based on the reflected light and the internal reference light.

[0020] The vertical rotation drive unit 22 and the horizontal rotation drive unit 24 are motors controlled by the scanner control unit. The vertical rotation drive unit 22 rotates the rotating mirror 21α in the vertical direction around an axis V. The horizontal rotation drive unit 24 rotates the support unit 2β in the horizontal direction around an axis H.

[0021] The vertical angle detector 23 and the horizontal angle detector 25 are rotary encoders. The vertical angle detector 23 measures the vertical rotation angle of the rotating mirror 21α. The horizontal angle detector 25 measures the horizontal rotation angle of the support part 2β. This allows the vertical angle and horizontal angle of the distance measurement optical axis to be detected.

[0022] The scanner control unit 26 includes at least one processor and at least one memory. The processor is, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The memory is, for example, an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The processor reads data and programs stored in the storage unit 29 or the like into the memory and executes processing for implementing the functions of the scanner 2.

[0023] In this specification, the processor is not limited to a processor that performs software processing for all the processes it executes. It may also be equipped with a dedicated hardware circuit (for example, an application specific integrated circuit (ASIC)) that performs hardware processing for at least a part of the processes it executes. In other words, the processor may be configured as a circuit including a combination of at least one processor that operates according to a computer program (software) and one or more dedicated hardware circuits that execute at least a part of the various processes.

[0024] The scanner control unit 26 calculates the distance to the irradiation point for each pulse of distance measuring light based on the time difference between the light emission timing of the light transmitting unit and the light reception timing of the light receiving unit (round trip time of the pulse light). It also calculates the irradiation angle of the distance measuring light at that time to calculate the angle of the irradiation point.

[0025] The scanner control unit 26 also includes, as functional units, a point cloud data acquisition unit 261 and a three-dimensional coordinate measurement unit 262. The point cloud data acquisition unit 261 controls the distance measurement unit 21, the rotating mirror 21α, the vertical rotation drive unit 22, and the horizontal rotation drive unit 24 to scan the distance measurement light in the entire circumference (360°) (full dome scan), acquire the coordinates of each irradiation point, and acquire point cloud data of the entire circumference. The point cloud data acquisition unit 261 acquires three-dimensional point cloud data (hereinafter referred to as 3D point cloud data) 71 of the inspection range and sends it to the data processing device 6. The three-dimensional coordinate measurement unit 262 realizes a target scan function that scans the periphery of a target at high density, measures the distance and angle of the target, and acquires the three-dimensional coordinates of the target.

[0026] The display unit 27 is, for example, a liquid crystal display. The operation unit 28 has a power key, numeric keys, a decimal point key, + / - keys, an execution key, a cursor movement key, etc. An operator can input operation instructions and information to the scanner 2 from the operation unit 28.

[0027] The storage unit 29 is, for example, a computer-readable storage medium such as a hard disk drive (HDD) or a flash memory. The storage unit 29 stores a program for executing the functions of the scanner control unit 26. The external storage device 30 is, for example, a memory card, and stores various data acquired by the scanner 2.

[0028] The communication unit 31 is a communication control device such as a network adapter, a network interface card, a LAN card, or a Bluetooth (registered trademark) adapter, and connects the scanner 2 to the eyewear device 4 and the data processing device 6 by wire or wirelessly. The scanner control unit 26 can transmit and receive information between the eyewear device 4 and the data processing device 6 via the communication unit 31.

[0029] 3. Eyewear equipment4 Fig. 4A is an external perspective view of the eyewear device 4, and Fig. 4B is a configuration block diagram of the eyewear device 4. The eyewear device 4 includes a display 41, a camera 50, and a control unit 42. The control unit 42 includes an eyewear control unit 43, an eyewear communication unit 44, a relative position detection sensor 45, a relative direction detection sensor 46, an eyewear storage unit 47, and an operation switch 48.

[0030] The display 41 is a goggle-lens type transmissive display that covers both eyes of the worker when worn by the worker. As an example, the display 41 is an optical see-through display using a half mirror, and displays an image received by the eyewear control unit 43 superimposed on the work site scenery.

[0031] Alternatively, display 41 may be a video see-through display that displays an image in which an image received by eyewear control unit 43 is superimposed on a frontal view image captured in real time by camera 50. In addition, the projection method may be a virtual image projection method or a retinal projection method.

[0032] The camera 50 is a digital camera equipped with a lens and an image sensor such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 50 is provided at the center of the front of the eyewear device 4, and captures an image of the front of the eyewear device 4 in real time. The camera 50 acquires image data 72 of the inspection target range, which is the range of the field of view of the eyewear device 4. The acquired image data 72 is sent to the data processing device 6. The image sensor has an orthogonal coordinate system with the imaging center of the camera 50 as the origin, and the local coordinates of each pixel are specified. The imaging center of the camera 50 is the center of the eyewear device 4, and the imaging optical axis is the line of sight of the eyewear device 4.

[0033] The eyewear communication unit 44 is a communication control device similar to the communication unit 31. The eyewear communication unit 44 connects to the eyewear device 4 in a wired or wireless manner, preferably in a wireless manner. The eyewear control unit 43 can transmit and receive information to and from the scanner 2 and the data processing device 6 via the eyewear communication unit 44.

[0034] The relative position sensor 45 performs wireless positioning using a GNSS (Global Navigation Satellite System) antenna, a Wi-Fi (registered trademark) access point, an ultrasonic oscillator, etc. installed at the observation site, and detects the position (local position) of the eyewear device 4 within the observation site.

[0035] The relative direction sensor 46 is a combination of a three-axis acceleration sensor or a gyro sensor and an inclination sensor. The relative direction sensor 46 detects the attitude (self-orientation) of the eyewear device 4 with the up-down direction as the Z-axis direction, the left-right direction as the Y-axis direction, and the front-back direction as the X-axis direction. As a result, the camera 50 is able to acquire images with its position and orientation known.

[0036] The eyewear storage unit 47 is a computer-readable storage medium such as a memory card, etc. The eyewear storage unit 47 stores programs for the eyewear control unit 43 to execute functions.

[0037] The operation switch 48 is, for example, a push button provided on the temple portion. The operation switch 48 includes, for example, a power button 48α for turning on / off the power of the eyewear device 4, a shooting button 48β for shooting a still image with the camera 50, and an image switching button 48γ for switching images.

[0038] Similar to scanner control unit 26 of scanner 2, eyewear control unit 43 includes at least one processor (e.g., CPU) and at least one memory (e.g., SRAM, DRAM, etc.). The processor reads out programs stored in eyewear storage unit 47 and deploys them in the memory, thereby realizing various functions of eyewear device 4.

[0039] The eyewear control unit 43 outputs information on the position and direction of the eyewear device 4 detected by the relative position sensor 45 and the relative direction sensor 46 to the data processing device 6. In addition, the data received from the data processing device 6 is displayed on the display 41 of the eyewear device 4, superimposed on the scenery of the inspection site.

[0040] 4. Data Processing Unit 6 FIG. 5 is a configuration block diagram of the data processing device 6. The data processing device 6 is a so-called computer, typically a personal computer, a server computer, etc., but may also be a tablet terminal, a smartphone, etc. The arithmetic processing unit 60 of the data processing device 6 corresponds to the system control unit in the claims. The system control unit may be a single computer like the data processing device 6, or may be a computer system in which multiple computers perform distributed processing. In this case, a part of the processing resources of one or more computers may be logically used. The system control unit may be configured as a part of the eyewear device 4, or may be configured as a part of the scanner 2. A part of the processing of the data processing device 6 may be configured to be performed by the eyewear device 4, or may be configured to be performed by the scanner 2.

[0041] The data processing device 6 includes at least an arithmetic processing unit 60 , a communication unit 63 , a display unit 64 , an input unit 65 , and a storage unit 66 .

[0042] The communication unit 63 is a communication control device equivalent to the communication unit 31 of the scanner 2, and enables the data processing device 6 to wirelessly communicate with the scanner 2 and the eyewear device 4. The arithmetic processing unit 60 can transmit and receive information to and from the scanner 2 and the eyewear device 4 via the communication unit 63.

[0043] The display unit 64 is, for example, a liquid crystal display. The input unit 65 is, for example, a keyboard, a mouse, etc., and enables an operator to input various commands, selections, decisions, etc.

[0044] The arithmetic processing unit 60 is a control arithmetic unit including at least one processor (e.g., CPU, GPU) and at least one memory (e.g., SRAM, DRAM, etc.) similar to the scanner control unit 26 of the scanner 2. The processor reads out a program stored in the storage unit 66 and loads it in the memory, thereby realizing various functions of the data processing device 6, in particular the functions of the following functional units.

[0045] The calculation processing unit 60 has, as functional units, a synchronous measurement unit 601, a point cloud data receiving unit 602, a reinforcement design data reading unit 603, an image data receiving unit 604, a point cloud composite image generating unit 605, a reinforcement state inspection unit 606, and a display data generating / output unit 607.

[0046] The synchronous measurement unit 601 synchronizes the scanner 2, eyewear device 4 (camera 50), and data processing device 6. Synchronization is a process that enables information including position coordinates, such as the positions and attitudes of the scanner 2 and eyewear device 4 (camera 50), and design data handled by the data processing device 6, to be managed in a common coordinate space with a common reference point as the origin. An example that is considered to be suitable is shown below, but synchronization may be achieved by any suitable method based on the knowledge of those skilled in the art.

[0047] First, for the inspection system 100, a reference point and a reference direction are set at the inspection site, and the scanner 2 and the data processing device 6 are synchronized. A known point (a point whose coordinates are known) or an arbitrary point at the site is selected as the reference point. A feature point other than the reference point is selected, and the direction from the reference point to the feature point is set as the reference direction. Using the target scan function of the scanner 2, the absolute coordinates of the scanner 2 are grasped by observing the reference point and the feature point using the method of intersection, and are transmitted to the data processing device 6. The synchronous measurement unit 601 recognizes the absolute coordinates of the reference point as (x, y, z) = (0, 0, 0) and the reference direction as a horizontal angle of 0°. This enables the data processing device 6 to manage the relative position and relative direction of the scanner 2 in a space with the reference point as the origin, with respect to the information from the scanner 2.

[0048] Next, the eyewear device 4 and the data processing device 6 are synchronized. The eyewear device 4 is placed at the reference point, and with the eyewear device 4 held horizontally, the line of sight direction of the eyewear device 4 is aligned with the reference direction, and (x, y, z) of the relative position sensor 45 is set to (0, 0, 0), and (roll, pitch, yaw) of the relative direction sensor is set to (0, 0, 0). This allows the data processing device 6 to manage the relative position and relative direction of the eyewear device 4 (camera 50) in a space with the reference point as the origin, for information from the eyewear device 4 (camera 50).

[0049] The point cloud data receiving unit 602 receives, via the communication unit 63, three-dimensional point cloud data of the entire inspection range acquired by performing a full dome scan with the scanner 2.

[0050] The bar arrangement design data reading unit 603 reads out 3D (three-dimensional) bar arrangement design data (hereinafter referred to as bar arrangement design data) 73, which will be described later, stored in the storage unit 66.

[0051] The image data receiving unit 604 receives the image data 72 acquired by the camera 50 via the communication unit 63 .

[0052] The point cloud composite image generating unit 605 generates a point cloud composite image by combining the 3D point cloud data 71 with the image data 72. FIG. 6 shows an image of the image data (A), the point cloud data (B), and the point cloud composite image (C). The image data 72 is data that includes color information, but may include shadows and the like. On the other hand, the point cloud data is not affected by shadows, but does not include color information. The coordinates of each point of the point cloud data are specified, and the actual dimensions are known. The point cloud composite image includes the color information of the image data and the information of the actual dimensions ascertained from the point cloud data. In other words, by using the point cloud composite image to identify the reinforcement state, the dimensions can be specified without setting a reference marker such as a ruler.

[0053] The reinforcement state inspection unit 606 inspects the reinforcement state from the point cloud composite image, and outputs three-dimensional reinforcement inspection result data (hereinafter referred to as 3D reinforcement inspection result data or reinforcement inspection result data) 75. The reinforcement state inspection unit 606 will be described in detail later.

[0054] The display data generating and outputting unit 607 generates display data to be displayed on the display 41 of the eyewear device 4 based on the bar arrangement inspection result data.

[0055] The storage unit 66 is, for example, an HDD, an SSD (Solid State Drive), etc. The above-mentioned 3D bar arrangement design data (hereinafter also referred to as bar arrangement design data) 73 is stored in the storage unit 66. In addition, when each functional unit of the arithmetic processing unit 60 is realized as software, the storage unit 66 stores a program for executing each function.

[0056] The reinforcement design data 73 is a detailed reinforcement drawing created using 3D CAD data. The detailed reinforcement drawing is a diagram showing the detailed reinforcement state of a reinforced concrete member. The reinforcement state includes, for example, the type of rebar (material and thickness), the spacing and number of rebars, the placement of spacers to prevent a shortage of rebar covering, the direction of the main bars, the type of rebar joints, the positions of the binding points, etc. The reinforcement design data 73 includes information on at least one of these.

[0057] The reinforcement design data 73 is generated in advance by a 3D reinforcement design data generating device 70 and stored in the storage unit 66. The 3D reinforcement design data generating device 70 is a computer including at least one processor and at least one memory equivalent to the data processing device 6.

[0058] Next, the details of the reinforcement state inspection unit 606 will be described with reference to Figs. 7 and 8. As shown in Fig. 7, the reinforcement state inspection unit 606 includes a pre-processing unit 611, a reinforcement state identification model 612, and a design data comparison unit 613.

[0059] The pre-processing unit 611 performs image processing on the point cloud composite image input as the inspection target to facilitate the recognition of rebars. For example, known image processing such as grayscale conversion, edge and line segment extraction, and brightness value averaging may be performed. In addition to this, the point cloud composite image may be enlarged or reduced to a predetermined scale. Since the composite image data is data that includes position coordinates, i.e., actual size information, it is possible to enlarge or reduce to a predetermined scale without simultaneously photographing a reference marker such as a ruler.

[0060] The reinforcement state identification model 612 is a trained model obtained by training a large number of training point cloud composite images created by capturing images of a large number of reinforcement states and acquiring point cloud data, as shown in Fig. 8. The training point cloud composite images are labeled with the type of rebar (material, shape, and thickness (diameter dimension)), the spacing and number of rebars, the arrangement of spacers to prevent insufficient rebar covering, the direction of the main bars, the type of rebar joints, the positions of the binding points, and the like. The training data are a large number of point cloud composite images generated in the same manner as the point cloud composite image by capturing images of various reinforcement states in a general reinforcement method. Image data may also be used.

[0061] When the point cloud composite image of the inspection range is input, the reinforcement state identification model 612 identifies the type of rebar (material and thickness), the spacing and number of rebars, the arrangement of spacers to prevent insufficient rebar covering, the direction of main bars, and the type of rebar joints in the inspection range, and outputs these events in association with their positions. Learning is performed by a computer having at least one processor and memory, similar to the data processing device 6. As a learning method, for example, deep learning using Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Boltzmann machines, etc. is used.

[0062] The types of rebar are designated according to the JIS standard, such as D3, D13, D38, SR295, etc., depending on the material, shape, and diameter. These designations may be used on the labels of the rebar types.

[0063] In addition, in the pre-processing, when the image data is enlarged / reduced to match the scale of the point cloud composite image, the accuracy of the reinforcement state identification is improved if image data of the same predetermined scale is used for the learning data used for learning.

[0064] When a point cloud composite image of the inspection range is input to the reinforcement condition identification model 612, reinforcement condition identification data 74 including the detected reinforcement condition information, i.e., the type of rebar (material, shape, and thickness (diameter dimension)), rebar spacing and number, arrangement of spacers to prevent insufficient rebar covering, main bar direction, type of rebar joint, position of binding points, and other information and their position information, is output, as shown in an example in the lower right of Figure 8.

[0065] The design data comparison unit 613 compares the point cloud composite image with the reinforcement design data 661, and outputs the differences (rebar arrangement errors) and their positions as reinforcement inspection result data 75. Specifically, the type of rebar, the interval and number of rebars, the arrangement of spacers to prevent insufficient rebar covering, the direction of the main bars, the type of rebar joints, differences in binding points, etc. are output in association with position information within the inspection range.

[0066] 5. Reinforcement Inspection Method (Processing of Reinforcement Inspection System 100) Next, a method for inspecting bar arrangement will be described with reference to Fig. 9. The method for inspecting bar arrangement will be described with reference to a flow chart of the outline of the flow of the method for inspecting bar arrangement using the inspection system 100 according to the present embodiment.

[0067] In step S01, the scanner 2 is installed at a known point and a full dome scan is performed to acquire point cloud data 71 of the inspection range (preferably the entire range). This entire inspection range does not strictly mean the entire inspection range, but may be the range within the inspection range that is required by the operator. The acquired point cloud data 71 is sent to the data processing device 6. If possible, the point cloud data 71 of the entire site may be acquired in a single full dome scan. The point cloud data 71 may be acquired using multiple scanners 2. Alternatively, the point cloud data 71 may be acquired by performing multiple full dome scans at different installation points.

[0068] Next, in step S02, an operator captures an image of a part of the range set in the inspection range (inspection target range) with the camera 50. The part of the range set in the inspection range is specifically the range of the angle of view of the camera 50, and is the range that the operator intends to capture for inspection. The acquired image data 72 is sent to the data processing device 6.

[0069] Next, in step S03, the data processing device 6 synthesizes the received 3D point cloud data 71 and image data 72 to generate a point cloud synthesized image.

[0070] Next, in step S04, the data processing device 6 executes an inspection of the reinforcement state using the point cloud composite image, and outputs the reinforcement inspection result data 75.

[0071] Next, in step S05, the data processing device 6 generates and outputs display data based on the reinforcement inspection result data 75. The inspection results may be output to the display 41 of the eyewear device 4, to the display unit 64 of the data processing device 6, or may be output from the data processing device 6 to an external device such as a printer as a form.

[0072] 6. Reinforcement Inspection Method (Processing of Data Processing Device 6) Fig. 10 is a flow diagram of the processing of the data processing device 6 in the above-mentioned bar arrangement inspection method. Fig. 11 is a detailed flow diagram of step S15.

[0073] When the process starts, first, in step S11, the point cloud data receiving unit 602 receives the point cloud data 71 of the entire circumference from the scanner 2 and stores it in the storage unit 66.

[0074] Next, in step S12, the reinforcement design data reading unit 603 reads the reinforcement design data 73 from the storage unit 66. The order of steps S11 and S12 is not limited to this, and the reinforcement design data may be read when the reinforcement state inspection unit 606 performs an inspection.

[0075] Next, in step S13, the image data receiving unit 604 receives, from the camera 50, the image data 72 of the inspection target range.

[0076] Next, in step S14, the point cloud composite image generating unit 605 generates a point cloud composite image by combining the image data 72 of the inspection target range and the point cloud data 71 corresponding to the inspection target range.

[0077] Next, in step S15, the reinforcement state inspection unit 606 compares the point cloud composite image of the inspection target range with the corresponding 3D reinforcement design data 73 to inspect the reinforcement state of the inspection target range.

[0078] Specifically, in step S21, the pre-processing unit 611 performs image processing on the point cloud composite image input as the inspection target to facilitate recognition of reinforcing bars.

[0079] Next, in step S22, the point cloud composite image of the inspection target range is input to the reinforcement condition identification model 612, the reinforcement condition of the inspection target range is identified, and the identified model is output as reinforcement condition identification data 74.

[0080] Next, in step S23, the 3D reinforcement design data 73 read in step S12 is compared with the reinforcement state identification data 74 output in step S22 to identify any parts with differences (reinforcement errors), which are associated with the positions of those parts and stored in the storage unit 66 as reinforcement inspection result data 75, and step S15 is terminated. The reinforcement inspection results obtained as a result of this are accumulated for each inspection range, and by inspecting the entire inspection site while moving within the inspection range, it is possible to obtain three-dimensional reinforcement inspection result data 75 for the entire inspection site.

[0081] Next, in step S16, the display data generating and outputting unit 607 generates display data to be displayed on the display of the eyewear device 4. Details will be described later.

[0082] 7. Details of display data generation FIG. 12 is a flow diagram showing detailed processing by the display data generating and outputting unit 607.

[0083] In step S21, the display data generating / outputting unit 607 generates a current three-dimensional model based on the current 3D point cloud data 71 acquired from the scanner 2 and stored in the storage unit 66, and sets the generated model as current 3D data.

[0084] Next, in step S22, the display data generation and output unit 607 associates the reinforcement inspection result data 75 output by the reinforcement state inspection unit 606 and stored in the memory unit 66 with the above-mentioned current three-dimensional model, and generates 3D inspection result display data.

[0085] Next, in step S23, the display data generating and outputting unit 607 generates 3D correction support data based on the 3D bar arrangement design data 73 stored in advance in the storage unit.

[0086] Then, in step S24, the display data generation and output unit 607 uses the 3D current state data, the 3D inspection result display data, and the 3D correction support data to generate a display image corresponding to the field of view of the display 41, and displays it on the display 41, superimposed on the actual object.

[0087] 13 and 14 are diagrams showing an image of the display image generated in this way. In the figures, the dashed lines indicate the actual object (reinforcing bar) seen within the field of view of the display 41, and the solid lines are the display image generated by the display data generation and output unit 607. Note that in reality, the display image is displayed superimposed on the actual object, but for convenience of explanation, it is drawn slightly shifted.

[0088] Fig. 13 shows an image of a worker wearing eyewear device 4 observing the reinforcement state while facing the reinforcing bars corresponding to the wall surface. Fig. 13(A) displays the current state. It is a current state image 41a generated based on 3D current state data, and is displayed superimposed on the actual object. The 3D current state data may include the reinforcement state identification result by the reinforcement state identification model 612, although this is not essential. As such an example, Fig. 13(A) displays the type of reinforcing bars (D13, D38) and the spacing between the reinforcing bars.

[0089] Fig. 13(B) shows an inspection result image 41b created based on the 3D inspection result display data. As an inspection result, the rebar at the right end is incorrectly arranged, and in addition to the 3D current state data, Fig. 13(B) shows that the vertical rebar at the right end is incorrectly arranged as D13, and is highlighted so that it can be seen that this is incorrect. Specifically, rebars that are properly arranged may be arranged in green or the like, and rebars with errors may be displayed in a conspicuous color such as red or yellow, or may flash.

[0090] Fig. 13(C) shows a correction support image 41c created based on the 3D correction support data. The correction support image 41c shows the correct reinforcement state based on the design reinforcement data to support the work of correcting the reinforcement arrangement error. Fig. 13(C) shows that the rebar on the right end is actually D38, and the rebar is highlighted to make it stand out.

[0091] In the eyewear device 4, the current state image 41a, the inspection result image 41b, and the correction support image 41c may be switchable under the control of the calculation processing unit 60 or the eyewear control unit 43. In this case, for example, the operator may be able to switch between them by pressing the image switching button 48γ.

[0092] Fig. 14 shows a display image when a worker wearing the eyewear device 4 observes the reinforcing bar arrangement state of the floor surface from diagonally above. Fig. 14(A) shows a current state image 41a generated based on 3D current state data, which shows the current state, and is displayed superimposed on the actual object.

[0093] Fig. 14(B) shows an inspection result image 41ba created based on the 3D inspection result display data. Inspection result image 41b highlights the insufficient covering portion by changing the color, etc., to show that the intersection of the central rebar sags and there is an insufficient covering as an inspection result. At this time, although not essential, as shown in Fig. 14(B), the content of the reinforcement arrangement error, i.e., "insufficient covering", may be displayed so that it is clear.

[0094] Fig. 14(C) shows a correction support image created based on the 3D correction support data. In Fig. 14(C), the correction support image 41c shows that the error can be eliminated by installing a spacer at the relevant location so that the amount of covering is appropriate, in order to support the work of correcting the reinforcement error.

[0095] In addition, the inspection result image 41b and the correction support image 41c may be related to errors in the spacing and number of reinforcing bars, errors in the direction of main reinforcing bars, errors in the type of reinforcing bar joints, errors in the positions of the binding points, and the methods of correcting these and the state of reinforcing bars after correction. Instead of displaying all of these, it may be possible to display only some of them. Alternatively, it may be possible to switch the display for each type of reinforcing bar error.

[0096] Here, a block diagram of the display system S according to the embodiment is shown in FIG. 15. The display system S includes a scanner (measuring instrument) 2, an eyewear device 4, and a data processing device 6. The scanner 2 functions as a measuring instrument for synchronizing the data of the absolute coordinate system handled by the eyewear device 4 and the data processing device 6. From this point of view, the scanner 2 as a measuring instrument related to the reinforcement inspection result display system S may include a three-dimensional coordinate measuring unit 262 in the scanner control unit 26. The data processing device 6 may include a synchronization measuring unit 601 for synchronizing with the eyewear device 4 and the scanner 2, and a display data generating and outputting unit 607. The data processing device 6 may include the acquired 3D point cloud data for the inspection range, the 3D reinforcement design data 73, and the reinforcement inspection result data 75.

[0097] The measuring instrument is not limited to the scanner 2, but may be any surveying instrument equipped with a three-dimensional coordinate measuring unit capable of acquiring the three-dimensional position coordinates of a measurement object. For example, it may be a total station having a distance measuring and angle measuring function. It may also be a camera equipped with two cameras and capable of acquiring the three-dimensional position coordinates of a measurement object by photogrammetry, as exemplified in JP 2021-77127 A.

[0098] Furthermore, the 3D point cloud data 71, the 3D reinforcement design data 73, and the 3D reinforcement inspection result data 75 do not necessarily need to be acquired by the inspection system 100 including the display system S as in this embodiment, but may be acquired separately and stored in advance in a storage unit. However, if they are acquired by the inspection system 100 including the display system S, it is advantageous because the inspection results can be displayed in real time.

[0099] 8. Effects In this way, according to the display system S of the present embodiment, the content and location of the reinforcement error are displayed so as to be identifiable by superimposing it on the actual object on the display 41 of the eyewear display device 4 worn by the worker on his / her head. As a result, the worker can check the reinforcement error by simply looking at the part he / she wants to check, without having to compare it with a form or tablet display, reducing the burden on the worker.

[0100] In addition, the details of the reinforcement errors are displayed so that they can be easily understood, making it easy to grasp the work required to correct them.

[0101] In particular, the display system S further displays the correct reinforcement state as an aid for correcting reinforcement errors, so that the worker can easily carry out correction work to achieve the correct reinforcement state.

[0102] Furthermore, according to the display system S of this embodiment, the display system S is included in the inspection system 100, and the eyewear display device 4 for displaying the reinforcement results is provided with a camera 50 for acquiring images of the inspection range. This allows the worker to simply take an image of the inspection range he or she wishes to check with the camera 50, and then, through a series of processes, check for reinforcement errors by overlaying the image on the display 41 over the actual object, which is very convenient.

[0103] 9. Variations FIG. 16 is a block diagram of a display system SA according to a modified example.

[0104] The display system SA includes a motion capture device 5 in addition to the components of the display system S. Moreover, the data processing device 6A includes a synchronous measurement unit 601A instead of the synchronous measurement unit 601.

[0105] The motion capture device 5 is a so-called magnetic type motion capture device. The motion capture device 5 includes a communication unit 51 that enables communication with the data processing device 6A, a plurality of magnetic three-dimensional position and orientation sensors 52 that are devices attached to the fingers of the worker, and a signal processing unit 53 that outputs signals detected by the three-dimensional position and orientation sensors 52 in time series as motion information of the worker to the data processing device. As the three-dimensional position and orientation sensor 52, for example, a magnetic position and orientation sensor disclosed in Japanese Patent Application Laid-Open No. 2007-236602 is suitable.

[0106] The motion capture device 5 has multiple three-dimensional position and orientation sensors 52 arranged on a flexible glove, and is capable of detecting delicate movements of the fingers.

[0107] 17(A), the motion capture device 5 assumes that the center of the sensor 52a located at the tip of the index finger is the origin, and obtains information on the Euler angles indicating the posture determined from the position coordinates (x, y, z) and the rotation angles around the X-axis, Y-axis, and Z-axis as viewed from a fixed reference point of the signal processing unit 53. The Z-axis is an axis that passes through the origin of the XY plane in FIG. 17(A) and is perpendicular to the XY plane.

[0108] In addition to the functions of the synchronous measurement unit 601, the synchronous measurement unit 601A also converts and manages information regarding position and direction received from the motion capture device A so that it matches the coordinate space of the synchronized scanner 2 and eyewear device 4. For synchronization of the motion capture device 5, the tip of the index finger of the worker wearing the motion capture device 5 is placed at the reference point, the tip of the index finger is directed to match the reference direction of the scanner 2, and the position coordinates and Euler angles are set to 0.

[0109] In this way, the eyewear device 4 can grasp the position touched by the hand of the worker wearing the motion capture device 5.

[0110] Fig. 18 is an example of display on the display 41 by the display system SA. Fig. 18(A) to (D) Fig. 18(A) shows a current state image 41a. Fig. 18(B) shows a current state image 41a in a state where the rebar at the right end is grasped by the hand wearing the motion capture device 5. The rebar touched by the motion capture device 5 is highlighted. The arrangement state of the rebar touched by the motion capture device 5 (the type and spacing of the rebar in the figure) is also displayed.

[0111] 18(C) and (D) show a correction support image 41c. In Fig. 18(C) and (D), the work of correcting the erroneous arrangement of the reinforcing bar touched by the motion capture device 5 is displayed.

[0112] In this way, the worker can easily identify the rebar he or she touched from among the many rebars displayed in the image, and can also easily grasp the work required to correct the reinforcing bar placement error.

[0113] In this modified example, a motion capture device is used to recognize hands, but this is not limited to the above. Hands can also be recognized from skin color and shape using image processing, making it possible to identify the rebar that has been touched by the hand.

[0114] 10. Another example of an inspection system The 3D point cloud data 71 and the bar arrangement inspection result data 75 used in the display system according to this embodiment do not necessarily have to be acquired by the inspection system 100 of the above embodiment. The three-dimensional point cloud data may be acquired by a laser scanner with the position and direction in the inspection range known, and the bar arrangement inspection result data 75 may be used if the bar arrangement inspection result obtained as three-dimensional data is available. For example, the data may be acquired using the inspection system 200 described below as the inspection system.

[0115] FIG. 19 is an external view showing a state in which a bar arrangement inspection system 200 according to the second embodiment is in use, and FIG. 20 is a block diagram.

[0116] The system 200 includes a scanner 2, a data processing device 206, a flying device 8 equipped with a camera 250, and a surveying instrument 9. The scanner 2, the data processing device 206, the flying device 8 equipped with a prism, and the surveying instrument 9 are wirelessly connected and can transmit and receive information to and from each other. The scanner 2 is the same as the scanner 2 according to the first embodiment.

[0117] The flight device 8 is an unmanned air vehicle (UAV) capable of autonomous flight. The flight device 8 includes a plurality of propellers 8b extending radially from a main body 8a, a prism 8c as a target, and a camera 250 for capturing image data of an inspection target. The flight device 8 can fly along a predetermined flight path or fly freely by remote control. The flight device 8 includes an IMU device and a timer (not shown) in the main body 8a. The positional relationship between the IMU (Inertial Measuring Unit) and the camera 250 is known, and the direction and attitude of the camera 250 can be known.

[0118] The surveying instrument 9 is a motor-driven total station equipped with an automatic tracking function. While flying the flying device 8 in the inspection range, image data of the inspection range can be acquired by synchronizing the image capturing timing of the camera 250 and the image capturing timing of the surveying instrument.

[0119] The data processing device 206 has a configuration equivalent to that of the data processing device 6, and is capable of acquiring the reinforcing bar inspection result data 75 equivalent to that of the inspection system 100, using the 3D point cloud data 71 acquired by the scanner 2 and the image data acquired by the camera 250.

[0120] The 3D point cloud data and image data of the entire inspection range may be acquired by a scanner whose position and orientation are known and a camera whose position and orientation are known, respectively, and are not limited to the above examples. For example, the camera may be replaced by an omnidirectional camera whose position and orientation (attitude) are known. Also, the scanner may be attached to an airborne device instead of a ground-based scanner.

[0121] The above describes preferred embodiments of the present invention. However, the above embodiments are merely examples of the present invention, and these can be combined based on the knowledge of those skilled in the art, and such combinations are also included in the scope of the present invention. [Explanation of symbols]

[0122] 4: Eyewear display device 31: Communications Department 41: Display 41a: Current state image 41b: Test result image 41c: Correction support image 44: Communications Department 45: Relative position sensor 46: Relative direction sensor 51: Communications Department 52a: Sensor 60: Control calculation section 63: Communications Department 70: 3D reinforcement design data generator 71: 3D point cloud data 73: Reinforcement design data 200: System 262: 3D coordinate measurement section S: Reinforcement inspection result display system

Claims

1. A measuring instrument having a three-dimensional coordinate measurement unit; an eyewear display device including a display, a relative position detection sensor that detects its own position, and a relative direction detection sensor that detects its own direction; and a coordinate space of information relating to the self-position and the self-orientation of the eyewear display device and a coordinate space of the measuring device in a space having a common reference point as an origin; The processor: generating a current three-dimensional model of the inspection range based on three-dimensional point cloud data of the inspection range acquired by the measuring device with the position and orientation known; generating three-dimensional inspection result display data by associating three-dimensional reinforcement inspection result data, in which reinforcement errors and their positions within the inspection range are associated with the three-dimensional model; A reinforcement bar arrangement inspection result display system characterized in that an inspection result image is superimposed on the actual object observed on the eyewear display device on the display, and the reinforcement bar arrangement error is displayed so that it can be recognized.

2. the processor generates correction support data for correcting the reinforcement error based on the three-dimensional reinforcement design data in the inspection range, 2. The reinforcement arrangement inspection result display system according to claim 1, wherein a correction support image for supporting an operation to correct the reinforcement arrangement error is displayed on the display based on the correction support data.

3. Further, a motion capture device is provided, The reinforcement bar arrangement inspection result display system according to claim 1, characterized in that the processor is capable of recognizing the worker's hand within the field of view of the eyewear display device using the motion capture device, and displays the reinforcement bar arrangement state of the rebar touched by the worker on the display.

4. Further comprising a motion capture device, The reinforcement bar arrangement inspection result display system according to claim 2, characterized in that the processor is capable of recognizing the worker's hand within the field of view of the eyewear display device, and displays on the display, as the correction support image, the work of correcting the reinforcement bar arrangement error for the rebar that the worker has touched.

5. The reinforcement inspection result display system described in Claim 1, characterized in that the processor is capable of recognizing the worker's hand within the field of view of the eyewear display device through image processing, and displays the reinforcement state of the reinforcing bars touched by the worker on the display.

6. The reinforcement inspection result display system described in Claim 2, characterized in that the processor is capable of recognizing the worker's hands within the field of view of the eyewear display device through image processing, and the correction assistance image on the display shows the work of correcting the reinforcement error for the rebar that the worker has touched.

7. The reinforcement inspection result display system described in Claim 1, characterized in that the three-dimensional reinforcement inspection result data is generated as a result of identifying the reinforcement state contained in a point cloud composite image obtained by combining the three-dimensional point cloud data of the inspection range and the image data of the inspection range using a reinforcement state identification model obtained by learning from a large number of training point cloud composite images created for various reinforcement states, and comparing it with the three-dimensional reinforcement design data of the inspection range.

8. Further comprising at least one camera that acquires image data of the inspection range with known coordinates and orientation; the measuring device is a three-dimensional laser scanner; the processor generates a point cloud composite image by combining the three-dimensional point cloud data of the inspection range and the image data of the inspection range; The reinforcement inspection result display system according to claim 1, characterized in that it identifies the reinforcement state and position of the reinforcing bars contained in the point cloud composite image, compares the point cloud composite image with three-dimensional reinforcement design data for the inspection range, and generates the three-dimensional reinforcement inspection result data.

Images