Bar arrangement inspection device, bar arrangement inspection method, and program
The reinforcement inspection device uses 3D reconstruction and image conversion to address occlusion issues in monocular camera systems, enabling precise detection and measurement of reinforcing bars in multiple layers.
Patent Information
- Application Number
- JP2023084228
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-12-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing reinforcement inspection systems using monocular cameras struggle with accurately detecting reinforcing bars in multiple layers due to occlusion issues, where lower layer bars are obscured by upper layers, leading to inaccurate detection.
A reinforcement inspection device that utilizes a monocular camera to capture images from different angles, calculates the camera's posture and movement based on marker changes, generates 3D point cloud data, creates a plane mask image to exclude non-reinforcing bar elements, and converts images to a frontal view for precise detection and measurement of reinforcing bars.
Enables accurate detection and measurement of reinforcing bars in each layer by overcoming occlusion, ensuring reliable inspection results through 3D reconstruction and image processing techniques.
Smart Images

Figure 2024167921000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a reinforcement inspection device, a reinforcement inspection method, and a program. [Background technology]
[0002] In the construction of reinforced concrete structures, a reinforcing bar inspection is performed to check whether the reinforcing bars, which are assembled together, are arranged as designed. For example, the reinforcing bar inspection system described in Patent Document 1 generates a superimposed image by superimposing multiple images of a marker placed on a specific layer of reinforcing bars taken from different angles by a monocular camera, and generates a target layer image of the specific layer of reinforcing bars in the superimposed image. The reinforcing bar inspection system then extracts the reinforcing bar area of the reinforcing bar based on the target layer image, calculates the number of center lines included within a specific range among the center lines of the reinforcing bar area, and judges whether the number of center lines matches the design data. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6801055 Summary of the Invention [Problem to be solved by the invention]
[0004] The reinforcing bar inspection system described in Patent Document 1 converts multiple images taken from different angles by a monocular camera into semi-transparent converted images, generates a superimposed image by superimposing these converted images, and generates a target layer image by leaving the parts of the superimposed image where objects in each converted image overlap and are displayed dark, and erasing the parts where the objects do not overlap. Therefore, there is a possibility that the reinforcing bar image of the upper layer and the reinforcing bar image of the lower layer may be misaligned and overlap, and in this case, there is a problem that the reinforcing bar cannot be detected accurately. For example, if the reinforcing bar of the upper layer occludes the reinforcing bar of the lower layer, the reinforcing bar of the lower layer hidden by the reinforcing bar of the upper layer cannot be detected.
[0005] The present disclosure is devised to solve the above-mentioned problems, and aims to provide a reinforcing bar inspection device that can detect reinforcing bars in each layer of reinforcing bars using images captured by a monocular camera. [Means for solving the problem]
[0006] The reinforcement bar inspection device according to the present disclosure includes an image acquisition unit that acquires multiple images of reinforcement bars having a plane made of multiple layers of rebars arranged in a lattice pattern, each captured from different angles by a monocular camera from the plane side including the rebars to which markers are attached; a three-dimensional restoration unit that calculates the shooting attitude and movement distance of the monocular camera based on changes in the size, position, and shooting angle of the markers between the captured images, and calculates three-dimensional point cloud data representing the reinforcement bar based on the calculated shooting attitude and movement distance of the monocular camera; a plane detection unit that uses the three-dimensional point cloud data to generate a plane mask image in which parts other than the rebars on the plane are masked; an image conversion unit that converts the captured image and the plane mask image into front-facing images in which the rebars are perpendicular to each other; a rebar detection unit that detects the rebars on the plane using the front-facing image of the captured image and the front-facing image of the plane mask image; and a measurement processing unit that measures the rebars on the plane based on the results of correlation processing between feature amounts of the detected rebars and feature amounts of rebars registered in advance. Effect of the Invention
[0007] According to the present disclosure, a plurality of images of reinforcement are taken from different angles by a monocular camera from the plane side including the rebars with markers, the shooting attitude and movement distance of the monocular camera are calculated based on the change in the markers between these images, and three-dimensional point cloud data representing the reinforcement is calculated based on the shooting attitude and movement distance of the monocular camera. Then, a plane mask image is generated using the three-dimensional point cloud data, the photographed images and the plane mask image are converted into front-facing images, the rebars are detected using the front-facing images of the photographed images and the front-facing images of the plane mask image, and the rebars are measured using the feature quantities of the detected rebars. In this way, the rebar arrangement inspection device according to the present disclosure can detect the rebars of each layer of reinforcement using the images photographed by the monocular camera. [Brief description of the drawings]
[0008] [Figure 1] 1 is a block diagram showing a configuration example of a bar arrangement inspection device according to a first embodiment. FIG. [Diagram 2] FIG. 1 is a schematic diagram showing an overview of the photographing process and the three-dimensional restoration process of reinforcement. [Diagram 3] 3A, 3B, and 3C are graphs showing functions indicating plane candidates and the relationship to outliers and inliers. [Figure 4] FIG. 1 is a schematic diagram showing an overview of a plane misselection and correction of a plane selection. [Diagram 5] FIG. 13 is a schematic diagram showing an overview of a conversion process to a front-facing image. [Figure 6] FIG. 13 is a schematic diagram showing an overview of a process for detecting reinforcing bars from a frontal image. [Figure 7] 7A, 7B, and 7C are schematic diagrams showing an overview of a process for identifying line segments corresponding to rebar portions in a planar mask image. [Figure 8] FIG. 13 is a schematic diagram showing an overview of characteristics of a line segment corresponding to a rebar portion in a planar mask image. [Figure 9] 9A, 9B, and 9C are schematic diagrams showing an outline of the correction process for line segments corresponding to reinforcing bar portions in a planar mask image. [Figure 10] FIG. 13 is a schematic diagram showing an outline of a process for measuring the distance between reinforcing bars in a captured image. [Figure 11] FIG. 13 is a schematic diagram showing an overview of a scaling process for an image of a reinforcing bar portion. [Figure 12] 12A and 12B are schematic diagrams showing an overview of a process for detecting an image of a reinforcing bar portion. [Figure 13] 1 is a schematic diagram showing an outline of a process for extracting feature amounts of a reinforcing bar portion from a captured image; [Figure 14] FIG. 13 is a schematic diagram showing an outline of a correction process for a frequency spectrum of luminance values in an image of a reinforcing bar portion. [Figure 15] FIG. 13 is a schematic diagram showing an outline of a correlation process between a frequency spectrum obtained from a frontal image and a frequency spectrum registered in advance. [Figure 16] 4 is a flowchart showing a bar arrangement inspection method according to the first embodiment. [Figure 17] 17A and 17B are block diagrams showing a hardware configuration for realizing the functions of the bar arrangement inspection device according to the first embodiment. [Figure 18] FIG. 11 is a block diagram showing a configuration example of a bar arrangement inspection device according to a second embodiment. [Figure 19] FIG. 13 is a schematic diagram showing an overview of a process for extracting a partial image of a reinforcing bar. [Figure 20] FIG. 1 is a schematic diagram showing an overview of the inference process of rebar diameter using a learning model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Embodiment 1 FIG. 1 is a block diagram showing a configuration example of a reinforcing bar inspection device 1 according to a first embodiment. In FIG. 1, the reinforcing bar inspection device 1 is a device that inspects at least one of the number of reinforcing bars, the spacing between adjacent reinforcing bars, the diameter of the reinforcing bars, and the spacing between nodes of the reinforcing bars in a reinforcing bar arrangement having a multi-layer plane made of reinforcing bars arranged in a lattice pattern. The reinforcing bar inspection device 1 is a tablet terminal, a smartphone, or a notebook-type personal computer (PC). As shown in FIG. 1, a monocular camera 2 is connected to the reinforcing bar inspection device 1 by wire or wirelessly. The reinforcing bar inspection device 1 is not limited to a device with an external monocular camera 2, and may have the monocular camera 2 built-in.
[0010] In a reinforcement inspection, an inspector carries a reinforcement inspection device 1 and moves around the reinforcement to be inspected, photographing the reinforcement with a monocular camera 2 connected to the reinforcement inspection device 1. The monocular camera 2, which moves with the inspector, can photograph the reinforcement from multiple angles. In addition, the moving object moving around the reinforcement is not limited to an inspector, but may be a vehicle, an unmanned aerial vehicle, etc.
[0011] The reinforcing bar inspection device 1 may be a component included in a server capable of communicating with a terminal device equipped with a monocular camera 2. For example, the terminal device can perform reinforcing bar inspection provided in the form of SaaS (Software as a Service). When performing reinforcing bar inspection in the form of SaaS, an information processing application for reinforcing bar inspection does not need to be installed on the terminal device. The information processing application for reinforcing bar inspection is executed on the server, and the terminal device is provided with measurement result information on a general-purpose web browser. The reinforcing bar inspection application is stored in a storage unit included in the server. Furthermore, an information processing application for bar arrangement inspection may be installed in the terminal device. In the terminal device in which the information processing application for bar arrangement inspection is installed, the application is executed to enable bar arrangement inspection.
[0012] The monocular camera 2 is a camera that is less expensive than a stereo camera, and as described above, is assumed to be a camera that is equipped on a tablet terminal, a smartphone, or a notebook PC. The bar arrangement inspection device 1 inputs images of the bar arrangement captured at different angles by the monocular camera 2, and can use these captured images to distinguish the reinforcing bars on the plane of each layer of the bar arrangement.
[0013] The reinforcing bar inspection device 1 is a device equipped with at least a calculation unit and a storage unit. The calculation unit controls the overall operation of the reinforcing bar inspection device 1. The calculation unit includes an image acquisition unit 11, a three-dimensional restoration unit 12, a plane detection unit 13, an image conversion unit 14, a reinforcing bar detection unit 15, and a measurement processing unit 16. The calculation unit executes an information processing application for reinforcing bar inspection, thereby realizing various functions of the image acquisition unit 11, the three-dimensional restoration unit 12, the plane detection unit 13, the image conversion unit 14, the reinforcing bar detection unit 15, and the measurement processing unit 16.
[0014] The storage unit stores an information processing application for the bar arrangement inspection and information used for the calculation processing of the calculation unit. The storage unit is a storage device provided in a computer functioning as the bar arrangement inspection apparatus 1, and includes a storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or a memory 104 in Fig. 17B described later. Note that the storage unit may be provided outside the bar arrangement inspection apparatus 1 as long as it is accessible by the bar arrangement inspection apparatus 1.
[0015] The image acquisition unit 11 acquires a plurality of images of a reinforcing bar arrangement having a plane of multiple layers of reinforcing bars arranged in a lattice pattern, each of which is photographed from different angles by the monocular camera 2 from the plane side including the reinforcing bars to which the markers are attached. FIG. 2 is a schematic diagram showing an overview of the photographing process and the three-dimensional restoration process of the reinforcing bar arrangement 3, and shows photographed images 21A, 21B, and 21C of the reinforcing bar arrangement 3 having a plane in which multiple reinforcing bars 4 are arranged in a lattice pattern. Markers 5 are laid at positions A, B, C, and D in the plane. The positions at which the markers 5 are laid are positions that define the inspection target area in the reinforcing bar arrangement 3. In FIG. 2, the inspection target area is a rectangular area defined by positions A, B, C, and D.
[0016] As described above, a plurality of markers 5 are provided on the plane of the reinforcement 3, and have a polygonal outer shape. When the markers 5 having a polygonal outer shape are photographed from different shooting angles, the outer shape appears distorted depending on the shooting angle. By analyzing this distortion, the shooting angle can be identified, and the shooting posture of the monocular camera 2 according to the shooting angle can be estimated.
[0017] 2, the marker 5 is an identification shape 6B drawn on a square background 6A, for example, an AR (augmented reality) marker. The captured images 21A, 21B, and 21C are captured by the monocular camera 2 at different capture angles so as to include all four markers 5.
[0018] The three-dimensional restoration unit 12 calculates the shooting attitude and movement distance of the monocular camera 2 based on the changes in size, position, and shooting angle of the marker 5 between the captured images, and calculates three-dimensional point cloud data representing the reinforcement 3 based on the calculated shooting attitude and movement distance of the monocular camera 2. For example, the three-dimensional reconstruction unit 12 executes SfM (Structure from Motion) on the captured images 21A, 21B, and 21C. SfM is a method for estimating the shape and position of the marker 5 in a three-dimensional space from the captured images 21A, 21B, and 21C. The three-dimensional reconstruction unit 12 identifies the position of the marker 5 from the captured images 21A, 21B, and 21C by performing image analysis such as pattern matching on each of the captured images 21A, 21B, and 21C. Next, the three-dimensional reconstruction unit 12 detects the contour of the base 6A of the marker 5 by binarizing the captured images 21A, 21B, and 21C, and detects the marker 5 as a feature point based on the contour.
[0019] The three-dimensional reconstruction unit 12 calculates the shooting position and shooting attitude of the monocular camera 2 based on the changes in size, position and shooting angle of the multiple markers 5 in the multiple captured images. Next, the three-dimensional reconstruction unit 12 calculates three-dimensional point cloud data using the shooting position and shooting attitude of the monocular camera 2 and the shooting position and movement distance of the monocular camera 2 calculated from a pair of mutually different captured images.
[0020] For example, the three-dimensional reconstruction unit 12 extracts the marker 5 as a feature point from the photographed images 21A, 21B, and 21C. Methods for extracting feature points include, for example, Scale Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF). Next, the three-dimensional reconstruction unit 12 performs feature point matching by comparing feature points in the pair of photographed images 21A and 21B and associating images with similar features. Then, similar feature point matching is performed for the pair of photographed images 21A and 21C and the pair of photographed images 21B and 21C.
[0021] The three-dimensional restoration unit 12 performs feature point matching on the pair of the photographed images 21B and 21C to associate the marker 5 at position A in the photographed image 21B with the marker 5 at position A in the photographed image 21C. The three-dimensional restoration unit 12 calculates the photographing position and photographing attitude of the monocular camera 2 in three-dimensional space based on the changes in the size, position and photographing angle of the associated marker 5.
[0022] For example, the three-dimensional reconstruction unit 12 determines the distance between the monocular camera 2 and the reinforcement 3 based on the size of the marker 5 in the captured image, whose outer shape is formed by a line segment connecting vertex P1 and vertex P2, a line segment connecting vertex P2 and vertex P3, a line segment connecting vertex P3 and vertex P4, and a line segment connecting vertex P4 and vertex P1, as shown in Fig. 2. Furthermore, the three-dimensional reconstruction unit 12 judges that a captured image in which the shape formed by the above-mentioned line segments is a square is a front view of the plane of the reinforcement 3 captured by the monocular camera 2. The three-dimensional reconstruction unit 12 determines that a captured image in which any of the four vertices P1 to P4 is not detected or the shape of the base 6A is detected as being distorted is a captured image in which the monocular camera 2 captured the plane of the reinforcement 3 from an oblique direction.
[0023] The three-dimensional reconstruction unit 12 estimates the relative positional relationship between the pair of captured images based on the positional relationship between corresponding feature points in the pair of different captured images. For example, when the two captured images are captured by translating the monocular camera 2, the three-dimensional reconstruction unit 12 estimates the amount of movement in the parallel direction between the distance between the corresponding feature points, and when the corresponding feature points are rotated, estimates the amount of rotation by calculating the angle difference between the corresponding feature points. Using the calculated amount of movement or rotation of the monocular camera 2, the three-dimensional reconstruction unit 12 calculates the capturing positions and capturing postures at which the monocular camera 2 captured the captured images 21A, 21B, and 21C.
[0024] Next, the three-dimensional restoration unit 12 calculates three-dimensional point cloud data using the shooting position and movement distance of the monocular camera 2. For example, the three-dimensional restoration unit 12 calculates three-dimensional point cloud data representing the reinforcement 3 by converting position information of each pixel of the captured images 21A, 21B, and 21C into a three-dimensional coordinate system using the shooting position and movement distance of the monocular camera 2.
[0025] The plane detection unit 13 generates a plane mask image using the three-dimensional point cloud data calculated by the three-dimensional reconstruction unit 12. The plane mask image is an image in which parts other than the reinforcing bars 4 on the plane of the reinforcement 3 are masked. For example, the plane detection unit 13 generates a first mask image based on the Harris operator. The Harris operator is a technology that generates a three-layer pyramid image from a captured image, extracts Harris features of each layer, and generates a plane mask image by threshold processing of the features. The three-layer pyramid image is composed of an image of the first layer, an image of the second layer, and an image of the third layer. The image of the first layer is an image that is magnified to the same size as the original captured image. The image of the second layer is an image (1 / 2 image) that is reduced to 1 / 2 the image of the first layer, and the image of the third layer is an image (1 / 4 image) that is reduced to 1 / 2 the image of the second layer.
[0026] The plane detection unit 13 extracts Harris features that indicate corner-likeliness from each of the first layer image, the second layer image, and the third layer image, and identifies parts corresponding to corners of objects in the images by comparing the Harris features with a predetermined threshold, and generates a mask image in which parts other than the identified parts are masked. Next, the plane detection unit 13 resizes the mask images of each layer generated based on the first layer image, the second layer image, and the third layer image, respectively, to their original size, and generates a first mask image by performing a logical OR operation on these images.
[0027] Next, the plane detection unit 13 generates a second mask image based on depth using the three-dimensional point cloud data. For example, the plane detection unit 13 separates a three-dimensional image (depth map) made up of the three-dimensional point cloud data into a shallow area and a deep area, and generates front and back mask images. Then, the plane detection unit 13 generates a second mask image for removing elements related to the background, for example, by leaving pixels with a predetermined brightness or higher in the front mask image and masking the other pixels.
[0028] The plane detection unit 13 generates a sampling mask for plane detection processing by combining the first mask image and the second mask image. Next, the plane detection unit 13 estimates a plane by RANSAC using the sampling mask and the depth map. In RANSAC, plane candidates are repeatedly estimated using point cloud data randomly sampled from the depth map using the sampling mask. The plane detection unit 13 detects, among the plane candidates estimated by RANSAC, the one with the most inliers, which are three-dimensional points whose distance from the plane candidate is within a set range, as the plane of each story of the reinforcement 3.
[0029] 3A, 3B, and 3C are graphs showing the relationship between functions P(1), P(2), and P(3) indicating plane candidates and outliers 31 and inliers 32. These graphs show the relationship between functions P(1), P(2), and P(3) indicating planes of each layer and outliers 31 and inliers 32, and show three-dimensional points in the XY coordinate system. Outliers 31 are three-dimensional points that are not included in an acceptable range, and inliers 32 are three-dimensional points that are included in an acceptable range.
[0030] In the RANSAC method, the number of three-dimensional points that are inliers 32 is counted for each parameter representing the functions P(1), P(2), and P(3), and the parameter with the largest count is determined as the optimal parameter. In other words, the plane candidate represented by the function to which the determined parameters are applied is determined as the plane estimation result. As is clear from Figures 3A, 3B, and 3C, the number of three-dimensional points that are inliers 32 is the largest for the parameters representing the function P(3), so the plane detection unit 13 detects the plane candidate represented by the function P(3) as the plane of the reinforcement 3.
[0031] The plane detection unit 13 estimates plane parameters representing plane candidates selected from a plurality of plane candidates determined using the three-dimensional point cloud data according to the number of three-dimensional points whose distance from the monocular camera 2 to the plane is within a set distance, and generates a plane mask image using the plane parameters. For example, the plane detection unit 13 identifies the foremost plane of the reinforcement 3, estimates plane parameters representing the identified plane, and generates a plane mask image using the estimated plane parameters. The criterion for determining the front side is that the closer an object is to the monocular camera 2, i.e., the closer it is to the front side, the larger it appears in the image and the less obstruction there is between the object and the monocular camera 2. Note that appearing larger in the image indicates a large number of three-dimensional points, and less obstruction between the object and the monocular camera 2 indicates less occlusion of the lower layer reinforcing bars by the upper layer reinforcing bars.
[0032] For example, the plane detection unit 13 randomly selects N three-dimensional points multiple times and performs the above process to estimate multiple plane candidates. Then, the plane detection unit 13 counts three-dimensional points whose distance from the monocular camera 2 is equal to or less than a threshold among the multiple plane candidates, and determines that the planes containing the most counted three-dimensional points are the front planes according to the criteria for front surface determination. That is, the plane candidate with the most counted three-dimensional points is determined to be the foreground plane.
[0033] However, due to insufficient adjustment of the criteria for determining the front side or the presence of an obstacle between the monocular camera 2 and the reinforcement 3, two or more plane candidates may be detected near the front side (top layer). FIG. 4 is a schematic diagram showing an overview of the erroneous selection of plane 41 and the correction of the selection of plane 41, showing the monocular camera 2 and the plane of the reinforcement 3 as viewed from above. For example, if the criteria for determining the front side is simply the proximity of the monocular camera 2 to the three-dimensional point 40, as shown in the left diagram of FIG. 4, a plane 42 tilted toward the monocular camera 2 may be determined as the frontmost plane of the reinforcement 3. Therefore, the plane detection unit 13 performs a process of correcting the plane 42 determined as the frontmost plane of the reinforcement 3 to the plane 41, as shown in the right diagram of FIG. 4.
[0034] In order to correct the plane 42 to the plane 41, the plane detection unit 13 calculates the plane parameters representing the plane to be inspected by repeatedly calculating the plane parameters of the plane candidate and updating a histogram indicating the amount of error of the plane based on the normal of the plane candidate. For example, the plane detection unit 13 estimates a plane using RANSAC and calculates a histogram based on the normal of the estimated plane candidate. This histogram visualizes the direction of the normal as a distribution of three-dimensional points. Next, the plane detection unit 13 recalculates plane parameters of a function indicating the plane candidate using the least squares method for the three-dimensional point cloud data. Then, the plane detection unit 13 recalculates a histogram of the normal of the plane candidate defined by the calculated plane parameters, and updates the histogram. By repeating these processes, the plane of the foreground of the reinforcement 3 is corrected from plane 42 to plane 41.
[0035] The plane detection unit 13 generates a plane mask image using plane parameters representing a plane (foreground) estimated from the three-dimensional point cloud data as described above. For example, the plane detection unit 13 generates a plane mask image from a pair of different captured images using the three-dimensional point cloud data. Hereinafter, it is assumed that the three-dimensional reconstruction unit 12 calculates first three-dimensional point cloud data (first depth map) using the pair of captured images 21A and 21B shown in FIG. 2, and calculates second three-dimensional point cloud data (second depth map) using the pair of captured images 21B and 21C.
[0036] The plane detection unit 13 performs the above-mentioned processing on the first three-dimensional point cloud data to estimate the foreground plane of the reinforcing bars 3, and generates a first plane mask image by thresholding the distance from the estimated plane to the three-dimensional point. For example, the plane detection unit 13 defines a proximity range a certain distance (threshold) away from the plane indicated by the function represented by the estimated plane parameters, and masks the three-dimensional points outside this proximity range to generate a first plane mask image in which everything on the plane other than the reinforcing bars 4 is masked. The plane detection unit 13 performs the same processing on the second three-dimensional point cloud data to generate a second plane mask image.
[0037] The plane detection unit 13 generates a plane mask image using three-dimensional point cloud data calculated from a pair of different captured images, and synthesizes these plane mask images to obtain a final plane mask image. For example, the plane detection unit 13 calculates one plane mask image by performing a sum operation on a first plane mask image and a second plane mask image. This allows the bar arrangement inspection device 1 to accurately generate a plane mask image for extracting rebars included in a plane.
[0038] The image conversion unit 14 converts the captured image and the planar mask image into a front-facing image. The front-facing image is an image in which the reinforcing bars 4 arranged in a lattice pattern are perpendicular to each other. Fig. 5 is a schematic diagram showing an overview of the conversion process to a frontal image. Captured image 21D is a three-dimensional image made up of three-dimensional point cloud data generated by three-dimensional restoration unit 12. As shown in Fig. 5, rebar 4 in captured image 21D appears larger the closer it is to monocular camera 2, and appears smaller the farther it is from monocular camera 2. That is, rebar 4 located closer to monocular camera 2 is captured in the lower image area of captured image 21D, and rebar 4 located farther from monocular camera 2 is captured in the upper image area.
[0039] Therefore, image conversion unit 14 specifies the four corner points of any rectangle among the multiple rebars 4 captured in captured image 21D, and estimates a facing transformation matrix that causes the rectangle defined by the four corner points to have a shape as viewed from the front of monocular camera 2. Next, image conversion unit 14 uses the facing transformation matrix to convert captured image 21D into captured image 21E, which is a facing image. Image conversion unit 14 scales all pixels in this facing image so that the distance from monocular camera 2 is constant. This allows the difference in size of the reinforcing bar 4 in the captured image 21D depending on the distance from the monocular camera 2 to be corrected. The image conversion unit 14 also performs the same process on the planar mask image to convert the planar mask image into a front-facing image.
[0040] The reinforcing bar detection unit 15 detects the reinforcing bar 4 on the plane by using the front-facing image of the photographed image and the front-facing image of the plane mask image. For example, the reinforcing bar detection unit 15 defines the outline of the area in which the plane of the reinforcing bar 3 is captured on the front-facing image of the plane mask image, sets a circle inscribed in the area defined by this outline, and sets the area surrounded by this circle as a circular mask image. Next, the reinforcing bar detection unit 15 performs a Hough transform on the generated circular mask image, specifies an angle parameter θ of the circular mask image in the Hough space, and counts the number of white pixels at the position indicated by the position parameter ρ of the circular mask image. The reinforcing bar detection unit 15 detects the position of the reinforcing bar 4 based on the count number at the position indicated by the position parameter ρ by controlling the angle parameter θ to rotate the circular mask image.
[0041] FIG. 6 is a schematic diagram showing an overview of the process of detecting the reinforcing bar 4 from the front image. The reinforcing bar detection unit 15 identifies the image portion of the reinforcing bar 4 using the plane mask image 51 converted into the front image. The reinforcing bar detection unit 15 then generates the plane mask image 51 by setting the pixel value of the identified image portion to "1" and the other image portions to "0". For example, the reinforcing bar detection unit 15 detects the position where the count number of white pixels in the plane mask image 51 corresponding to the front of the reinforcing bar 3 is equal to or greater than a threshold value as the position of the reinforcing bar 4. As shown in FIG. 6, the reinforcing bar detection unit 15 can detect the position of the reinforcing bar 4 in the X direction and the Y direction by rotating the plane mask image 51 and counting the number of white pixels. This allows the reinforcing bar detection unit 15 to detect the position information of the reinforcing bars arranged vertically and the position information of the reinforcing bars arranged horizontally on the plane of each layer of the reinforcing bar 3.
[0042] The reinforcing bar detection unit 15 performs a process of identifying line segments corresponding to reinforcing bars in the planar mask image 51. FIGS. 7A, 7B, and 7C are explanatory diagrams showing a process of identifying line segments corresponding to reinforcing bar portions in the planar mask image 51. As shown in FIG. 7A, when the planar mask image 51 has white pixel regions A, B, and C, it is unclear which region corresponds to the image of the reinforcing bar 4. The reinforcing bar detection unit 15 calculates, from the planar mask image 51, the length D(1) of region A, the length D(2) of region B, and the length D(3) of region C on the coordinate axis passing through the white pixel regions A, B, and C, the distance D(4) between region A and region B, and the distance D(5) between region B and region C.
[0043] As shown in FIG. 7B, the reinforcing bar detection unit 15 determines that between the intervals D(4) and D(5) that are spaced apart by a threshold value or more, they do not correspond to image areas of the reinforcing bar 4, and determines that between the intervals that are spaced apart by less than the threshold value, they are image areas that correspond to the same reinforcing bar 4. 7C , the reinforcing bar detection unit 15 determines that the image area indicated by the line segment 52, which passes through area A and area B separated by an interval D(4) less than the threshold and has a length D(6), is an image area corresponding to the same reinforcing bar 4. In addition, since D(5) is equal to or greater than the threshold, the reinforcing bar detection unit 15 determines that there is no image area corresponding to the reinforcing bar 4 between area B and area C in the planar mask image 51. The reinforcing bar detection unit 15 assigns position information of the portion of the planar mask image 51 that is determined to be the image region of the reinforcing bar 4 to the front-facing image of the captured image, and outputs this front-facing image to the measurement processing unit 16.
[0044] Furthermore, the reinforcing bar detection unit 15 may correct the edge positions of line segments corresponding to reinforcing bar portions in the front-facing image of the planar mask image based on the brightness edges in the front-facing image of the captured image. In this case, the rebar detection unit 15 inputs a planar mask image converted into a front-on image and a captured image converted into a front-on image, and generates a planar mask image in which the contour areas of the lines corresponding to the rebar parts are corrected.
[0045] FIG. 8 is a schematic diagram showing an overview of the characteristics of the line segments corresponding to the reinforcing bar portions in the planar mask image 51. The reinforcing bar detection unit 15 converts the captured image converted into a frontal image into an image 60 that can be rotated around the midpoint of the line segment corresponding to the reinforcing bar portion. The diagram on the left side of FIG. 8 shows this image 60. For example, in the image 60 shown in the diagram on the left side of FIG. 8, the midpoint of the line segment corresponding to the reinforcing bar portion is a point of x coordinate=Cx'' and can be rotated around this point. The rectangular area enclosed by the dashed line includes the line segment corresponding to this reinforcing bar portion.
[0046] The reinforcing bar detection unit 15 corrects the angle and position of the line segment corresponding to the reinforcing bar portion in the image 60 on the basis of the luminance gradient of the image 60. This correction is performed for each reinforcing bar in the image 60. The diagram in the center of FIG. 8 shows a histogram 61 obtained for each X coordinate by integrating the luminance values of each pixel in the pixel row aligned in the Y direction of the rectangular area shown in the diagram on the left side of FIG. 8. In the rectangular area, the area outside the reinforcing bar portion is a portion with a low luminance value corresponding to the background, and the line segment corresponding to the reinforcing bar portion has a high luminance value. The diagram on the right side of FIG. 8 shows the result of quadratic differentiation of the histogram 61 shown in the diagram in the center of FIG. 8, and strong peaks 62 and 63 appear at the boundary between the line segment corresponding to the reinforcing bar portion and the background portion.
[0047] 9A, 9B, and 9C are schematic diagrams showing an outline of the correction process of the line segment corresponding to the reinforcing bar portion in the image 60. FIG. 9A shows the state in which the image 60 is gradually rotated. The reinforcing bar detection unit 15 calculates the luminance gradient in the direction perpendicular to the line segment while rotating the image 60 around the center point of the line segment corresponding to the reinforcing bar portion, and determines the rotation angle at which the luminance gradient is maximum as the final angle to be reflected in the planar mask image 51 converted into a frontal image. FIG. 9B and FIG. 9C show rectangular areas 60A and 60B at different rotation angles and waveforms showing the luminance gradients (waveforms obtained by second-order differentiation of the histogram 61 shown in FIG. 8).
[0048] At the rotation angle of rectangular region 60A, rectangular region 60A including the reinforcing bar portion is inclined with respect to the Y axis, so there is no peak in the brightness gradient in the X direction, and the boundary between the line segment corresponding to the reinforcing bar portion and the masked portion of the background may become blurred. On the other hand, at the rotation angle of rectangular region 60B, rectangular region 60B including the reinforcing bar portion is aligned with the Y axis, so a peak corresponding to the edge portion appears in the brightness gradient in the X direction, and the boundary between the line segment corresponding to the reinforcing bar portion and the background is clear. Reinforcing bar detection unit 15 reflects the correction of the edge position of the line segment corresponding to the reinforcing bar portion in planar mask image 51 by rotating planar mask image 51 at a rotation angle that results in rectangular region 60B.
[0049] Although the correction for the reinforcing bar portion along the Y direction has been described, the edge positions of the line segments corresponding to the reinforcing bar portion along the X direction can also be corrected by similar processing. This allows the reinforcing bar arrangement inspection device 1 to generate a plane mask image 51 that can accurately extract the reinforcing bars 4 included in the plane.
[0050] The measurement processing unit 16 measures the reinforcing bars 4 in the reinforcing bars 3 by using the feature amount of the reinforcing bars 4 detected by the reinforcing bar detection unit 15. For example, the measurement processing unit 16 measures the interval between adjacent reinforcing bars on the plane of the inspection target. FIG. 10 is a schematic diagram showing an overview of the process of measuring the interval between reinforcing bars in a photographed image 21E converted into a front-facing image. As shown in the left diagram of FIG. 10, the measurement processing unit 16 sets a rectangle 112 circumscribing an image area 111 corresponding to the plane of the front of the reinforcing bars 3 in the photographed image 21E. Then, the measurement processing unit 16 calculates an intersection point P between a line segment L2 that passes through the center 113 of the rectangle 112 and is perpendicular to a line segment L1 having an average gradient of each reinforcing bar 4 detected from the image area 111 by the reinforcing bar detection unit 15, and a line segment 4A corresponding to the reinforcing bar portion whose edge position has been corrected.
[0051] Next, the measurement processing unit 16 calculates representative points p1 and p2 in the two-dimensional image (photographed image 21E) shown on the left side of FIG. 10 by multiplying the photographed image 21E by a rotation inverse matrix. The representative point p1 is a point included in the reinforcing bar in the two-dimensional image, and the representative point p2 is a point included in the reinforcing bar adjacent to the reinforcing bar including the representative point p1 in the two-dimensional image. As shown in the diagram on the right side of FIG. 10, the measurement processing unit 16 converts the representative points p1 and p2 of the two-dimensional image into points P1 and P2 in an area 111A of a three-dimensional coordinate space corresponding to the image area 111. The area 111A is an area corresponding to the plane of the front surface of the reinforcement 3. The measurement processing unit 16 can obtain the interval between the reinforcing bars 4 by measuring the interval between the points P1 and P2.
[0052] The measurement processing unit 16 may perform scaling to match the image scale between the captured image of the reinforcing bar 4 detected by the reinforcing bar detection unit 15 and the preregistered captured image of the reinforcing bar 4 so that the distance from the monocular camera 2 to the reinforcing bar 4 is constant. This enables the reinforcing bar inspection device 1 to accurately compare feature amounts of the reinforcing bar 4 detected from captured images taken at different positions and attitudes.
[0053] Fig. 11 is a schematic diagram showing an overview of the scaling process of a captured image 21E of a reinforcing bar portion. The reinforcing bar inspection device 1 sequentially registers inspection result information of the reinforcing bars 4 of the reinforcing bars 3 in an inspection result database stored in a memory 104 shown in Fig. 17B. The inspection result information includes not only the inspection results such as reinforcing bar diameter, reinforcing bar spacing, and number of reinforcing bars, but also the images used for these measurements. When a new photograph of the reinforcement 3 is taken, the measurement processing unit 16 performs a correlation calculation with the already registered inspection result information, and registers new inspection result information obtained from the photographed image that is evaluated to have a high correlation with the already registered inspection result information in the inspection result database.
[0054] For example, as shown in FIG. 11, the evaluation position is assumed to be 1.5 m. The evaluation position is a position for evaluating the correlation between the already registered result and the newly inspected result, and the distance to the reinforcing bar to be inspected is a predetermined value (for example, 1.5 m). The photographed image 21E is an image of the reinforcing bar to be inspected photographed at a photographing distance f, which is converted into a frontal image. The photographing distance f is assumed to be shorter than the above-mentioned distance at the evaluation position. When the measurement processing unit 16 inspects the reinforcing bar shown in the photographed image 21E, it generates a photographed image 21F by reducing the image size of the photographed image 21E to the image size when photographed at the evaluation position, and registers the photographed image 21F in the inspection result database by including it in the inspection result information.
[0055] The captured image 21G is an image of the reinforcing bar to be inspected, captured at a shooting distance F, converted into a frontal image. The shooting distance F is longer than the distance at the evaluation position. When the measurement processing unit 16 inspects the reinforcing bar shown in the captured image 21G, it generates a captured image 21H by enlarging the image size of the captured image 21G to the image size when captured at the evaluation position. Then, the measurement processing unit 16 performs a correlation calculation with the registered captured image 21F, and newly registers the inspection result information obtained from the captured image 21H evaluated to have a high correlation in the inspection result database.
[0056] Furthermore, the measurement processing unit 16 may calculate the intersections where the rebars 4 cross each other vertically based on the position parameters of the rebars 4 on the plane to be inspected, and extract feature quantities of the rebars 4 between the intersections from the front-facing image of the captured image. This allows only the rebars 4 between the intersections to be inspected, so that the reinforcing bar inspection device 1 can eliminate the influence of other rebars 4 assembled together with the rebar 4 to be inspected and accurately inspect the rebar 4 to be inspected.
[0057] 12A and 12B are schematic diagrams showing an overview of the process of detecting an image of a reinforcing bar portion. FIG. 12A shows a vertical reinforcing bar 4 to be inspected and horizontal reinforcing bars 4(0), 4(1), and 4(2) arranged so as to intersect with the vertical reinforcing bar. FIG. 12B shows a captured image 21H of the reinforcing bar converted into a frontal image. The measurement processing unit 16 sets a detection line for each reinforcing bar based on the position information of the reinforcing bar detected by the reinforcing bar detection unit 15. For example, as shown in FIG. 12B, the measurement processing unit 16 sets a detection line LB in the longitudinal direction of the reinforcing bar based on the position information of the reinforcing bar, and further sets detection lines LC1 and LC2 within a range of ±1 pixel and ±2 pixels based on the detection line LB.
[0058] The measurement processing unit 16 identifies the positions of the reinforcing bar 4 and the reinforcing bars 4(0), 4(1), and 4(2) within the range including the detection line LB and the detection lines LC1 and LC2, and calculates the intersections between the reinforcing bar 4 and each of the reinforcing bars 4(0), 4(1), and 4(2). The measurement processing unit 16 then sets a value of 0 to the pixels of a portion of the reinforcing bar 4 that is within a 10% length range from the intersection, of the length of the reinforcing bar 4 including the intersection, and removes it from the inspection target, and the remaining portion within a 80% length range is the segment of the reinforcing bar 4 to be inspected. Note that if the number of pixels in the segment of the reinforcing bar 4 to be inspected is less than the specified number of pixels (for example, 512 pixels), the measurement processing unit 16 assumes that the reinforcing bar 4 has not been clearly photographed, and sets a value of 0 to the pixels of that segment and removes it from the inspection target.
[0059] Furthermore, the measurement processing unit 16 acquires a scanning line consisting of the luminance values of a plurality of pixels along the longitudinal direction of the reinforcing bar 4 in the frontal image, for each pixel in a direction perpendicular to the longitudinal direction of the reinforcing bar 4. Then, the measurement processing unit 16 calculates multiple frequency spectra by frequency-converting the luminance values of the scanning lines for each pixel, and sets the median of the amount of change in luminance values for each frequency extracted from these frequency spectra as the feature amount of the reinforcing bar 4. This enables the reinforcing bar inspection device 1 to accurately extract the features of the reinforcing bar 4 from the captured image.
[0060] Fig. 13 is a schematic diagram showing an overview of the process of extracting feature amounts of a reinforcing bar portion from a captured image. The diagram on the left side of Fig. 13 shows captured image 21H of the reinforcing bar converted into a frontal image, and the image area of the reinforcing bar 4 is area 141 surrounded by a dashed line. Based on the position information of the reinforcing bar detected by reinforcing bar detection unit 15, measurement processing unit 16 sets scanning lines i=0, 1, 2, and 3 along the longitudinal direction of the reinforcing bar 4 within area 141. Scanning lines i=0, 1, 2, and 3 are set in a direction perpendicular to the longitudinal direction of the reinforcing bar 4 (the width direction of the reinforcing bar).
[0061] In the actual environment around the reinforcing bar 3, the reinforcing bar 4 in the reinforcing bar 3 may not have a uniform brightness due to the direction of sunlight, etc. In this case, the amount of change in luminance value in the image area of the reinforcing bar portion is also not uniform. For example, when sunlight is shining from the side of scanning line i=3, as shown in the center diagram of FIG. 13, for scanning lines i=0, 1, 2, and 3, the overall luminance value of the pixel row along scanning line i=3 is high, but the amount of change in luminance value along the scanning line is small. On the other hand, the pixel row along scanning line i=0 is on the shade side of the sunlight, so the overall luminance value is low, but the amount of change in luminance value along the scanning line is large.
[0062] Therefore, the measurement processing unit 16 performs frequency transformation (e.g., FFT; Fast Fourier Transform) on the luminance values of pixel rows along each of the scanning lines i=0, 1, 2, and 3 to calculate the frequency spectrum of each scanning line. The diagram on the right side of Fig. 13 shows frequency spectra SP0, SP1, SP2, and SP3 of the scanning lines i=0, 1, 2, and 3. Since the amount of change in luminance values along the scanning line i=0 is large, the frequency spectrum SP0 is a spectrum with a large amplitude. In contrast, since the amount of change in luminance values along the scanning line i=3 is small, the frequency spectrum SP3 is a spectrum with a small amplitude.
[0063] The measurement processing unit 16 sets segments seg0, seg1, seg2, seg3, and seg4 for each certain frequency range for the frequency spectrum of the scan lines i=0, 1, 2, and 3. Then, the measurement processing unit 16 calculates the median of the amount of change in luminance value along the longitudinal direction of the rebar for each scan line and each frequency band (for each segment).
[0064] The measurement processing unit 16 sets segments for each certain frequency range in the frequency spectrum, compares the sum of the changes in luminance value for each segment with a judgment reference value, and replaces the sum of the segments lower than the judgment reference value with the sum that is the median value for the same segment among the multiple frequency spectra. FIG. 14 is a schematic diagram showing an outline of the correction process of the frequency spectrum of the luminance value in the image of the rebar part. The diagram on the left side of FIG. 14 illustrates a frequency spectrum SP1 corresponding to the scanning line i=1 and a frequency spectrum SP3 corresponding to the scanning line i=3. The same frequency band of the frequency spectrum SP1 and the frequency spectrum SP3 is divided into segments seg0, seg1, seg2, seg3, and seg4, respectively. The partial waveform of the region A1 in the frequency spectrum SP3 is a spectrum showing the amount of change in the luminance value in the frequency band of the segment seg2. Moreover, the partial waveform of the region A2 in the frequency spectrum SP1 is a spectrum showing the amount of change in the luminance value in the frequency band of the segment seg2.
[0065] Next, the measurement processing unit 16 calculates the sum of the amount of change in luminance value in each of the segments seg0, seg1, seg2, seg3, and seg4 in the frequency spectra SP0, SP1, SP2, and SP3. Then, the measurement processing unit 16 calculates the median of the sums in the frequency spectra SP0, SP1, SP2, and SP3 among the sums for each segment. The diagram in the center of FIG. 14 is an enlarged view of partial spectra in the regions A1 and A2. As shown in the diagram in the center of FIG. 14, the frequency spectrum SP3 has a small amount of change in luminance value in the frequency band of the segment seg2, and the spectrum SP3A has small fluctuations. On the other hand, the frequency spectrum SP1 has a large amount of change in luminance value in the frequency band of the segment seg2, and the spectrum SP1A has large fluctuations.
[0066] Furthermore, the measurement processing unit 16 calculates the median value of the sum of the amounts of change in luminance value of the segment seg2 in the frequency spectra SP0, SP1, SP2, and SP3. The median of the sum of the amounts of change in luminance values in segment seg2 of frequency spectra SP0, SP1, SP2, and SP3 is the amount of change in luminance values in segment seg2 of frequency spectrum SP1.
[0067] Such a change in brightness value is strongly influenced by the actual environment in which the reinforcing bar 3 is placed, such as the direction of sunlight, and ideally, for the same reinforcing bar 4, the change in brightness value should be similar for scan lines i=0, 1, 2, and 3. In other words, when the change in brightness value in a frequency spectrum segment is less than the threshold, it is considered that the change in brightness value is small due to the influence of the actual environment when the reinforcing bar is photographed. For this reason, in order to calculate the change in brightness value that accurately represents the characteristics of the reinforcing bar, it is necessary to compensate for the decrease in the change in brightness value.
[0068] For example, when the amount of change in luminance value in spectrum SP3A in segment seg2 of frequency spectrum SP3 is less than a threshold, the measurement processing unit 16 searches for the median of the sum of the amounts of change in luminance value in segment seg2 of frequency spectra SP0, SP1, SP2, and SP3. Here, the amount of change in luminance value in spectrum SP1A in segment seg2 of frequency spectrum SP1 is the median. Therefore, the measurement processing unit 16 performs a process of replacing the spectrum SP3A in the segment seg2 of the frequency spectrum SP3 with the spectrum SP1A, as shown in the right diagram of Fig. 14. This reduces the influence of the actual environment, enabling the bar arrangement inspection device 1 to accurately extract the features of the reinforcing bar 4.
[0069] Furthermore, the measurement processing unit 16 performs frequency conversion on the luminance values of the scanning lines i=0, 1, 2, and 3 for each segment between the intersections of the reinforcing bars 4 shown in FIG. 12A to calculate frequency spectra SP0, SP1, SP2, and SP3. Then, the measurement processing unit 16 calculates the median of the amount of change in luminance value for each frequency of these frequency spectra as the feature amount of the reinforcing bar 4. In other words, the frequency spectra SP0, SP1, SP2, and SP3 are combined into a frequency spectrum consisting of the median of the amount of change in luminance value for each frequency. Furthermore, the measurement processing unit 16 may synthesize the frequency spectrum after the synthesis in each segment into a frequency spectrum consisting of a median value of the amount of change in luminance value for each frequency.
[0070] The measurement processing unit 16 calculates a correlation coefficient between the median of the amount of change in luminance value at the peak frequency in the frequency spectrum (the frequency spectrum after the synthesis) obtained from the captured image 21H converted into a frontal image, and the median of the amount of change in luminance value at the peak frequency in the frequency spectrum obtained from the pre-registered image of the reinforcing bar 4. Then, the measurement processing unit 16 calculates a correlation evaluation value by adding the median of the amount of change in luminance value at the peak frequency weighted according to the value of the correlation coefficient, and newly registers the frequency spectrum of the reinforcing bar 4 whose correlation evaluation value exceeds the threshold value.
[0071] Fig. 15 is a schematic diagram showing an outline of correlation processing between a frequency spectrum obtained from a photographed image 21H and a frequency spectrum registered in advance. In Fig. 15, the frequency spectrum obtained from the photographed image 21H obtained in the current examination is the spectrum indicated by the dashed line. The frequency spectrum registered in advance is the spectrum indicated by the solid line. The measurement processing unit 16 performs a correlation calculation (1) that evaluates the correlation between frequency spectra over the entire frequency range, a correlation calculation (2) that evaluates the correlation between first peaks (Peak A and Peak B) that appear first in the frequency spectrum, and a correlation calculation (3) that evaluates the correlation between second peaks that appear next in the frequency spectrum.
[0072] In the correlation calculation (1), the measurement processing unit 16 calculates a correlation score (1) indicating the correlation between both spectra in a correlation evaluation range expressed as (min(PeakA,PeakB)-4) to 256. Here, PeakA is the intensity of the first peak that first appears in a pre-registered frequency spectrum, and PeakB is the intensity of the first peak that first appears in a frequency spectrum obtained in the current test. In the correlation calculation (2), the measurement processing unit 16 calculates a correlation score (2) indicating the correlation between both spectra within a correlation evaluation range represented by (PeakA)±10. In the correlation calculation (3), the measurement processing unit 16 calculates a correlation score (3) indicating the correlation between both spectra within a correlation evaluation range represented by (PeakA×2)±10.
[0073] The measurement processing unit 16 determines weighting coefficients according to the respective values of the correlation score (1), the correlation score (2), and the correlation score (3). Then, the measurement processing unit 16 calculates a total correlation score, for example, according to the following formula. If the total correlation score is equal to or greater than a threshold, the measurement processing unit 16 determines that the inspection result information using the frequency spectrum obtained in the current inspection is similar to previously registered inspection result information, and registers the inspection result information in the inspection result database. This allows the reinforcement inspection device 1 to newly register highly reliable inspection results that are highly correlated with already registered inspection results. Total correlation score = (correlation score (1) x 0.5) + (correlation score (2) x 0.2) + (correlation score (3) x 0.3)
[0074] FIG. 16 is a flowchart showing the bar arrangement inspection method according to the first embodiment. The image acquisition unit 11 acquires a plurality of images of the reinforcing bars 3 taken by the monocular camera 2 from different angles from the plane side including the reinforcing bars 4 with the markers 5 attached thereto (step ST1). The three-dimensional restoration unit 12 calculates the shooting attitude and movement distance of the monocular camera 2 based on the changes in size, position and shape of the marker 5 between the captured images, and calculates three-dimensional point cloud data representing the reinforcement 3 based on the shooting attitude and movement distance of the monocular camera 2 (step ST2). The plane detection unit 13 uses the three-dimensional point cloud data to generate a plane mask image in which parts of the plane other than the reinforcing bars 4 are masked (step ST3). The image conversion unit 14 converts the photographed image and the plane mask image into a frontal image in which the multiple reinforcing bars 4 are perpendicular to each other (step ST4). The reinforcing bar detection unit 15 detects reinforcing bars 4 on a plane using a front-facing image of the captured image and a front-facing image of the plane mask image in step ST5, and the measurement processing unit 16 measures the reinforcing bars 4 in the reinforcement 3 using features of the detected reinforcing bars 4 (step ST6). By executing this bar arrangement inspection method, the bar arrangement inspection device 1 can detect the reinforcing bars 4 of each layer of the bar arrangement 3 using the images captured by the monocular camera 2.
[0075] Next, a hardware configuration for realizing the functions of the bar arrangement inspection device 1 will be described. The functions of the image acquisition unit 11, three-dimensional restoration unit 12, plane detection unit 13, image conversion unit 14, reinforcing bar detection unit 15, and measurement processing unit 16 included in the reinforcing bar inspection device 1 are realized by a processing circuit. That is, the reinforcing bar inspection device 1 includes a processing circuit for executing each process from step ST1 to step ST6 shown in Fig. 16. The processing circuit may be dedicated hardware, or may be a CPU (Central Processing Unit) that executes a program stored in a memory.
[0076] Fig. 17A is a block diagram showing a hardware configuration for realizing the functions of the bar arrangement inspection device 1. Fig. 17B is a block diagram showing a hardware configuration for executing software for realizing the functions of the bar arrangement inspection device 1. In Figs. 17A and 17B, the input interface 100 is an interface that relays image information output from the monocular camera 2 to the bar arrangement inspection device 1. The output interface 101 is an interface that relays the bar arrangement inspection results output from the bar arrangement inspection device 1 to a downstream device.
[0077] When the processing circuit is the dedicated hardware processing circuit 102 shown in FIG. 17A, the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the image acquisition unit 11, three-dimensional restoration unit 12, plane detection unit 13, image conversion unit 14, rebar detection unit 15 and measurement processing unit 16 provided in the reinforcement inspection device 1 may be realized by separate processing circuits, or these functions may be combined and realized by a single processing circuit.
[0078] 17B, the functions of the image acquisition unit 11, the three-dimensional reconstruction unit 12, the plane detection unit 13, the image conversion unit 14, the reinforcing bar detection unit 15, and the measurement processing unit 16 included in the reinforcing bar inspection device 1 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104.
[0079] The processor 103 reads out and executes the programs stored in the memory 104 to realize the functions of the image acquisition unit 11, the three-dimensional reconstruction unit 12, the plane detection unit 13, the image conversion unit 14, the reinforcing bar detection unit 15, and the measurement processing unit 16 of the reinforcing bar inspection device 1. For example, the reinforcing bar inspection device 1 includes a memory 104 for storing a program that, when executed by the processor 103, results in the processing of steps ST1 to ST6 shown in FIG. 16 being executed. These programs cause a computer to execute the procedures or methods of the processing performed by the image acquisition unit 11, the three-dimensional reconstruction unit 12, the plane detection unit 13, the image conversion unit 14, the reinforcing bar detection unit 15, and the measurement processing unit 16. The memory 104 may be a computer-readable storage medium in which programs for causing a computer to function as the image acquisition unit 11, the three-dimensional reconstruction unit 12, the plane detection unit 13, the image conversion unit 14, the reinforcing bar detection unit 15, and the measurement processing unit 16 are stored.
[0080] Memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as a Random Access Memory (RAM), a Read Only Memory (ROM), a flash memory, an Erasable Programmable Read Only Memory (EPROM), an Electrically-EPROM (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD, etc.
[0081] Some of the functions of the image acquisition unit 11, three-dimensional restoration unit 12, plane detection unit 13, image conversion unit 14, rebar detection unit 15 and measurement processing unit 16 included in the reinforcement inspection device 1 may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware. For example, the image acquisition unit 11 has its functions realized by a processing circuit 102, which is dedicated hardware, and the three-dimensional restoration unit 12, plane detection unit 13, image conversion unit 14, rebar detection unit 15 and measurement processing unit 16 have their functions realized by a processor 103 reading and executing programs stored in a memory 104. Thus, the processing circuitry may implement the above-described functions through hardware, software, firmware or a combination of these.
[0082] As described above, the reinforcing bar inspection device 1 according to the first embodiment includes an image acquisition unit 11 that acquires a plurality of images of the reinforcing bars 3 taken from different angles by the monocular camera 2 from the plane side including the reinforcing bars 4 to which the markers 5 are attached, a three-dimensional restoration unit 12 that calculates the shooting attitude and movement distance of the monocular camera 2 based on the change in the size, position, and shooting angle of the marker 5 between the shot images, and calculates three-dimensional point cloud data representing the reinforcing bars 3 based on the calculated shooting attitude and movement distance of the monocular camera 2, a plane detection unit 13 that generates a plane mask image in which parts other than the reinforcing bars 4 on the plane are masked using the three-dimensional point cloud data, an image conversion unit 14 that converts the shot image and the plane mask image into a frontal image, a reinforcing bar detection unit 15 that detects the reinforcing bars 4 on the plane using the frontal image of the shot image and the frontal image of the plane mask image, and a measurement processing unit 16 that measures the reinforcing bars 4 in the reinforcing bars 3 using the feature amounts of the detected reinforcing bars 4. As a result, the reinforcing bar inspection device 1 can detect the reinforcing bars 4 in each layer of the reinforcing bars 3 using the images taken by the monocular camera 2.
[0083] In the bar arrangement inspection device 1 according to the first embodiment, a plurality of markers 5 are provided on the plane of the bar arrangement 3, and have a polygonal outer shape. When the marker 5 having a polygonal outer shape is photographed from different shooting angles, the outer shape appears distorted depending on the shooting angle, and by analyzing this distortion, it is possible to estimate the shooting posture of the monocular camera 2 according to the shooting angle.
[0084] In the reinforcement inspection device 1 according to the first embodiment, the three-dimensional restoration unit 12 calculates the position and shooting attitude of each of the multiple markers 5 based on changes in size, position and shooting angle of each of the multiple markers 5 in multiple captured images, calculates the shooting position and movement distance of the monocular camera 2 between captured images using the position and shooting attitude of the markers 5, and calculates three-dimensional point cloud data using the shooting position and movement distance of the monocular camera 2 calculated from a pair of mutually different captured images. This enables the reinforcement inspection device 1 to accurately calculate three-dimensional point cloud data that represents the reinforcement 3.
[0085] In the bar arrangement inspection device 1 according to the first embodiment, the plane detection unit 13 generates a plane mask image from a pair of different captured images using three-dimensional point cloud data. This enables the bar arrangement inspection device 1 to generate a plane mask image that can accurately extract rebars included in the plane.
[0086] In the bar arrangement inspection device 1 according to the first embodiment, the plane detection unit 13 estimates plane parameters representing a plane candidate selected from a plurality of plane candidates determined using the three-dimensional point cloud data according to the number of three-dimensional points whose distance from the monocular camera 2 to the plane is within a set distance, and generates a plane mask image using the plane parameters. This enables the bar arrangement inspection device 1 to generate a plane mask image that can accurately extract the reinforcing bars included in the plane.
[0087] In the bar arrangement inspection device 1 according to the first embodiment, the plane detection unit 13 calculates plane parameters indicating the plane of the inspection target by repeatedly calculating the plane parameters of the plane candidate and updating the histogram indicating the amount of error of the plane based on the normal of the plane candidate. This enables the bar arrangement inspection device 1 to generate a plane mask image that can accurately extract the reinforcing bars included in the plane.
[0088] In the reinforcing bar inspection device 1 according to the first embodiment, the plane detection unit 13 generates a plane mask image using three-dimensional point cloud data calculated from a pair of different captured images, and synthesizes these plane mask images to obtain a final plane mask image. This enables the reinforcing bar inspection device 1 to generate a plane mask image that can accurately extract reinforcing bars included in the plane.
[0089] In the bar arrangement inspection device 1 according to the first embodiment, the reinforcing bar detection unit 15 corrects the edge positions of the line segments corresponding to the reinforcing bar portions in the front-facing image of the plane mask image based on the brightness edges in the front-facing image of the captured image. This enables the bar arrangement inspection device 1 to generate a plane mask image that can accurately extract the reinforcing bars included in the plane.
[0090] In the reinforcing bar inspection device 1 according to the first embodiment, the measurement processing unit 16 performs scaling to match the image scale of the captured image of the detected reinforcing bar 4 with the image scale of the captured image of the reinforcing bar 4 registered in advance, so that the distance from the monocular camera 2 to the reinforcing bar 4 becomes constant. This enables the reinforcing bar inspection device 1 to accurately compare the feature amounts of the reinforcing bar 4 detected from captured images taken at different shooting positions and attitudes.
[0091] In the reinforcing bar inspection device 1 according to the first embodiment, the measurement processing unit 16 calculates the intersections where the reinforcing bars 4 cross each other vertically based on the position parameters of the reinforcing bars 4 on the plane to be inspected, and extracts feature values of the reinforcing bars 4 between the intersections from the front-facing image of the captured image. This enables the reinforcing bar inspection device 1 to accurately inspect the reinforcing bars 4 to be inspected by eliminating the influence of other reinforcing bars 4 assembled together with the reinforcing bars 4 to be inspected.
[0092] In the reinforcing bar inspection device 1 according to the first embodiment, the measurement processing unit 16 acquires a scanning line consisting of the luminance values of multiple pixels along the longitudinal direction of the reinforcing bar 4 in the frontal image for each pixel in the direction perpendicular to the longitudinal direction of the reinforcing bar 4, calculates multiple frequency spectra by frequency-converting the luminance values of the scanning line for each pixel, and regards the median of the changes in luminance values calculated from these frequency spectra as the feature amount of the reinforcing bar 4. This enables the reinforcing bar inspection device 1 to accurately extract the features of the reinforcing bar 4 from the captured image.
[0093] In the reinforcing bar inspection device 1 according to the first embodiment, the measurement processing unit 16 sets segments for each certain frequency range in the frequency spectrum, compares the sum of the change in brightness value for each segment with a criterion value, and replaces the sum of the segments lower than the criterion value with the median sum of the same segment among the multiple frequency spectra. This enables the reinforcing bar inspection device 1 to accurately extract the features of the reinforcing bar 4 from the captured image.
[0094] In the reinforcing bar inspection device 1 according to embodiment 1, the measurement processing unit 16 calculates a correlation coefficient between the median amount of change in luminance value at the peak frequency in the frequency spectrum obtained from the captured image and the median amount of change in luminance value at the peak frequency in the frequency spectrum obtained from a pre-registered image of the reinforcing bar 4, calculates a correlation evaluation value by adding the median amount of change in luminance value at the peak frequency weighted according to the value of the correlation coefficient, and newly registers the frequency spectrum of the reinforcing bar 4 whose correlation evaluation value exceeds a threshold value. This allows the reinforcing bar inspection device 1 to newly register inspection results that are highly correlated with already registered inspection results and are highly reliable.
[0095] The reinforcing bar inspection method according to the first embodiment includes step ST1 in which an image acquisition unit 11 acquires a plurality of images of the reinforcing bar 3 taken from different angles by the monocular camera 2 from a plane side including the reinforcing bar 4 to which the marker 5 is attached, step ST2 in which a three-dimensional reconstruction unit 12 calculates the shooting attitude and travel distance of the monocular camera 2 based on changes in the size, position, and shape of the marker 5 between the captured images, and calculates three-dimensional point cloud data representing the reinforcing bar 3 based on the calculated shooting attitude and travel distance of the monocular camera 2, and step ST3 in which a plane detection unit 13 The method includes step ST3 of generating a plane mask image in which parts of the plane other than the reinforcing bars 4 are masked using the three-dimensional point cloud data, step ST4 of converting the captured image and the plane mask image into a front-facing image in which a plurality of reinforcing bars 4 are perpendicularly intersecting, step ST5 of detecting the reinforcing bars 4 on the plane using the front-facing image of the captured image and the front-facing image of the plane mask image by the image conversion unit 14, and step ST6 of measuring the reinforcing bars 4 in the reinforcing bars 3 using feature amounts of the detected reinforcing bars 4 by the measurement processing unit 16. By executing the reinforcing bar inspection method by the reinforcing bar inspection device 1, the reinforcing bars 4 in each layer of the reinforcing bars 3 can be detected using the image captured by the monocular camera 2.
[0096] By having a computer execute the program according to the first embodiment, it is possible to provide a bar arrangement inspection device 1 that can detect reinforcing bars 4 in each layer of bar arrangement 3 using images captured by a monocular camera 2.
[0097] Embodiment 2 FIG. 18 is a block diagram showing a configuration example of a bar arrangement inspection device 1A according to the second embodiment. In Fig. 18, the reinforcing bar inspection device 1A is a device that inspects at least one of the number of reinforcing bars, the spacing between adjacent reinforcing bars, the diameter of the reinforcing bars, or the spacing between nodes of the reinforcing bars in a reinforcing bar arrangement having a plane of multiple layers made of reinforcing bars arranged in a lattice pattern. The reinforcing bar inspection device 1A is a tablet terminal, a smartphone, or a notebook PC. As shown in Fig. 18, a monocular camera 2 is connected to the reinforcing bar inspection device 1A by wire or wirelessly. The reinforcing bar inspection device 1A is not limited to a device with an external monocular camera 2, and may have the monocular camera 2 built-in.
[0098] In a bar arrangement inspection, an inspector carries a bar arrangement inspection device 1A and moves around the area of the bar arrangement to be inspected, photographing the bar arrangement with a monocular camera 2 connected to the bar arrangement inspection device 1A. The monocular camera 2, which moves with the inspector, can photograph the bar arrangement from multiple shooting angles. In addition, the moving object moving around the reinforcement is not limited to an inspector, but may be a vehicle, an unmanned aerial vehicle, etc.
[0099] The bar arrangement inspection device 1A may be a component included in a server capable of communicating with a terminal device equipped with a monocular camera 2. For example, the terminal device can perform bar arrangement inspection provided in the form of SaaS (Software as a Service). When performing bar arrangement inspection in the form of SaaS, an information processing application for bar arrangement inspection does not need to be installed on the terminal device. The information processing application for bar arrangement inspection is executed on the server, and the terminal device is provided with measurement result information on a general-purpose web browser. The bar arrangement inspection application is stored in a storage unit included in the server. Furthermore, an information processing application for bar arrangement inspection may be installed in the terminal device. In the terminal device in which the information processing application for bar arrangement inspection is installed, the application is executed to enable bar arrangement inspection.
[0100] The reinforcing bar inspection device 1A includes at least a calculation unit and a memory unit. The calculation unit controls the overall operation of the reinforcing bar inspection device 1A. The calculation unit includes an image acquisition unit 11, a three-dimensional restoration unit 12, a plane detection unit 13, an image conversion unit 14, a reinforcing bar detection unit 15, and a measurement processing unit 16A. The calculation unit executes an information processing application for reinforcing bar inspection to realize various functions of the image acquisition unit 11, the three-dimensional restoration unit 12, the plane detection unit 13, the image conversion unit 14, the reinforcing bar detection unit 15, and the measurement processing unit 16A. The measurement processing unit 16A is connected to a learning device 71. The learning device 71 includes a learning unit 711 and a memory unit 712.
[0101] The storage unit stores an information processing application for the bar arrangement inspection and information used for the calculation processing of the calculation unit. The storage unit is a storage device provided in a computer functioning as the bar arrangement inspection apparatus 1A, and includes a storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), or the memory 104 in Fig. 17B. Note that the storage unit may be provided outside the bar arrangement inspection apparatus 1 as long as it is accessible by the bar arrangement inspection apparatus 1A.
[0102] The measurement processing unit 16A divides the image of the reinforcing bar 4 extracted from the frontal image into multiple partial images, and infers the diameter of the reinforcing bar 4 using a learning model that outputs the diameter of the reinforcing bar 4 when multiple partial images are input. For example, the measurement processing unit 16A receives input of multiple partial images obtained by dividing an image of a single reinforcing bar 4, and infers the diameter of the reinforcing bar using a learning model that infers the diameter of the reinforcing bar as inspection information of the reinforcing bar.
[0103] The learning device 71 generates a learning model that outputs a rebar diameter when a plurality of partial images obtained by dividing a photographed image of a rebar are input. The learning unit 711 generates the learning model by performing learning to infer the rebar diameter from the learning data. The learning data is a data set including a plurality of partial image data obtained by dividing a photographed image of a rebar and a correct answer label, which is input data for the learning model. The correct answer label is information indicating the rebar diameter corresponding to each of the plurality of partial image data. The learning model is stored in the storage unit 712. For example, deep learning, neural network, genetic programming, functional logic programming, or support vector machine is used as the learning algorithm.
[0104] FIG. 19 is a schematic diagram showing an outline of the process of extracting partial images 2H(1), 2H(2), and 2H(3) of the reinforcing bar 4. As shown in FIG. 19, the measurement processing unit 16A extracts images of the reinforcing bar 4 along the longitudinal direction from the captured image 2H converted into a frontal image. Furthermore, the measurement processing unit 16A sequentially extracts partial images 2H(1), 2H(2), and 2H(3) of the same size from the image of the reinforcing bar 4. The partial images 2H(1), 2H(2), and 2H(3) are, for example, square images with the same number of pixels in both the vertical and horizontal directions, and the reinforcing bar diameter is D. In FIG. 19, the measurement processing unit 16A performs these processes for all reinforcing bars in the plane of each story of the reinforcing bar 3.
[0105] Fig. 20 is a schematic diagram showing an outline of inference processing of rebar diameter by a learning model. In Fig. 20, the learning model is generated using learning data that is a set of multiple partial images extracted from an image of a rebar and a correct answer label indicating the diameter D of the rebar that is assigned to each partial image. The measurement processing unit 16A inputs multiple partial image data into the learning model to perform inference, and calculates, for each partial image, the probability that a reinforcing bar 4 with diameter D is captured and the probability that something other than the reinforcing bar 4 is captured.
[0106] For example, the inference result data 151 is a result of calculating the probability that each type of reinforcing bar is captured and the probability that something other than reinforcing bars (NON) is captured in the partial image 2H(1) shown in FIG. 19. There are 16 types of reinforcing bars based on the JIS standard, for example. The 16 types of reinforcing bars are given "names" such as D4, D5, D6, D8, D10, D13, D16, D19, D22, D25, D29, D32, D35, D38, D41, and D51. The names indicate the diameter of the reinforcing bar rounded off from its nominal diameter.
[0107] The nominal diameter of a rebar designated D10 is 9.53 mm, that of a rebar designated D13 is 12.7 mm, and that of a rebar designated D16 is 15.9 mm. Note that rebars generally used as the framework of buildings are of D10 or higher.
[0108] The inference result data 152 is a result of calculating the probability that each type of reinforcing bar is captured and the probability that something other than a reinforcing bar (NON) is captured in the partial image 2H(2) shown in Fig. 19. Such inference result data is calculated for all partial image data extracted from the same reinforcing bar image.
[0109] Next, the learning model calculates inference result data 153 for the image of the rebar by averaging the inference result data for all partial image data extracted from the same image of the rebar. For example, the inference result data 153 includes an average value of the probability that each type of rebar is captured and an average value of the probability that something other than a rebar (NON) is captured.
[0110] The measurement processing unit 16A determines the reinforcing bar diameter D based on the inference result data 153. In the example of Fig. 20, the type of the reinforcing bar 4 in the photographed image 2H shown in Fig. 19 is determined to be D16, that is, the reinforcing bar 4 has a diameter D of 15.9 (mm). By using a learning model, the measurement processing unit 16A can accurately detect the reinforcing bar diameter D and can determine the type of reinforcing bar based on the reinforcing bar diameter D.
[0111] In addition, the learning model may be one that outputs the distance between adjacent rebars 4 when multiple partial images of a rebar 4 are input, or one that infers the distance between nodes in a single rebar 4.
[0112] As described above, in the bar arrangement inspection device 1A according to the second embodiment, the measurement processing unit 16A divides the image of the reinforcing bar 4 extracted from the frontal image into multiple partial images, and infers the diameter of the reinforcing bar 4 using a learning model that outputs the diameter of the reinforcing bar 4 when multiple partial images are input. The use of the learning model by the measurement processing unit 16A enables the bar arrangement inspection device 1A to accurately infer inspection information for the reinforcing bar 4.
[0113] Various aspects of the present disclosure are summarized below as appendices.
[0114] (Appendix 1) An image acquisition unit that acquires a plurality of images of a reinforcing bar arrangement having a plane of multiple layers made of reinforcing bars arranged in a lattice pattern, the images being taken from different angles by a monocular camera from the plane side including the reinforcing bars to which markers are attached; a three-dimensional restoration unit that calculates a shooting attitude and a moving distance of the monocular camera based on changes in size, position and shooting angle of the marker between the captured images, and calculates three-dimensional point cloud data representing the reinforcement based on the calculated shooting attitude and moving distance of the monocular camera; a plane detection unit that generates a plane mask image by masking parts other than the reinforcing bars on a plane using the three-dimensional point cloud data; an image conversion unit that converts the captured image and the planar mask image into a frontal image in which rebars are perpendicular to each other; a reinforcing bar detection unit that detects reinforcing bars on a plane using the front-facing image of the captured image and the front-facing image of the plane mask image; A measurement processing unit that measures the reinforcing bars in the reinforcing bar arrangement using the detected feature amount of the reinforcing bars. A reinforcing bar inspection device characterized by the above. (Appendix 2) The marker is provided in a plurality on the plane of the reinforcement and has a polygonal outer shape. 2. The reinforcing bar inspection device according to claim 1 , (Appendix 3) The three-dimensional reconstruction unit calculates the shooting position and shooting attitude of the monocular camera based on changes in size, position, and shooting angle of the plurality of markers in the plurality of captured images, and calculates the three-dimensional point cloud data using the shooting position and movement distance of the monocular camera calculated from a pair of the captured images that are different from each other. 3. The reinforcing bar inspection device according to claim 1 or 2, (Appendix 4) The plane detection unit generates the plane mask image from a pair of the captured images that are different from each other by using the three-dimensional point cloud data. 4. The reinforcing bar inspection device according to claim 1, (Appendix 5) The plane detection unit estimates plane parameters representing a plane candidate selected from a plurality of plane candidates determined using the three-dimensional point cloud data according to the number of three-dimensional points whose distance from the monocular camera to the plane is within a set distance, and generates the plane mask image using the plane parameters. 5. The reinforcing bar inspection device according to claim 4, (Appendix 6) The plane detection unit calculates the plane parameters representing the plane to be inspected by repeating calculation of the plane parameters of the plane candidate and updating a histogram indicating an error amount of the plane based on a normal of the plane candidate. 6. The reinforcing bar inspection device according to claim 5, (Appendix 7) The plane detection unit generates the plane mask image using the three-dimensional point cloud data calculated from each pair of the captured images that are different from each other, and synthesizes these plane mask images to obtain the final plane mask image. 5. The reinforcing bar inspection device according to claim 4, (Appendix 8) The reinforcing bar detection unit corrects edge positions of line segments corresponding to reinforcing bar portions in the front-facing image of the planar mask image based on brightness edges in the front-facing image of the captured image. 8. The reinforcing bar inspection device according to claim 1, (Appendix 9) The measurement processing unit performs scaling to match an image scale between the captured image of the detected reinforcing bar and the captured image of the preregistered reinforcing bar so that the distance from the monocular camera to the reinforcing bar is constant. 9. The reinforcing bar inspection device according to claim 1, (Appendix 10) The measurement processing unit calculates intersections where the reinforcing bars cross each other vertically based on position parameters of the reinforcing bars on the plane to be inspected, and extracts feature amounts of the reinforcing bars between the intersections from the front-facing image of the captured image. 10. The reinforcing bar inspection device according to claim 1, (Appendix 11) The measurement processing unit acquires a scanning line consisting of the luminance values of a plurality of pixels along the longitudinal direction of the rebar in the frontal image for each pixel in a direction perpendicular to the longitudinal direction of the rebar, calculates a plurality of frequency spectra by frequency-converting the luminance values of the scanning line for each pixel, and determines the median of the amount of change in the luminance values extracted from these frequency spectra as a feature of the rebar. 11. The reinforcing bar inspection device according to claim 10, (Appendix 12) The measurement processing unit sets segments for each certain frequency range in the frequency spectrum, compares a sum of changes in luminance value for each segment with a reference value, and replaces the sum of the segment that is lower than the reference value with the sum of the same segment among the plurality of frequency spectra that is the median. 12. The reinforcing bar inspection device according to claim 11, (Appendix 13) The measurement processing unit calculates a correlation coefficient between the median of the luminance values at the frequency of the peak in the frequency spectrum obtained from the captured image and the median of the amount of change in the luminance values at the frequency of the peak in the frequency spectrum obtained from a previously registered image of the reinforcing bar, calculates a correlation evaluation value by adding the median of the amount of change in the luminance values at the frequency of the peak weighted according to the value of the correlation coefficient, and newly registers the frequency spectrum of the reinforcing bar for which the correlation evaluation value exceeds a threshold value. 12. The reinforcing bar inspection device according to claim 11, (Appendix 14) The measurement processing unit divides the image of the reinforcing bar extracted from the frontal image into a plurality of partial images, and infers the diameter of the reinforcing bar using a learning model that outputs the diameter of the reinforcing bar when the plurality of partial images are input. 14. The reinforcing bar inspection device according to claim 1, (Appendix 15) A method for inspecting a reinforcing bar using a reinforcing bar inspection device, comprising the steps of: An image acquisition unit acquires a plurality of images of a reinforcing bar arrangement having a multi-layer plane made of reinforcing bars arranged in a lattice pattern, the images being taken from a plane side including the reinforcing bars to which markers are attached, from different angles using a monocular camera; a three-dimensional reconstruction unit calculating a shooting attitude and a moving distance of the monocular camera based on changes in size, position and shape of the marker between the captured images, and calculating three-dimensional point cloud data representing the reinforcement based on the calculated shooting attitude and moving distance of the monocular camera; A step in which a plane detection unit generates a plane mask image by masking parts other than the rebar on the plane using the three-dimensional point cloud data; An image conversion unit converts the captured image and the planar mask image into a frontal image in which rebars are perpendicular to each other; A reinforcing bar detection unit detects reinforcing bars on a plane using the front-facing image of the captured image and the front-facing image of the plane mask image; A step in which the measurement processing unit measures the reinforcing bar on a plane based on a result of correlation processing between the detected feature amount of the reinforcing bar and the feature amount of the reinforcing bar registered in advance. A reinforcing bar inspection method comprising: (Appendix 16) Computer, An image acquisition unit that acquires a plurality of images of a reinforcing bar arrangement having a plane of multiple layers made of reinforcing bars arranged in a lattice pattern, the images being taken from different angles by a monocular camera from the plane side including the reinforcing bars to which the markers are attached; a three-dimensional restoration unit that calculates a shooting attitude and a moving distance of the monocular camera based on changes in size, position and shape of the marker between the captured images, and calculates three-dimensional point cloud data representing the reinforcement based on the calculated shooting attitude and moving distance of the monocular camera; a plane detection unit that generates a plane mask image by masking parts other than the reinforcing bars on a plane using the three-dimensional point cloud data; an image conversion unit that converts the captured image and the planar mask image into a frontal image in which the reinforcing bars are perpendicular to each other; a reinforcing bar detection unit that detects reinforcing bars on a plane using the front-facing image of the captured image and the front-facing image of the plane mask image; a measurement processing unit that measures the reinforcing bars on a plane based on a result of correlation processing between the detected feature amount of the reinforcing bars and the feature amount of the reinforcing bars registered in advance; A program to function as a
[0115] It is possible to combine the embodiments, modify any of the components of the embodiments, or omit any of the components of the embodiments. [Explanation of symbols]
[0116] 1,1A Reinforcement inspection device, 2 Monocular camera, 2H(1),2H(2),2H(3) Partial image, 2H,21A,21B,21C,21D,21E,21F,21G,21H Photographed image, 3 Reinforcement, 4 Reinforcement, 4A,52 Line segment, 5 Marker, 6A Base, 6B Identification shape, 11 Image acquisition unit, 12 Three-dimensional restoration unit, 13 Plane detection unit, 14 Image conversion unit, 15 Reinforcement detection unit, 16,16A Measurement processing unit, 31 Outlier, 32 Inlier, 40 Three-dimensional point, 41,42 Plane, 51 Plane mask image, 60 Image, 60A,60B Rectangular area, 61 Histogram, 62 Peak, 71 Learning device, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 processor, 104 memory, 111 image area, 111A, 141 area, 112 rectangle, 113 center, 151, 152, 153 inference result data, 711 learning unit, 712 memory unit.
Claims
1. An image acquisition unit that acquires a plurality of images of a reinforcing bar arrangement having a plane of multiple layers made of reinforcing bars arranged in a lattice pattern, the images being taken from different angles by a monocular camera from the plane side including the reinforcing bars to which markers are attached; a three-dimensional restoration unit that calculates a shooting attitude and a moving distance of the monocular camera based on changes in size, position and shooting angle of the marker between the captured images, and calculates three-dimensional point cloud data representing the reinforcement based on the calculated shooting attitude and moving distance of the monocular camera; a plane detection unit that generates a plane mask image by masking parts other than the reinforcing bars on a plane using the three-dimensional point cloud data; an image conversion unit that converts the captured image and the planar mask image into a frontal image in which rebars are perpendicular to each other; a reinforcing bar detection unit that detects reinforcing bars on a plane using the front-facing image of the captured image and the front-facing image of the plane mask image; A measurement processing unit that measures the reinforcing bars in the reinforcing bar arrangement using the detected feature amount of the reinforcing bars. A reinforcing bar inspection device characterized by the above.
2. The marker is provided in a plurality on the plane of the reinforcement and has a polygonal outer shape.
2. The reinforcing bar inspection device according to claim 1 .
3. The three-dimensional reconstruction unit calculates the shooting position and shooting attitude of the monocular camera based on changes in size, position, and shooting angle of the plurality of markers in the plurality of captured images, and calculates the three-dimensional point cloud data using the shooting position and movement distance of the monocular camera calculated from a pair of the captured images that are different from each other.
3. The reinforcing bar inspection device according to claim 2 .
4. The plane detection unit generates the plane mask image from a pair of the captured images that are different from each other by using the three-dimensional point cloud data.
4. The reinforcing bar inspection device according to claim 3.
5. The plane detection unit estimates plane parameters representing a plane candidate selected from a plurality of plane candidates determined using the three-dimensional point cloud data according to the number of three-dimensional points whose distance from the monocular camera to the plane is within a set distance, and generates the plane mask image using the plane parameters.
5. The reinforcing bar inspection device according to claim 4.
6. The plane detection unit calculates the plane parameters representing the plane to be inspected by repeating calculation of the plane parameters of the plane candidate and updating a histogram indicating an error amount of the plane based on a normal of the plane candidate.
6. The reinforcing bar inspection device according to claim 5.
7. The plane detection unit generates the plane mask image using the three-dimensional point cloud data calculated from each pair of the captured images that are different from each other, and synthesizes these plane mask images to obtain the final plane mask image.
5. The reinforcing bar inspection device according to claim 4.
8. The reinforcing bar detection unit corrects edge positions of line segments corresponding to reinforcing bar portions in the front-facing image of the planar mask image based on brightness edges in the front-facing image of the captured image.
2. The reinforcing bar inspection device according to claim 1 .
9. The measurement processing unit performs scaling to match an image scale between the captured image of the detected reinforcing bar and the captured image of the preregistered reinforcing bar so that the distance from the monocular camera to the reinforcing bar is constant.
2. The reinforcing bar inspection device according to claim 1 .
10. The measurement processing unit calculates intersections where the reinforcing bars cross each other vertically based on position parameters of the reinforcing bars on the plane to be inspected, and extracts feature amounts of the reinforcing bars between the intersections from the front-facing image of the captured image. The bar arrangement inspection device according to claim 9 .
11. The measurement processing unit acquires a scanning line consisting of the luminance values of a plurality of pixels along the longitudinal direction of the rebar in the frontal image for each pixel in a direction perpendicular to the longitudinal direction of the rebar, calculates a plurality of frequency spectra by frequency-converting the luminance values of the scanning line for each pixel, and determines the median of the amount of change in the luminance values extracted from these frequency spectra as a feature of the rebar. The bar arrangement inspection device according to claim 10 .
12. The measurement processing unit sets segments for each certain frequency range in the frequency spectrum, compares a sum of changes in luminance value for each segment with a reference value, and replaces the sum of the segment that is lower than the reference value with the sum of the same segment among the plurality of frequency spectra that is the median. The bar arrangement inspection device according to claim 11.
13. The measurement processing unit calculates a correlation coefficient between the median of the luminance values at the frequency of the peak in the frequency spectrum obtained from the captured image and the median of the amount of change in the luminance values at the frequency of the peak in the frequency spectrum obtained from a previously registered image of the reinforcing bar, calculates a correlation evaluation value by adding the median of the amount of change in the luminance values at the frequency of the peak weighted according to the value of the correlation coefficient, and newly registers the frequency spectrum of the reinforcing bar for which the correlation evaluation value exceeds a threshold value. The bar arrangement inspection device according to claim 11.
14. The measurement processing unit divides the image of the reinforcing bar extracted from the frontal image into a plurality of partial images, and infers the diameter of the reinforcing bar using a learning model that outputs the diameter of the reinforcing bar when the plurality of partial images are input. The bar arrangement inspection device according to claim 9 .
15. A method for inspecting a reinforcing bar using a reinforcing bar inspection device, comprising the steps of: An image acquisition unit acquires a plurality of images of a reinforcing bar arrangement having a multi-layer plane made of reinforcing bars arranged in a lattice pattern, the images being taken from a plane side including the reinforcing bars to which markers are attached, from different angles using a monocular camera; a three-dimensional reconstruction unit calculating a shooting attitude and a moving distance of the monocular camera based on changes in size, position and shape of the marker between the captured images, and calculating three-dimensional point cloud data representing the reinforcement based on the calculated shooting attitude and moving distance of the monocular camera; A step in which a plane detection unit generates a plane mask image by masking parts other than the rebar on the plane using the three-dimensional point cloud data; An image conversion unit converts the captured image and the planar mask image into a frontal image in which rebars are perpendicular to each other; A reinforcing bar detection unit detects reinforcing bars on a plane using the front-facing image of the captured image and the front-facing image of the plane mask image; A step in which a measurement processing unit measures the reinforcing bars in the reinforcing bars by using the detected feature amounts of the reinforcing bars. A reinforcement inspection method comprising:
16. Computer, an image acquisition unit that acquires a plurality of images of a reinforcing bar arrangement having a plane of multiple layers made of reinforcing bars arranged in a lattice pattern, the images being taken from different angles by a monocular camera from the plane side including the reinforcing bars to which the markers are attached; a three-dimensional restoration unit that calculates a shooting attitude and a moving distance of the monocular camera based on changes in size, position and shape of the marker between the captured images, and calculates three-dimensional point cloud data representing the reinforcement based on the calculated shooting attitude and moving distance of the monocular camera; a plane detection unit that generates a plane mask image by masking parts other than the reinforcing bars on a plane using the three-dimensional point cloud data; an image conversion unit that converts the captured image and the planar mask image into a frontal image in which the reinforcing bars are perpendicular to each other; a reinforcing bar detection unit that detects reinforcing bars on a plane using the front-facing image of the captured image and the front-facing image of the plane mask image; a measurement processing unit that measures the reinforcing bars in the reinforcing bar arrangement by using the detected feature values of the reinforcing bars; A program to function as a
Citation Information
Patent Citations
Reinforcement inspection system and marker
JP6801055B1
Cited By
Target object extraction system and target object extraction method
WO2026163447A1
