Object extraction system, object extraction method, and object extraction program

Abstract

To accurately determine a plane that represents an extraction object.SOLUTION: An object extraction system that extracts information on an object from an image, includes: an arithmetic operation unit which executes arithmetic processing; and a storage unit which can be accessed by the arithmetic operation unit. The arithmetic operation unit acquires a distance image including distance information measured for each pixel, divides the distance image into multiple regions, counts the number of pixels for each distance in the divided area, extracts the pixels within a prescribed distance range from the maximum value of the shortest distance from an imaging point among the maximum values of the counted number of pixels, calculates the first plane representing the extracted pixels for each divided area, groups the divided areas using the relation between the first planes calculated for each area, selects one of the groups generated by the grouping, and calculates the second plane representing the extracted pixels in the area of the selected group.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an object extraction system that extracts a plane of the position of an object shown in a distance image.

Background Art

[0002] In the past, problems have occurred at construction sites where the number, diameter, spacing, etc. of reinforcing bars are constructed differently from the design, and the inspection of reinforcing bars has been tightened, increasing the burden in on-site work. To reduce this burden, three-dimensional data of reinforcing bars is obtained using sensors such as cameras. The point cloud represented by the obtained three-dimensional coordinates contains information other than the reinforcing bars to be measured, and it is necessary to extract the point cloud representing the target reinforcing bars.

[0003] As background art in this technical field, there is the following prior art. In Patent Document 1 (Japanese Unexamined Patent Application Publication No. 2015-1146), from an image obtained by photographing a reinforcing bar to be inspected with a photographing device, images of a plurality of joints adjacent to the reinforcing bar to be inspected are detected, and based on the detected images of the plurality of joints, the distance between adjacent joints is derived. For reinforcing bars of a plurality of predetermined diameters, the diameter of the reinforcing bar corresponding to the derived distance is read from a secondary storage unit that stores in advance the association between the distance between adjacent joints and the diameter of the corresponding reinforcing bar, and a reinforcing bar inspection support device for specifying the diameter of the reinforcing bar to be inspected is described.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the aforementioned known techniques, point cloud data of the rebar positions on a plane is extracted using a histogram of distance to the object and the number of pixels. This involves rotating the point cloud data three-dimensionally so that the camera's optical axis and the normal direction of the plane are parallel. In other words, the extraction of the plane of the rebar positions using a histogram assumes that the camera's optical axis and the plane are perpendicular. However, it is difficult to accurately position the camera so that the optical axis is perpendicular to the plane of the rebar positions, and errors occur due to the misalignment between the optical axis and the normal direction of the plane. Furthermore, depending on the site environment, it may not be possible to position the camera so that the optical axis is perpendicular to the plane of the rebar positions.

[0006] Therefore, there is a need for a system that can accurately determine the plane representing the object to be extracted, even without positioning the camera perpendicular to the object. [Means for solving the problem]

[0007] A typical example of the invention disclosed in this application is as follows: an object extraction system for extracting object information from an image, comprising a calculation unit that performs calculation processing and a storage unit accessible by the calculation unit, wherein the calculation unit acquires a distance image including distance information measured for each pixel, divides the distance image into a plurality of regions, counts the number of pixels for each distance in the divided regions, extracts pixels within a predetermined distance range from the maximum value of the shortest distance from the shooting point among the maximum values ​​of the counted number of pixels, calculates a first plane representing the extracted pixels for each divided region, groups the divided regions using the relationship between the first planes calculated for each region, selects one of the groups generated by the grouping, and calculates a second plane representing the extracted pixels within the region of the selected group. [Effects of the Invention]

[0008] According to one aspect of the present invention, the plane representing the object to be extracted can be accurately determined. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the configuration of the object extraction system in this embodiment. [Figure 2] This figure shows an example of the data structure of the distance image used in this embodiment. [Figure 3] This is a flowchart of a conventional object extraction process. [Figure 4] This diagram shows the designation of three points on a reinforcing bar in a conventional image. [Figure 5] This is a top view of a conventional reinforcing bar. [Figure 6] This figure shows the extracted region, which is presumed to be conventional reinforcing steel. [Figure 7] This is a flowchart of the object extraction process in this embodiment. [Figure 8] This figure shows the definition of a region on an image in this embodiment. [Figure 9] This figure shows an example of a histogram of the number of pixels and distance within a region in this embodiment. [Figure 10] This figure shows an example of grouped regions in this embodiment. [Figure 11] This is a top view of the extracted coordinates obtained in this embodiment. [Figure 12] This figure shows an example of the screen for setting the number of station transport devices to be allocated in this embodiment. [Figure 13] This figure shows the extracted region that is presumed to be reinforcing steel in this embodiment. [Modes for carrying out the invention]

[0010] An embodiment of the present invention will be described below.

[0011] Figure 1 is a block diagram showing the configuration of the object extraction system 100 in this embodiment.

[0012] The object extraction system 100 of this embodiment is composed of a computer having a processor (CPU) 1, a memory 2, an auxiliary storage device 3, and a communication interface 4. The object extraction system 100 may have an input interface 5 and an output interface 8.

[0013] The processor 1 is an arithmetic unit that executes programs stored in the memory 2. By the processor 1 executing various programs, the functions of each functional unit of the object extraction system 100 are realized. Note that part of the processing performed by the processor 1 when executing a program may be executed by another arithmetic unit (for example, hardware such as an ASIC or FPGA).

[0014] The memory 2 includes a ROM which is a non-volatile memory element and a RAM which is a volatile memory element. The ROM stores immutable programs (for example, BIOS), etc. The RAM is a high-speed and volatile memory element such as a DRAM (Dynamic Random Access Memory), and temporarily stores the programs executed by the processor 1 and the data used during program execution.

[0015] The auxiliary storage device 3 is, for example, a large-capacity and non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). Also, the auxiliary storage device 3 stores the data used by the processor 1 during program execution and the programs executed by the processor 1. That is, the programs are read from the auxiliary storage device 3, loaded into the memory 2, and executed by the processor 1 to realize the various functions of the object extraction system 100.

[0016] The communication interface 4 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0017] Input interface 5 is an interface to which input devices such as a keyboard 6 and a mouse 7 are connected and to receive input from the operator. Output interface 8 is an interface to which output devices such as a display device 9 and a printer (not shown) are connected and to output the program execution results in a format that the user can see. In addition, a user terminal connected to the object extraction system 100 via a network may provide the input and output devices. In this case, the object extraction system 100 may have the functionality of a web server, and the user terminal may access the object extraction system 100 using a predetermined protocol (e.g., http).

[0018] The program executed by processor 1 is provided to object extraction system 100 via removable media (such as CD-ROM or flash memory) or a network, and stored in non-volatile auxiliary storage device 3, which is a non-temporary storage medium. For this reason, object extraction system 100 should have an interface for reading data from removable media.

[0019] The object extraction system 100 is a computer system that operates on a single physical computer, or on multiple computers configured logically or physically, and may operate on a virtual computer built on multiple physical computer resources. For example, the multiple programs that implement the functions of the object extraction system 100 may each operate on separate physical or logical computers, or multiple programs may be combined and operate on a single physical or logical computer.

[0020] Figure 2 shows an example of the data structure of image 401, which includes distance information, as handled in this embodiment.

[0021] Distance image data has a data structure in which each pixel contains four pieces of information: Red, Green, Blue, and Depth (distance). Distance image data can be acquired using imaging devices that can acquire distance data for each pixel, such as TOF cameras, LiDAR, and stereo cameras. Alternatively, distance image data may be generated by combining image data and distance data of the same resolution acquired by multiple devices (for example, a camera that outputs RGB pixel data and a laser rangefinder that outputs point cloud distance data).

[0022] Although Figure 2 illustrates a color image with RGB luminance data for each pixel, RAW data with one luminance data per pixel, or a monochrome image with black and white luminance data per pixel, are also acceptable as long as each pixel has distance data.

[0023] Next, the conventional object extraction procedure will be explained with reference to Figures 3 to 6. In the plane estimation method of this conventional object extraction procedure, the operator specifies three points on the object.

[0024] Figure 3 is a flowchart of the conventional object extraction process using plane estimation.

[0025] The object extraction system 100 receives a specification of three points on the object to be extracted (e.g., reinforcing bars) in an image showing the object to be extracted, and stores the coordinates of the specified pixels on the image (301). As shown in Figure 4, the operator specifies three points 403 on the reinforcing bars 501 and 502 in image 401.

[0026] Next, the object extraction system 100 uses the camera position and captured data to convert the coordinates of the three specified points 403 on the image into three-dimensional coordinates (302).

[0027] Next, the object extraction system 100 calculates the normal vectors of the planes representing the three coordinates in three-dimensional space (303).

[0028] Next, the object extraction system 100 defines a region b with a predetermined depth Δ in the normal direction relative to a plane containing the three coordinates (304). For example, as shown in Figure 5, a rectangular parallelepiped region b507 is defined within a predetermined range Δ506 in front of and behind a plane 504 containing the specified point 403 (a plane defined by the normal vector 505 calculated from the three specified points 403).

[0029] Next, the object extraction system 100 extracts pixels contained in region b (305). For example, as shown in Figure 6, by extracting pixels located within region b 507, it is possible to extract regions that are estimated to be reinforcing bars 501 and 502.

[0030] Figure 7 is a flowchart of the object extraction process performed by the object extraction system 100 in this embodiment. The object extraction process shown in Figure 7 is performed when the processor 1 of the object extraction system 100 executes a program.

[0031] First, the object extraction system 100 divides the area of ​​image 401 from which to extract objects into a predetermined number or size of regions 803. Region 803 is a rectangular area smaller than a predetermined size, and it is preferable to divide the area from which to extract objects into equal parts. Then, the number of pixels in the divided regions 803 is counted at each distance, the maximum value of the number of pixels is searched for, the maximum value with the smallest distance is identified, and pixels included in a predetermined distance before and after that maximum value are extracted (701). As shown in Figure 2, image 401 contains brightness information and distance information for each RGB color for each pixel. Also, in image 401, the camera and the grid formed by the reinforcing bars 501 and 502 do not need to be directly facing each other.

[0032] For example, as shown in Figure 8, when an operator identifies an area in image 401 in which the reinforcing bars 501 and 502 to be extracted are visible, the object extraction system 100 divides the identified area into a region 803 of a predetermined size. Alternatively, when an operator specifies a single point in image 401 in which the reinforcing bars 501 and 502 to be extracted are visible, a region 803 of a predetermined size including that point may be defined, and the adjacent regions 803 on all sides (up, down, left, and right) may be defined to cover the entire image 401. While the region 803 may cover the entire image 401, it does not have to be defined in the areas of predetermined width on all sides (the outer edges of the image). This is because the outer edges of the image may contain objects other than the object to be extracted (e.g., the left and right walls, the ceiling above, and the floor below), and it is desirable to avoid detecting these objects.

[0033] Then, in a certain region 803, the number of pixels is counted for each distance. Typically, an image is taken with the object to be extracted in the foreground, and the pixels of the object to be extracted are included in larger numbers than the pixels of other objects. Therefore, as shown in Figure 9, when the number of pixels is plotted on the vertical axis and distance on the horizontal axis in a histogram, the pixels of the reinforcing bars 501 and 502, which are the objects to be extracted, are located at the peaks of the short distance on the histogram. For this reason, pixels included in a predetermined distance 903, before and after the maximum value 901 with the smallest distance, are extracted. If the object to be extracted is reinforcing bars, the predetermined distance 903 should be set to the maximum diameter of the reinforcing bar. Note that the maximum value 902, other than the maximum value 901 with the smallest distance, is an object that exists in the background of the object to be extracted.

[0034] Next, the object extraction system 100 uses the camera position and captured data to convert the image coordinates of the extracted pixels into three-dimensional coordinates (702).

[0035] Next, the object extraction system 100 calculates the normal vector of the plane representing the extracted pixels in three-dimensional space and calculates the centroid of the three-dimensional coordinates of the extracted pixels within region 803 (703). For example, it is preferable to calculate the normal vector of the plane that minimizes the sum of the distances from the extracted pixels. Alternatively, the centroid (i.e., the average value of the coordinates) of the extracted pixel group within region 803 may be calculated using the RANSAC method, and vectors from the calculated centroid to each pixel may be calculated, and the normal vector of the plane may be calculated using the eigenvectors of these pixel groups. Then, the plane having the calculated normal vector and passing through the centroid of the pixel group becomes the plane representing the three-dimensional coordinates of the pixels.

[0036] Next, the object extraction system 100 groups adjacent regions 803 using the calculated plane and point cloud relationships between the regions 803 (704). For example, if the average of the distance between the plane representing a first region 803 and the centroid of the pixel group extracted in the adjacent second region 803, and the distance between the plane representing the second region 803 and the centroid of the pixel group extracted in the first region 803 is smaller than a predetermined threshold, the first region 803 and the second region 803 may be grouped into one group. Alternatively, instead of the average of the two distances, one distance (for example, the distance between the plane representing the already grouped first region 803 and the centroid of the pixel group extracted in the adjacent second region 803) may be used. The predetermined threshold may be, for example, twice the maximum diameter of the reinforcing bar.

[0037] Other methods may be used to group regions 803. For example, if the angle between the normal vectors calculated for two adjacent regions 803 is smaller than a predetermined threshold, those two regions 803 may be grouped into one group. Alternatively, if the distance between the centroids of the pixel groups extracted within two adjacent regions 803 is smaller than a predetermined threshold, those two regions 803 may be grouped into one group.

[0038] Alternatively, one could use the multiple methods described above to calculate a single indicator and then determine whether to group the data comprehensively.

[0039] In this way, by grouping Region 803s based on their relationships, the plane representing the location of an object can be accurately determined. Furthermore, Region 803s that do not contain rebar or those that primarily show rebar in the background can be excluded.

[0040] Next, the object extraction system 100 selects one region group from the grouped region groups (705). Typically, since the object to be extracted is photographed in the foreground, it is good to select the region 803 with the shortest distance. If the object to be extracted is in the background, the region 803 with the second closest distance may be selected. For example, as shown in Figure 10, the region 803 containing reinforcing bars 501 and 502 is grouped into the same group shown in gray, and the other regions 803 are excluded.

[0041] Next, the object extraction system 100 calculates the normal vector of a plane representing the coordinates of the extracted pixels within the selected region 803 (706). For example, as shown in Figure 11, the plane is calculated from the coordinates 1104 of the extracted pixels within the selected region 803. For example, the normal vector may be calculated using the RANSAC method, similar to step 703. The plane calculated in step 706 allows the position of the object to be extracted to be identified.

[0042] Next, the object extraction system 100 defines a region b with a predetermined depth Δ in the normal direction based on a plane containing the three coordinates (707). For example, as shown in Figure 12, a rectangular parallelepiped region b1207 is defined within a predetermined range Δ1206 in front of and behind the plane 1203 (normal vector 1205) calculated from the coordinates of pixel 1204 in step 706.

[0043] Next, the object extraction system 100 extracts pixels contained in region b (708). For example, as shown in Figure 13, by extracting pixels located within region b1207, it is possible to extract regions that are estimated to be reinforcing bars 501 and 502.

[0044] As described above, according to the embodiment of the present invention, even if the camera is not directly facing the plane consisting of the object to be extracted (e.g., a reinforcing bar grid), the plane representing the object to be extracted can be accurately determined. Furthermore, since pixels within a predetermined range are extracted from the plane consisting of the object to be extracted, the object can be extracted stably regardless of the user, eliminating reliance on individual expertise.

[0045] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0046] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0047] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other storage media such as IC cards, SD cards, and DVDs.

[0048] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0049] 1...Processor, 2...Memory, 3...Auxiliary storage device, 4...Communication interface, 5...Input interface, 6...Keyboard, 7...Mouse, 8...Output interface, 9...Display device, 100...Object extraction system, 401...Image, 403...Specified point, 501, 502...Reinforcement bars, 803...Region, 901, 902...Maximum value,

Claims

1. An object extraction system that extracts information about an object from an image, It comprises an arithmetic unit that performs calculation processing and a storage unit that the arithmetic unit can access, The aforementioned arithmetic unit, A distance image containing distance information measured for each pixel is obtained. The aforementioned distance image is divided into multiple regions, In the divided region, count the number of pixels for each distance, From the maximum values ​​of the counted pixels, pixels within a predetermined distance range from the maximum value at the shortest distance from the shooting point are extracted. For each of the divided regions, a first plane representing the extracted pixels is calculated. Using the relationship between the first planes calculated for each of the aforementioned regions, the divided regions are grouped together. An object extraction system characterized by selecting one of the groups generated by the aforementioned grouping and calculating a second plane representing the extracted pixels within the region of the selected group.

2. The object extraction system according to claim 1, The object extraction system is characterized in that the calculation unit selects the group that is the shortest distance from the shooting point from among the groups generated by the grouping.

3. The object extraction system according to claim 1, The object extraction system is characterized in that the calculation unit extracts pixels that are located at a predetermined distance from the calculated second plane as pixels of the object.

4. The object extraction system according to claim 1, The object extraction system is characterized in that the calculation unit groups the first region and the second region into a single group when the distance between the centroids of the pixel groups extracted in the plane representing the first region and the adjacent second region is smaller than a predetermined threshold.

5. The object extraction system according to claim 1, The object extraction system is characterized in that the calculation unit groups two adjacent regions into one group if the angle between the normal vectors of the first plane calculated in each of the two adjacent regions is smaller than a predetermined threshold.

6. The object extraction system according to claim 1, The object extraction system is characterized in that, if the distance between the centroids of the pixel groups extracted in each of the two adjacent regions is less than a predetermined threshold, the calculation unit groups the two regions into a single group.

7. An object extraction system is an object extraction method for extracting information about an object from an image, The object extraction system is comprised of a computer having an arithmetic unit that performs calculation processing and a storage unit accessible by the arithmetic unit. The aforementioned object extraction method is: The calculation unit acquires a distance image including distance information measured for each pixel, The calculation unit divides the distance image into multiple regions, The calculation unit counts the number of pixels for each distance in the divided region, The calculation unit extracts pixels from the maximum value of the counted pixel count that are within a predetermined distance range from the maximum value of the shortest distance from the shooting point, The calculation unit calculates a first plane representing the extracted pixels for each of the divided regions, The calculation unit groups the divided regions using the relationships between the first planes calculated for each region, The object extraction method is characterized in that the calculation unit selects one of the groups generated by the grouping and calculates a second plane representing the extracted pixels within the region of the selected group.

8. An object extraction system is an object extraction program that extracts information about an object from an image, The object extraction system is comprised of a computer having an arithmetic unit that performs calculation processing and a storage unit accessible by the arithmetic unit. The aforementioned object extraction program is A procedure for obtaining a distance image that includes distance information measured for each pixel, A procedure for dividing the aforementioned distance image into multiple regions, The procedure for counting the number of pixels at each distance in the divided region, A procedure for extracting pixels within a predetermined distance range from the maximum value of the shortest distance from the shooting point among the aforementioned counted maximum values ​​of pixels, A procedure for calculating a first plane representing the extracted pixels for each of the divided regions, A procedure for grouping the divided regions using the relationship between the first planes calculated for each region, An object extraction program characterized by causing the calculation unit to perform the following steps: selecting one of the groups generated by the aforementioned grouping and calculating a second plane representing the extracted pixels within the region of the selected group.

Images