Object extraction system and object extraction method

Abstract

To improve the accuracy of object extraction. [Solution] An object extraction system for extracting information about an object from an image, comprising a calculation unit that performs calculation processing and a storage unit accessible by the calculation unit, wherein the calculation unit acquires a distance image including distance information for each measured pixel, sets a plurality of reference points within the photographed object, calculates a plane including the set plurality of reference points, extracts pixels included in a predetermined range in the normal direction of the calculated plane, compares the color of the extracted pixels with the colors of the pixels of the set plurality of reference points, and determines whether the extracted pixels are the object or noise based on the comparison result of the pixel colors.

Description

Technical Field

[0001] The present invention relates to an object extraction system for extracting an object shown in a distance image.

Background Art

[0002] In the past, problems have occurred where the number, diameter, spacing, etc. of reinforcing bars are constructed differently from the design at a construction site. The inspection of reinforcing bars has become stricter, and the burden in on-site work has increased. To reduce this burden, three-dimensional data of reinforcing bars is acquired using a sensor such as a camera. The point cloud represented by the acquired three-dimensional coordinates contains information other than the reinforcing bars to be measured, and it is necessary to extract the point cloud representing the target reinforcing bars.

[0003] [[ID= (16)]]

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The rebar inspection system described in Patent Document 1 extracts point cloud data from a plane consisting of a rebar grid, based on three points on the rebar input by the user. However, the three-dimensional data acquired by the sensor often contains noise, and especially in bright environments such as outdoors, objects that are not actually on the rebar plane may be extracted as being on the rebar plane, and non-rebar objects may be judged as rebar. [Means for solving the problem]

[0006] A typical example of the invention disclosed in this application is as follows: an object extraction system for extracting information about an object from an image, comprising a calculation unit that performs calculation processing and a storage unit accessible by the calculation unit, wherein the calculation unit acquires a distance image including distance information for each measured pixel, sets a plurality of reference points within the photographed object, calculates a plane including the set plurality of reference points, extracts pixels included in a predetermined range in the normal direction of the calculated plane, compares the color of the extracted pixels with the colors of the pixels of the set plurality of reference points, and determines whether the extracted pixels are the object or noise based on the comparison result of the pixel colors. [Effects of the Invention]

[0007] According to one aspect of the present invention, the extraction accuracy of the target object (e.g., reinforcing bars) can be improved. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the configuration of an object extraction system in an embodiment of the present invention. [Figure 2] This is a flowchart of the object extraction process in an embodiment of the present invention. [Figure 3] This figure shows three points specified on an image of an embodiment of the present invention. [Figure 4] This figure shows a region extracted from an image according to an embodiment of the present invention. [Figure 5] This figure shows the normalization of the luminance value in an embodiment of the present invention. [Figure 6] This is a flowchart of the RGB value determination process in an embodiment of the present invention. [Figure 7] This figure shows a point on an image with normalized luminance values ​​according to an embodiment of the present invention. [Figure 8] This figure shows an image from an embodiment of the present invention with noise removed. [Modes for carrying out the invention]

[0009] Figure 1 is a block diagram showing the configuration of the object extraction system 100 in this embodiment.

[0010] The object extraction system 100 in this embodiment includes a processor (CPU) 101, memory 102, auxiliary storage device 103, communication interface 104, input interface 105, operation unit 107, output interface 108, and display device 109. The object extraction system 100 is preferably implemented in a terminal device such as a tablet. A camera 106 capable of acquiring three-dimensional data such as a distance image containing distance information for each pixel is connected to the object extraction system 100. The camera 106 may be built into the terminal device as long as it can acquire three-dimensional data. The camera 106 only needs to be able to capture a color image and preferably outputs an RGB image in which color information is represented by a set of red, green, and blue pixel values, or an HLS image in which color information is represented by a set of hue, lightness, and saturation pixel values.

[0011] The processor 101 is an arithmetic unit that executes programs stored in the memory 102. The execution of various programs by the processor 101 enables the functions of each functional unit of the object extraction system 100. Note that some of the processing performed by the processor 101 through program execution may be performed by other arithmetic units (e.g., hardware such as ASICs or FPGAs).

[0012] Memory 102 includes a non-volatile memory element called ROM and a volatile memory element called RAM. ROM stores immutable programs (e.g., BIOS). RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory) and temporarily stores programs executed by the processor 101 and data used during program execution.

[0013] The auxiliary storage device 103 is a high-capacity, non-volatile storage device such as a flash memory (SSD) or a magnetic storage device (HDD). The auxiliary storage device 103 also stores data used by the processor 101 when executing a program, and the program executed by the processor 101. In other words, the program is read from the auxiliary storage device 103, loaded into memory 102, and executed by the processor 101, thereby realizing each function of the object extraction system 100.

[0014] The communication interface 104 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0015] The input interface 105 is connected to input devices such as the control unit 107 and the camera 106, and receives distance images captured by the camera 106 and input from the operator. The output interface 108 is connected to output devices such as the display device 109 and is an interface that outputs the program execution results in a format that can be viewed by the user.

[0016] The program executed by the processor 101 is provided to the object extraction system 100 via removable media (such as a CD-ROM or flash memory) or a network, and is stored in a non-volatile auxiliary storage device 103, which is a non-temporary storage medium. For this reason, the object extraction system 100 should have an interface for reading data from the removable media.

[0017] The object extraction system 100 may be implemented in a computer system configured on a plurality of logically or physically configured computers rather than a terminal device, and may operate on a virtual computer built on a plurality of physical computer resources. For example, the plurality of programs that implement the functions of the object extraction system 100 may each operate on a separate physical or logical computer, or a plurality of them may be combined and operate on one physical or logical computer. In this case, a user terminal connected to the object extraction system 100 via a network may provide an input device and an output device.

[0018] FIG. 2 is a flowchart of an object extraction process executed by the object extraction system 100 of the present embodiment. The object extraction process shown in FIG. 2 is performed when the processor 101 of the object extraction system 100 executes a program.

[0019] The object extraction system 100 receives the specification of three or more points on the object to be extracted in an image in which the object to be extracted (for example, a reinforcing bar) appears, and stores the coordinates of the three or more specified points on the image (701). As shown in FIG. 3, for the specification of three or more points on the image, the user specifies three points 411, 412, and 413 on the reinforcing bars 501 and 502 in the image 301. Further, the object extraction system 100 receives the setting of the noise removal threshold α by the user, and stores the set threshold α (702). The order of the processes in steps 701 and 702 may be arbitrary.

[0020] Next, the object extraction system 100 uses the position of the camera and the shooting data (shooting direction and angle of view) to convert all the coordinates of the distance image into three-dimensional coordinates (703).

[0021] Next, the object extraction system 100 calculates the normal vector of the plane including the three points specified by the user in the three-dimensional space (704).

[0022] Next, the object extraction system 100 defines a rectangular region b with a predetermined depth Δ in the normal direction, based on a plane containing the three points specified by the user (705).

[0023] Next, the object extraction system 100 extracts pixels contained in region b (706). For example, as shown in Figure 4, when pixels whose distance data is within region b507 are extracted, region 202, which is estimated to be reinforcing bars 501 and 502, and region 203, which is estimated to be noise, are extracted.

[0024] Next, the object extraction system 100 normalizes the brightness values ​​of the pixels included in region b (707). The process of normalizing the brightness values ​​is performed, for example, by the procedure shown in Figure 5. First, the RGB image 301 is converted to an HLS image 302. Then, the brightness values, which are represented by 8 bits, are converted to a smaller number of bits (for example, 7 bits). For example, by performing a linear transformation using the following formula so that X=0 becomes Y=64 and X=255 becomes Y=191, an HLS image 303 is obtained in which the brightness values ​​in the range from 0 to 255 are normalized to the range from 64 to 191. Note that in an HLS image, the median brightness value represents the primary color, and in this transformation, the midpoint of the range, 128, remains 128. Y = (X - 128) × 0.5 + 128

[0025] Then, the HLS image 303 with normalized luminance values ​​is converted to an RGB image 304 (see Figure 5(D)). In this way, an RGB image 304 with a narrower range of luminance values ​​and corrected overexposure and underexposure can be obtained.

[0026] Alternatively, instead of normalizing luminance values ​​using HLS images, brightness values ​​may be normalized using HSV images, or luminance values ​​of RGB images may be normalized using other algorithms.

[0027] Furthermore, as mentioned above, normalization of luminance values ​​is effective for images with high luminance; therefore, when dealing with low-luminance images taken indoors or in the shade, step 707 may be omitted.

[0028] An RGB value determination process is performed (708) which compares the difference between the RGB values ​​of all pixels in region b and the RGB values ​​of three user-specified points with a threshold α.

[0029] The details of the RGB value determination process 708 will be explained with reference to Figures 6, 7, and 8.

[0030] First, the RGB values ​​of one pixel (for example, the leftmost column of the top row) 421, which is determined to be within region b, are obtained from the corrected RGB image 304 (7081).

[0031] Then, the RGB values ​​obtained in step 7081 are compared with the RGB values ​​of the first point 411 specified by the user (7082). If the difference in RGB values ​​is less than a predetermined threshold, it is determined that the pixel 421 is reinforcing steel, and the RGB value determination process for the pixel 421 is terminated.

[0032] On the other hand, if it is determined in step 7082 that the difference in RGB values ​​is greater than or equal to a predetermined threshold, the RGB values ​​obtained in step 7081 are compared with the RGB values ​​of the second point 412 specified by the user (7083). If the difference in RGB values ​​is less than the predetermined threshold, it is determined that the pixel 421 is reinforcing steel, and the RGB value determination process for the pixel 421 is terminated.

[0033] On the other hand, if it is determined in step 7083 that the difference in RGB values ​​is greater than or equal to a predetermined threshold, the RGB values ​​obtained in step 7081 are compared with the RGB values ​​of the third point 413 specified by the user (7084). If the difference in RGB values ​​is less than the predetermined threshold, it is determined that the pixel 421 is reinforcing steel, and the RGB value determination process for the pixel 421 is terminated.

[0034] On the other hand, if it is determined in step 7084 that the difference in RGB values ​​is greater than or equal to a predetermined threshold, the RGB value of pixel 421 is determined to be noise rather than rebar, because it is not similar to the RGB values ​​of any of the points 411, 412, and 413 specified by the user. The value of pixel 421 is then set to the background value (for example, 0, which indicates black).

[0035] In steps 7082-7084, the RGB value comparison may involve comparing the hue of each pixel using the three colors R, G, and B, and determining whether the difference of at least one of the three colors is below a threshold, or whether the difference of all three colors is below a threshold. Alternatively, it may be determined whether the distance between two points in the RGB color space is below a predetermined threshold. The thresholds for each color may be the same or different. The thresholds may be set in advance during system construction, be user-selectable, or automatically set according to the environment in which the image was captured. For example, the threshold may be determined by the average brightness value of all pixels in the image, and the threshold may be increased for brighter images.

[0036] Furthermore, the RGB values ​​of points 411, 412, and 413 specified by the user as the basis for comparison may be the RGB values ​​of points 411, 412, and 413 themselves, or they may be the average value of the RGB values ​​within a predetermined range surrounding points 411, 412, and 413.

[0037] Furthermore, although the colors are compared using the RGB image 304 converted from the HLS image 303, it is also possible to compare the color of the first point 411 with the colors of points 411, 412, and 413 specified by the user in the HLS image 303 without converting to an RGB image 304.

[0038] In this way, pixels whose RGB values ​​are similar to any of the three points specified by the user are identified as rebar. Therefore, even if the colors of the three rebar points specified by the user differ depending on their position in the image or the orientation of the rebar, it is possible to accurately distinguish between rebar and noise. The denoised image shown in Figure 8 is then obtained.

[0039] Although Figure 6 shows the processing for a single pixel within region b, the processing shown in Figure 6 is repeatedly performed for all pixels within region b to determine whether every pixel in region b is rebar or noise.

[0040] As described above, the object extraction system 100 of the embodiment of the present invention focuses on the color information of points on the reinforcing bars specified by the user in order to remove noise other than the reinforcing bar area, which is a problem in the prior art. The color of the reinforcing bars and the color of the background (for example, the formwork installed behind the reinforcing bars) are different. Also, generally, reinforcing bar grids are made up of reinforcing bars of the same color. Therefore, by removing pixels that are different in color from the three points on the reinforcing bars input by the user as noise among the pixels included in the three-dimensional data acquired from the sensor, the reinforcing bar area can be extracted and the accuracy of reinforcing bar extraction can be improved. Furthermore, various forms of color information can be used for judgment, such as RGB values, HLS values, and RGB values ​​obtained by normalizing the brightness value.

[0041] Using RGB values ​​for color information has a noise reduction effect on low-luminance images taken indoors or in the shade. However, in high-luminance images taken in sunlight, overexposure occurs due to sunlight reflection from rebar, and underexposure occurs in rebar in the shade.

[0042] When HLS values ​​are used for color information, color is represented by hue, brightness, and saturation. Areas affected by sunlight will have high or low brightness values, and noise can be detected by focusing on the hue.

[0043] By using HLS values ​​for color information to normalize luminance values ​​and then determining noise using RGB values, it is possible to suppress the effects of sunlight, similar to HLS values, while maintaining the high noise reduction effectiveness of RGB values. For example, as shown in the example described later, an RGB image is converted to an HLS image, the luminance value L is normalized, and then converted back to an RGB image. For normalization, it is preferable to compress the range of possible luminance values ​​from 8 bits (0-255) to 7 bits (64-191) with the same center. Through normalization, blown-out highlights due to sunlight reflection and crushed black areas in shadows can be brought closer to the original colors, and noise can be removed by comparing the pixel colors in the RGB image after normalizing the luminance values. In particular, it is possible to easily extract objects (e.g., rebar) even in bright environments such as outdoors.

[0044] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and the present invention is not necessarily limited to having all the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, configurations of other embodiments may be added to the configuration of one embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0045] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0046] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other storage media such as IC cards, SD cards, and DVDs.

[0047] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0048] 100 Object Extraction System 101 Processors 102 memory 103 Auxiliary storage device 104 Communication Interface 105 Input Interfaces 106 Camera 107 Operation section 108 Output Interfaces 109 Display devices 301 RGB image captured by camera 302 HLS images 303 HLS image after normalization of luminance values 304 Corrected RGB image 411, 412, 413 User-specified points 421 pixels 501, 502 Reinforcement bars

Claims

1. An object extraction system that extracts information about an object from an image, It comprises an arithmetic unit that performs calculation processing and a storage unit that the arithmetic unit can access, The aforementioned arithmetic unit, A distance image containing distance information for each measured pixel is obtained. Set multiple reference points within the photographed object, A plane containing the aforementioned set reference points is calculated, Pixels included in a predetermined range in the normal direction of the calculated plane are extracted, The color of the extracted pixel is compared with the color of the pixels of the set plurality of reference points. An object extraction system characterized by determining whether the extracted pixels are objects or noise based on the color comparison results of the pixels.

2. A computer system according to claim 1, The aforementioned arithmetic unit, If the difference between the color of the extracted pixel and the color of the pixel at the set reference point is less than a predetermined threshold, the pixel is determined to be the object. An object extraction system characterized in that, if the difference between the color of the extracted pixel and the color of the pixel at the set reference point is greater than or equal to a predetermined threshold, the pixel is determined to be noise.

3. A computer system according to claim 1, The aforementioned arithmetic unit, The brightness values ​​of the extracted pixels and the brightness values ​​of the pixels at the set reference points are normalized. An object extraction system characterized by comparing the color of a pixel whose brightness value has been normalized with the hue of a pixel at a reference point whose brightness value has been normalized.

4. A computer system according to claim 3, The aforementioned distance image is an RGB image represented by three colors: red, green, and blue. The aforementioned arithmetic unit, The acquired distance image is converted into an HLS image represented by hue, brightness, and saturation. The brightness of the pixels extracted in the converted HLS image and the brightness value of the pixels at the set reference point are normalized. The HLS image with normalized luminance values ​​is converted into an RGB image with normalized luminance values. An object extraction system characterized by comparing the color of the extracted pixels with the color of the pixels at the set reference points in an RGB image in which the luminance values ​​have been normalized.

5. A computer system according to claim 1, The object extraction system is characterized in that the threshold for determining the color difference of the aforementioned pixels is set according to the environment in which the image was taken.

6. An object extraction system is an object extraction method for extracting information about an object from an image, The object extraction system comprises a calculation unit that performs calculation processing and a storage unit that the calculation unit can access. The aforementioned object extraction method is: The calculation unit acquires a distance image including the measured distance information for each pixel, The calculation unit sets a plurality of reference points within the photographed object, The calculation unit calculates a plane that includes the set plurality of reference points, The calculation unit extracts pixels that are included in a predetermined range in the normal direction of the calculated plane, The calculation unit compares the color of the extracted pixel with the color of the pixels of the set plurality of reference points, The object extraction method is characterized in that the calculation unit determines whether the extracted pixel is the object or noise based on the result of comparing the colors of the pixels.

Images