Object extraction system and object extraction method

Abstract

An object extraction system for extracting information on an object from an image comprises: a calculation unit that executes calculation processing; and a storage unit that is accessible by the calculation unit. The calculation unit acquires a distance image including measured distance information for each pixel, sets a plurality of reference points in the captured object, calculates a plane including the plurality of set reference points, extracts pixels included in a predetermined range in a normal direction of the calculated plane, compares the colors of the extracted pixels with the colors of the pixels at the plurality of set reference points, and determines, on the basis of the results of comparison between the colors of the pixels, whether the extracted pixels correspond to the object or noise.

Description

Object Extraction System and Object Extraction Method Incorporation by Reference

[0001] This application claims the priority of Japanese Patent Application No. 2024-160625, which was filed on September 18, 2024, and incorporates its content by reference into this application.

[0002] The present invention relates to an object extraction system for extracting an object shown in a distance image.

[0003] In the past, problems have occurred at construction sites where the number, diameter, spacing, etc. of reinforcing bars are constructed differently from the design, and the inspection of reinforcing bars has become stricter, increasing the burden on on-site work. To reduce this burden, three-dimensional data of reinforcing bars is acquired using sensors such as cameras. The point cloud represented by the acquired three-dimensional coordinates contains information other than the reinforcing bars to be measured, and it is necessary to extract the point cloud representing the target reinforcing bars.

[0004] As background art in this technical field, there is the following prior art. Patent Document 1 (Japanese Unexamined Patent Application Publication No. 2021-85838) describes a reinforcement inspection system for inspecting reinforcing bars in building construction, which includes a first image acquisition unit that acquires a distance image of the reinforcing bars, a second image acquisition unit that acquires a still image of the reinforcing bars, and a control unit that performs arithmetic processing on the distance image and the still image. The control unit selects the reinforcing bars to be measured, sets a measurement line that serves as a reference for measuring the reinforcing bars, extracts the edges of the reinforcing bars to be measured on the measurement line using the still image, determines the coordinates of the extracted edges from the distance image, and measures at least one of the spacing and diameter of the reinforcing bars using the determined coordinates of the edges.

[0005] In the reinforcement inspection system described in Patent Document 1, point cloud data on a plane composed of a reinforcing bar grid is extracted based on three points on the reinforcing bar input by the user. However, the three-dimensional data acquired by the sensor often contains noise. Especially in bright environments such as outdoors, objects that are not actually on the reinforcing bar plane may be extracted as if they are on the reinforcing bar plane, and objects that are not reinforcing bars may be determined to be reinforcing bars.

[0006] A typical example of the invention disclosed in this application is as follows: an object extraction system for extracting information about an object from an image, comprising a calculation unit that performs calculation processing and a storage unit accessible by the calculation unit, wherein the calculation unit acquires a distance image including distance information for each measured pixel, sets a plurality of reference points within the photographed object, calculates a plane including the set plurality of reference points, extracts pixels included in a predetermined range in the normal direction of the calculated plane, compares the color of the extracted pixels with the colors of the pixels of the set plurality of reference points, and determines whether the extracted pixels are the object or noise based on the comparison result of the pixel colors.

[0007] According to one aspect of the present invention, the extraction accuracy of the target object (e.g., reinforcing bars) can be improved. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

[0008] This is a block diagram showing the configuration of the object extraction system in an embodiment of the present invention. This is a flowchart of the object extraction process in an embodiment of the present invention. This is a diagram showing three specified points on an image in an embodiment of the present invention. This is a diagram showing the region extracted from an image in an embodiment of the present invention. This is a diagram showing the normalization of brightness values ​​in an embodiment of the present invention. This is a flowchart of the RGB value determination process in an embodiment of the present invention. This is a diagram showing points on an image with normalized brightness values ​​in an embodiment of the present invention. This is a diagram showing an image from which noise has been removed in an embodiment of the present invention.

[0009] Figure 1 is a block diagram showing the configuration of the object extraction system 100 in this embodiment.

[0010] The object extraction system 100 in this embodiment includes a processor (CPU) 101, memory 102, auxiliary storage device 103, communication interface 104, input interface 105, operation unit 107, output interface 108, and display device 109. The object extraction system 100 is preferably implemented in a terminal device such as a tablet. A camera 106 capable of acquiring three-dimensional data such as a distance image including distance information for each pixel is connected to the object extraction system 100. The camera 106 may be built into the terminal device as long as it can acquire three-dimensional data. The camera 106 only needs to be able to capture a color image and may output an RGB image in which color information is represented by a set of red, green, and blue pixel values, or an HLS image in which color information is represented by a set of hue, lightness, and saturation pixel values.

[0011] The processor 101 is an arithmetic unit that executes programs stored in the memory 102. By executing various programs, the processor 101 realizes the functions of each functional unit of the object extraction system 100. Note that some of the processing performed by the processor 101 when executing programs may be performed by other arithmetic units (for example, hardware such as ASICs or FPGAs).

[0012] The memory 102 includes a non-volatile memory element called ROM and a volatile memory element called RAM. The ROM stores immutable programs (e.g., BIOS). The RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory) and temporarily stores programs executed by the processor 101 and data used during program execution.

[0013] The auxiliary storage device 103 is a high-capacity, non-volatile storage device such as a flash memory (SSD) or a magnetic storage device (HDD). The auxiliary storage device 103 also stores data used by the processor 101 when executing a program, and the program executed by the processor 101. In other words, the program is read from the auxiliary storage device 103, loaded into memory 102, and executed by the processor 101 to realize each function of the object extraction system 100.

[0014] The communication interface 104 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0015] The input interface 105 is connected to input devices such as the control unit 107 and the camera 106, and receives distance images captured by the camera 106 and input from the operator. The output interface 108 is connected to output devices such as the display device 109, and is an interface that outputs the program execution results in a format that can be viewed by the user.

[0016] The program executed by the processor 101 is provided to the object extraction system 100 via removable media (such as a CD-ROM or flash memory) or a network, and is stored in a non-volatile auxiliary storage device 103, which is a non-temporary storage medium. For this reason, the object extraction system 100 should have an interface for reading data from the removable media.

[0017] The object extraction system 100 may be implemented not on a terminal device, but on a computer system consisting of multiple logically or physically configured computers, or it may operate on a virtual computer built on multiple physical computer resources. For example, the multiple programs that realize the functions of the object extraction system 100 may each operate on separate physical or logical computers, or multiple programs may be combined and operate on a single physical or logical computer. In this case, a user terminal connected to the object extraction system 100 via a network may provide input and output devices.

[0018] Figure 2 is a flowchart of the object extraction process performed by the object extraction system 100 in this embodiment. The object extraction process shown in Figure 2 is performed when the processor 101 of the object extraction system 100 executes a program.

[0019] The object extraction system 100 receives the designation of three or more points on the object to be extracted (e.g., reinforcing bars) in an image containing the object to be extracted, and stores the coordinates of the three or more designated points on the image (701). As shown in Figure 3, the user designates three points 411, 412, and 413 on the reinforcing bars 501 and 502 in image 301. The object extraction system 100 also receives the user's setting of a noise reduction threshold α and stores the set threshold α (702). The order of processing in steps 701 and 702 is arbitrary.

[0020] Next, the object extraction system 100 uses the camera position and shooting data (shooting direction and field of view) to convert all coordinates of the depth image into three-dimensional coordinates (703).

[0021] Next, the object extraction system 100 calculates the normal vector of a plane containing the three points specified by the user in three-dimensional space (704).

[0022] Next, the object extraction system 100 defines a rectangular parallelepiped region b with a predetermined depth Δ in the normal direction, based on a plane containing the three points specified by the user (705).

[0023] Next, the object extraction system 100 extracts pixels contained in region b (706). For example, as shown in Figure 4, when pixels whose distance data is within region b 507 are extracted, region 202 is estimated to be reinforcing bars 501 and 502, and region 203 is estimated to be noise.

[0024] Next, the object extraction system 100 normalizes the brightness values ​​of the pixels included in region b (707). The process of normalizing the brightness values ​​is performed, for example, by the procedure shown in Figure 5. First, the RGB image 301 is converted to an HLS image 302. Then, the brightness values, which are represented by 8 bits, are converted to a smaller number of bits (for example, 7 bits). For example, by performing a linear transformation using the following formula so that X=0 becomes Y=64 and X=255 becomes Y=191, an HLS image 303 is obtained in which the brightness values ​​in the range from 0 to 255 are normalized to the range from 64 to 191. Note that in an HLS image, the median brightness value represents the primary color, and in this transformation, the central value of 128 remains 128. Y = (X - 128) × 0.5 + 128

[0025] Then, the HLS image 303 with normalized brightness values ​​is converted into an RGB image 304 (see Figure 5(D)). In this way, an RGB image 304 with a narrower range of brightness values ​​and corrected overexposure and underexposure can be obtained.

[0026] Alternatively, instead of normalizing luminance values ​​using HLS images, brightness values ​​may be normalized using HSV images, or luminance values ​​of RGB images may be normalized using other algorithms.

[0027] Furthermore, as mentioned above, normalization of luminance values ​​is effective for images with high luminance; therefore, when dealing with low-luminance images taken indoors or in the shade, step 707 may be omitted.

[0028] An RGB value determination process is performed (708) which compares the difference between the RGB values ​​of all pixels in region b and the RGB values ​​of each of the three user-specified points with a threshold α.

[0029] The details of the RGB value determination process 708 will be explained with reference to Figures 6, 7, and 8.

[0030] First, the RGB value of one pixel (for example, the leftmost column of the top row) 421, which is determined to be within region b, is obtained from the corrected RGB image 304 (7081).

[0031] Then, the RGB values ​​obtained in step 7081 are compared with the RGB values ​​of the first point 411 specified by the user (7082). If the difference in RGB values ​​is smaller than a predetermined threshold, it is determined that the pixel 421 is reinforcing steel, and the RGB value determination process for the pixel 421 is terminated.

[0032] On the other hand, if it is determined in step 7082 that the difference in RGB values ​​is greater than or equal to a predetermined threshold, the RGB value obtained in step 7081 is compared with the RGB value of a second point 412 specified by the user (7083). If the difference in RGB values ​​is less than the predetermined threshold, it is determined that the pixel 421 is reinforcing steel, and the RGB value determination process for the pixel 421 is terminated.

[0033] On the other hand, if it is determined in step 7083 that the difference in RGB values ​​is greater than or equal to a predetermined threshold, the RGB value obtained in step 7081 is compared with the RGB value of the third point 413 specified by the user (7084). If the difference in RGB values ​​is less than the predetermined threshold, it is determined that the pixel 421 is reinforcing steel, and the RGB value determination process for the pixel 421 is terminated.

[0034] On the other hand, if it is determined in step 7084 that the difference in RGB values ​​is greater than or equal to a predetermined threshold, the RGB value of pixel 421 is not similar to the RGB values ​​of any of the points 411, 412, and 413 specified by the user. Therefore, it is determined that pixel 421 is noise and not rebar, and the value of pixel 421 is set to the background value (for example, 0, which indicates black).

[0035] In steps 7082 to 7084, the RGB value comparison may be performed by comparing the hue of each pixel using the three colors R, G, and B, and determining whether the difference of at least one of the three colors is below a threshold, or by determining whether the difference of all three colors is below a threshold. Alternatively, it may be determined whether the distance between two points in the RGB color space is below a predetermined threshold. The thresholds for each color may be the same or different. The thresholds may be set in advance during system construction, be selectable by the user, or automatically set according to the environment in which the image was captured. For example, the threshold may be determined by the average value of the brightness of all pixels in the image, and the threshold may be increased for brighter images.

[0036] Furthermore, the RGB values ​​of points 411, 412, and 413 specified by the user as the basis for comparison may be the RGB values ​​of points 411, 412, and 413 themselves, or they may be the average value of the RGB values ​​within a predetermined range surrounding points 411, 412, and 413.

[0037] Furthermore, although the colors are compared using the RGB image 304 converted from the HLS image 303, it is also possible to compare the color of the first point 411 with the colors of points 411, 412, and 413 specified by the user in the HLS image 303 without converting to an RGB image 304.

[0038] In this way, pixels whose RGB values ​​are similar to any of the three points specified by the user are identified as rebar. Therefore, even if the colors of the three rebar points specified by the user differ depending on their position in the image or the direction in which the rebar is arranged, it is possible to accurately distinguish between rebar and noise. The denoised image shown in Figure 8 is then obtained.

[0039] Although Figure 6 shows the processing for a single pixel within region b, the processing shown in Figure 6 is repeatedly performed for all pixels within region b to determine whether every pixel in region b is rebar or noise.

[0040] As described above, the object extraction system 100 of the embodiment of the present invention focuses on the color information of points on the reinforcing bars specified by the user in order to remove noise other than the reinforcing bar area, which is a problem in the prior art. The color of the reinforcing bars and the color of the background (for example, the formwork installed behind the reinforcing bars) are different. Also, generally, reinforcing bar grids are made up of reinforcing bars of the same color. Therefore, by removing pixels that differ in color from the three points on the reinforcing bars input by the user as noise from the three-dimensional data acquired from the sensor, the reinforcing bar area can be extracted and the accuracy of reinforcing bar extraction can be improved. Furthermore, various forms of color information can be used for judgment, such as RGB values, HLS values, and RGB values ​​normalized from luminance values.

[0041] Using RGB values ​​for color information has a noise reduction effect on low-luminance images taken indoors or in the shade. However, in high-luminance images taken in sunlight, overexposure occurs due to sunlight reflection from rebar, and underexposure occurs in rebar in the shade.

[0042] When using the HLS value for color information, color is represented by hue, luminance, and saturation. Therefore, in areas affected by sunlight, the luminance value will be high or low, and noise can be determined by focusing on the hue.

[0043] When normalizing the luminance value using the HLS value for color information and determining noise with RGB values, it is possible to suppress the influence of sunlight like the HLS value while maintaining the high characteristic of noise removal for RGB values. For example, as in the embodiments described later, convert the RGB image to an HLS image, normalize the luminance value L, and then convert it back to an RGB image. The normalization may compress the range of possible luminance values from 0 to 255 in 8 bits to a range of 64 to 191 in 7 bits with the same center. Through the normalization process, it is possible to bring the white spots due to sunlight reflection and the black crushed areas in the shadow closer to the primary colors, and noise can be removed by comparing the pixel colors in the RGB image after normalizing the luminance value. In particular, it is possible to easily extract an object (for example, a reinforcing bar) even in a bright environment such as outdoors.

[0044] Note that the present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Also, the configuration of another embodiment may be added to the configuration of one embodiment. Also, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations may be made.

[0045] Also, each of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing a part or all of them with an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0046] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, hard disk, SSD (Solid State Drive), or a recording medium such as an IC card, SD card, DVD.

[0047] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. An object extraction system for extracting information about an object from an image, comprising: a calculation unit that performs calculation processing; and a storage unit accessible by the calculation unit, wherein the calculation unit acquires a distance image including distance information for each measured pixel; sets a plurality of reference points within the photographed object; calculates a plane including the set plurality of reference points; extracts pixels included in a predetermined range in the normal direction of the calculated plane; compares the color of the extracted pixels with the colors of the pixels of the set plurality of reference points; and determines whether the extracted pixels are the object or noise based on the comparison result of the pixel colors.

2. A computer system according to claim 1, wherein the calculation unit determines that a pixel is an object if the difference between the color of the extracted pixel and the color of the pixel at the set reference point is less than a predetermined threshold, and determines that a pixel is noise if the difference between the color of the extracted pixel and the color of the pixel at the set reference point is greater than or equal to a predetermined threshold.

3. A computer system according to claim 1, wherein the calculation unit normalizes the brightness value of the extracted pixels and the brightness value of the pixels at the set reference point, and compares the color of the pixels whose brightness value has been normalized with the hue of the pixels at the reference point whose brightness value has been normalized.

4. A computer system according to claim 3, wherein the distance image is an RGB image represented by three colors: red, green, and blue; the calculation unit converts the acquired distance image into an HLS image represented by hue, brightness, and saturation; normalizes the brightness of the pixels extracted in the converted HLS image and the brightness values ​​of the pixels at the set reference points; converts the HLS image with normalized brightness values ​​into an RGB image with normalized brightness values; and compares the color of the extracted pixels with the color of the pixels at the set reference points in the RGB image with normalized brightness values.

5. The computer system according to claim 1, wherein the threshold for determining the color difference of the pixels is set according to the environment in which the image was taken.

6. An object extraction method for extracting object information from an image, wherein the object extraction system comprises a calculation unit that performs calculation processing and a storage unit accessible by the calculation unit, and the object extraction method is characterized in that the calculation unit acquires a distance image including distance information for each measured pixel, the calculation unit sets a plurality of reference points within the photographed object, the calculation unit calculates a plane including the set plurality of reference points, the calculation unit extracts pixels included in a predetermined range in the normal direction of the calculated plane, the calculation unit compares the color of the extracted pixels with the colors of the pixels of the set plurality of reference points, and the calculation unit determines whether the extracted pixels are the object or noise based on the comparison result of the pixel colors.

Images