Fixing device identification device

Abstract

To provide a fixture identification apparatus which can accurately identify an image of a fixture based on a captured image of a plurality of fixtures fitted to a plate.SOLUTION: A fixture identification apparatus 10 according to the present invention has a corner detection image generating unit 12 for generating a corner detection image GA, a binary processing image generating unit 13 for generating binary processing image GB, a masking processing image generating unit 14 for comparing the corner detection image GA and the binary processing image GB to generate from the corner detection image GA a masking processed image GC obtained by masking processing an image gd of the corner parts, which is obtained by removing images gc of overlapped parts overlapped with images gb of identified parts identified by adaptive binary processing, among a plurality of corner part images ga detected in the corner detection image GA, and a fixture specifying unit 15 for specifying a machine screw image 6 based on the image gc of the plurality of overlapped parts in the masking processed image GC.SELECTED DRAWING: Figure 3

Description

Technical Field

[0004] ,

[0006] , , , , ,

[0005] , , , , ,

[0001] The present invention relates to a fixture identification device that identifies an image of a fixture from a captured image including a plurality of fixtures driven into a plate material.

Background Art

[0002] Conventionally, as shown in Patent Document 1, a plate material such as a gypsum board used for an inner wall material or a ceiling material is attached to a base via a plurality of fixtures such as screws. The plurality of fixtures are driven into the plate material at a prescribed interval along at least the periphery of the plate material. After the construction of the plate material, it is inspected whether the interval between the fixtures is within the prescribed interval.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, such inspection can be performed visually, but it is also assumed that an image of a plate material into which fixtures are driven is captured and the interval between the fixtures is measured from the captured image. However, it takes time to visually identify the image of the fixture from a captured image including a plurality of fixtures driven into the plate material, and even if the shape of the fixture is learned by machine learning and used, it is difficult to say that the identification accuracy is high.

[0005] The present invention has been made in view of such points, and an object thereof is to provide a fixture identification device that can accurately identify an image of a fixture from a captured image including a plurality of fixtures driven into a plate material.

Means for Solving the Problems

[0006] In view of the above problems, the fixing device identification device according to the present invention is a fixing device identification device that identifies images of fixing devices from an image captured including a plurality of fixing devices embedded in a plate material, and is characterized by comprising: a corner detection image generation unit that generates a corner detection image by performing corner detection on the image captured; a binarization image generation unit that generates a binarized image by performing adaptive binarization processing on the image captured; a masking image generation unit that compares the corner detection image and the binarization image and generates a masking image from the corner detection image by performing a masking process that leaves images of overlapping parts that overlap with the images of identified parts identified by adaptive binarization processing among the images of a plurality of corner parts detected in the corner detection image, and deletes the images of other corner parts; and a fixing device identification unit that identifies the image of the fixing device based on the plurality of images of the overlapping parts that remain in the masking image.

[0007] According to the present invention, the corner detection image generation unit generates a corner detection image by performing corner detection on an image captured by an imaging device. The generated corner image can detect not only images of the fixing device, but also images of the corners of the board material, dot-like patterns other than the board material, dirt, etc., as corner images.

[0008] The binarized image generation unit generates a binarized image by performing adaptive binarization on the captured image. Adaptive binarization does not fix thresholds such as image density, but rather changes the threshold for each pixel according to the surrounding image. Therefore, it is possible to identify images of identified parts according to the difference in grayscale within the captured image. Consequently, the images of identified parts obtained in this way include not only images of the fixtures, but also images of the edges of the plate material and images of the ruled lines on the plate material.

[0009] Here, the inventors considered that when comparing the corner detection image and the binarized image, both would contain images of the fixing device. Therefore, by comparing the corner image detected by the corner detection image with the identified portion image identified by the adaptive binarization process, they obtained a new finding that the image located at a common position between the corner image and the identified portion image in the entire image area is highly likely to be an image of the fixing device.

[0010] Based on these findings, the masking image generation unit compares the corner detection image with the binarized image and generates a masked image from the corner detection image. This masked image retains the overlapping portion of the corner area images that overlap with the identified portion identified by adaptive binarization, while masking the remaining corner area images.

[0011] As a result, the masked image is an image from which images other than the fastener image (specifically, images of the edges of the board material, images of the marking lines on the board material, images of elongated stains, and images of small dot-like patterns) have been removed from the corner image. The masked image does not reflect the images of the edges of the board material, images of the marking lines on the board material, images of elongated stains, and images of small dot-like patterns that were identified by adaptive binarization. In this way, the images of multiple overlapping parts included in the masked image are highly likely to be images of the fastener. However, round or nearly round stains and corners of the board material may not be removed from the masked image.

[0012] Therefore, the fixing device identification unit identifies the image of the fixing device based on the multiple overlapping portions remaining in the masked image. This allows for accurate identification of the fixing device image using the captured image.

[0013] Here, the fixing device identification unit may identify (determine) an image of a fixing device if, for example, the number of images in the overlapping portion and the number of vertical and horizontal arrangements of the overlapping portion (or the aspect ratio of the overlapping portion) are within a predetermined range. However, in a more preferred embodiment, the fixing device identification unit is a machine learning unit that has learned the characteristics of the fixing device based on the shape of the fixing device, and the machine learning unit identifies the image of the fixing device based on the multiple overlapping portions. Specifically, the machine learning unit may directly identify the image of the fixing device from the overlapping portion of the masked image, or it may, for example, cut out (extract) the original image (captured image) corresponding to the positions of the multiple overlapping portions, use this extracted image as the input image, and determine whether the extracted image contains an image of the fixing device.

[0014] According to this embodiment, the fixing device identification unit can more accurately and quickly identify images of fixing devices from multiple overlapping parts by utilizing a machine learning unit that has learned to identify fixing devices based on the shape of the fixing device.

[0015] In a more preferred embodiment, the machine learning unit extracts an image within a certain range that includes the overlapping portion for each overlapping portion, and identifies the image of the fixing device from each extracted image. Note that "extracting an image within a certain range that includes the overlapping portion" may be done from the masked image, but here it is preferable to extract an image within a certain range that includes the overlapping portion from the original captured image.

[0016] According to this embodiment, the machine learning unit does not identify the image of the fixture for the entire image, but rather identifies the image of the fixture for each extracted image, thereby reducing the computational load and enabling rapid identification of the fixture image. By using the extracted images from the original captured image, the fixture can be identified more accurately and quickly.

[0017] More preferably, the masking processing image generation unit performs an adaptive binarization process on the masking processing image, and the fixture identification unit identifies the image of the fixture using the masking processing image subjected to the adaptive binarization process.

[0018] According to this aspect, the masking processing image generated by the masking processing image generation unit is an image obtained by masking the corner detection image. Therefore, shades are imparted to the pixels of the image of the corner portion included in the masking processing image according to the corner-likeness. Since the pixels constituting the image of the fixture have a greater difference in shade than the pixels of other dot-like overlapping images, when an adaptive binarization process is performed on the masking processing image, other dot-like overlapping images can be removed. Therefore, the fixture identification unit can more accurately identify the image of the fixture using the masking processing image subjected to the adaptive binarization process.

Advantages of the Invention

[0019] According to the present invention, an image of a fixture is accurately identified from a captured image including a plurality of fixtures driven into a plate material.

Brief Description of the Drawings

[0020] [Figure 1] It is a schematic diagram of a state where screws (fixtures) to be identified in the present embodiment are driven into a gypsum board. [Figure 2] It is a schematic diagram of a screw pitch identification device according to the present embodiment. [Figure 3] It is a control block diagram of the identification device shown in FIG. 1. [Figure 4] It is a conceptual diagram showing the whole corner detection image. [Figure 5] It is a conceptual diagram showing the whole binarized image. [Figure 6] It is a conceptual diagram for explaining the masking process. (a) is a partial image of the corner detection image, (b) is a partial image of the binarized image, and (c) is a partial image of the masking processing image. [Figure 7]This is a conceptual diagram showing the entire masking-processed image. [Figure 8] (a) is a diagram for explaining an image of an overlapping part before adaptive binarization processing by the fixture identification part shown in FIG. 3, and (b) is a diagram for explaining an image of the overlapping part after adaptive binarization processing. [Figure 9] (a) is a diagram for explaining an image before fixture identification by the fixture identification part (machine learning part) shown in FIG. 3, and (b) is a diagram for explaining an image after fixture identification. [Figure 10] This is the entire image after identifying the screws. [Figure 11] This is a flowchart of operations using the identification device shown in FIG. 3.

Mode for Carrying Out the Invention

[0021] The identification device 10 according to the present embodiment will be described below with reference to FIGS. 1 to 11.

[0022] 1. Regarding the plate material and the fixture In the present embodiment, the fixture identification device 1(hereinafter referred to as the “identification device 10”)is a device that identifies a plurality of fixtures driven into a rectangular plate material. The identification device 10 measures the distance between adjacent fixtures among the plurality of fixtures. Here, the fixtures are driven in at a predetermined interval along at least the periphery of the plate material. Examples of the combination of the plate material and the fixture include eaves boards (exterior finishing materials under the eaves) and nails for fixing them, structural plywood in wooden houses and nails for fixing them, floor underlay materials and nails / screws for fixing them, and the like.

[0023] In the following embodiment, as shown in FIG. 1, the gypsum board 5 is exemplified as the plate material, and the screw 6A is exemplified as the fixture. Therefore, the gypsum board 5 corresponds to the “plate material” in the present invention, and the screw 6A corresponds to the “fixture” in the present invention. Therefore, the pitch of the screw 6A (screw pitch) corresponds to the “interval between fixtures”.

[0024] The fasteners used to secure the gypsum board 5 may be nails or staples. The gypsum board 5 is used as an interior wall material or ceiling material, etc. The gypsum board 5 is fixed with screws 6A along the long and short sides of the gypsum board 5 at a pitch below that specified by law, etc. The screw pitch here refers to the distance between adjacent screws 6A, 6A along the long and short sides of the gypsum board 5.

[0025] Here, multiple screws 6A, 6A, ... are driven into the gypsum board 5 at a specified pitch along the periphery, inside the periphery of the gypsum board 5. Furthermore, in this embodiment, multiple screws 6A, 6A, ... are driven into the gypsum board 5 at a specified pitch along the long side direction of the gypsum board 5, in the center in the width direction of the gypsum board 5. After the installation of the gypsum board 5, the screw pitch is checked to ensure that it is within the specified pitch. The screw pitch is the distance between the centers of adjacent screws 6A along the long side direction and the short side direction when the gypsum board 5 is viewed from the front.

[0026] The screw identification device 10 according to this embodiment is a device that identifies screws 6A driven into the gypsum board 5 after the gypsum board 5 has been installed. The identification device 10 ultimately extracts an image 6 of the screws 6A (hereinafter referred to as "screw image 6") from an image G1 including the gypsum board 5 captured by the imaging device 20, and estimates the pitch of adjacent screws 6A.

[0027] 2. Hardware configuration of identification device 10 As shown in Figure 2, the identification device 10 comprises, as hardware, a storage device 10A which is composed of ROM, RAM, etc., and which stores the conditions of the gypsum board 5, a screw pitch estimation program, etc., and a processing unit 10B which executes the screw pitch estimation program.

[0028] An input device 31 and an output device 32 are connected to the identification device 10. In this embodiment, the input device 31 and the output device 32 may be integrated into a single touch panel display. The input device 31 receives data such as the specifications of the gypsum board 5 and a program for estimating the screw pitch. In this embodiment, the input device 31 receives image data captured by the imaging device 20. The data input by the input device 31 is stored in the storage device 10A. The output device 32 displays the image data captured by the imaging device 20, the calculation results calculated by the arithmetic device 10B, etc.

[0029] In this embodiment, the identification device 10 consisted of a storage device 10A and an arithmetic unit 10B, but it may also include, for example, an input device 31 and an output device 32. In addition to the input device 31 and the output device 32, the identification device 10 may further include an imaging device 20, and these may be integrated into a mobile terminal such as a smartphone or tablet.

[0030] 3. Software configuration of identification device 10 In this embodiment, as shown in Figure 3, the identification device 10 comprises at least a brightness uniformization processing unit 11, a corner detection image generation unit 12, a binarization processing image generation unit 13, a masking processing image generation unit 14, and a fixing device identification unit 15.

[0031] 3-1. About the brightness uniformization processing unit 11 The brightness equalization processing unit 11 equalizes the brightness difference for each pixel that makes up the captured image captured by the imaging device 20. For example, the brightness equalization processing unit 11 equalizes the brightness of each pixel so that the brightness of each pixel falls within a predetermined range. Specifically, the brightness equalization processing unit 11 may multiply the brightness of each pixel by a correction coefficient corresponding to the difference between the maximum and minimum brightness values ​​of the pixels in the imaging device 20.

[0032] By using the captured image whose brightness has been made uniform by this brightness uniformization processing unit 11, the pixels of the screw image, which vary depending on the imaging conditions, can be made uniform. Furthermore, in the corner detection image GA described later, the corner image ga becomes easier to detect as the screw image 6, and in the binarized image GB, the screw 6A becomes easier to identify as the image of the identification part. This improves the accuracy of identifying the screw image 6. Note that if the screw 6A can be identified by the following series of steps, the brightness uniformization processing unit 11 may be omitted, and furthermore, noise reduction may be performed on the captured image using a median filter or the like before the brightness uniformization processing.

[0033] 3-2. About the corner detection image generation unit 12 The corner detection image generation unit 12 generates a corner detection image GA as shown in Figure 4 by performing corner detection on the captured image. The corner detection image generation unit 12 generates the corner detection image GA from the captured image that has been processed to equalize brightness. In the corner detection image GA, the generated corner portion image includes not only the screw image 6, but also images 7a other than the screw image 6, such as the corner of the gypsum board 5, dot-like patterns other than the gypsum board 5, dirt, etc., and these can be detected as corner portion images ga(6, 7a).

[0034] The corner detection method used by the corner detection image generation unit 12 is a general detection method using a Harris corner detector or a Moravec corner detector. Pixels in images where corners were not detected (for example, flat images or edge images (linear images)) are assigned a certain color (for example, black), while pixels containing images where corners were detected are assigned a color to indicate how likely the image is to be a corner. More specifically, in this detection method, each pixel is assigned a value indicating its corner-likeness. Pixels constituting flat surfaces or edges are assigned values ​​close to zero or negative values, while pixels constituting the image 6 of the bis are assigned large positive values. When assigning values ​​as shades, one possible method is to set values ​​less than or equal to zero (black), and for positive values ​​greater than that, to a grayscale image where the maximum value is 255 (white).

[0035] 3-3. About the binarized image generation unit 13 The binarized image generation unit 13 generates a binarized image GB by performing adaptive binarization processing on the captured image. In this embodiment, as shown in Figure 5, the binarized image generation unit 13 generates a binarized image GB from a captured image whose brightness has been uniformized. Adaptive binarization processing does not fix thresholds such as image density, but rather changes the threshold for each pixel according to the surrounding image. Therefore, it is possible to identify the image gb of the identified portion according to the difference in grayscale within the captured image.

[0036] Adaptive binarization may be performed depending on the size of the image region containing the image 6 of the screw. In Figure 5, the white areas are images of the parts that were not identified, and the black areas are images of the parts identified by adaptive binarization. Therefore, adaptive binarization is a method that determines the binarization threshold for each pixel, and can optimally binarize images even if the brightness changes in parts. Consequently, the image gb of this identified part includes not only the image 6 of the screw, but also the image of the outline 7b of the gypsum board 5 and the image 7c of the ruled lines of the gypsum board 5.

[0037] 3-4. About the masking image generation unit 14 The masking image generation unit 14 compares the corner detection image GA (see Figure 6(a)) with the binarized image GB (see Figure 6(b)). Specifically, it generates a masked image GC from the corner detection image GA by performing a masking process that leaves the overlapping image gc (specifically, the overlapping pixels) that overlaps with the identified image gb (6, 7b, 7c) identified by adaptive binarization processing among the multiple corner images ga(6, 7a) detected by the corner detection image GA, and deletes the images gd of the other corner parts.

[0038] When comparing the corner detection image GA with the binarized image GB in this way, both contain the image 6 of the bis. When comparing the corner image ga(6, 7a) detected by the corner detection image GA with the identified image gb(6, 7b, 7c) identified by the adaptive binarization process, the image that is located at a common position between the corner image ga and the identified image gb in the entire image region is highly likely to be the image 6 of the bis. Specifically, the masking image generation unit 14 uses the corner detection image GA as a mask image for the original (base) image and the adaptive binarized image, leaving the pixels in the same position in the original image as they are for pixels that are white (255) in the mask image, and changing the pixels in the same position in the original image to black (0) for pixels that are black (0) in the mask image.

[0039] Thus, as shown in Figure 6(c), the masked image GC is an image obtained by removing images other than the screw image (specifically, the image of the edge of the gypsum board 5, the image of some of the marking lines on the gypsum board 5, the image of the elongated stain, and the image of the small dot-like pattern) from the corner image ga(6, 7a). In this masked image GC, the image of the edge of the gypsum board 5, the image of some of the marking lines on the gypsum board 5, the image of the elongated stain, and the image of the small ceiling pattern are generally not reflected in the identified portion image gb identified by adaptive binarization. However, as shown in Figure 7, in the masked image GC, the screw image 6 makes up a large proportion of the multiple overlapping portion images gc, but other images 7a, such as round or nearly round stains, and the corner of the gypsum board 5, may not be able to be removed.

[0040] Therefore, in this embodiment, the masking image generation unit 14 may further perform the adaptive binarization process described above on the masking image GC. The masking image GC generated by the masking image generation unit 14 is an image obtained by masking the corner detection image GA. Therefore, the pixels of the corner portion images ga(6, 7a) included in the masking image GC are assigned varying shades according to their corner-likeness. As shown in Figure 8(a), the pixels constituting the screw image 6 have a larger difference in shade compared to the pixels of other point-like overlapping images. Therefore, by performing adaptive binarization on the masking image GC, other point-like overlapping images 7a can be removed, as shown in Figure 8(b). As a result, the fixing device identification unit 15, which will be described later, can more accurately identify the screw image 6 using the adaptively binarized masking image GC.

[0041] 3-5. Regarding the fixing device specific part (screw specific part) 15 The fastener identification unit 15 identifies the image 6 of the screw from the images gc of multiple overlapping portions remaining in the masked image CG. The fastener identification unit 15 is a machine learning unit 15A that has learned the features of the screw 6A based on the shape of the screw 6A, and the machine learning unit 15A identifies the image 6 of the screw (see Figure 9(b)) based on the images gc of multiple overlapping portions (see Figure 9(a)). The machine learning unit 15A extracts an image of a certain range including the overlapping portion for each overlapping portion, and identifies the image 6 of the screw from the extracted image ge for each extracted image ge. Specifically, as shown in Figures 9(a) and (b), the machine learning unit 15A may directly identify the image 6 of the screw from the overlapping portion of the masked image CG, however, as will be described later, it is preferable to cut out (extract) the original image (captured image) corresponding to the position of the multiple overlapping portions, use this extracted image as the input image, and determine whether the extracted image contains the image 6 of the screw.

[0042] The machine learning unit 15A may use a support vector machine (SVM) or the like to learn the features of the screw image 6 from the captured image, using the screw image 6 captured by the imaging device 20 and the shape features of the screw image 6 in this image (for example, multiple points along the circumference which is the outer edge of the screw image 6) as training data. This makes it possible to extract the screw image 6 from any overall image that includes the screw image 6. Alternatively, a cascade classifier using features such as Haar-like features may be used to identify the gypsum board itself, and then the screw image 6 may be extracted from the image within the identified area using the machine learning method described above.

[0043] By utilizing the machine learning unit 15A, which has learned to identify screws based on their shape, the screw image 6 can be identified more accurately and quickly from multiple overlapping areas. In particular, the machine learning unit 15A does not identify the screw image 6 in the original captured image, but rather identifies the screw image 6 for each extracted image extracted from the original image, thus reducing the computational load and enabling rapid identification of the screw image 6. In this way, as shown in Figure 10, an overall image GD with the screw image 6 identified can be obtained, and the screw pitch can be calculated from this overall image GD.

[0044] The following flowchart, using the identification device 10, will be explained with reference to Figure 11. First, in step S1, the imaging device 20 captures an image of the area including the gypsum board 5 as the area to be inspected, and acquires an overall image.

[0045] Next, in step S2, noise reduction is performed on the captured image using a median filter or the like. Then, in step S3, the brightness equalization processing unit 11 equalizes the brightness difference for each pixel that makes up the captured image.

[0046] Next, in step S4, corner detection is performed on the captured image to generate a corner detection image GA from the captured image whose brightness has been uniformized. In parallel with step S4, in step S5, a binarized image GB for masking is generated from the captured image whose brightness has been uniformized. Note that steps S4 and S5 may be performed in either order.

[0047] Next, in step S6, the masking image generation unit 14 generates a masking image GC from the corner detection image GA by keeping the image gc of the overlapping portion that overlaps with the image gb (6, 7b, 7c) of the identified portion identified by adaptive binarization processing, and masking the image gd of the other corner portions.

[0048] Furthermore, in step S7, the adaptive binarization process described above is further applied to the masked image GC to enhance the image of the screw candidate portion and remove the image of the other portion. In step S8, the fastener identification unit 15 generates an extracted image that includes the overlapping portion within a certain range for each overlapping portion, and in step S9, the fastener identification unit 15 (machine learning unit 15A) identifies the screw image 6 from the multiple overlapping portion images gc.

[0049] In step S10, the positions of the screws are identified and the screw pitch is calculated for the masked image GC in which the screw image 6 has been identified. At this time, the image of the gypsum board 5 and the image of the screws 6 are extracted from the overall image including the gypsum board 5 captured by the imaging device 20, and the estimated contour line of the gypsum board 5 is extracted from the extracted image of the gypsum board 5 using edge processing or the like. Here, the position of the screws 6A relative to the overall image G is identified. At this time, the images of the gypsum board 5 and screws 6A may be projected onto an image viewed from the normal direction of the gypsum board 5. For the obtained image, groups of screws 6A in a vertical column are identified, and the screw pitch of adjacent screws 6A is calculated. Finally, in step S11, it is determined whether the calculated screw pitch falls within a specified range.

[0050] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and various design modifications can be made without departing from the spirit of the invention as described in the claims. [Explanation of Symbols]

[0051] 5: Board material (gypsum board), 6A: Fixing device (screw), 6: Image of fixing device (image of screw), 10: Fixing device identification device, 11: Brightness uniformization processing unit, 12: Corner detection image generation unit, 13: Binarization processing image generation unit, 14: Masking processing image generation unit, 15: Fixing device identification unit, GA: Corner detection image, ga: Image of corner area, GB: Binarization processing image, gb: Image of identified area, GC: Masking processing image, gc: Image of overlapping area, ge: Extracted image

Claims

1. A fastener identification device that identifies images of multiple fasteners embedded in a plate from an image captured including the fasteners, A corner detection image generation unit generates a corner detection image by performing corner detection on the aforementioned captured image, A binarized image generation unit generates a binarized image by performing adaptive binarization processing on the captured image, A masking image generation unit compares the corner detection image with the binarized image and generates a masked image from the corner detection image by performing a masking process that keeps the overlapping images of the identified portion identified by adaptive binarization among the multiple corner portion images detected in the corner detection image, and deletes the images of the other corner portions. A fixing device identification device comprising: a fixing device identification unit that identifies an image of the fixing device based on a plurality of images of the overlapping portions remaining in the masked image.

2. The aforementioned fixing device identification unit is a machine learning unit that has learned the characteristics of the fixing device based on the shape of the fixing device. The fixing device identification device according to claim 1, characterized in that the machine learning unit identifies the image of the fixing device based on the images of the plurality of overlapping portions.

3. The fixing device identification device according to claim 2, characterized in that the machine learning unit extracts an image within a certain range including the overlapping portion for each overlapping portion, and identifies an image of the fixing device from each extracted image.

4. The masking image generation unit performs adaptive binarization on the masking image. The fixing device identification device according to claim 1, characterized in that the fixing device identification unit identifies the image of the fixing device using the masked image that has undergone adaptive binarization processing.

Images