Methods for training / operating object detection algorithm
A camera-based system with machine learning algorithms efficiently detects and categorizes protruding nails in pallets, improving safety and repair efficiency by accurately identifying nail defects.
Patent Information
- Application Number
- JP2025094151
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-10-11
- Filing Date
- 2025-06-05
- Publication Date
- 2025-08-07
AI Technical Summary
Detecting protruding nails in wooden pallets is challenging, as they pose a risk of injury during manual handling and are difficult to identify using existing methods.
A system utilizing cameras and machine learning algorithms to classify and detect protruding nails in pallets, involving image classification to identify support blocks and object detection to pinpoint nail defects, with a two-step process to filter and analyze images for accuracy.
Effectively identifies and categorizes protruding nails and other defects in pallets, enhancing safety by reducing manual inspection risks and improving the efficiency of pallet repair and reuse.
Smart Images
Figure 2025116254000001_ABST
Abstract
Description
[Technical Field]
[0001]
[0002] The present invention relates to a method for training / operating an object detection algorithm. [Background technology]
[0002] Related Applications
[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 262,452, filed October 13, 2021, the entire contents of which are incorporated herein by reference.
[0003]
[0003] Wooden pallets are used to transport a variety of bulk goods and equipment required in manufacturing and warehousing operations. In high-volume manufacturing industries, pallet pools offer industry-wide cost savings over one-way pallets.
[0004] After bulk goods and equipment are offloaded from pooled pallets, the pallets are returned to a pallet inspection and repair facility. If the repairs are too extensive, the damaged pallet is discarded. Otherwise, if the damage is minor, the pallet is repaired and painted before returning to service. Summary of the Invention
[0005] The pallet inspection system includes a conveyor for moving pallets to be inspected. The pallets include top and bottom decks separated by spaced support blocks disposed therebetween, and nails are used to secure the top and bottom decks to the support blocks. A plurality of cameras are positioned to generate images of the pallets as they are moved on the conveyor. A processor is coupled to the plurality of cameras and receives the images for processing. The processing includes running a first algorithm on the images to tag images having visible support blocks therein and running a second algorithm on the tagged images to detect nails with exposed tips.
[0006] The pallet inspection system may further include a first sensor positioned in front of the plurality of cameras and a second sensor positioned behind the plurality of cameras, and the processor may be configured to receive a first set of images of the pallet in response to the first sensor being activated, and to receive a second set of images of the pallet in response to the second sensor being activated.
[0007] The first and second sensors may comprise photoelectric sensors, and the first and second algorithms may be executed by a processor to detect the exposed-tip nail in real time.
[0008]
[0008] The plurality of support blocks are spaced apart to form a pair of outer rows and a central row therebetween, the outer rows including corner support blocks, and each camera may be focused on the support blocks of one of the rows and not on the support blocks of the other row.
[0009]
[0009] The plurality of cameras may be divided into first and second camera sets, with the first camera set directed toward an entrance to the pallet inspection system and the second camera set directed toward an exit to the pallet inspection system.
[0010]
[0010] The first camera set may provide a front oblique view of the multiple support blocks, and the second camera set may provide a rear oblique view of the multiple support blocks, with the first and second camera sets collectively providing images of all sides of each individual support block.
[0011] The first set of cameras may include a camera adjacent to each side of the conveyor, and the second set of cameras may include a camera adjacent to each side of the conveyor, each camera being configured as a color camera.
[0012] The first algorithm may include an image classification algorithm, and the second algorithm may include an object detection algorithm.
[0013] The image classification algorithm may be configured to classify each image as one of a block image corresponding to an in-focus support block, a non-block image corresponding to an out-of-focus support block, or a background image corresponding to neither an out-of-focus support block nor an out-of-focus support block. The block images may be tagged by the image classification algorithm.
[0014]
[0014] The object detection algorithm may be configured to place a bounding box around the support blocks in each block image and, in response to an exposed nail tip being detected, to place a bounding box around the exposed nail tip.
[0015]
[0015] The object detection algorithm may be configured to detect other nail defects in addition to exposed-tip nails, which are ignored by the processor.
[0016] Another aspect is directed to a method for manufacturing pallets as described above. The method includes operating a conveyor to move a pallet to be inspected, the pallet including a top deck and a bottom deck separated by a plurality of spaced support blocks disposed therebetween, and nails are used to secure the top deck and the bottom deck to the plurality of support blocks. The method further includes operating a plurality of cameras positioned to generate images of the pallet as it is moved on the conveyor and receiving the images for processing. The processing may include running a first algorithm on the images to tag images having visible support blocks therein and running a second algorithm on the tagged images to detect nails with exposed tips.
[0017] Another aspect is directed to a method for training an object detection algorithm as described above. The method includes creating a database of images of pallets with protruding nails and other nail defects, defining different categories to be detected in the database, and annotating images in the database corresponding to the different detected categories. The model is trained using machine learning to learn a function that generates a mapping between the annotated images and the different detected categories. The method further includes analyzing output data from the model and optimizing the model based on the analyzed output data.
[0018] Yet another aspect is directed to a method for operating an object detection algorithm as described above. The method includes receiving images of a pallet to be inspected and executing a machine learning model trained to learn a function that generates a mapping between annotated images of pallets with protruding nails and other nail defects detected. The annotated images correspond to different categories detected. The method further includes identifying objects by their locations in the received images that correspond to the different categories detected, providing confidence values for the categories detected in the received images, and identifying pallets with protruding nails based on the confidence values. [Brief explanation of the drawings]
[0019] [Figure 1] FIG. 1 is a top perspective view of a wooden pallet in which various aspects of the present disclosure may be implemented. [Figure 2]
[0020] FIG. 2 is a bottom perspective view of the wooden pallet shown in FIG. [Figure 3]
[0021] FIG. 3 is a partial cross-sectional view of the wooden pallet shown in FIG. 1 with protruding nails. [Figure 4]
[0022] FIG. 4 is a block diagram of a pallet inspection system in which various aspects of the present disclosure may be implemented. [Figure 5]
[0023] FIG. 5 is a more detailed block diagram of the protruding nail detection station shown in FIG. 4 for detecting protruding nails in a wooden pallet. [Figure 6]
[0024] FIG. 6 shows images produced by the protruding nail detection station shown in FIG. 5 that have been classified by an image classification algorithm as a block image, a non-block image, or a background image. [Figure 7] FIG. 7 shows images produced by the protruding nail detection station shown in FIG. 5 that have been classified by an image classification algorithm as a block image, a non-block image, or a background image. [Figure 8] FIG. 8 shows images produced by the protruding nail detection station shown in FIG. 5 that have been classified by an image classification algorithm as a block image, a non-block image, or a background image. [Figure 9]
[0025] FIG. 9 is a block diagram of an exemplary object detection algorithm used by the protruding nail detection station shown in FIG. 5 to detect protruding nails in a wooden pallet. [Figure 10]
[0026] FIG. 10 is a flow diagram for training the object detection algorithm shown in FIG. [Figure 11]
[0027] FIG. 11 is an annotated image used to train the object detection algorithm shown in FIG. [Figure 12] FIG. 12 is an annotated image used to train the object detection algorithm shown in FIG. [Figure 13] FIG. 13 is an annotated image used to train the object detection algorithm shown in FIG. [Figure 14]
[0028] FIG. 14 is a flow diagram for operating the object detection algorithm shown in FIG. [Figure 15]
[0029] FIG. 15 is the image output from the object detection algorithm shown in FIG. [Figure 16] FIG. 16 is an image output from the object detection algorithm shown in FIG. [Figure 17] FIG. 17 is an image output from the object detection algorithm shown in FIG. [Figure 18]
[0030] FIG. 5 is a flow diagram for operating the pallet inspection system shown in FIG. 4. DETAILED DESCRIPTION OF THE INVENTION
[0020]
[0031] This description is made with reference to the accompanying drawings, in which exemplary embodiments are shown. However, many different embodiments may be used, and thus this description should not be construed as limited to the specific embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. Like numbers refer to like elements throughout.
[0021]
[0032] Nail defects on pallets are of particular concern when the nail defect is a protruding nail. A protruding nail is when the tip of the nail is exposed. While the form factor of the pallet is not affected, protruding nails need to be detected during pallet inspection so the defect can be corrected.
[0022]
[0033] Typically, when workers manually handle pallets, their hands are positioned within one or more pockets on the pallet. Each side of the pallet has a pocket formed between a top deck and a bottom deck separated by a support block. Reaching a hand into a pocket with protruding nails can result in injury.
[0023]
[0034] 1-3, an exemplary wooden pallet 10 is described. The wooden pallet 10 is for discussion of typical nail placement within the wooden pallet 10. The illustrated wooden pallet 10 is not limiting, as other wooden pallet configurations are readily available.
[0024]
[0035] The wooden pallet 10 includes a bottom deck 20, a top deck 30, and a plurality of wooden support blocks 40, 46 coupled between the bottom and top decks. The support blocks 40, 46 form a gap 50 (i.e., pocket) between the bottom and top decks 20, 30 for receiving a lifting member, such as the tip of a forklift.
[0025]
[0036] The top deck 30 includes a pair of spaced-apart wooden end deck boards 32 and a wooden intermediate deck board 34 disposed between the end deck boards 32. The top deck 30 also includes a pair of spaced-apart wooden connector boards 36 and a wooden intermediate connector board 37. The connector boards 36 and the intermediate connector board 37 are perpendicular to the end deck boards 32 and the intermediate deck boards 34. The end deck boards 32 and the intermediate deck boards 34 are placed on the connector boards 36 and directly connected to the support blocks 40, 46 via nails 70.
[0026]
[0037] The bottom deck 20 includes bottom deck boards 22, 26 oriented in the same direction as the end deck boards 32 and middle deck boards 34 of the top deck 30. The bottom deck boards 22, 26 are also directly connected to the support blocks 40, 46 via nails 70.
[0027]
[0038] The support blocks include corner support blocks 40 and center support blocks 46 between the corner support blocks 40. There are a total of nine support blocks 40, 46 arranged in three rows. Each outer row includes a pair of outer support blocks 40 and a single center support block 46, and the center row includes all of the center support blocks 46. The actual number of support blocks may vary based on the configuration and size of the wooden pallet 10.
[0028]
[0039] The corner support blocks 40 and the center support block 46 each have a rectangular shape. In other configurations, one or more of the support blocks 40, 46 may have a non-rectangular shape, such as a circular shape.
[0029]
[0040] A partial cross-sectional view of the wooden pallet 10 is provided in Figure 3 to show the placement of nails 70 within the support blocks 40, 46. When the wooden pallet 10 is formed, the nails enter the support blocks 40, 46 from the bottom and top decks 20, 30.
[0030]
[0041] A protruding nail 80 can occur when a nail operator places a nail 70 off-center, exposing the tip 72. A protruding nail 80 can also occur when a nail 70 is driven into the location of an existing nail and bounces off the existing nail, exposing the tip 72. Wear and tear on the wooden pallet 10 can also cause portions of the support blocks 40, 46 to break, exposing the nail tip 72. Protruding nails 80 are not limited to protruding from the support blocks 40, 46. Protruding nails 80 can also protrude from other regions or areas of the pallet 10, such as protruding from one of the connector boards 36.
[0031]
[0042] Detecting protruding nails 80 in wooden pallets 10 is challenging. As described in detail below, a camera is used to collect images of each wooden pallet 10 during inspection. The images are sent to a processing unit for processing. The processing unit runs a machine learning object detection algorithm that is trained to detect protruding nails 80. How the algorithm is trained and executed is also described in detail below. The processing unit may be, for example, a graphics processing unit (GPU), a central processing unit (CPU), or an edge computing device.
[0032]
[0043] An overview of the pallet inspection system 140 is described with reference to Figure 4. Pooled pallets 10 from different customers are returned to the pallet pooling company for inspection and repair if necessary before the pallets 10 are returned to service.
[0033]
[0044] Prior to inspection, the pallets 10 are provided to a stack infeed 152, which aligns individual stacks of pallets before being delivered to a tipper / accumulator 154. The tipper / accumulator 154, for example, provides a steady stream of spaced apart pallets 10 to a conveyor. As the pallets 10 move on the conveyor, they pass through a preparatory screening line 156.
[0034]
[0045] At the preparatory screening line 156, each pallet 10 is visually inspected by a human operator to remove any loose debris or trash that may affect the inspection. If necessary, the human operator will also make minor repairs. In some cases, as indicated by block 158, the pallet 10 may be discarded during the preparatory screening line 156 if it is too badly damaged.
[0035]
[0046] From the preliminary screening line 156, individual pallets 10 are moved for inspection to a protruding nail detection station 160. The pallets 10 are inspected for pallet sorting to detect protruding nails 80.
[0036]
[0047] Pallet classification by the protruding nail detection station 160 is to determine whether the pallet 10 being inspected belongs to the pallet pooling company that operates the pallet inspection system 140. A first set of cameras 162 in the protruding nail detection station 160 generates images of the top and bottom of each pallet 10. The images may be video images or still images. The images may be color or monochrome and are provided to a processing unit 170. The processing unit 170 executes a first algorithm for pallet classification 172. The first algorithm may be, for example, a machine learning (ML) image classification algorithm.
[0037]
[0048] An image classification algorithm compares the generated image data to an expected profile. The expected profile is used to classify the pallet. The image classification algorithm labels or tags each image as to whether the pallet 10 belongs to a pallet pooling company. For example, if 75 images of a pallet 10 are generated and the majority of these images are tagged as belonging to a pallet pooling company, the pallet 10 is classified as such.
[0038]
[0049] The image classification algorithms associated with the first set of cameras 162 can also detect certain obvious nail defects, such as raised nails and free standing nails. In a raised nail, the head of the nail extends slightly above the top deck. A raised nail can potentially affect product placed on the pallet. In a free standing nail, the head and body of the nail are visible, but the tip is not.
[0039]
[0050] The detection of protruding nails by the protruding nail detection station 160 determines the protruding nails 80 in the pallet 10. A second set of cameras 164 in the protruding nail detection station 160 generates images of the sides of each pallet 10. The images are in color and are provided to the processing unit 170. Alternatively, the images may be monochrome. The processing unit 170 executes a second algorithm trained to detect protruding nails 80. The second algorithm may be, for example, a machine learning (ML) object detection algorithm.
[0040]
[0051] Pallet sorting line 180 accepts pallets 10 as they exit protruding nail detection station 160. Pallet sorting line 180 interrogates processing unit 170 to determine the pallet classification. If the inspected pallet 10 does not belong to the pallet pooling company operating pallet inspection system 140, the pallet 10 is sent to discard line 182.
[0041]
[0052] In the illustrated embodiment, the protruding nail detection station 160 includes first and second sets of cameras 162, 164. In an alternative embodiment, the cameras may be combined such that the same set of cameras used for pallet sorting is also used for protruding nail detection; that is, the cameras are not mutually exclusive.
[0042]
[0053] Alternatively, if the inspected pallet 10 belongs to a pallet pooling company that operates a pallet inspection system 140, the pallet sort line 180 queries the processing unit 170 to determine whether the pallet 10 is good or bad.
[0043]
[0054] If the inspected pallet 10 is bad, this means that the pallet needs repair and is sent to repair line 184. After repair, the pallet 10 is sent to paint line 186 for painting before being returned to service. If the inspected pallet 10 is good, this means that the pallet does not need repair and instead is sent to paint line 186 before being returned to service.
[0044]
[0055] Referring to FIG. 5, the operation of the protruding nail detection station 160 to detect protruding nails 70 will be described. The conveyor 105 moves the pallets 10 through the protruding nail detection station 160 in the direction of the arrow shown. The conveyor 105 includes a first sensor 110 at the entrance to the protruding nail detection station 160 and a second sensor 112 at the exit of the protruding nail detection station 160. In other embodiments, a single sensor may be used. The first and second sensors 110, 112 may be configured, for example, as photoelectric sensors. Each photoelectric sensor includes a transmitter and a receiver on opposite sides of the conveyor 105. The transmitter transmits a light signal, which may be visible or infrared, to the receiver. A pallet 10 is detected when the light beam is blocked from reaching the receiver from the transmitter.
[0045]
[0056] Outputs from the first and second sensors 110, 112 are provided to a controller 130. In response to receiving outputs from the first and second sensors 110, 112, the controller 130 is configured to trigger a second set of cameras 164 as the pallet is being moved on the conveyor 105. The second set of cameras 164 is divided into a first set of cameras 164(1), 164(2) and a second set of cameras 164(1), 164(2).
[0046]
[0057] The cameras in the first camera set 164(1), 164(2) are triggered by the first sensor 110. In response to the first sensor 110 detecting the arrival of the pallet 10, a first stop 111 in the path of the pallet 10 is dropped below the conveyor 105, allowing the pallet 10 to enter the protruding nail detection station 160. Once the pallet 10 enters the protruding nail detection station 160, the controller 130 activates or triggers the cameras in the first camera set 164(1), 164(2) to provide images to the processing unit 170 for processing.
[0047]
[0058] The images produced by the first set of cameras 164(1), 164(2) are a subset of the overall images received by GPU 170 for palette 10. The remaining images received by processing unit 170 are produced by the second set of cameras 164(3), 164(4).
[0048]
[0059] The cameras in the first camera set 164(1), 164(2) are positioned to view a front perspective view of the support blocks 40, 46. The cameras in the second camera set 164(3), 164(4) are positioned to view a rear perspective view of the support blocks 40, 46. Collectively, the combined images are of all sides of the individual support blocks 40, 46.
[0049]
[0060] The cameras in the second camera set 164(3), 164(4) are triggered by the second sensor 112. In response to the second sensor 112 detecting the exit of the pallet 10, a second stop 113 in the path of the pallet 10 is dropped under the conveyor 105, allowing the pallet 10 to exit the protruding nail detection station 160. Once the pallet 10 exits the protruding nail detection station 160, the controller 130 activates or triggers the cameras in the second camera set 164(3), 164(4) to provide images to the processing unit 170 for processing.
[0050]
[0061] Having two different triggers ensures that a complete set of images is acquired of the support blocks 40, 46. If the first and second camera sets 164(1)-164(4) are triggered simultaneously by a single photoelectric sensor and the conveyor 105 jams and the pallet 10 becomes stuck, no images on the rear perspective view of the support blocks 40, 46 will be provided to the processing unit 170.
[0051]
[0062] The first set of cameras 164(1), 164(2) includes six cameras, and the second set of cameras 164(3), 164(4) also includes six cameras, half of which are on either side of the conveyor 105. Although twelve cameras are used, a different number of cameras may be used in other embodiments.
[0052]
[0063] The support blocks 40, 46 on each pallet 10 are spaced apart to form a pair of outer rows with a center row between them. The illustrated pallet includes nine support blocks 40, 46, with three support blocks 40, 46 in each row. The rows are parallel to the conveyor 105. Each of the outer rows includes a pair of corner support blocks 40 and a center support block 46. The center row includes only a center support block 46.
[0053]
[0064] Each camera in the first and second camera sets 164(1)-164(4) is focused on a particular row of support blocks. For example, camera group 164(1) includes three cameras. The first camera is focused on the outer row of support blocks 40, 46 closest to camera group 164(1). The second camera is focused on the center row of support blocks 46. The third camera is focused on the outer row of support blocks 40, 46 farthest from camera group 164(1). Similarly, each camera in camera group 164(2) on the other side of conveyor 105 is focused on a particular row of support blocks 40, 46.
[0054]
[0065] As mentioned above, the cameras in the first camera set 164(1), 164(2) are positioned to view a front oblique view of the support blocks 40, 46. Typically, there is a delay between the six cameras in the camera group 164(1), 164(2) when they are focused on a particular row of support blocks 40, 46.
[0055]
[0066] The pair of cameras focused on the outer row of support blocks 40, 46 closest to each camera are focused first. This pair of cameras may be triggered immediately in response to the first sensor 110 detecting the arrival of a pallet 10. The pair of cameras focused on the center row of support blocks 46 then focuses because the center support block 46 is further away from the cameras. Consequently, this pair of cameras may be delayed, for example, 25 milliseconds, before being triggered by the controller 130 to generate an image. The pair of cameras focused on the outer row of support blocks 40, 46 furthest from the cameras are focused after the center row of support blocks 46 is focused because they are the furthest away from the cameras. Consequently, this pair of cameras may be delayed, for example, 50 milliseconds, before being triggered by the controller 130 to generate an image. Once the cameras are triggered to provide an image to the processing unit 170, the triggering continues for a predetermined period of time. The predetermined period of time may be, for example, between 3 and 4 seconds and may vary based on the speed of the conveyor 105.
[0056]
[0067] The cameras in the second camera set 164(3), 164(4) are configured similarly to the cameras in the first camera set 164(1), 164(2). Each camera in the second camera set 164(3), 164(4) is similarly focused on a particular column of the support blocks. As described above, the cameras in the second camera set 164(3), 164(4) are positioned to view a rear perspective view of the support blocks 40, 46.
[0057]
[0068] In response to the second sensor 112 detecting that the pallet 10 has exited the protruding nail detection station 160, the controller 130 may immediately trigger the pair of cameras focused on the outer rows of the support blocks 40, 46 closest to the individual cameras, and may delay triggering the pair of cameras focused on the center row of the support blocks 46 and the pair of cameras focused on the outer rows of the support blocks 40, 46 farthest from the individual cameras.
[0058]
[0069] The operation of the processing unit 170 to detect protruding nails 80 from received images is a two-step process. The first step is to determine the images that have support blocks 40, 46 in focus with one of the cameras. The second step is to analyze only those images that have support blocks 40, 46 in focus therein for protruding nails 80.
[0059]
[0070] The rationale for determining images with in-focus support blocks is that as nails 70 are driven into support blocks 40, 46, protruding nails 80 will be adjacent to support blocks 40, 46. The number of images analyzed for protruding nails 80 is significantly reduced by the first step.
[0060]
[0071] Each camera may generate approximately 40 images per 10 pallets. Using 12 cameras, this equates to approximately 480 images per pallet. The actual number of images generated is variable and can vary from one deployment location to another. The processing unit 170 runs a first algorithm on the 480 images. The first algorithm can be, for example, an image classification algorithm 132. The image classification algorithm 132 is trained using artificial intelligence (AI) and machine learning (ML) to determine images that have in-focus support blocks 40, 46 therein.
[0061]
[0072] If an image has a support block 40, 46 in focus within it, it is tagged by the image classification algorithm 132. The image classification algorithm 132 acts as a pre-filter for the images provided to the processing unit 170 by the second set of cameras 164.
[0062]
[0073] The tagged images are provided to a second algorithm, which may be, for example, an object detection algorithm 134. Images that are not tagged are not passed to the object detection algorithm 134. As an example, of the 480 images, approximately one-half to one-third of the images may not be tagged.
[0063]
[0074] The implementation of the object detection algorithm 134 is computationally more extensive than the implementation of the image classification algorithm 132. Reducing the number of images run by the object detection algorithm 134 simplifies the overall process for determining protruding nails 80.
[0064]
[0075] The image classification algorithm 132 is trained to classify each image as a block image, a non-block image, or a background image. Block images correspond to in-focus support blocks 40, 46. Non-block images correspond to out-of-focus support blocks 40, 46. Background images correspond to neither in-focus nor out-of-focus support blocks 40, 46.
[0065]
[0076] 6-8, images generated by the protruding nail detection station 160 and classified by the image classification algorithm 132 are described. Image 200 in FIG. 6 is a blocked image because the support block 40 is in focus. Image 202 in FIG. 7 is a non-blocked image because the support block 40 is out of focus. Image 204 in FIG. 8 is a background image because no in-focus or out-of-focus support block is visible. Background images are typically generated when the pallet 10 first arrives at the protruding nail detection station 160 and when the pallet 10 leaves the protruding nail detection station 160.
[0066]
[0077] The image classification algorithm 132 classifies each image using a percentage. The classification percentage changes as the pallet 10 moves on the conveyor 105. The classification percentage is displayed for each image and totals 100%. A percentage number is assigned by the image classification algorithm 132 to each of the three possible classifications. In response to a classification percentage exceeding a threshold, such as 75%, the image classification algorithm 132 classifies the image accordingly.
[0067]
[0078] Image 200 has a 100% classification as a block image, but 0% classification as a non-block image or background image. Image 202 has a 99.9511% classification as a non-block image, 0.0333% classification as a block image, and 0.0156% classification as a background image. Image 204 has an 82.796% classification as a background image, 17.0867% classification as a non-block image, and 0.1173% classification as a block image. In this image 204, the pallet 10 is partially visible as it exits the protruding nail detection station 160.
[0068]
[0079] Tagged images with the support blocks 40, 46 in focus are generally referred to as tagged images 200. These images are passed to an object detection algorithm 134 for processing. Instead of tagging the images, the object detection algorithm 134 is trained to find objects within the tagged images 200. The located objects are the support blocks and the visible nails overlapping the support blocks. The visible nails 70 may be protruding nails 80, or they may be nails 70 where the body of the nail is visible but the tip is not.
[0069]
[0080] As described below, the object detection algorithm 134 is trained using annotated images that include several different categories to be detected, each corresponding to a particular type of object to be detected and located. By learning the different categories, the object detection algorithm 134 can distinguish between protruding nails that should be corrected and other types of nail defects that do not need to be corrected. The different categories provide contextual details to the object detection algorithm 134.
[0070]
[0081] The object detection algorithm 134 may operate based on artificial intelligence (AI) and machine learning (ML) to determine objects in the image 200 having the support blocks 40, 46 focused thereon. In one example, the object detection algorithm 134 may be a single-shot detector (SSD) 210, as shown in FIG. 9. Other types of object detectors may be used as alternatives to the illustrated SSD 210, which detects multiple objects in the image 200 by taking only a single shot of the image 200. The SSD 210 is an open-source algorithm that has been modified to detect protruding nails 80.
[0071]
[0082] The SSD 210 generally includes a base VGG-16 network 212 followed by multi-box convolutional layers 214 and 216. The base VGG-16 network 212 is used to extract features in the image 200. The convolutional layers 214 are for detection, and the convolutional layers 216 support object detection at multiple scales because the sizes of these layers progressively decrease. The convolutional models for detection are different for each feature layer.
[0072]
[0083] The multi-box convolutional layers 214, 216 are applied to multiple feature maps from later stages of the network, which helps perform detection at multiple scales: the prediction of bounding boxes and confidence of different objects in the image 200 is done not by one, but by multiple feature maps of different sizes representing multiple scales.
[0073]
[0084] The problem of detecting protruding nails can be addressed in different ways. One method is to use object detection using bounding boxes, as described below. Another method may use a segmentation approach that segments pixels to define the outline of protruding nail pixels in the image. Yet another method may train an algorithm to detect regions using both bounding boxes and segmentation pixels.
[0074]
[0085] The training of the object detection algorithm 134 will now be described with reference to flow diagram 250 of Figure 10. From the beginning (block 252), a database of images of pallets 10 with protruding nails 70 and other nail defects is created in block 254. The number of pallets 10 may be quite large, for example, 100 or more. Images from all 12 cameras for each pallet 10 are collected and stored in a database. Each pallet 10 will have approximately 480 images associated with it.
[0075]
[0086] The different categories to be detected are defined in block 256. These categories include support blocks, protruding nails, partially visible nails, clinched nails, freestanding nails, and splinters. Other types of categories can be defined to help the object detection algorithm 134 detect protruding nails 80.
[0076]
[0087] The images in the database are annotated in block 258 to reflect the different categories to be detected. Individual images are manually reviewed, and if the image has a category that is detected by the object detection algorithm 134, the image is annotated with a bounding box and labeled accordingly.
[0077]
[0088] Annotated example images are provided in Figures 11-13. A green bounding box 202a is placed around each support block 40, 46. A red bounding box 204a is used for the protruding nail 80. A yellow bounding box is used in response to the detected nail not being a protruding nail. In Figure 12, for example, a yellow bounding box 206 is placed around the nail 70, where the body of the nail is visible but the tip is not. In Figure 13, the support block 40 is split, revealing a pair of protruding nails 80.
[0078]
[0089] Detecting split support blocks is helpful. Sometimes it is hard to distinguish between split support blocks and colored nails. Usually the nails are displayed in gray / block color. However, if the split support block is not detected early and the pallet is sent for painting, the exposed nails will be painted the same color as the block.
[0079]
[0090] In addition to detecting nail defects and split support blocks, images can be annotated to identify other types of defects. For example, images can be annotated to detect stickers and stretch wrap remaining on the pallet 10. Typically, individual pallets 10 are visually inspected by a human operator to remove any stray debris or debris that may affect the inspection. Adding sticker and shrink wrap detection helps identify areas that may not be captured during a visual inspection of an individual pallet 10.
[0080]
[0091] The model is trained in block 260 using machine learning to learn a function that generates a mapping between the annotated images and the different categories to be detected. In some embodiments, the machine learning may be based on a neural network to train the model. In other embodiments, a neural network is not used to train the model. We are not limited to a single algorithm or set of algorithms, as neural network-based models / algorithms or other types of non-neural network-based models / algorithms may be used.
[0081]
[0092] Output data from the model is analyzed in block 262. The model is optimized based on the analyzed output data in block 264. The method ends in block 266.
[0082]
[0093] The operation of the object detection algorithm 134 will now be described with reference to the flow diagram 300 of Figure 14. From the start (block 302), an image of the pallet 10 to be inspected is received in block 304. The image is an image 200 that has been tagged by the image classification algorithm 132 as having support blocks 40, 46 in focus therein.
[0083]
[0094] At block 306, the method includes executing a machine learning model trained to learn a function that generates a mapping between annotated images of pallets 10 with protruding nails 80 and other nail defects to be detected, where the annotated images correspond to different categories to be detected, as described above.
[0084]
[0095] Detected objects corresponding to different categories to be detected in image 200 are identified by location in block 308. Bounding boxes are placed around each identified object. Example images with detected objects are provided in Figures 15-17.
[0085]
[0096] In image 320 of Figure 15, a bounding box 330 is placed around support block 46. Support block 46 is detected even though a chunk of wood has been removed. Separate bounding boxes 332, 334 are used for protruding nail 80. Similarly, in image 322 of Figure 16, a bounding box 336 is placed around split support block 46. Separate bounding boxes 338, 340 are used for protruding nail 80.
[0086]
[0097] As another example, in image 324 of Figure 17, a bounding box 342 is placed around the split support block 46. A pair of bent-fastened nails 71 are detected where the tips are exposed. In this case, bounding boxes 344, 346 are placed around the bent-fastened nails 71. Although the body of the nail 70 is visible in image 324, the nail is ignored because the nail tip is not exposed.
[0087]
[0098] For each bounding box, the pixel-by-pixel location is determined by the object detection algorithm 134. For example, image 320 in FIG. 15 is 1,000 pixels by 1,000 pixels. For example, the 0,0 coordinate is the bottom left of image 320. The x-axis of bounding box 330 around support block 46 begins at 250 pixels, and the y-axis of bounding box 330 begins at 100 pixels. The width and height of bounding box 330 are then determined.
[0088]
[0099] Similarly, for the bounding box 334 showing the protruding nail 80, the x-axis begins at 720 pixels and the y-axis of the bounding box 334 begins at 110 pixels. The width and height of the bounding box 334 are then determined. This is done for each bounding box.
[0089]
[0100] The method further includes providing a confidence value at block 310 for the category detected in the received image 320, 322, 324. The confidence value for each support block is 99%. The confidence value also has a detection category associated with it. The working "block" is used to refer to the support blocks 40, 46.
[0090]
[0101] A "protruding nail" label and a confidence value for each detected protruding nail are also provided. When the object detection algorithm 134 is trained to contextually observe each image, protruding nails 80 typically have high confidence values. In examples, the confidence values range from 87% to 99%. A "protruding nail" label and a confidence value are provided for the bent-fixed nail 71 in image 324.
[0091]
[0102] Pallets 10 with protruding nails 80 are identified based on the confidence value in block 312. If the confidence value exceeds a threshold, such as 75%, the individual bounding boxes are labeled accordingly.
[0092]
[0103] Another aspect is directed to a method for operating a pallet inspection system 140 as described above. Referring now to flow diagram 350 of FIG. 18 , from a start (block 352), the method includes operating the conveyor 105 to move a pallet 10 to be inspected in block 354. As described above, the pallet 10 includes a top deck 30 and a bottom deck 20 separated by a plurality of spaced-apart support blocks 40, 46 located therebetween. Nails 70 are used to secure the top and bottom decks 30, 20 to the support blocks 40, 46.
[0093]
[0104] Cameras 162, 164 positioned adjacent to the conveyor 105 are operated in block 356 to generate images of the pallets 10 as they are moved on the conveyor 105. The images are received in block 358 by processing unit 170 for processing.
[0094]
[0105] The processing includes running a first algorithm 132 on the images at block 360 to tag the images having support blocks 40, 46 visible therein. At block 362, a second algorithm 134 is run on the tagged images to detect nails 70 with exposed tips 72. The first algorithm 132 is an image classification algorithm and the second algorithm 134 is an object detection algorithm.
[0095]
[0106] Many modifications and other embodiments will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. It is therefore understood that the foregoing description is not limited to the illustrative embodiments, and that modifications and other embodiments are intended to be included within the scope of the appended claims.
Claims
1. 1. A method for training an object detection algorithm, comprising: creating, via a processor, a database of images of pallets having protruding nails and other nail defects; defining, via said processor, different categories to be detected in said database; annotating, via the processor, the images in a database corresponding to the different categories to be detected; training, via the processor, a model using machine learning to learn a function that generates a mapping between the annotated images and the different categories to be detected; analyzing, via said processor, output data from said model; and optimizing, via the processor, the model based on the analyzed output data.
2. 2. The method of claim 1, wherein the defined different categories to be detected include at least one of support blocks, protruding nails, partially visible nails, bent and fixed nails, freestanding nails, and split support blocks.
3. The method of claim 1 , wherein the annotating comprises placing a separate bounding box around each defined category that appears in the image.
4. 1. A method for operating an object detection algorithm, comprising: receiving, via a processor, an image of a pallet to be inspected; executing, via the processor, a machine learning model trained to learn a function that generates a mapping between annotated images of pallets with protruding nails and other nail defects to be detected, the annotated images corresponding to different categories to be detected; Identifying, via the processor, objects by their locations within the received image that correspond to the different categories to be detected; providing, via the processor, confidence values for the categories detected in the received image; and identifying, via the processor, the pallets having protruding nails based on the confidence values.
5. The method of claim 4 , wherein identifying an object by location includes placing a bounding box around the object.
6. The method of claim 4 , further comprising labeling each detected category along with providing the confidence value.
7. Each image is then sorted by an image classification algorithm. a block image corresponding to the in-focus support block; a non-block image corresponding to the out-of-focus support block, or a background image that does not correspond to either an in-focus support block or an out-of-focus support block, the block image being tagged by the image classification algorithm; 6. The method of claim 3 or 5, wherein placing the bounding boxes further comprises placing a first bounding box around the support block in each block image by the object detection algorithm, and placing a second bounding box around the exposed nail tip in response to detecting the exposed nail tip, wherein the first bounding box and the second bounding box are different colors from each other.
8. The method of claim 7 , wherein the image is a color image.