Pallet inspection system and related methods

Abstract

The pallet inspection system includes a conveyor for moving a pallet to be inspected. The pallet includes a top deck and a bottom deck separated by a plurality of spaced apart support blocks located therebetween, and nails are used to secure the top deck and the bottom deck to the plurality of support blocks. A camera is positioned to generate images of the pallet as it moves on the conveyor. A processor is coupled to the camera and receives the images for processing. The processing includes running a first algorithm on the images to tag images having support blocks visible in the images and running a second algorithm on the tagged images to detect nails having exposed tips.

Description

Technical Field

[0001]

[0002] The present invention relates to pallet inspection, and more particularly to determining nail defects in pallets.

Background Art

[0002] Related Application

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 262,452, filed Oct. 13, 2021, which is hereby incorporated by reference in its entirety.

[0003]

[0003] Wooden pallets are used to transport various bulk goods and equipment required in manufacturing and warehousing operations. In mass production industries, pallet pooling provides cost savings across the industry compared to single-use pallets.

[0004]

[0004] After bulk goods and equipment are offloaded from pooled pallets, the pallets are returned to a pallet inspection and repair facility. If the repairs are too extensive, damaged pallets are discarded. Otherwise, if the damage is minor, the pallets are repaired and painted before being returned to service.

Summary of the Invention

[0005]

[0005] A pallet inspection system includes a conveyor for moving the pallets to be inspected. The pallets include a top deck and a bottom deck separated by spaced support blocks disposed therebetween, and nails are used to secure the top deck and bottom deck to the support blocks. A plurality of cameras are arranged to generate images of the pallets as they are moved along the conveyor. A processor is coupled to the plurality of cameras and receives the images for processing. The processing includes executing a first algorithm on the images to tag images having support blocks visible therein, and executing a second algorithm on the tagged images to detect nails with exposed tips.

[0006]

[0006] The pallet inspection system may further include a first sensor disposed in front of the plurality of cameras and a second sensor disposed behind the plurality of cameras. The processor may be configured to receive a first set of images of the pallet in response to activation of the first sensor and to receive a second set of images of the pallet in response to activation of the second sensor.

[0007]

[0007] The first and second sensors comprise photoelectric sensors. The first and second algorithms may be executed by the processor to detect nails with exposed tips in real time.

[0008]

[0008] The plurality of support blocks are spaced apart to form a pair of outer columns and a central column therebetween, and the outer columns include corner support blocks. Individual cameras may be focused on one support block of a column and may not be focused on support blocks of other columns.

[0009]

[0009] The plurality of cameras may be divided into first and second camera sets, the first camera set being directed toward the entrance of the pallet inspection system and the second camera set being directed toward the exit of the pallet inspection system.

[0010]

[0010] The first camera set may provide a front perspective of the plurality of support blocks, the second camera set may provide a rear perspective of the plurality of support blocks, and the first and second camera sets collectively provide images of all sides of the individual support blocks.

[0011]

[0011] The first camera set may include cameras adjacent to each side of the conveyor, and the second camera set may include cameras adjacent to each side of the conveyor. Each camera may be configured as a color camera.

[0012]

[0012] The first algorithm may include an image classification algorithm, and the second algorithm may include an object detection algorithm.

[0013]

[0013] The image classification algorithm may be configured to classify each image as one of a block image corresponding to a focused support block, a non-block image corresponding to an unfocused support block, or a background image corresponding to neither an unfocused support block nor an unfocused support block. The block image may be tagged by the image classification algorithm.

[0014]

[0014] The object detection algorithm may be configured to place a bounding box around a support block within an individual block image and, in response to a detected exposed nail tip, place a bounding box around the exposed nail tip.

[0015]

[0015] The object detection algorithm may be configured to detect other nail defects in addition to nails with exposed tips, and the other nail defects are ignored by the processor.

[0016]

[0016] Another aspect is directed to a method for manufacturing a pallet as described above. The method includes operating a conveyor to move a pallet to be inspected, the pallet including a top deck and a bottom deck separated by a plurality of spaced support blocks disposed therebetween, and nails being used to secure the top deck and the bottom deck to the plurality of support blocks. The method further includes operating a plurality of cameras arranged to generate an image of the pallet as it moves on the conveyor and receiving the image for processing. The processing may include executing a first algorithm on the image, tagging an image having a support block visible therein, and executing a second algorithm on the tagged image to detect nails with exposed tips.

[0017]

[0017] Another aspect is directed to a method for training an object detection algorithm as described above. The method includes creating a database of images of pallets having protruding nails and other nail defects, defining different categories detected in the database, and annotating the images in the database corresponding to the different categories detected. The model is trained using machine learning to learn a function that generates a mapping between the annotated images and the different categories detected. The method further includes analyzing the output data from the model and optimizing the model based on the analyzed output data.

[0018]

[0018] Yet another aspect is directed to a method for operating an object detection algorithm as described above. The method includes receiving an image of a pallet to be inspected and executing a machine learning model trained to learn a function that generates a mapping between an annotated image of a pallet having a protruding nail and other nail defects detected. The annotated images correspond to different categories detected. The method further includes identifying an object based on its position in the received image corresponding to the different categories detected, providing a confidence value for the category detected in the received image, and identifying a pallet having a protruding nail based on the confidence value.

Brief Description of the Drawings

[0019]

Figure 1

[0019] FIG. 1 is a top perspective view of a wooden pallet on which various aspects of the present disclosure may be implemented.

Figure 2

[0020] FIG. 2 is a bottom perspective view of the wooden pallet shown in FIG. 1.

Figure 3

[0021] FIG. 3 is a partial cross-sectional view of the wooden pallet shown in FIG. 1 having a protruding nail.

Figure 4

[0022] FIG. 4 is a block diagram of a pallet inspection system on which various aspects of the present disclosure may be implemented.

Figure 5

[0023] Figure 5 is a more detailed block diagram of the protruding nail detection station shown in Figure 4 for detecting protruding nails in a wooden pallet.

Figure 6

[0024] Figure 6 is an image generated by the protruding nail detection station shown in Figure 5, which is classified as a block image, a non-block image, or a background image by an image classification algorithm.

Figure 7

[0024] Figure 7 is an image generated by the protruding nail detection station shown in Figure 5, which is classified as a block image, a non-block image, or a background image by an image classification algorithm.

Figure 8

[0024] Figure 8 is an image generated by the protruding nail detection station shown in Figure 5, which is classified as a block image, a non-block image, or a background image by an image classification algorithm.

Figure 9

[0025] Figure 9 is a block diagram of an exemplary object detection algorithm used by the protruding nail detection station shown in Figure 5 to detect protruding nails in a wooden pallet.

Figure 10

[0026] Figure 10 is a flowchart for training the object detection algorithm shown in Figure 9.

Figure 11

[0027] Figure 11 is an annotated image used to train the object detection algorithm shown in Figure 9.

Figure 12

[0027] Figure 12 is an annotated image used to train the object detection algorithm shown in Figure 9.

Figure 13

[0027] Figure 13 is an annotated image used to train the object detection algorithm shown in Figure 9.

Figure 14

[0028] Figure 14 is a flowchart for operating the object detection algorithm shown in Figure 9.

Figure 15

[0029] Figure 15 is an image output from the object detection algorithm shown in Figure 9.

Figure 16

[0029] Figure 16 is the image output from the object detection algorithm shown in Figure 9.

Figure 17

[0029] Figure 17 is the image output from the object detection algorithm shown in Figure 9.

Figure 18

[0030] It is a flowchart for operating the pallet inspection system shown in Figure 4.

Embodiments for Carrying Out the Invention

[0020]

[0031] This specification is made with reference to the accompanying drawings in which exemplary embodiments are shown. However, many different embodiments may be used and thus this specification should not be construed as being limited to the specific embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. Like numbers refer to like elements throughout.

[0021]

[0032] Nail defects in pallets are a concern, especially when the nails are protruding nails. A protruding nail is a case where the tip of the nail is exposed. Although the form factor of the pallet is not affected, it is necessary to detect protruding nails during pallet inspection so that the defects can be corrected.

[0022]

[0033] Typically, when an operator manually handles a pallet, the operator's hand is located within one or more pockets on the pallet. Each side of the pallet has pockets formed between a top deck and a bottom deck separated by support blocks disposed therebetween. Inserting a hand into a pocket with a protruding nail can cause injury to the operator.

[0023]

[0034] Referring to FIGS. 1 through 3, an exemplary wooden pallet 10 is described. The wooden pallet 10 is for the purpose of discussion regarding the general nail placement within the wooden pallet 10. The illustrated wooden pallet 10 is not limiting as other wooden pallet configurations are readily available.

[0024]

[0035] The wooden pallet 10 includes a bottom deck 20, a top deck 30, and a plurality of wooden support blocks 40, 46 coupled between the bottom deck and the top deck. The support blocks 40, 46 form a gap 50 (i.e., a pocket) between the bottom and top decks 20, 30 to receive a lifting member such as the tip of a forklift.

[0025]

[0036] The top deck 30 includes a pair of wooden end deck boards 32 spaced apart and a wooden intermediate deck board 34 disposed between the end deck boards 32. Also included within the top deck 30 are a pair of wooden connector boards 36 spaced apart and a wooden intermediate connector board 37. The connector boards 36 and the intermediate connector board 37 are orthogonal to the end deck boards 32 and the intermediate deck board 34. The end deck boards 32 and the intermediate deck board 34 are disposed on the connector boards 36 and are directly coupled to the support blocks 40, 46 via nails 70.

[0026]

[0037] The bottom deck 20 includes bottom deck boards 22, 26 oriented in the same direction as the end deck boards 32 and the intermediate deck board 34 of the top deck 30. The bottom deck boards 22, 26 are also directly coupled to the support blocks 40, 46 via nails 70.

[0027]

[0038] The support blocks include corner support blocks 40 and center support blocks 46 between the corner support blocks 40. There are a total of nine support blocks 40, 46 arranged in three columns. Each of the outer columns includes a pair of outer support blocks 40 and a single center support block 46, and the center column includes all the center support blocks 46. The actual number of support blocks can vary based on the configuration and size of the wooden pallet 10.

[0028]

[0039] The corner support blocks 40 and the center support blocks 46 each have a rectangular shape. In other configurations, one or more of the support blocks 40, 46 may have a non-rectangular shape such as circular.

[0029]

[0040] A partial cross-sectional view of the wooden pallet 10 is provided in FIG. 3 to show the arrangement of nails 70 within the support blocks 40, 46. When the wooden pallet 10 is formed, the nails enter the support blocks 40, 46 from the bottom and top decks 20, 30.

[0030]

[0041] Protruding nails 80 can occur when a nailing operator places the nail 70 off-center and the tip 72 is exposed. Protruding nails 80 also occur when the nail 70 is driven into the position of an existing nail and bounces back, causing the tip 72 to be exposed. Also, due to wear or damage to the wooden pallet 10, a part of the support blocks 40, 46 may be damaged and the tip 72 of the nail may be exposed. The protruding nails 80 are not limited to protruding from the support blocks 40, 46. The protruding nails 80 may protrude from other areas or regions of the pallet 10, for example, from one of the connector boards 36.

[0031]

[0042] Detecting protruding nails 80 within the wooden pallet 10 is challenging. As will be described in detail below, a camera is used to collect images of individual wooden pallets 10 during inspection. The images are sent to a processing unit for processing. The processing unit executes a machine learning object detection algorithm trained to detect protruding nails 80. How the algorithm is trained and executed will also be described in detail below. The processing unit can be, for example, a graphics processing unit (GPU), a central processing unit (CPU), or an edge computing device.

[0032]

[0043] An overview of the pallet inspection system 140 is described with reference to FIG. 4. Pooled pallets 10 from different customers are returned to a pallet pooling company for inspection and repaired if necessary before the pallets 10 are put back into service.

[0033]

[0044] Before inspection, the pallets 10 are provided to a stack infeed 152. The stack infeed 152 aligns the individual stacks of pallets before delivering them to a tipper / accumulator 154. The tipper / accumulator 154 provides, for example, a steady flow of spaced-apart pallets 10 to a conveyor. As the pallets 10 move on the conveyor, they pass through a preparation screening line 156.

[0034]

[0045] In the preparation screening line 156, the individual pallets 10 are visually inspected by a human operator to remove foreign debris or dust that could affect the inspection. If necessary, the human operator will also perform minor repairs. In some cases, as indicated by block 158, the pallets 10 can be discarded during the preparation screening line 156 if they are too severely damaged.

[0035]

[0046] From the preparation screening line 156, the individual pallets 10 are sent to a protruding nail detection station for inspection160 It is moved to. The pallet 10 is inspected for pallet classification and detects protruding nails 80.

[0036]

[0047] The pallet classification by the protruding nail detection station 160 is to determine whether the pallet 10 being inspected belongs to the pallet pooling company operating the pallet inspection system 140. The protruding nail detection station 160 The first set of cameras 162 within generates upper and lower images of each pallet 10. The images can be video images or still images. The images can be color or monochrome and are provided to the processing unit 170. The processing unit 170 executes a first algorithm for pallet classification 172. The first algorithm can be, for example, a machine learning (ML) image classification algorithm.

[0037]

[0048] The image classification algorithm compares the generated image data with an expected profile. The expected profile is used for pallet classification. The image classification algorithm labels or tags individual images as to whether the pallet 10 belongs to the pallet pooling company. For example, if 75 images of the pallet 10 are generated and most of these images are tagged as belonging to the pallet pooling company, the pallet 10 is classified as such.

[0038]

[0049] The image classification algorithm associated with the first set of cameras 162 can also detect certain obvious nail defects such as raised nails and free standing nails. In the case of raised nails, the head of the nail extends slightly above the top deck. Raised nails can potentially affect the products placed on the pallet. In the case of free standing nails, the head and body of the nail are visible but the tip is not.

[0039]

[0050] The detection of the protruding nails by the protruding nail detection station 160 is to determine the protruding nails 80 on the pallet 10. A second set of cameras 164 within the protruding nail detection station 160 generates images of the sides of the individual pallets 10. The images are in color and are provided to the processing unit 170. Alternatively, the images may be monochrome. The processing unit 170 executes a second algorithm trained to detect the protruding nails 80. The second algorithm can be, for example, a machine learning (ML) object detection algorithm.

[0040]

[0051] The pallet sorting line 180 receives the pallet 10 when it exits the protruding nail detection station 160. The pallet sorting line 180 queries the processing unit 170 to determine the pallet classification. If the inspected pallet 10 does not belong to the pallet pooling company operating the pallet inspection system 140, the pallet 10 is sent to the waste line 182.

[0041]

[0052] In the illustrated embodiment, the protruding nail detection station 160 includes first and second sets of cameras 162, 164. In an alternative embodiment, the cameras can be combined such that the same set of cameras used for pallet classification is also used for the detection of protruding nails. That is, the cameras are not mutually exclusive.

[0042]

[0053] Alternatively, if the inspected pallet 10 belongs to the pallet pooling company operating the pallet inspection system 140, the pallet sorting line 180 queries the processing unit 170 to determine whether the pallet 10 is good or bad.

[0043]

[0054] If the inspected pallet 10 is defective, this means that the pallet needs repair and is sent to the repair line 184. After repair, the pallet 10 is sent to the painting line 186 for painting before being returned to service. If the inspected pallet 10 is good, this means that the pallet does not need repair and is instead sent to the painting line 186 before being returned to service.

[0044]

[0055] Referring to FIG. 5, the operation of the protruding nail detection station 160 for detecting protruding nails 70 is described. The conveyor 105 moves the pallet 10 through the protruding nail detection station 160 in the direction of the arrow shown. The conveyor 105 includes a first sensor 110 at the entrance of the protruding nail detection station 160 and a second sensor 112 at the exit of the protruding nail detection station 160. In other embodiments, a single sensor may be used. The first and second sensors 110, 112 may be configured as, for example, photoelectric sensors. Each photoelectric sensor includes a transmitter and a receiver on opposite sides of the conveyor 105. The transmitter transmits an optical signal, which may be visible or infrared, to the receiver. The pallet 10 is detected when the light beam is blocked from reaching the receiver from the transmitter.

[0045]

[0056] The outputs from the first and second sensors 110, 112 are provided to the controller 130. In response to receiving the outputs from the first and second sensors 110, 112, the controller 130 is configured to trigger a second set of cameras 164 as the pallet is being moved on the conveyor 105. The second set of cameras 164 is divided into a first set of cameras 164(1), 164(2) and a second set of cameras 164(1), 164(2).

[0046]

[0057] The cameras of the first camera sets 164(1), 164(2) are triggered by the first sensor 110. In response to the first sensor 110 detecting the arrival of the pallet 10, the first stopper 111 in the path of the pallet 10 is dropped under the conveyor 105, allowing the pallet 10 to enter the protruding nail detection station 160. When the pallet 10 enters the protruding nail detection station 160, the controller 130 activates or triggers the cameras in the first camera sets 164(1), 164(2) to provide images to the processing unit 170 for processing.

[0047]

[0058] The images generated by the first camera sets 164(1), 164(2) are a subset of the overall image received by the GPU 170 for the pallet 10. The remaining images received by the processing unit 170 are generated by the second camera sets 164(3), 164(4).

[0048]

[0059] The cameras of the first camera sets 164(1), 164(2) are arranged to view the front perspectives of the support blocks 40, 46. The cameras of the second camera sets 164(3), 164(4) are arranged to view the rear perspectives of the support blocks 40, 46. Collectively, the combined images are all sides of the individual support blocks 40, 46.

[0049]

[0060] The cameras of the second camera sets 164(3), 164(4) are triggered by the second sensor 112. In response to the second sensor 112 detecting the departure of the pallet 10, the second stopper 113 within the path of the pallet 10 is dropped under the conveyor 105, allowing the pallet 10 to exit the protruding nail detection station 160. When the pallet 10 exits the protruding nail detection station 160, the controller 130 activates or triggers the cameras in the second camera sets 164(3), 164(4) to provide images to the processing unit 170 for processing.

[0050]

[0061] Having two different triggers ensures that a complete set of images of the support blocks 40, 46 is obtained. If the first and second camera sets 164(1)-164(4) are triggered simultaneously by a single photosensor and the conveyor 105 jams and the pallet 10 stops moving, the images on the rear perspective views of the support blocks 40, 46 will not be provided to the processing unit 170.

[0051]

[0062] The first camera sets 164(1), 164(2) include six cameras, and the second camera sets 164(3), 164(4) also include six cameras. Half of the cameras are on both sides of the conveyor 105. Although twelve cameras are being used, in other embodiments, a different number of cameras may be used.

[0052]

[0063] The support blocks 40, 46 on the individual pallets 10 are spaced apart to form a pair of outer columns and a central column therebetween. The illustrated pallet includes nine support blocks 40, 46 having three support blocks 40, 46 in individual rows. The individual columns are parallel to the conveyor 105. Each of the outer columns includes a pair of corner support blocks 40 and center support blocks 46. The central column includes only center support blocks 46.

[0053]

[0064] The individual cameras in the first and second camera sets 164(1)-164(4) are focused on specific columns of the support blocks. For example, camera group 164(1) includes three cameras. The first camera is focused on the outer column of the support blocks 40, 46 closest to camera group 164(1). The second camera is focused on the central column of the support blocks 46. The third camera is focused on the outer column of the support blocks 40, 46 farthest from camera group 164(1). Similarly, the individual cameras in camera group 164(2) on the opposite side of the conveyor 105 are focused on specific columns of the support blocks 40, 46.

[0054]

[0065] As described above, the cameras of the first camera sets 164(1), 164(2) are arranged to view the front perspectives of the support blocks 40, 46. Typically, there is a delay between the six cameras of the camera groups 164(1), 164(2) when a particular row of the support blocks 40, 46 is in focus.

[0055]

[0066] A pair of cameras focused on the outer columns of the support blocks 40, 46 closest to the individual cameras are focused first. This pair of cameras can be triggered immediately in response to the first sensor 110 detecting the arrival of the pallet 10. Since the center support block 46 is further away from the cameras, next, a pair of cameras focused on the central column of the support block 46 are focused. As a result, this pair of cameras can be delayed by 25 milliseconds before being triggered by the controller 130, for example, to generate an image. The pair of cameras focused on the outer columns of the support blocks 40, 46 furthest from the cameras are focused after the central column of the support block 46 has been focused because they are the furthest from the cameras. As a result, this pair of cameras can be delayed by 50 milliseconds before being triggered by the controller 130, for example, to generate an image. When the cameras are triggered and an image is provided to the processing unit 170, the trigger continues for a predetermined period. The predetermined period can be, for example, between 3 and 4 seconds and may vary based on the speed of the conveyor 105 belt.

[0056]

[0067] The cameras in the second camera sets 164(3), 164(4) are configured similarly to the cameras in the first camera sets 164(1), 164(2). The individual cameras in the second camera sets 164(3), 164(4) are similarly focused on specific columns of the support blocks. As described above, the cameras of the second camera sets 164(3), 164(4) are arranged to view the rear perspectives of the support blocks 40, 46.

[0057]

[0068] In response to the second sensor 112 detecting that the pallet 10 exits the protruding nail detection station 160, the controller 130 may immediately trigger a pair of cameras focused on the outer columns of the support blocks 40, 46 closest to the individual cameras, and may delay triggering the pair of cameras focused on the central column of the support block 46 and the pair of cameras focused on the outer columns of the support blocks 40, 46 farthest from the individual cameras.

[0058]

[0069] The operation of the processing unit 170 for detecting the protruding nails 80 from the received images is a two-step process. The first step is to determine an image having the support blocks 40, 46 on which one of the cameras is focused. The second step is to analyze only the images having the support blocks 40, 46 on which the focus is set for the protruding nails 80.

[0059]

[0070] A reasonable reason for determining an image having a focused support block is that since the nails 70 are driven into the support blocks 40, 46, the protruding nails 80 will be adjacent to the support blocks 40, 46. The number of images analyzed for the protruding nails 80 is significantly reduced by the first step.

[0060]

[0071] Each individual camera can generate approximately 40 images per pallet 10. Using 12 cameras, this corresponds to approximately 480 images per pallet. The actual number of images generated is variable and can be changed from one deployment location to another. The processing unit 170 executes a first algorithm on the 480 images. The first algorithm can be, for example, the image classification algorithm 132. The image classification algorithm 132 is trained using artificial intelligence (AI) and machine learning (ML) to determine the images having the support blocks 40, 46 on which the focus is set.

[0061]

[0072] When the image has in-focus support blocks 40, 46 therein, it is tagged by the image classification algorithm 132. The image classification algorithm 132 functions as a pre-filter for the images provided to the processing unit 170 by the second set of cameras 164.

[0062]

[0073] The tagged images are provided to a second algorithm, which can be, for example, the object detection algorithm 134. Untagged images are not passed to the object detection algorithm 134. As an example, out of 480 images, about half to one-third of the images may not be tagged.

[0063]

[0074] The execution of the object detection algorithm 134 is more computationally extensive than the execution of the image classification algorithm 132. Reducing the number of images executed by the object detection algorithm 134 simplifies the overall process for determining the protruding nail 80.

[0064]

[0075] The image classification algorithm 132 is trained to classify individual images as block images, non-block images, or background images. Block images correspond to in-focus support blocks 40, 46. Non-block images correspond to out-of-focus support blocks 40, 46. Background images do not correspond to either in-focus support blocks 40, 46 or out-of-focus support blocks 40, 46.

[0065]

[0076] Referring to FIGS. 6-8, an image generated by the protruding nail detection station 160 and classified by the image classification algorithm 132 is described. The image 200 in FIG. 6 is a block image because the focus is on the support block 40. The image 202 in FIG. 7 is a non-block image because the focus is not on the support block 40. The image 204 in FIG. 8 is a background image because no focused or unfocused support block is visible. Background images are typically generated when the pallet 10 first arrives at the protruding nail detection station 160 and when the pallet 10 exits the protruding nail detection station 160.

[0066]

[0077] The image classification algorithm 132 classifies individual images using percentages. The classification percentages change as the pallet 10 moves on the conveyor 105. The classification percentages are displayed for individual images and total 100%. The percentage numbers are assigned by the image classification algorithm 132 to each of the three possible classifications. For example, in response to a classification percentage exceeding a threshold such as 75%, the image classification algorithm 132 classifies the image accordingly.

[0067]

[0078] In the image 200, the classification as a block image is 100%, while the classifications as a non-block image or a background image are each 0%. In the image 202, it is classified as a non-block image at 99.9511%, classified as a block image at 0.0333%, and classified as a background image at 0.0156%. In the image 204, it is classified as a background image at 82.796%, classified as a non-block image at 17.0867%, and classified as a block image at 0.1173%. In this image 204, the pallet 10 is partially visible when exiting the protruding nail detection station 160.

[0068]

[0079] Tagged images having in-focus support blocks 40, 46 are generally referred to as tagged image 200. These images are passed to object detection algorithm 134 for processing. Instead of tagging the images, object detection algorithm 134 is trained to find objects within tagged image 200. The objects to be placed are support blocks and visible nails overlapping the support blocks. Visible nail 70 may be a protruding nail 80, or may be a nail 70 where the body of the nail is visible but the tip is not visible.

[0069]

[0080] As described below, object detection algorithm 134 is trained using annotated images that include several different categories to be detected, and each category corresponds to a specific type of object to be detected and found. By learning different categories, object detection algorithm 134 can distinguish between protruding nails to be repaired and other types of nail defects that do not need to be repaired. Different categories provide context details to object detection algorithm 134.

[0070]

[0081] Object detection algorithm 134 can operate based on artificial intelligence (AI) and machine learning (ML) to determine objects within image 200 having in-focus support blocks 40, 46 therein. In one example, object detection algorithm 134 can be a single-shot detector (SSD) 210, as shown in FIG. 9. Other types of object detectors can be used as an alternative to the illustrated SSD 210. In SSD 210, only a single shot of image 200 is taken to detect multiple objects within image 200. SSD 210 is an open-source algorithm modified to detect protruding nail 80.

[0071]

[0082] The SSD210 generally has a base VGG-16 network 212, followed by multi-box convolutional layers 214, 216. The base VGG-16 network 212 is used to extract features within the image 200. The convolutional layer 214 is for detection, and the convolutional layer 216 helps with object detection at multiple scales because the sizes of these layers gradually decrease. The convolutional model for detection varies for each feature layer.

[0072]

[0083] The multi-box convolutional layers 214, 216 are applied to multiple feature maps from the later stages of the network. This helps in performing detections at multiple scales. The prediction of bounding boxes and the confidence of different objects within the image 200 are done not by one, but by multiple feature maps of different sizes representing multiple scales.

[0073]

[0084] The problem of detecting protruding nails can be addressed in different ways. One way is to use object detection using bounding boxes as described below. Another way is to use a segmentation approach that segments pixels to define the contours of protruding nail pixels within the image. Yet another way is that an algorithm can be trained to detect regions using both bounding boxes and segmentation pixels.

[0074]

[0085] Now, referring to the flowchart 250 of FIG. 10, the training of the object detection algorithm 134 will be described. From the beginning (block 252), a database of images of pallets 10 with protruding nails 70 and other nail defects is created in block 254. The number of pallets 10 can be quite large, for example, more than 100. Images from all 12 cameras for each individual pallet 10 are collected and stored in the database. Approximately 480 images are associated with each individual pallet 10.

[0075]

[0086] The different categories to be detected are defined in block 256. These categories include support blocks, protruding nails, partially visible nails, clinched nails, standing nails without support, and splinters. Other types of categories can be defined to assist the object detection algorithm 134 in detecting protruding nails 80.

[0076]

[0087] The images in the database are annotated in block 258 to reflect the different categories to be detected. Individual images are manually reviewed, and if the image has the categories detected by the object detection algorithm 134, the image is annotated with a bounding box and labeled accordingly.

[0077]

[0088] Exemplary images with annotations are provided in FIGS. 11 - 13. The green bounding boxes 202a are placed around the individual support blocks 40, 46. For protruding nails 80, red bounding boxes 204a are used. In response to the detected nail not being a protruding nail, yellow bounding boxes are used. In FIG. 12, for example, the yellow bounding box 206 is placed around a nail 70 where the body of the nail is visible but the tip is not. The support block 40 in FIG. 13 is split, and a pair of protruding nails 80 are visible.

[0078]

[0089] The detection of split support blocks is useful. Sometimes, it can be difficult to distinguish between split support blocks and colored nails. Usually, nails are displayed in gray / block color. However, if the split support blocks are not detected early and the pallet is sent for painting, the exposed nails will be painted the same color as the blocks.

[0079]

[0090] In addition to detecting nail defects and split support blocks, the image can be annotated to identify other types of defects. The image can be annotated, for example, to detect stickers and stretch wrapping remaining on the pallet 10. Typically, individual pallets 10 are visually inspected by a human operator to remove errant pieces or debris that could affect the inspection. Adding the detection of stickers and shrink wrapping helps to identify areas that may not be captured during the visual inspection of individual pallets 10.

[0080]

[0091] The model is trained in block 260 using machine learning to learn a function that generates a mapping between the annotated image and the different categories to be detected. In some embodiments, the machine learning can be based on a neural network for training the model. In other embodiments, the neural network is not used to train the model. Since a neural network-based model / algorithm, or another type of non-neural network-based model / algorithm can be used, we are not limited to a single or a set of algorithms.

[0081]

[0092] The output data from the model is analyzed in block 262. The model is optimized in block 264 based on the analyzed output data. The method ends in block 266.

[0082]

[0093] Now, with reference to the flowchart 300 of FIG. 14, the operation of the object detection algorithm 134 will be described. Starting from start (block 302), an image of the pallet 10 to be inspected is received in block 304. The image is the image 200 tagged by the image classification algorithm 132 as having the support blocks 40, 46 in focus therein.

[0083]

[0094] In block 306, the method includes executing a machine learning model trained to learn a function that generates a mapping between an annotated image of pallet 10 with protruding nail 80 and other nail defects to be detected. The annotated images correspond to different categories to be detected, as described above.

[0084]

[0095] Detected objects corresponding to different categories to be detected within image 200 are identified by location in block 308. Bounding boxes are placed around the identified individual objects. Exemplary images with detected objects are provided in FIGS. 15 - 17.

[0085]

[0096] In the image 320 of FIG. 15, the bounding box 330 is placed around support block 46. Support block 46 is detected even though the wood mass has been removed. For the protruding nail 80, separate bounding boxes 332, 334 are used. Similarly, in the image 322 of FIG. 16, the bounding box 336 is placed around the split support block 46. For the protruding nail 80, separate bounding boxes 338, 340 are used.

[0086]

[0097] As another example, in the image 324 of FIG. 17, the bounding box 342 is placed around the split support block 46. A pair of bent and fixed nails 71 are detected at the part where the tip is exposed. In this case, the bounding boxes 344, 346 are placed around the bent and fixed nails 71. The body of nail 70 is visible within image 324, but since the tip of the nail is not exposed, this nail is ignored.

[0087]

[0098] For each individual bounding box, the pixel-level position is determined by the object detection algorithm 134. For example, the image 320 in FIG. 15 is 1,000 pixels × 1,000 pixels. For example, the 0,0 coordinates are at the lower left of the image 320. The x-axis of the bounding box 330 around the support block 46 starts at 250 pixels, and the y-axis of the bounding box 330 starts at 100 pixels. Then, the width and height of the bounding box 330 are determined.

[0088]

[0099] Similarly, for the bounding box 334 showing the protruding nail 80, the x-axis starts at 720 pixels, and the y-axis of the bounding box 334 starts at 110 pixels. Then, the width and height of the bounding box 334 are determined. This is performed for each bounding box.

[0089]

[0100] This method further includes providing a confidence value at block 310 for the categories detected in the received images 320 , 322, 324. The confidence value for each individual support block is 99%. The confidence value also has the detection category associated with it. The working "blocks" are used to indicate the support blocks 40, 46.

[0090]

[0101] The "protruding nail" label and the confidence value for each detected individual protruding nail are also provided. When the object detection algorithm 134 is trained to observe individual images contextually, the protruding nail 80 typically has a high confidence value. In the example, the confidence value ranges from 87% to 99%. For the bent and fixed nail 71 in the image 324, the "bent and fixed nail" label and the confidence value are provided.

[0091]

[0102] The pallet 10 having the protruding nail 80 is identified based on the confidence value at block 312. When the confidence value exceeds a threshold such as 75%, the individual bounding boxes are labeled accordingly.

[0092]

[0103] Another aspect is directed to a method for operating the pallet inspection system 140 as described above. Referring now to the flowchart 350 of FIG. 18, starting at (block 352), the method includes operating the conveyor 105 at block 354 to move the pallet 10 to be inspected. As described above, the pallet 10 includes a top deck 30 and a bottom deck 20 separated by a plurality of spaced support blocks 40, 46 positioned therebetween. Nails 70 are used to secure the top and bottom decks 30, 20 to the support blocks 40, 46.

[0093]

[0104] Cameras 162, 164 disposed adjacent to the conveyor 105 are operated at block 356 to generate an image of the pallet 10 as the pallet 10 is moved on the conveyor 105. The image is received at block 358 by the processing unit 170 for processing.

[0094]

[0105] The processing includes executing a first algorithm 132 on the image at block 360 to tag the image having the support blocks 40, 46 visible therein. At block 362, a second algorithm 134 is executed on the tagged image to detect nails 70 having exposed tips 72. The first algorithm 132 is an image classification algorithm and the second algorithm 134 is an object detection algorithm.

[0095]

[0106] Numerous variations and other embodiments will occur to those skilled in the art who benefit from the teachings presented in the foregoing description and the related drawings. Accordingly, the foregoing description is not limited to the exemplary embodiments, and it is understood that variations and other embodiments are intended to be included within the scope of the appended claims.

Claims

1. A pallet inspection system, a conveyor configured to move a pallet to be inspected, the pallet including a top deck and a bottom deck separated by a plurality of spaced support blocks located therebetween, and nails being used to fix the top deck and the bottom deck to the plurality of support blocks, the conveyor; a plurality of cameras arranged to generate an image of the pallet as the pallet is moved on the conveyor; a processor coupled to the plurality of cameras and configured to receive the image for processing, the processing including: executing a first algorithm on the image to tag the image with support blocks visible in the image; executing a second algorithm on the tagged image to detect nails having exposed tips, the first algorithm including an image classification algorithm and the second algorithm including an object detection algorithm, the image classification algorithm configured to classify an individual image as a block image corresponding to a focused support block, a non-block image corresponding to an unfocused support block, or a background image corresponding to neither a focused support block nor an unfocused support block, with the block image being tagged by the image classification algorithm, The object detection algorithm is configured to place a first bounding box around the support block within each individual block image and, in response to detecting the tip of an exposed nail, place a second bounding box around the tip of the exposed nail, wherein the first bounding box and the second bounding box have different colors from each other, a pallet inspection system.

2. The pallet inspection system according to claim 1, further comprising a first sensor disposed in front of the plurality of cameras and a second sensor disposed behind the plurality of cameras, wherein the processor is configured to: receive a first set of images of the pallet in response to activation of the first sensor and receive a second set of images of the pallet in response to activation of the second sensor.

3. The pallet inspection system according to claim 2, wherein the first sensor and the second sensor include optoelectronic sensors.

4. The pallet inspection system according to claim 1, wherein the first algorithm and the second algorithm are executed by the processor such that nails having exposed tips are detected in real time.

5. The pallet inspection system according to claim 1, wherein the plurality of support blocks are spaced apart to form a pair of outer columns and a central column therebetween, the outer columns include corner support blocks, and each of the plurality of cameras is focused on one of the support blocks in the columns and not on the support blocks in the other columns.

6. The pallet inspection system according to claim 1, wherein the plurality of cameras are divided into a first and a second camera set, the first camera set is directed towards the entrance of the pallet inspection system station, and the second camera set is directed towards the exit of the pallet inspection system station.

7. The first camera set provides a front perspective of the plurality of support blocks, the second camera set provides a rear perspective of the plurality of support blocks, and the first camera set and the second camera set collectively provide images of all sides of individual support blocks. The pallet inspection system according to claim 6.

8. The first camera set includes cameras adjacent to each side of the conveyor, and the second camera set includes cameras adjacent to each side of the conveyor. The pallet inspection system according to claim 6.

9. Each of the plurality of cameras is configured as a color camera. The pallet inspection system according to claim 1.

10. The object detection algorithm is configured to detect other nail defects in addition to nails with exposed tips, and the other nail defects are ignored by the processor. The pallet inspection system according to claim 1.

11. A method for operating a pallet inspection station, comprising: Operating a conveyor to move a pallet to be inspected, the pallet including a top deck and a bottom deck separated by a plurality of spaced support blocks located therebetween, and nails being used to secure the top deck and the bottom deck to the plurality of support blocks. The operating; Operating a plurality of cameras arranged to generate an image of the pallet as the pallet moves on the conveyor; Receiving the image via a processor for processing, the processing comprising: Executing a first algorithm on the image to tag the image having support blocks visible in the image; Executing a second algorithm on the tagged image to detect a nail having an exposed tip; The first algorithm includes an image classification algorithm, and the second algorithm includes an object detection algorithm; The image classification algorithm classifies individual images as A block image corresponding to a focused support block, A non-block image corresponding to an unfocused support block, or As one of a background image corresponding to neither a focused support block nor an unfocused support block, Accompanied by the block image being tagged by the image classification algorithm, and is configured to classify; The object detection algorithm is configured to place a first bounding box around the support block within an individual block image and, in response to the exposed tip of the nail being detected, place a second bounding box around the exposed tip of the nail, the first bounding box and the second bounding box being different in color from each other.

12. The pallet inspection station further includes a first sensor disposed in front of the plurality of cameras and a second sensor disposed behind the plurality of cameras, and the method includes Receiving a first set of images of the pallet in response to activation of the first sensor; Receiving a second set of images of the pallet in response to activation of the second sensor; The method according to claim 11, further comprising.

13. The plurality of support blocks are spaced apart to form a pair of outer columns and a central column therebetween, the outer columns each include a corner support block, and each of the plurality of cameras is focused on one of the support blocks of one of the columns and not focused on the support blocks in the other column, the method according to claim 11.

Images