Small package detection and intelligent sorting

Abstract

To provide a sorting conveyor and a method for sorting packages to selected destinations.SOLUTION: A neural network executed by a multi-core classification processor 36 uses captured images of package units conveyed on an infeed conveyor 18, 19 to classify each package unit. A control processor 34 controls a sorting conveyor 26 to direct each package unit to a destination depending on the package unit's classification.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention generally relates to a power-driven conveyor, and more particularly to a conveyor that uses artificial intelligence to classify small packages and sort the small packages to their respective destinations according to the classification.

Background Art

[0002] In the small package handling industry, sorting conveyors are used to separate individual small packages from a load flow of stacked small packages that are facing in a disorderly direction and have various sizes and shapes. However, before the small packages can be sorted to their appropriate destinations, it is necessary to separate the small packages from each other. Conveyors use various techniques to separate the load flow into individual small packages. However, sometimes the conveyor fails to separate all the small packages, and manual intervention may be required. Also, extra-large boxes can be problematic as they can cause congestion, and if this is recognized, the boxes are removed manually. However, extra-large plastic bags are flexible and may not cause congestion. Therefore, removing such bags from the sorting machine reduces the overall throughput.

Summary of the Invention

[0003] A conveyor system embodying the features of the present invention comprises a feed conveyor section that transports separated parcel units downstream in the transport direction. A parcel unit detector detects parcel units at detection positions along the feed conveyor section. An imaging system captures images of parcel units within the target zone of the feed conveyor section. A computer processing system executes program instructions to track the position of each parcel unit as it is transported along the feed conveyor section, and program instructions to provide each image of a parcel unit as input to a classifier. The classifier is trained to recognize a set of parcel units and to assign a classification to each parcel unit that corresponds to one of the members of the set of parcel units that the classifier is trained to recognize. A downstream conveyor section receives parcel units from the feed conveyor. The computer processing system executes program instructions to control the downstream conveyor section to transport each parcel unit according to its classification.

[0004] A separate conveyor system is provided with a feed conveyor section that transports small packages downstream in the transport direction at a transport speed. An occupied area detector is placed at a detection position along the feed conveyor section. A camera is placed along the feed conveyor section downstream of the detection position to capture an image of the capture area on the feed conveyor section. A sorting conveyor section receives small packages from the feed conveyor section and selectively sorts the small packages to multiple destinations. The control processor executes: (1) a program instruction to operate the occupied area detector and detect the occupied area of ​​small packages moving forward in the transport direction past the detection position; and (2) a program instruction to control the sorting conveyor section and the feed conveyor section. The classification processor, communicating with the control processor, executes: (1) a program instruction to calculate the position of each parcel unit on the feed conveyor section from the area they occupy as they move along the conveyor and pass through the detected position; (2) a program instruction to control the camera to capture images of one or more parcel units and generate images of one or more parcel units within the capture area if the calculated position of one or more parcel units is within the target zone of the capture area on the feed conveyor section; (3) a program instruction to crop the images into one or more cropped images and associate each cropped image with one or more parcel units within the capture area; and (4) a program instruction to use artificial intelligence to classify the cropped images into multiple classifications and assign one of the classifications to each of the one or more parcel units. The control processor controls the sorting conveyor section to sort each of the one or more parcel units to a destination according to the classification of the parcel unit.

[0005] In another embodiment, a processor-implemented method for sorting parcels being transported on a feed conveyor section includes: (a) detecting multiple parcel units moving forward on the feed conveyor section at transport speed in the transport direction past a detection position; (b) calculating the position of each of the multiple parcel units detected as passing the detection position on the feed conveyor section as they move along the conveyor; (c) capturing images of one or more of the multiple parcel units within an image acquisition area on the feed conveyor section to generate images of one or more of the multiple parcel units within the acquisition area; (d) cropping the images into one or more cropped images and associating each cropped image with one or more of the multiple parcel units within the image acquisition area; (e) classifying the cropped images into several categories using artificial intelligence and assigning one of the categories to each of the one or more multiple parcel units; and (f) sorting each of the one or more multiple parcel units to a destination according to the classification of the parcel unit. [Brief explanation of the drawing]

[0006] [Figure 1] This is a top view of a transport system that embodies the features of the present invention. [Figure 2] Figure 1 is a flowchart of program instructions executed by the multicore classification processor and control processor for the conveyor system. [Figure 3] Figure 2 is an enlarged flowchart of the TCP connection management task. [Figure 4] Figure 2 is an enlarged flowchart of the small package tracking task. [Figure 5] Figure 2 shows an enlarged flowchart of the image processing task and the small package classification task. [Figure 6] This is a schematic diagram of an exemplary neural network used to help explain the operation of a neural network that can be used in the conveyor system shown in Figure 1. [Modes for carrying out the invention]

[0007] Figure 1 shows a conveyor system that embodies the features of the present invention for intelligently sorting small packages. The conveyor 10 includes a feed conveyor section 12 that transports small packages 13 downstream in the transport direction 14. A small package unit detector 16 is positioned along the feed conveyor section 12 at the detection location. In this example, the feed conveyor section 12 is illustrated as a series of two conveyors: a first conveyor 18 upstream of the small package unit detector 16 and a second conveyor 19 downstream of the small package unit detector. Both conveyors 18 and 19 can be implemented as, for example, belt conveyors, slat conveyors, chain conveyors, or roller conveyors. The feed conveyor section 12 can be implemented as a single conveyor or as more than two conveyors. In this two-conveyor configuration, the parcel unit detector 16 emits a curtain of light from the transmitter array through the gap 20 between the two conveyors 18 and 19 to the receiver array. The receivers and transmitters are positioned on both sides of the conveyor section 12, one below and the other above. The resulting curtain of light spreads across the width of the conveyor section 12. Parcels 13 passing through the curtain of light block the transmitting beam, preventing it from reaching the receiver array. Once a parcel 13 passes through the parcel unit detector 16, its occupied area, i.e., its projection onto the feed conveyor section, can be determined from the pattern of blocked receivers. Thus, in this configuration, the parcel unit detector 16 is an occupied area detector. Other types of parcel unit detectors, such as laser rangefinders or cameras, can be used instead.

[0008] Downstream of the small package unit detector 16, along the feed conveyor section 12, is an imaging system including a camera 22. The camera 22 has an acquisition area 24 that covers a continuous range on the transport side of the feed conveyor section 12, spanning its width. The imaging system captures images of the small packages 13 as they pass through the acquisition area 24. The small packages 13 leaving the feed conveyor section 12 are received on the downstream sorting conveyor section 25. The sorting conveyor section 25 can be implemented as a sorting conveyor 26, for example, as a roller belt with rollers that operate and rotate selectively to redirect goods to one of two flanking conveyors 28, 29. The sorting conveyor 26 and the flanking conveyors 28, 29 may be, as two examples, powered roller conveyors or modular plastic belt conveyors. Furthermore, the feed conveyor section 12 and the sorting conveyor 26 can be realized by a single conveyor belt. In one form, the sorting conveyor 26 is an INTRALOX® Series 7000 activated-roller-belt conveyor manufactured and sold by Intralox, LLC, Harahan, Louisiana, USA. Similarly, the flanking conveyors 28 and 29 can be made of, for example, roller belts, flat belts, or modular plastic belts. Slats This can be implemented by a conveyor, chute, or roller conveyor. The sorting conveyor, by selectively operating its rollers, redirects the parcels to one of three destinations: (1) a right-hand flanking conveyor 28; (2) a left-hand flanking conveyor 29; or (3) a destination 30 where a human worker 32 is positioned to decide how to dispose of the received parcel units.

[0009] The output of the occupied area detector 16 is sent to a computer processing system 33, which includes a control processor 34 programmed to determine the occupied area of ​​the packages 13 passing through the detector at the detection location. (The computer processing system 33 includes a program memory that stores constants and program instructions executed by one or more processors, and a volatile data memory that stores calculations, tables, and other temporary or modifiable information.) Some packages may not be separated from other packages. Their overlapping, i.e., stacked packages may have an occupied area that is different in shape from the occupied area of ​​a single separated package. For this reason, the occupied area detector 16 detects the occupied area of ​​a separated package unit that contains a single separated package, and the occupied area of ​​a group of overlapping or stacked packages whose occupied area is defined by a single contour line.

[0010] The control processor 34 executes stored program instructions that define the individual occupied area of ​​each package unit passing through the detection position. For example, each occupied area can be defined by the coordinates of its corners or by its centroid calculated by the control processor 34. The position of each package unit on the feed conveyor can always be described by its coordinates in an xy coordinate system, which for example uses the detection position as the reference position x=0 and the right edge 35 of the feed conveyor section 12 as the reference position y=0.

[0011] In any coordinate system, the x-axis is parallel to the transport direction 14, and the y-axis is perpendicular to the transport direction. The control processor 34 can also be programmed to perform the tasks of: (a) controlling the speeds of the feed conveyor section 12, the sorting conveyor 26, and the first and second conveyors 28, 29; (b) receiving input reporting the speeds of various conveyors; and (c) controlling the communication network between the control processor itself and the classification processor 36 used with the imaging system. Similar to the control processor 34, the classification processor 36 is contained within the computer processing system 33 and may include an external or same-chip image processing unit (GPU). The control processor 34 may perform control of the conveyors 12, 26, 28, 29 via an external physical programmable logic controller (PLC) 38 or via an internal virtual PLC, or control may be shared between the external physical PLC and the internal virtual PLC. For example, an external PLC may be able to control the engines that drive the feed and sorting conveyor sections and read sensors that report the belt speed, while the virtual PLC receives the output of the occupied area detector. In any case, the PLC 38 is considered part of the control processor 34 and part of the computer processing system 33.

[0012] The operation of the control processor 34 and the classification processor 36 will be described with reference to the flowcharts in Figures 2 to 5.

[0013] Figure 3 shows in more detail the TCP (Transmission Control Protocol) connection manager task 40, which is performed by the control processor 34 (Figure 1). This task controls the communication network 42 (Figure 1) which is responsible for communication between the control processor 34, the classification processor 36, and the PLC 38. The TCP connection manager task 40 has three subtasks: (1) a read task 44; (2) a connection monitor task 46; and (3) a write event handler task 48.

[0014] The reading task 44 reads message 50, parses message 52, and executes message 54. In the case of a message from the control processor 34 indicating that the occupied area corresponding to a small package unit has been identified by the occupied area detector, the occupied area and its corresponding coordinates are added to the feed conveyor status table, which represents the most recently calculated position of any identified occupied area of ​​the small package unit on the feed conveyor 19 downstream of the detection position.

[0015] The write event handler 48 processes any event 56 that occurs, frames a message indicating the occurrence of the event along with any related data 58 related to the event, and sends the message frame 60 over the communication network so that it can be read by the intended recipient. An example is given below.

[0016] The connection monitoring task 46 checks and verifies that all devices or nodes, such as the control processor 34, the classification processor 36, and the PLC 38, are connected. This task sends a heartbeat message 62 over the communication network. This task determines whether the message has been received by the intended receiving device 64. If there are no message transmission failures, the control processor knows that the network is not compromised 66. If there are transmission failures, the control processor attempts to reconnect to the disconnected device 68.

[0017] The classification processor 36 executes stored program instructions, including a timer-tick task 70, at a periodic rate set by the timer, for example, every 75ms, as shown in Figure 2. The timer-tick task 70 commands several subtasks, which track small package units on the feed conveyor section, capture images of the small package units, and classify the small package units. First, the timer-tick task 70 clears old subtasks 72 and prepares them for new execution. The classification processor 36 executes a small package tracking task 74 for each small package unit identification occupied area in the feed conveyor status table. In this form, the classification processor 36 has multiple cores, which can execute the small package tracking task 74 simultaneously in parallel. The task for each detected small package unit is executed in one or another of these cores, or in a dedicated thread in one core running multiple threads. By executing the parcel tracking task 74 and subsequent tasks in parallel on separate cores or threads, the conveyor system can handle high parcel throughput rates. Similarly, the control processor 34 could potentially be a multi-core processor.

[0018] As shown in more detail in Figure 4, the parcel tracking task 74 for each parcel unit on the feed conveyor section calculates the time interval since the task was last performed for that parcel unit 76. Task 74 uses the transport speed information to calculate the distance the parcel unit has advanced in the transport direction since the last update 78, and updates the coordinates of the position of that parcel unit to its current position 80.

[0019] After the location of the small package unit is updated, the multi-core classification processor 36 (Figure 1) executes an in-target task 82 in parallel, which determines when the small package unit is in the target zone 84 within the image acquisition area 24, thereby enabling the camera 22 to operate and acquire images of the small package unit within the target zone. Before executing the in-target task 82, the classification processor 36 first limits the execution of the in-target task to small package units that have not yet been imaged 86.

[0020] Each of the target narrowing tasks 82 first determines whether a package unit is within the target zone 84 (Figure 1) by comparing its coordinates to the coordinate range of the target zone 88. If it is determined that the package unit is within the target zone, the task adds the package unit to the target list 90; otherwise, it does nothing 92. The classification processor then checks the target list as shown in Figure 2 94. If one or more package units are newly added to the target list, the classification processor 36 signals the camera 22 (Figure 1) to capture an image of the package unit within the target zone 84 96. If there are no new package units in the target list, nothing is done 98, and the timer task 70 completes.

[0021] Using all the new images, the classification processor 36 executes the target object processing task 100 shown in detail in FIG. 5 in parallel. Each target object processing task 100 includes an image cropping task 102 that first calculates the relative pixel coordinates of the parcel units in the captured image. For example, the pixel coordinates of the corners of the parcel unit can be used as the relative pixel coordinates. Based on those coordinates, the image cropping task 102 then performs a coordinate rotation 106 of the image including the parcel unit to determine the orientation of the parcel unit so that effective cropping can be performed. As a result of the coordinate rotation, the coordinates of the parcel units in the captured image are rotated. The rotated captured image is then cropped into smaller rectangular regions 108, and these regions each enclose the rotated parcel unit image in the captured image. The pixels within the rectangular region define the cropped image, and they are then stored 110.

[0022] For each of the cropped images, a classification task 112 is executed in parallel by the core of the classification processor. Each cropped image includes a rectangular pixel array of RGB values of 1 byte (0 to 255). First, each classification task 112 preprocesses 114 the cropped image into a format for the specific classification program being used. For example, it may be necessary to change the dimensions of the pixel array and pad the array with zeros to fill it. Examples of classification programs include: AlexNet; Inception Network v1, v2, v3, v4 ; MobileNet; PNASNet; SqueezeNet; ResNet, etc. The pixel array is supplied as an input P to a neural network 116 (FIG. 6), and this neural network classifies 118 (FIG. 5) the cropped image of the parcel unit. Once the parcel unit is classified, the classification task stores 120 this classification in the input conveyor state table.

[0023] In this format, the classification processor uses artificial intelligence in the form of a neural network as the classifier. However, other artificial intelligence techniques, such as cascaded classifiers using Haar features, fully connected networks, convolutional neural networks, support vector machines, Bayesian neural networks, k-NN networks, Parzen neural networks, and fuzzy logic, may be used as classifiers for classifying package units. Neural network 116 in Figure 6 represents an exemplary neural network illustrating how artificial intelligence is used to classify package units. The input P is the RGB values ​​of pixels in the pixel array of a preprocessed cropped image. The input is multiplied by a weighting coefficient w, which differs with each line connecting the input P to the neurons N1 of the first layer in the hidden layer 122. Each neuron N1 of the first layer has an activation value, which is the normalized sum of the products of the input value P and the weights connected to it, plus a bias term. Normalization is achieved by a mathematical function that maps the sum to a limited range, for example, from 0 to 1. Similarly, the activation value of the first hidden layer N1 is applied to the neuron N2 of the second hidden layer. The activation value of neuron N2 of the second hidden layer is then used to calculate the output (A~E). A real neural network used to classify package units may have more than two hidden layers, with approximately six neurons shown for each hidden layer.

[0024] Outputs A to E represent various classification sets per piece of small cargo. The classification task assigns the classification with the maximum activation value among the five outputs to each piece of small cargo. For example, A = plastic bag; B = recognized single piece of small cargo that is not a plastic bag; C = stack of small cargos with a overlap less than a predetermined percentage (e.g., 25%); D = stack of small cargos with a overlap greater than the predetermined percentage; E = unrecognized small cargo. Of course, other sets of small cargo unit classifications can be used to identify other types of small cargos or characteristics of small cargos, such as surface texture, cracks or breaks, water stains, specific colors, specific visible indicators, and even wrinkled plastic bags where machine-readable indicators such as barcodes are unreadable.

[0025] The neural network classifier is trained by feeding the network a large number of clipped images of small cargo units corresponding to the output classifications. This training adjusts the weights w and biases of the neurons in each layer to minimize the cost function, that is, the difference between the desired output for the image (0 for all outputs other than the output corresponding to that image, and 1 for the output corresponding to that image) and the output calculated by the neural network. In the training process, the weights and biases are iteratively adjusted for each training image by the conventional error backpropagation method. Training is typically performed offline rather than in real time.

[0026] As shown in FIG. 5, the classification determined by the neural network for each small cargo unit is sent to the control processor 124, and invalid small cargo units and previously classified small cargo units are removed from the table of items to be classified 126. Metadata such as the clipped image, time stamp, and pixel coordinates of the small cargo unit are sent to the storage device 130 for offline analysis 128.

[0027] In Figure 1, the control processor 34 checks the position of the parcel units 13 on the feed conveyor section 12, which has been updated by the parcel tracking task of the classification processor. Once the parcel units 13 reach the sorting conveyor section 25, the control processor 34 takes over the calculation of the position of each parcel unit and the determination of its destination and trajectory. The destination of each parcel unit is determined by its classification. For example, unrecognized parcel units (classification E) and stacked parcels with more than 25% overlap (classification D) are sent to destination 30 where a human worker 32 is stationed. It is also possible that any parcel unit of any classification exceeding a predetermined maximum size can be sent to destination 30 where a person is stationed. In this configuration, the trajectory of those parcel units is straight in the transport direction 14. Polybags (Category A), recognized single packages that are not polybags (Category B), and stacked packages with less than 25% overlap (Category C) are redirected to one or the other sorting destination on the flanking conveyors 28, 29, which transport the packages downstream for further processing. To control the trajectory 132 of each package unit 13, the control processor 34 selectively operates roller belts or sections of rollers on a powered roller conveyor, or shoes or pushers on a shoe sorter to redirect the package units to their designated destinations.

[0028] The present invention has been described in detail in terms of an exemplary form using a computer processing system comprising two processors and a PLC. This system provides redundancy in the event of a failure of the sorting processor. In the event of failure, the control processor can sort the packages based solely on the area occupied by each package unit, without the additional benefit of the packages being sorted. Alternatively, if the control processor fails, the PLC can simply sort the packages until the control processor recovers, creating a balanced flow on two flanking conveyors temporarily staffed by human workers. However, other forms are possible. For example, the control processor and sorting processor can be implemented by a single multi-core processor within the computer processing system. Another example is that the sorting processor does not necessarily have to be a multi-core processor executing task instructions in parallel on individual cores or threads. The sorting processor could potentially be a single-core processor executing one task instruction at a time.

[0029] In an alternative configuration, the camera's image acquisition area may be positioned along the feed conveyor section upstream of the occupied area detector. In this case, the classification processor may perform the task of continuously acquiring images and storing the acquired images with timestamps in an image table in the processor's volatile memory. The control processor may detect occupied areas with an occupied area detector downstream of the image acquisition area. Using information on the feed conveyor speed and the distance from the image acquisition area to the occupied area detection position, the classification processor may associate each occupied area with the package unit image contained in the acquired images in the image table. The classification processor may then rotate, crop, preprocess, and classify the package units as shown in Figure 5 before they leave the feed conveyor section and enter the sorting conveyor section.

[0030] Furthermore, to detect the occupied area or position of a parcel unit, the imaging system can be used as an occupied area detector, eliminating the need for a separate, dedicated occupied area detector. The classification processor may perform tasks beyond those necessary to classify parcel units from their images, such as acting as a parcel unit detector, which involves recognizing individual parcel units as they move along the feed conveyor section, tracking their positions, and sorting them to their assigned destinations.

Claims

1. A feed conveyor section that transports small packages downstream in the transport direction at a transport speed; An occupied area detector is positioned at a detection location along the feed conveyor section; A camera is positioned downstream of the detection position along the feed conveyor section to capture images of the capture area on the feed conveyor section; A sorting conveyor section that receives the small package units from the aforementioned feed conveyor section and selectively sorts the small package units to multiple destinations; A computer processing system including a control processor and a classification processor that communicates with the control processor; A conveyor system including, The control processor executes program instructions that operate the occupied area detector to detect the occupied area of ​​the small package unit as it moves forward in the transport direction past the detection position, and program instructions that control the sorting conveyor section and the feed conveyor section. The classification processor provides program instructions to track the position of each of the small package units on the feed conveyor section based on the occupied area detected at the detection position as they move forward in the transport direction; A program command to control the camera to capture an image of the package unit and generate one or more cropped images corresponding to each of the package units from the images within the capture area, when the calculated position of the package unit is within the target zone of the capture area; and A program instruction that uses artificial intelligence functioning as a classifier to classify the cropped image into one of several categories and assign that category to each of the parcel units; Execute, A conveyor system in which the control processor controls the sorting conveyor section to sort the parcel units to destinations according to the classification assigned by the classification processor.

2. The conveyor system according to claim 1, wherein the classification processor classifies the cropped images in a neural network trained to recognize various small package units, and transmits the classification to the control processor.

3. The conveyor system according to claim 1, wherein the control processor includes a virtual or separate programmable logic controller for controlling the sorting conveyor section and the feed conveyor section.

4. The conveyor system according to claim 1, wherein the classification processor is a multi-core processor having a plurality of cores that execute the program instructions in parallel.

Images