Apparatus and method for reception and validation of returnable objects

Abstract

The present disclosure relates to an apparatus for the reception and validation of returnable objects and a method for the reception and validation of an object as to be accepted or rejected. The present disclosure relies on the use of a camera module and a light source to extract features of an object such as texture, shape, colour, and appearance, alongside other sensors such as load cell and metal detection to validate if an object should be accepted and passed into a reception chamber to further processing. The present disclosure could be used as part of a reverse vending machine (RVM) or on smart bins which allow the reception of objects meeting certain criteria.

Description

APPARATUS AND METHOD FOR RECEPTION AND VALIDATION OF RETURNABLE OBJECTSBACKGROUND

[0001] The present disclosure relates to returnable objects, and particularly to validation of returnable objects. More particularly, the present disclosure relates to an apparatus and method for reception and validation of returnable objects.SUMMARY

[0002] According to the present disclosure, there is provided an apparatus for the reception and validation of returnable objects, such as recyclable objects and reusable containers, comprising:

[0003] - an enclosed sensing area having an opening for the insertion of the returnable objects and a sensing plate for receiving the returnable objects;

[0004] - a camera module having a camera with a field of view covering the sensing plate and being configured to generate and provide an image of the returnable object being positioned on the sensing plate, the image containing pixels with each pixel having values representing the colour and / or the brightness of the respective pixel;

[0005] - a light source pointing towards the sensing plate ensuring sufficient lightning of the returnable object being positioned on the sensing plate;

[0006] - at least one sensor being configured to determine physical properties of the returnable object being received by the sensing plate and to provide signals characterizing the physical properties;

[0007] - a central processing unit being in communication with the camera module and the at least one sensor, wherein the central processing unit is operable to process the images provided by the camera module and the signals provided by the at least one sensor, wherein the central processing unit comprises a machine learning algorithm running at least one convolutional neural network, the at least one convolutional neural network being configured to process the images from the camera module, wherein the central processing unit is configured to specify, based on theprocessing of the images by the at least one convolutional neural network and the signals provided by the at least one sensor, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected.

[0008] Throughout this specification the term “returnable object” refers to any object that has to be taken after the object has served a purpose. Examples of returnable objects include but are not limited to recyclable objects such as bottles, cans and tetra packs; and reusable containers such as coffee cups, lunch boxes and bowls.

[0009] The approach of the current present disclosure is simple, requires no manipulations of the scanned objects and uses standard computing elements which are available off-the-shelf, in order to validate the receipted object with high accuracy. Thereby, only the images from the camera modules are needed, without any preprocessing or enhancement of the images. The present disclosure just relies on the visual appearance to obtain metrics from the processed item, such as type, depth, contour, in combination with some of its physical properties determined by the additional sensor. Consequently, any kind of object can be recognised and validated, e.g., transparent and non-transparent materials. Further, crushed objects can be distinguished from uncrushed objects by processing the 3D reconstruction. So, the present disclosure focuses on the smart usage of the convolutional neural network and the sensor signals to process the information available, avoiding the need to transmit or take into account external data. Due to the flexibility of the convolutional neural network the validation is easily extendable to various objects to be returned.

[0010] In further embodiments of the present disclosure the apparatus might further include:

[0011] - A single camera with a field of view covering the entire sensing area and providing colour or grayscale imagery.

[0012] - A single light source located behind the camera to ensure there is enough light to visually distinguish the scanned objects.

[0013] - A load cell to measure the weight of the scanned objects located under the scanning area.

[0014] - A metal detector to measure the presence of a metallic object in the sensing area located at the entrance of the sensing area.

[0015] In one embodiment of the present disclosure, the light source and the camera are placed in a way such that the field of view of the camera covers the entire sensing area and the light source does not blind the camera.

[0016] In another embodiment of the present disclosure, the sensing area is defined as a plate where the returned object sits while it is scanned. This plate sits on top of a load cell that is used to measure the weight of the object.

[0017] In another embodiment of the present disclosure, the metal detector is placed in a way such that the returned object has to pass through the metal detector.

[0018] In another embodiment of the present disclosure, the CPU is configured to process the data from the camera and other sensors.

[0019] In another embodiment of the present disclosure, the CPU comprises a machine learning algorithm to extract information from the scanned object such as outline, type of object and 3D shape.

[0020] In another embodiment of the present disclosure, the machine learning algorithm uses convolutional neural networks (CNNs) to extract semantic and depth information from the scanned object.

[0021] In another embodiment of the present disclosure, the CPU extracts optical flow from the camera feed to analyse the motion of the scanned object.

[0022] Further, the present disclosure relates to a method for reception and validation of returnable objects comprising the steps of:

[0023] - receipt of the returnable object and positioning of the returnable object on the sensing plate such that a light source pointing towards the sensing plate lightens the returnable object;

[0024] - generating and providing an image of the returnable object being positioned on the sensing plate using a camera module;

[0025] - determining physical properties of the returnable object being positioned on the sensing plate via at least one sensor and providing sensor signals characterizing the determined physical properties of the returnable object;

[0026] - processing the images provided by the camera module via a convolutional neural network and the signals provided by the at least one sensor,

[0027] - specifying, based on the processing of the images by the at least one convolutional neural network and the signals provided by the at least one sensor, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected.

[0028] In another embodiment of the present disclosure the method includes the steps of:

[0029] - Processing the camera imagery to estimate the type, shape and size of the scanned object on real time;

[0030] - Processing data from the metal detector and load cell data;

[0031] - Comparing the data from the sensors to a database of valid objects;

[0032] - Flagging the object as accepted if the sensor data matches the information on the database;

[0033] - Processing the camera images and producing a semantic segmentation of the scanned object and a 3D representation of the scanned object;

[0034] - Processing the 3D representation of the object and other sensor data against the information on the database and accepting the object if it matches certain criteria;

[0035] - Accepting the object by allowing it to travel to the reception chamber;

[0036] Additional features of the present disclosure will become apparent to those skilled in the art upon consideration of illustrative embodiments exemplifying the best mode of carrying out the disclosure as presently perceived.BRIEF DESCRIPTIONS OF THE DRAWINGS

[0037] The detailed description particularly refers to the accompanying figures in which:

[0038] Figure 1 is a schematic view of an apparatus for reception and validation of returnable objects and

[0039] Figure 2 shows a process flow illustrating the method of the present disclosure.DETAILED DESCRIPTION

[0040] Referring to the drawings there is illustrated an apparatus for the reception and validation of returnable objects 6 according to the present disclosure generally by the reference numeral 1. The apparatus 1 comprises an enclosed sensing area 3, an opening 2 where objects 6 enter the sensing area 3, a camera module 5 with a wide angle lens that allows the camera module 5 to view the entire sensing plate 10 being positioned in the sensing area 3, a light source 8 illuminating the sensing area 3, a metal detector 7 placed at the opening 2, a load cell 4 placed under the sensing plate 10. A central processing unit (CPU) 9 is in communication with the camera module 5, metal detector 7 and load cell 4 and is operable to process the data received from the camera module 5 and sensors 4, 7 to deem the object 6 in the sensing area 3 as accepted or rejected.

[0041] The camera module 5 is operable to constantly monitor the sensing plate 10 for new objects 6 placed in the sensing area 3, in particular, without further user activation. The images from the camera module 5 are recorded and processed by the CPU 9 producing an image segmentation and 3D representation of the scanned object 6. Thus, just the images, e.g., RGB images or grayscale images, containing pixels with each pixel having values for the colours (e.g. RGB values) and / or the brightness is transmitted to and processed by the CPU 9.

[0042] The process flow is outlined in Figure 2 and begins with the camera module 5 detecting an object 6 placed on the sensing plate 10 in the sensing area 3. The CPU 9 then uses the images from the camera module 5 and runs a first convolutional neural network (CNN), e.g., implemented in a machine learning algorithm on the CPU 9, to obtain a segmented image just out of the images provided by the camera module 5.

[0043] Thereby, the image to be analyzed is fed as input into the first convolutional neural network. In Convolutional Layers (CL) of the first convolutionalneural network, filters (also called kernels) are applied to the image to be analyzed. The filters perform a mathematical operation called convolution, which detects patterns or features like edges, textures, or simple shapes. The respective filter preferably slides over the image in steps (e.g., a certain pixel window), producing a feature map that highlights where specific features are located in the image. In several of such convolutional layers, different filters are used in order to extract patterns or features of different complexity in the respective image.

[0044] Combining and processing several of these convolutional layers within the first convolutional neural network according to suitable methods, a segmented image is obtained, where the outline of the scanned object 6 is highlighted and a semantic class is assigned to the scanned object 6. In this context, assigning a semantic class to the scanned object 6 is to be understood as obtaining a label, in particular, for each pixel of the image to be analyzed, wherein the label indicates to which semantic class this pixel belongs, e.g., bottle, can, container, tetra pack, coffee cup, lunch box, bowl, hands of the user, background, and the like. It is easily possible to add further classes, depending on the specific application of the apparatus 1.

[0045] A separate second convolutional neural network (CNN), e.g., also implemented in the machine learning algorithm on the CPU 9, is used to extract a 3D reconstruction of the object 6, as well just out of the images provided by the camera module 5. The obtained 3D reconstruction is used to identify if the object 6 is damaged and to extract the dimensions of the object 6. Consequently, additional information about the depth or contour of the respective object 6 can be estimated. This information can be used to differentiate a crushed object from an uncrushed object, for example, or even to distinguish between different objects, e.g., based on their dimensions.

[0046] The 3D reconstruction of the object 6 is preferably done via image recognition and image processing referred to as “single-view 3D-reconstruction”, where the 3D shape is predicted from a single 2D image. This prediction can be achieved by deep learning models that are trained on large datasets of 3D objects and their corresponding 2D images. However, other models or techniques may be used as well.

[0047] The CPU 9 then uses the semantic class provided by the first convolutional neural network alongside with the dimensions provided by the second convolutional neural network via the 3D reconstruction and readings from the load cell 4 and the metal detector 7 to validate if the object 6 is to be flagged accepted or rejected. Further data or information from additional sensors might be used by the machine learning algorithm or the CPU 9 to validate the scanned object 6.

[0048] Thereby, the CPU 9 matches the processed data (semantic class, 3D reconstruction, dimension, etc) and the read data from the load cell 4 and metal detector 7 and additional sensors (if applicable) assigned to the returned object 6 against a database of valid data for a given object 6 of the respective semantic class and dimension. If the CPU 9 specifies that the processed and read data, which is based on the obtained measurements, are within some given threshold the object 6 is deemed as accepted and passed to a reception chamber. Otherwise, it is flagged as rejected and returned to the user.

[0049] Comparative work relies on the use of a single camera, however it requires complex lighting setups making use of sub light sources with different spectral compositions to gather information from the processing object and optical beam splitters. This requires additional computing power and requires frequent cleaning of the surfaces compared with a single camera with a single light source.

[0050] The comparative work requires the scanned objects to be placed on a specific arrangement of “bottom first” and requires dedicated circuitry for handling the video data.

[0051] These comparative approaches, although workable, are cumbersome, complex and require frequent maintenance of the scanners resulting in increased costs.

Claims

CLAIMS1. An apparatus for the reception and validation of returnable objects, comprising: an enclosed sensing area having an opening for the insertion of the returnable objects and a sensing plate for receiving the returnable objects; a camera module having a camera with a field of view covering the sensing plate and being configured to generate and provide an image of the returnable object being positioned on the sensing plate, the image containing pixels with each pixel having values representing the color and / or the brightness of the respective pixel; a light source pointing towards the sensing plate ensuring sufficient lightning of the returnable object being positioned on the sensing plate; at least one sensor being configured to determine physical properties of the returnable object being received by the sensing plate and to provide signals characterizing the physical properties; a central processing unit being in communication with the camera module and the at least one sensor, wherein the central processing unit is operable to process the images provided by the camera module and the signals provided by the at least one sensor, wherein the central processing unit comprises a machine learning algorithm running at least one convolutional neural network, the at least one convolutional neural network being configured to process the images from the camera module, wherein the central processing unit is configured to specify, based on the processing of the images by the at least one convolutional neural network and the signals provided by the at least one sensor, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected.

2. The apparatus of claim 1, wherein the at least one convolutional neural network is configured to assign a semantic class to the returnable object contained in the image provided by the camera module and to extract a 3D reconstruction of the returnable object contained in the image provided by the camera module, wherein thecentral processing unit is configured to specify, based on the semantic class, the 3D reconstruction and the signals provided by the at least one sensor, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected.

3. The apparatus of claim 2, wherein the central processing unit is configured to match the semantic class, the 3D reconstruction and the signals from the at least one sensor against a database of valid data for a given returnable object to validate, whether the returnable object being positioned on the sensing plate is to be flagged accepted or rejected.

4. The apparatus of claim 2 or 3, wherein the machine learning algorithm contains a first convolutional neural network, being configured to assign the semantic class to the returnable object contained in the image provided by the camera module.

5. The apparatus according to any one of the claims 2 to 4, wherein the machine learning algorithm contains a second convolutional neural network, being configured to extract the 3D reconstruction of the returnable object contained in the image provided by the camera module.

6. The apparatus according to any one of the claims 2 to 5, wherein the at least one convolutional neural network being configured to assign the semantic class “bottle” or “can” or “container” or “tetra pack” or “coffee cup” or “lunch box” or “bowl” to the returnable object contained in the image.

7. The apparatus according to any one of the preceding claims, wherein the at least one sensor is a load cell being operatively connected to the sensing plate, the load cell being configured to measure the weight of the returnable object oncereceived on the sensing plate, and the load cell being configured to provide signals characterizing the weight of the returnable object.

8. The apparatus according to any one of the preceding claims, wherein the at least one sensor is a metal detector located in the sensing area, e.g., next to the opening, the metal detector being configured to determine if a returnable object inserted into the sensing area is metallic, and the metal detector being configured to provide signals characterizing whether the returnable object is metallic or not.

9. The apparatus according to any one of the preceding claims, wherein the camara module comprises a single camera.

10. The apparatus according to any one of the preceding claims, wherein the light source is located behind the camera module in respect to its field of view.

11. The apparatus according to any one of the preceding claims, wherein the central processing unit is configured to extract optical flow from the images provided by the camera module for analysing the motion of the scanned object.

12. A method for the reception and validation of returnable objects comprising the steps of: receiving the returnable object and positioning of the returnable object on the sensing plate such that a light source pointing towards the sensing plate lightens the returnable object; generating and providing an image of the returnable object being positioned on the sensing plate using a camera module;determining physical properties of the returnable object being positioned on the sensing plate via at least one sensor and providing sensor signals characterizing the determined physical properties of the returnable object; processing the images provided by the camera module via a convolutional neural network and the signals provided by the at least one sensor; and specifying, based on the processing of the images by the at least one convolutional neural network and the signals provided by the at least one sensor, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected.

13. The method of claim 12, wherein the processing of the images by the at least one convolutional neural network includes assigning a semantic class to the returnable object contained in the image provided by the camera module and to extract a 3D reconstruction of the returnable object contained in the image provided by the camera module, wherein specifying, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected is carried out based on the semantic class, the 3D reconstruction and the signals provided by the at least one sensor.

14. The method of claim 13, wherein specifying, whether the returnable object being positioned on the sensing plate is considered as accepted or rejected, includes matching or comparing the semantic class, the 3D reconstruction and the signals from the at least one sensor against a database of valid data for a given returnable object.

15. The method according to claim 13 or 14, wherein assigning a semantic class to the returnable object contained in the image provided by the camera module is carried out by a first convolutional neural network and extracting a 3D reconstruction of the returnable object contained in the image provided by the camera module is carried out by a second convolutional neural network.

16. The method according to any one of the claims 12 to 15, wherein extracting a 3D reconstruction of the returnable object contained in the image provided by the camera module includes estimating a depth or a contour or a dimension of the returnable object.

17. The method according to any one of the claims 12 to 16, wherein assigning a semantic class to the returnable object contained in the image provided by the camera module includes obtaining a label for each pixel of the image to be analysed, wherein the label indicates to which semantic class this pixel belongs, e.g., bottle, can, container, hands of the user, background, and the like.

Images