Method and system for fraud detection during isolation of returnable objects

Abstract

The present invention relates to a method and system for the detection of fraud in the form of the removal of a returnable object while it is travelling from a validation area to an acceptance region The method and system rely on the usage of a (single) camera and a light source to estimate the optical flow live from the images taken by the camera and then uses flow vectors to identify if the returnable object has reached the acceptance region or isolation area or it has been removed by a user. The invention could be used as part of a reverse vending machine (RVM) or on smart bins accepting objects, such as containers, matching certain criteria to prevent fraud.

Description

[0001] Method and system for fraud detection during isolation of returnable objects

[0002] The present invention relates to a method and system for the detection of fraud in the form of the removal of a returnable object while it is travelling from a validation area to an acceptance region. The method and system rely on the usage of a (single) camera and a light source to estimate the optical flow live from the images taken by the camera and then uses (optical) flow vectors to identify if the returnable object has reached the acceptance region or isolation area or it has been removed by a user. The invention could be used as part of a reverse vending machine (RVM) or on smart bins accepting objects, such as containers, matching certain criteria to prevent fraud.

[0003] Background

[0004] The relevance of the current invention is highlighted on reverse vending machines operating autonomously to receive returnable objects as part of a deposit-return-scheme. In these schemes, a reverse vending machine may be located in an area with limited monitoring. A fraudulent user may attempt to take advantage of this limited monitoring and try to get a deposit for an object that was not returned. Some deposit return schemes may use special marks on the object to validate its authenticity. Optionally they may use some form of devaluation method, such as compaction, or store the object without any further processing; those steps, although effective, require the intermediate step of isolating the returnable object from the user.

[0005] Through this specification, isolation refers to the action of making the returned object inaccessible to the user while it is devalued or moved to another location.

[0006] The isolation step is a transition between a receiving area of the returnable object and the acceptance region. One example of an isolation step is the sliding of a tray where the returnable object is sitting during valuation, allowing the object to displace via gravity into an acceptance region situated under the receiving area as shown in Figure 1. Other mechanisms such as rollers, conveyor belts or doors that move the returned object away from the user could also be part of the isolation step. The reverse vending machines must be capable of detecting on its own, when a user interferes with the isolation of an object that has been deemed as accepted. Then it should void the return or show a warning to the user. Prior art, such as EP 3 360 100 A1 rely on the signal from a compactor load sensor to verify that the object has been compacted. However, if the reverse vending machine has no compactor, the load sensor signal is not present.

[0007] The work of EP 2 538 394 A1 uses a feature extraction algorithm based on edge information to estimate the velocity of the returned object. Thereby, it is evaluated if there is any inconsistency in the position or motion of one feature on a returned object with respect to another feature on the same returned object. Although effective in scenarios where the main motion is produced by the accepted object, the edge information is unreliable for cases where multiple objects move in different directions at the same time as part of the accepting process resulting in a constant changing of the image gradients.

[0008] Summary of the invention

[0009] According to the present invention, there is provided a system for fraud detection during isolation of a returned object, such as recyclable objects and reusable containers, comprising: a receiving area or validation area for the positioning of said returned object, the receiving area having an opening for the insertion of said returned object, wherein the returned object is validated to be accepted or rejected in this receiving area; an acceptance region or isolation region where the returned object can be devalued or stored or moved to another location, while it is inaccessible to the user; a transfer device between the receiving area and the acceptance region, the transfer device being adapted or configured to transfer the returned object from the receiving area to the acceptance region; a camera module having at least one camera with a field of view covering the (entire) receiving area and being adapted or configured to generate and provide images (colour or grayscale imagery) containing the returned object being positioned in the receiving area, the images each containing pixels with each pixel having values representing the colour and / or the brightness of the respective pixel; optionally a single or multiple light sources located around the camera to ensure there is enough light to visually distinguish the returned object being positioned in the receiving area; optionally additional sensors to gather information about the returned object, examples of those sensors are: ultrasound, metal detection, weight and infrared; a central processing unit being in (direct or indirect) communication with the camera module responsible for the processing of the images from the camera and the sensors; an optical flow algorithm on the central processing unit being adapted or configured to produce multiple (optical) flow vectors for the receiving area, at least while the returned object is displaced to the acceptance region, wherein these multiple flow vectors each represent the movement of a specific pixel (or a group of pixels) between at least two consecutive images provided by the camera module, thereby representing pixel-wise motion, wherein at least one of these multiple flow vectors is representing the movement of a pixel (or a group of pixels) associated to the returned object for estimating or analysing the actual motion of the returned object, wherein the optical flow algorithm or the central processing unit is further adapted or configured to:

[0010] - determine, based on the flow vectors, in particular several of such flow vectors, if there is a motion of the returned object towards the opening of the receiving area and / or if there is a motion of the returned object towards to the acceptance region, and

[0011] - flag the motion of the returned object as fraud if the flow vectors, in particular several of such flow vectors, indicate that the returned object is leaving the receiving area via the opening and / or that the motion of the returned object is not directed to the acceptance region; and an optional machine learning algorithm to segment the representation of the returned object contained in the camera images using machine learning, e.g., using a convolutional neural network.

[0012] Throughout this specification the term “returned object” or “returnable object” refers to any object that has to be taken after the object has served a purpose. Examples of returnable objects include but are not limited to recyclable objects such as bottles, cans and tetra packs; and reusable containers such as coffee cups, lunch boxes and bowls.

[0013] The approach of the present disclosure is simple, as it only requires camera information to track the journey of the returned object once it is validated and is displaced to the acceptance region. Secondary data such as compactor load, weight sensor, infrared or other sensors could be used alongside the camera information, however they are deemed as optional. This allows for automated decision making based on the camera and sensor data.

[0014] Contrary to the known prior art, by using the optical flow or the multiple flow vectors of several pixels in the receiving area, it can be clearly distinguished between the movement of different objects in the receiving area and movement of these different objects into different directions at the same time by analysing the optical flow or the several flow vectors. This allows to securely determine a fraudulent movement of the returned object, by identifying, based upon several flow vectors associated to the receiving area, if the returned object is moving in the direction of the opening or entrance or not. Thus, the fraudulent behaviour is determined by considering the motion of the returned object in relation to its surrounding, which is evaluated based on the multiple flow vectors corresponding to the (entire) receiving area.

[0015] In an embodiment of the invention, the optical flow algorithm or the central processing unit is further configured to flag the motion of the returned object as fraud by processing the flow vectors characterizing the optical flow on the receiving area considering a motion of the transfer device for transferring the returned object from the receiving area to the acceptance region. This makes the fraud detection more secure as the known movement of the transfer device is taken into account.

[0016] In one embodiment of the invention, the light source and the camera are placed in a way such that the field of view of the camera covers the entire receiving area where the returned object sits and the light source does not blind the camera.

[0017] In another embodiment of the invention, the receiving area includes or comprises a sliding tray as a transfer device where the returned object sits while it is scanned by the camera for validation purposes. Other forms of the transfer device in the receiving area may include or comprise a conveyor belt, a door or rollers.

[0018] In another embodiment of the invention, the sliding tray in the receiving area can slide to allow the returned object to pass via gravity to the acceptance region. In further embodiments, the driven conveyor belt, the door and / or the rollers carry the returned object to the acceptance region.

[0019] In another embodiment of the invention, the acceptance region is located out of the reach of the user when the returned object is positioned in the acceptance region, e.g., it is located more than 2 meters away from the opening. This prevents removing the returned object by fraud after it has reached the acceptance region.

[0020] In another embodiment of the invention, the central processing unit is configured to process the data from the camera and any additional sensors of the system. This allows to provide a more reliable decision on a fraudulent behaviour. In another embodiment of the invention, the central processing unit optionally uses the machine learning algorithm to locate the returned object across the images on the camera feed and / or to filter the flow vectors used to determine the motion of the returned object. This allows to identify the returned object or the flow vectors linked to it in a more secure way.

[0021] Further, according to the present invention, a method for fraud detection during isolation of returnable objects is provided, e.g., by using the system according to the invention, comprising the steps of: receiving the returned object through an opening (or bringing the system into a receiving state) for positioning of the returned object in a receiving area; transferring the returned object from the receiving area to an acceptance region via a transfer device, e.g., one the returned object has been validated in the receiving region; generating and providing at least two consecutive images of the returned object or a camera feed of several images using a camera module while the returned object is intended to be transferred from the receiving area to the acceptance region via the transfer device, e.g., once it has been validated as a permissible object; determine multiple flow vectors for the receiving area while the returned object is intended to be transferred from the receiving area to the acceptance region, wherein these multiple flow vectors each represent the movement of a specific pixel or a group of pixels between at least two consecutive images provided by the camera module, wherein at least one of these multiple flow vectors is representing the movement of a pixel or a group of pixels associated to the returned object while the returned object is intended to be transferred from the receiving area to the acceptance region for estimating the actual motion of the returned object; determine, based on the flow vectors, if there is a motion of the returned object towards the opening of the receiving area and / or if there is a motion of the returned object towards to the acceptance region, and flag the motion of the returned object as fraud if the flow vectors indicate that the returned object is moving towards the opening of the receiving area and / or that the motion of the returned object is not directed to the acceptance region.

[0022] Optionally, object detection can be used to verify that the flow vectors correspond to an object of the same type as the one being accepted.

[0023] In another embodiment, the method includes the steps for: Processing the camera frames in real time and estimate the optical flow from the live feed from the camera.

[0024] Filtering the flow vectors to determine the ones corresponding to the returned object and the ones corresponding to other motion.

[0025] Flag the returnable object as pulled out if the container left the receiving area through the entrance.

[0026] The features of the system according to the invention can also be implemented in the method, and vice versa.

[0027] Brief description of the drawings

[0028] The invention will be more clearly understood by the following description of some of the embodiment thereof, given by a way of example only, with reference to the accompanying drawings. In which:

[0029] Figure 1 is a schematic view of the receiving area and acceptance region on the reverse vending machine;

[0030] Figure 2A shows the sliding tray when the returned object is scanned;

[0031] Figure 2B shows the sliding tray when the tray is open to accept the returned object and the object travels to the acceptance region;

[0032] Figure 2C shows the sliding tray as it is closed once the object is in the acceptance region;

[0033] Figure 3A shows the flow vectors when the sliding tray is sliding and the object is travelling towards the acceptance region;

[0034] Figure 3B shoes the flow vectors when the sliding tray is sliding and the object is removed by a user while it is travelling towards the acceptance region;

[0035] Figure 4 shows a process flow illustrating the method of the invention;

[0036] Referring to the drawings there is an illustrated sample of a system 1 for fraud detection during isolation of a returned object 4. Through this specification, isolation refers to the action of making the returned object 4 inaccessible to the user while it is devalued or moved to another location. Examples of returned objects 4 include but are not limited to recyclable objects such as bottles, cans, and tetra packs; and reusable containers such as coffee cups, lunch boxes and bowls.

[0037] The system 1 comprises - a receiving area 3 or validation area where the returned object 4 is validated, e.g., against information on an existing database of returnable objects;

[0038] - a sliding tray 2 being positioned in or being part of the receiving area 3 where the returned object 4 sits while it is validated;

[0039] - an acceptance region 8 where the returned object 4 can be devalued or stored as required;

[0040] - a camera module 5 having at least one camera, preferably a single camera, with a wide- angle field of view that allows the camera module 5 to generate images showing the entire receiving area 3, in particular, the entire returned object 4 being positioned in the receiving area 3;

[0041] - a light source 6 illuminating the receiving area 3 and an opening 7 deemed as the entrance for the returned objects 4 for allowing the user to place the returned object 4 in the receiving area 3.

[0042] The camera module 5 comprises a high-speed camera operable to constantly grab frames or images containing the returned object 4 and feed them to a central processing unit 9 (CPU) of the system 1. Using an optical flow algorithm, the central processing unit 9 is responsible for estimating the optical flow out of a sequence of images from the camera module 5 in real time producing flow vectors V for the receiving area 3. This allows estimating the motion of depicted objects, in particular the returned objects 4, between consecutive images (frames) of the image sequence. Thereby, the image sequence under consideration contains at least two such images from the camera module 5.

[0043] Estimating or analysing the optical flow by the optical flow algorithm on the central processing unit 9 relies on techniques like image processing and computer vision, known to the skilled in the art, considering the motion at the level of individual pixels, e.g., by using "Dense Optical Flow". Thereby, optical flow measures how the image intensities (brightness and color) of the respective pixels change from one image to the next, which indicates movement in the image area under consideration. This helps understand the speed and direction of moving objects or scenes in the sequence of images.

[0044] The flow vectors V resulting from this optical flow analysis represent the movement of the pixels between two consecutive images (frames) of the image sequence under consideration. The calculation of the flow vectors V is typically done via motion estimation within the optical flow algorithm, known to the skilled in the art.

[0045] Figure 2A, 2B, 2C show a sample setup for accepting the returned object 4, wherein in Fig. 2A the user placed the returned object 4 in the receiving area 3, in this exemplary embodiment onto the sliding tray 2, through the opening 7 for item validation. During normal operation, the subsequent opening of the sliding tray 2 once the item validation is completed produces a motion that results in flow vectors V (from optical flow analysing as described above) similar to Figure 3A, where the vertically aligned flow vectors V are mainly caused from the motion of the sliding tray 2. Opening the sliding tray 2 automatically causes a transfer of the returned object 4 into the acceptance region 8 as shown in Fig. 2B. In the shown embodiment, motion of the returned object 4 moving into the acceptance region 8 is essentially aligned perpendicular to the motion of the sliding tray 2. Thus, flow vectors V representing the motion of the returned object 4 are oriented into the plane when illustrated similar to Figs. 3A, 3B.

[0046] If a user commits fraud, the removal of the returned object 4 out of the receiving area 3 produces flow vectors V as exemplarily shown in Figure 3B, where the flow vectors V are mainly caused from the motion of the returned object 4 leaving the receiving area 3 or validation area through the opening 7 resulting in horizontally aligned flow vectors towards the entrance. The flow vectors V during the removal of the returned object 4 in fraud can be mixed in case the removal takes place at the same time as opening or closing of the sliding tray 2, as can be seen from the vertically aligned flow vectors V in Fig. 3B. In this case, the sliding tray 2 moves in one direction and the returned object 4 moves in a different direction, e.g., perpendicular to the movement of the sliding tray 2.

[0047] Thereby, the sliding tray 2 is just an exemplary embodiment of a transfer device T that is adapted to transfer the returned object 4 from the receiving area 3 into the acceptance region 8. Other forms of the transfer device T include conveyor belts (not shown), a door (not shown) or rollers (not shown). Preferably, the sliding tray 2 (or any other form of transfer device T) is always assumed to be moved perpendicularly to the movement of a returned object 4 entering or leaving the receiving area 3 (through the entrance in fraud or into the acceptance region 8 for devaluating or storing). This helps to clearly distinguish their movements or flow vectors V in the images. However, the movement of the sliding tray 2 (or any other form of transfer device T) may also be oriented into the direction of the opening 7, wherein, in this case, the calculation of the flow vectors V from the images may additionally include a segmentation step via image processing in the central processing unit 9 to differentiate which flow vectors V belong to the returned object 4 and which one belong to the sliding tray 2 (or the transfer device T).

[0048] By considering the known “reference” movement of the sliding tray 2 (or the transfer device T), or the respective “reference” flow vectors V, the fraudulent movement of the returned object 4 can be detected by the central processing unit 9 during processing / analysing the respective flow vectors V. Also, other typical or atypical movements or reactions can be taken into account when assessing fraudulent behaviour. Thereby, the central processing unit 9 can also make use of a convolutional neural network (CNN) to filter or classify the flow vectors V, making the detection of a fraudulent movement of the returned object 4 more reliable.

[0049] An exemplary process flow is illustrated in Figure 4 and begins with a returned object 4 to be scanned, e.g., by the camera module 5, and validated, whether it is to be accepted or not. Thereby, signals from additional sensors, gathering information about the returned object 4 can be taken into account, e.g., ultrasonic sensors, metal detectors, weight sensors or infrared sensors.

[0050] If the central processing unit 9 validates that the returned object 4 is valid, the accepting process starts. In the exemplary embodiment in Figure 1 , the sliding tray 2 opens (Fig. 2B) to allow the displacement of the returned object 4 into the acceptance region 8. During this state, optical flow is calculated across the live frames provided by the high-speed camera 5. The optical flow can be calculated with frame skipping (skip frames to reduce the complexity of the motion) to enhance robustness when dealing with large motions. However, this is not required. If required, the convolutional neural network implemented on the central processing unit 9 (within a machine learning algorithm) for object detection or segmentation can be used to filter the flow vectors V used to determine the motion of the returned object 4.

[0051] The flow vectors V are analysed by the central processing unit 9, for example, and then if they correspond to the motion of a returned object 4 being displaced in the direction of the opening 7 of the receiving area 3, fraud is flagged and the associated deposit is not credited to the user. In any other case, the user is credited for the returned object 4. Once the sliding tray 2 is fully closed, the system 1 is ready to process the next returned object 4.

Claims

1. A system (1) for fraud detection during isolation of a returned object (4), comprising: a receiving area (3) for the positioning of said returned object (4), the receiving area(3) having an opening (7) for the insertion of said returned object (4); an acceptance region (8) where the returned object (4) can be devalued or stored; a transfer device (T) between the receiving area (3) and the acceptance region (8), the transfer device (T) being adapted to transfer the returned object (4) from the receiving area (3) to the acceptance region (8); a camera module (5) having at least one camera with a field of view covering the receiving area (3) and being adapted to generate and provide images containing the returned object (4) being positioned in the receiving area (3), the images each containing pixels; and- a central processing unit (9) being in communication with the camera module (5), wherein the central processing unit (9) comprises an optical flow algorithm being adapted to determine multiple flow vectors (V) for the receiving area (3), wherein these multiple flow vectors (V) each represent the movement of a specific pixel or a group of pixels between at least two consecutive images provided by the camera module (5), wherein at least one of these multiple flow vectors (V) is representing the movement of a pixel or a group of pixels associated to the returned object (4) for estimating the actual motion of the returned object(4), wherein the optical flow algorithm or the central processing unit (9) is further adapted to:- determine, based on the flow vectors (V), if there is a motion of the returned object (4) towards the opening (7) of the receiving area (3) and / or if there is a motion of the returned object (4) towards to the acceptance region (8), and- flag the motion of the returned object (4) as fraud if the flow vectors (V) indicate that the returned object (4) is moving towards the opening (7) of the receiving area (3) and / or that the motion of the returned object (4) is not directed to the acceptance region (8).

2. The system (1) according to claim 1 , wherein it further comprises:- a light source (6) pointing towards the receiving area (3) ensuring lightning of the returned object (4) being positioned in the receiving area (3), the light source (6) preferably being positioned such that the light source (6) does not blind the at least one camera of the camera module (5), e.g., being positioned out of the field of view of the at least one camera.

3. The system (1) according to claim 1 or 2, wherein it further comprises:- at least one additional sensor being adapted to determine physical properties of the returned object (4) being positioned in the receiving area (3) and to provide signals characterizing said physical properties.

4. The system (1) according to claim 3, wherein the central processing unit (9) is adapted to receive the signals from the at least one additional sensor characterizing said physical properties and to process these signals as well as the images provided by the camera module (5) to validate if the returned object (4) being positioned in the receiving area (3) is to be accepted and to be moved to the acceptance region (8) or to be rejected.

5. The system (1) according to any one of the preceding claims, wherein the camera module (5) is oriented such that the field of view of the camera covers the entire receiving area (3).

6. The system (1) according to any one of the preceding claims, wherein the optical flow algorithm of the central processing unit (9) is adapted to calculate the optical flow or the flow vectors (V) with frame skipping.

7. The system (1) according to any one of the preceding claims, wherein the optical flow algorithm or the central processing unit (9) is further configured to flag the motion of the returned object (4) as fraud by processing the flow vectors (V) characterizing the optical flow on the receiving area (3) considering a motion of the transfer device (T) for transferring the returned object (4) from the receiving area (3) to the acceptance region (8).

8. The system (1) according to any one of the preceding claims, wherein the central processing unit (9) further comprises a machine learning algorithm running at least one convolutional neural network, said at least one convolutional neural network being adapted to process the images from the camera module (5) to locate the pixels assigned to the returned object (4) in the images and / or to filter the flow vectors (V) used to determine the motion of the returned object (4) for verifying that the processed flow vectors (V) correspond to the returned object (4) intended to be transferred from the receiving area (3) to the acceptance region (8).

9. The system (1) according to any one of the preceding claims, wherein the transfer device (T) is a sliding tray (2) being positioned in the receiving area (3), wherein the sliding tray (2) is adapted to slide to allow the returned object (4) to be transferred into the acceptance region (8) via gravity once the sliding tray (2) slides open.

10. The system (1) according to any one of the preceding claims, wherein the transfer device (T) is a conveyor belt, a door and / or contains rollers.

11. The system (1) according to any one of the preceding claims, wherein the acceptance region (8) is located such that the returned object (4) is not accessible to a user through the opening (7) once the returned object (4) is located inside the acceptance region (8), wherein, for example, the acceptance region (8) can be covered or separated from the receiving area (3) and / or is located at least 2 meters away from the receiving area (3) or the opening (7).

12. A method for fraud detection during isolation of a returned object (4), comprising the steps of:- receiving the returned object (4) through an opening (7) for positioning of the returned object (4) in a receiving area (3);- transferring the returned object (4) from the receiving area (3) to an acceptance region (8) via a transfer device (T);- generating and providing at least two images of the returned object (4) using a camera module (5) while the returned object (4) is intended to be transferred from the receiving area(3) to the acceptance region (8) via the transfer device (T);- determine multiple flow vectors (V) for the receiving area (3) while the returned object(4) is intended to be transferred from the receiving area (3) to the acceptance region (8), wherein these multiple flow vectors (V) each represent the movement of a specific pixel or a group of pixels between at least two consecutive images provided by the camera module (5), wherein at least one of these multiple flow vectors (V) is representing the movement of a pixel or a group of pixels associated to the returned object (4) while the returned object (4) is intended to be transferred from the receiving area (3) to the acceptance region (8) for estimating the actual motion of the returned object (4);- determine, based on the flow vectors (V), if there is a motion of the returned object (4) towards the opening (7) of the receiving area (3) and / or if there is a motion of the returned object (4) towards to the acceptance region (8), and- flag the motion of the returned object (4) as fraud if the flow vectors (V) indicate that the returned object (4) is moving towards the opening (7) of the receiving area (3) and / or that the motion of the returned object (4) is not directed to the acceptance region (8).

13. The method according to claim 12, wherein it further comprises the step of:- process the images from the camera module (5) via a machine learning algorithm running at least one convolutional neural network to locate the pixels or the group of pixels assigned to the returned object (4) in the images and / or to filter the flow vectors (V) used to determine the motion of the returned object (4) for verifying that the processed flow vectors(V) correspond to the returned object (4) intended to be transferred from the receiving area (3) to the acceptance region (8).

14. The method according to claim 12 or 13, wherein flagging the motion of the returned object (4) as fraud considers a motion of the transfer device (T) for transferring the returned object (4) from the receiving area (3) to the acceptance region (4), wherein the motion of the transfer device (T) is considered via the processed flow vectors (V).

Images