Food waste detection method and system

A camera and scale-based system with image recognition and user feedback effectively identifies and quantifies food waste, enhancing waste management efficiency and inventory optimization.

JP7580693B2Active Publication Date: 2024-11-12WASTIQ BV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2021534291
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-12-14
Filing Date
2019-12-13
Publication Date
2024-11-12
Estimated Expiration
2039-12-13

AI Technical Summary

Technical Problem

Food waste is often generated in places like restaurants and is not efficiently managed, leading to inefficiencies in planning, inventory, and environmental impact.

Method used

A system using cameras and scales to detect food-related products before disposal, employing image recognition algorithms to identify and weigh waste, with optional remote server processing and user feedback for improved detection.

Benefits of technology

Enables accurate identification and quantification of food waste, allowing for data analysis and recommendations to minimize waste, and can be integrated with inventory management systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580693000001
    Figure 0007580693000001
  • Figure 0007580693000002
    Figure 0007580693000002
  • Figure 0007580693000003
    Figure 0007580693000003
Patent Text Reader

Abstract

A system (1) for detecting food-related products (2) before they are discarded, the system comprising one or more cameras (11), a display unit (12), a computing device (13) communicatively connected to the camera and the display, and a scale (3) communicatively connected to the computing device and holding a trash can (31), wherein the camera captures an image or video of the product when the product is within the field of view of the camera and before the product enters the trash can, the scale is configured to weigh the product in the trash can, the computing device obtains information about the product from the captured image or video by applying an image recognition algorithm, receives the weight from the scale, and generates and outputs data on the display unit, the data being based on the information about the product and the weight.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a system and method for detecting food-related products, and an indication unit for use in the system. [Background technology]

[0002]

[0002] Food establishments are often faced with food waste because they must throw away food that has passed its expiration date or that is left over after consumption or cooking. Examples of such establishments are restaurants, where food waste can occur because customers leave food on their plates, the kitchen has leftovers after cooking meals, or food in the inventory has passed its expiration date.

[0003]

[0003] Food waste needs to be reduced. Insights into food waste can be used, for example, in restaurants to optimize planning, allocation and inventory management, resulting in more efficient food purchases and more environmentally friendly behavior. Other examples of places that can benefit from insights into food waste are caterers, the catering industry, hospitals, medical institutions, and generally all places involved in the preparation of food. Summary of the Invention

[0004]

[0004] According to one aspect of the present invention, a system for detecting food-related products before they are discarded is proposed. The system may comprise one or more cameras. The system may further comprise a display unit. The system may further comprise a computing device communicatively connected to the one or more cameras and the display unit. The system may further comprise a scale communicatively connected to the computing device. The scale may be configured to hold a trash bin. For example, the scale is separable from the trash bin, with the trash bin simply resting on the scale. The scale may be integral with the trash bin. The trash bin may be a recycling bin. The one or more cameras may be configured to capture an image or video of the food-related product before it enters the trash bin when the food-related product is within the field of view of the one or more cameras. Advantageously, this allows food remnants to be detected before they are mixed with other food waste in the trash bin. The scale may be configured to capture weight information of the food-related product when the food-related product is in the trash bin. The computing device may be configured to capture information about the food-related product from the captured image or video by applying an image recognition algorithm. The image recognition algorithm may be executed locally on the computing device or may be executed remotely on a remote server to which the computing device may be communicatively connected. The computing device may be configured to receive weight information from the scale. The computing device may be configured to generate and output data on a display unit, the data being based on the information about the food-related product and the weight information.

[0005]

[0005] Food related products are typically food leftovers, but may also include other discarded objects such as plastic, paper, napkins, cardboard, (disposable) cutlery, etc. Food related products may include containers, plates, or other tray items on which disposables are placed, which may be detected along with the disposables and input into image recognition algorithms to improve detection of food leftovers or other disposables.

[0006] In one embodiment, the computing device may be communicatively connected to a remote server. The computing device may be configured to transmit captured images or videos to the remote server for application of image recognition algorithms. The computing device may be configured to receive information about the food-related product from the remote server. The remote server may be implemented as a cloud computing server or a cloud computing service.

[0007] In one embodiment, the computing device may be further configured to store one or more of the information about the food-related product, the weight information, the output data, and the time stamp in a data storage on a remote server, which allows for analyzing or mapping food waste over time and also allows for generating recommendations regarding minimizing food waste detected over time.

[0008]

[0008] In one embodiment, the computing device may be configured to present one or more questions on the display unit about one or more objects in the image or video if the image recognition algorithm is unable to identify one or more of the food-related products from the captured image or video. The display unit may comprise a user interface, preferably in the form of a touch screen interface, for receiving user input in response to the one or more questions. The responses may be used by the image recognition algorithm to improve detection of the one or more objects.

[0009]

[0009] In one embodiment, the one or more cameras may be configured to automatically capture images or footage when the food-related product is within the field of view of the one or more cameras or depth sensors and is in a substantially fixed position in the dynamic minimum time required for successful detection. Upon successful detection of the ingredient, audiovisual feedback may be provided to the user. The fixed position may be any position within the field of view, and is typically defined by where the user holds the food-related product under the one or more cameras before discarding it in a trash bin.

[0010]

[0010] In one embodiment, the output data may include a ratio of different food related products. The different food related products may be detected by an image recognition algorithm. The ratio may be based on weight information. Thus, by combining the weight information with the image detection algorithm, the ratio of the different food related products presented to the camera and scale may be obtained.

[0011] In one embodiment, the one or more cameras may comprise stereoscopic cameras for obtaining 3D information about the food-related product from images or footage.

[0012] In one embodiment, the image recognition algorithm may be configured to obtain volumetric information from the 3D information. The computing device may be configured to obtain a weight estimate of the food-related product based on the volumetric information. The stereo camera may replace the scale. The weight estimate may be used instead of the weight information. Thus, using a stereo camera allows the system to be implemented without a scale.

[0013] In one embodiment, the one or more cameras may comprise a hyperspectral imaging camera for obtaining substance information about food-related products from images or videos, non-limiting examples of substance information being fat, protein, and sugar levels in food leftovers.

[0014]

[0014] In one embodiment, the system may further include a depth sensor, such as an ultrasonic depth sensor or a laser-based depth sensor, for detecting when a food-related product is within the field of view of the one or more cameras. The depth sensor may be used in combination with the one or more cameras or may be used alone to detect when a food-related product is within the field of view of the one or more cameras, thereby triggering the one or more cameras to capture an image or video of the food-related product. The depth sensor is typically located next to the one or more cameras.

[0015] In one embodiment, the field of view may be located in an area substantially around a downward line of sight from one or more cameras.

[0016]

[0016] In one embodiment, the display unit may include a housing for housing one or more cameras. The housing may include an exterior side that is angled from the horizontal. The cameras may be located on the exterior side within the housing such that the line of sight is tilted vertically at the angle. The line of sight is perpendicular to the exterior side. The angle may be between 15 and 45 degrees, preferably between 15 and 30 degrees, and more preferably between 15 and 25 degrees.

[0017] In one embodiment, the housing of the display unit may further include a computing device.

[0018] In one embodiment, the housing may further comprise a visual indicator that indicates where the food related product is presented to the one or more cameras.

[0019] In one embodiment, the visual indicator may change its color when a food-related product is registered by one or more cameras.

[0020] In one embodiment, the housing may further include an audible indicator that provides audible feedback.

[0021] In one embodiment, the audible indicator may emit a sound when a food-related product is registered by one or more cameras.

[0022] In one embodiment, the scale may include at least one sloping sidewall that allows the trash can to roll onto and off the scale.

[0023]

[0023] In one embodiment, The sloping sidewall forms an integral part with the top of the scale. do.

[0024]

[0024] In one embodiment, the housing (100) and the scale (3) are connected by a vertically aligned support structure to fix the housing at a vertical distance from the scale.

[0025] According to one aspect of the invention, a display unit in a housing is proposed, the housing further comprising one or more cameras and a computing device, for use in a system having one or more of the above features.

[0025]

[0026] According to one aspect of the present invention, a method for detecting food-related products before they are discarded is proposed. The method may include capturing images or videos of the food-related products using one or more cameras when the food-related products are within the field of view of the one or more cameras and before the food-related products are discarded in a trash bin. The method may further include capturing weight information of the food-related products using a scale when the food-related products are in the trash bin, the scale being configured to hold the trash bin. The method may further include capturing information about the food-related products from the captured images or videos in a computing device by applying an image recognition algorithm. The method may further include generating and outputting data on a display unit by the computing device, the data may be based on the information about the food-related products and the weight information.

[0026]

[0027] In one embodiment, the method may further include transmitting the captured image or video from the computing device to a remote server for applying an image recognition algorithm. The method may further include receiving information regarding the food-related product from the remote server in the computing device.

[0027]

[0028] In one embodiment, the method may further include presenting one or more questions on the display unit about one or more objects in the image or video if the image recognition algorithm is unable to identify one or more of the food-related products from the captured image or video. The method may further include receiving user input from a user interface of the display unit in response to the one or more questions, the response being used by the image recognition algorithm to improve detection of the one or more objects.

[0028]

[0029] Hereinafter, the embodiments will be described in more detail, but it should be understood that these embodiments should not be construed as limiting the scope of protection of the present disclosure. [Brief description of the drawings]

[0029]

[0030] Embodiments will now be described, by way of example only, with reference to the accompanying schematic drawings in which corresponding reference symbols indicate corresponding parts and in which:

[0030] [Figure 1]

[0031] 1 illustrates a system according to an exemplary embodiment of the present invention. [Diagram 2]

[0032] 1 shows a schematic side view of a housing and camera of an exemplary embodiment of the present invention. [Diagram 3]

[0033] 1 illustrates an elevation view of a display unit and camera within a housing in an exemplary embodiment of the present invention. [Figure 4]

[0034] 1 shows a block diagram of method steps of an exemplary embodiment of the present invention; [Diagram 5]

[0035] 1 illustrates another exemplary embodiment of a display unit within a housing. [Figure 6A]

[0036] 1 shows an elevation view of an exemplary scale. [Figure 6B]

[0037] 1 illustrates a side view of an exemplary scale.

[0031]

[0038] These diagrams are for illustrative purposes only and do not serve as a limitation on the scope or protection defined in the claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032]

[0039] FIG. 1 shows an exemplary system 1 for detecting food-related products 2 before being thrown away in a waste bin 31. The system 1 preferably includes a housing 100, which includes one or more cameras 11, a display unit 12, a computing unit 13, and a communication module 14. Less preferably, it is possible to have the one or more cameras 11 separate from the housing 100 and / or the computing device 13 separate from the housing 100. The system 1 may include a scale 3 configured to hold the waste bin 31 and to weigh the food-related products 2 when they are placed in the waste bin 31. The scale 3 may be integrated with the waste bin 31. Preferably, the waste bin 31 is a recycling receptacle that allows the food-related products 2 to be recycled after being thrown away. The scale 3 may include a communication module 32 for communicating with the computing device 13, typically via the communication module 14.

[0033]

[0040] The display unit 12 can typically present full-color bitmap images representative of the food detected among the food-related products 2. The display unit 12 can be configured to display a graphical user interface, for example in the form of any selectable button objects or any other user interface elements selectable through a touch screen interface of the display unit 12. The computing device 13 can be any suitable CPU, GPU, and / or NPU based computer, for example in the form of a Raspberry Pi™ computer. Preferably, the computing device 13 is a small form factor or single board computer to minimize the size requirements of the computing device 13. The communication module 14 may be integrated with the computing device 13. The communication module 14 can be any suitable wireless or wired communication module. The communication module 14 can include a number of different communication interfaces, for example a Bluetooth™ interface for short-range communication with the communication module 32 of the scale 3, and a Wi-Fi or LAN interface for communication with the remote server 4. In the example of a Wi-Fi or LAN interface, the communications would typically further include a router (not shown) for connecting to the Internet 5, a local area network, or other suitable network.

[0034]

[0041] In an exemplary embodiment, the housing 100 and scale 3 may be connected by a pole or other vertically aligned support structure to fix the housing 100 at a vertical distance from the scale 3. This allows the housing 100 and scale 3 to be moved as a single unit and positioned at a desired location. The vertically aligned support structure may be used to guide or house electrical and / or data cables for electrical or data / signal connections between the scale 3 and components within the housing 100.

[0035]

[0042] The remote server 4 typically includes a data storage 41 and a communication module 42. The remote server 4 may be a standalone server, may be implemented as a cloud computing server, may be implemented as a cloud computing service, or may be any other computer system. The data network 5 may be a local area network, a wide area network, the Internet, or any other suitable network.

[0036]

[0043] FIG. 2 shows an exemplary embodiment of a camera setup in a housing 100. In FIG. 2, a side view of the housing 100 including the camera 11 is shown. The housing 100 can be mounted to a wall 6. In FIG. 2, the housing 100 has an exterior side 102 on the underside of the housing 100. The exterior side 102 can be disposed at an angle α from a horizontal plane 103, and thus the underside of the housing 100 can be disposed at an angle α. The camera 11 can be mounted on the surface side 102. Since the line of sight 101 from the camera is typically in a direction perpendicular to the surface side 102, the line of sight 101 can thus be tilted vertically at an angle α. In particular, when the housing 100 is mounted to a wall 6, the tilted line of sight 101 can improve the field of view by eliminating the covering of a large portion of the wall 6 that cannot present the food-related product 2 below the camera 11. Additionally, the tilted line of sight 101 allows the product to be slightly viewed from the side, thereby improving the detectability of the food-related product 2.

[0037]

[0044] Figure 3 illustrates an exemplary display unit 12 that may be installed within housing 100. Figure 3 also illustrates a camera 11 that may be installed within housing 100. In the example of Figure 3, the underside of the housing on which camera 11 is installed is positioned at an angle that may tilt the line of sight of camera 11 from the vertical, as described in Figure 2.

[0038]

[0045] In one embodiment, the food-related products (preferably all) that end up in the trash bin 31 may first be captured by a smart camera 11 above the trash bin 31 and then by a digital scale 3 placed below the trash bin 31. If something moves within the field of view of the camera, the camera may automatically take a photo or capture a video as soon as it detects that the object is completely within the field of view and stable (stops moving) in a fixed position within the field of view. The object may include a plate or container on which the food-related product 2 is located. The object may be a human hand holding the food-related product 2. If the detection of the ingredient is successful, audiovisual feedback may be provided to the user. The captured image may be transmitted to the cloud 4, where the ingredient detection may be performed by an image recognition algorithm. Alternatively, the image recognition algorithm may be executed locally, for example in the computing device 13. The model used by the image recognition algorithm may detect / recognize one or more food-related products 2 on the image and send the result back to the computing device 13 for local feedback on the display unit 12. When the waste 2 is dumped in the bin 31, the digital scale 3 can capture the weight and transmit this information to the computing device 13 and / or the cloud 4. The weight and image processing results can be linked together with the component recognized ratios and the results thus obtained can be sent back to the computing device where the results can be displayed on the display unit 12. The results can also be stored in a data storage 41 such as a cloud database. Preferably, the results are stored with a timestamp or any other indication of date and / or time. This data can be used at any point in time to generate a dashboard showing actionable results.

[0039]

[0046] Since there are only a limited number of items that can be captured in the camera's field of view, waste streams can be relatively easily detected. The inventors currently focus on food waste and have identified four types of food waste streams in restaurants. However, in practice, not all restaurants separate their waste, so the tool also detects other types of waste, e.g., plastic, paper, cardboard, and cutlery. This gives the inventors' detection model the opportunity to go beyond just food waste.

[0040]

[0047] Various types of food waste streams may be captured by the system 1, examples of which include: (i) expired products from inventory, (ii) processed kitchen waste, which may be detectably presented to the camera 11 on metal containers, in trash bins or pans or large plastic containers, (iii) cutting waste, which may be detectably presented to the camera 11 on cutting plates or larger plastic or metal trash bins, (iv) restaurant waste, which may be detectably presented to the camera 11 on smaller plates or bowls (e.g., porcelain).

[0041]

[0048] The state or condition of the food-related product 2 can also be detected in the images and used for waste stream analysis. For example, in a restaurant environment, the following detection criteria can be used: (i) for expired products from inventory, the product is untouched and therefore easier to detect, (ii) processed foods from the kitchen may be chopped or ground or may contain liquid side dishes, (iii) for cutting waste from the kitchen, it may contain inedible skins and bones, (iv) cooking waste from a restaurant may contain leftovers or produce parts of a dish, which may be partially eaten and mixed and therefore more difficult to detect. The system 1 can detect the state of the condition to further improve the detection of the food-related product 2 and / or generate recommendations on minimizing food waste.

[0042]

[0049] In an exemplary embodiment, the camera 11 may be installed approximately 50 cm above the dustbin 31 or another base platform. In another exemplary embodiment, the camera 11 may be installed approximately 70 cm above the dustbin 31 or another base platform. The registration of the food-related product 2 may take place between this platform and the camera 11, for example 40 cm below the camera 11, which may be detected by a depth sensor. The depth sensor may thus be used to trigger the camera 11 to start registering the food-related product 2. As shown in FIG. 3, the camera 11 may be installed at a slight angle α with respect to a surface 102 on which the camera 11 may be mounted. For example, referring to FIG. 2, the angle α may be any angle between 15 degrees and 45 degrees from the horizontal plane 103, allowing the camera 11 to have a better perspective, both in taking an image not obstructed by the plane on which the machine is mounted, and to slightly view the product from the side.

[0043]

[0050] In one embodiment, the camera 11 can be completely detached from the display unit 12 to make the system more suitable when space is limited. This may also allow the camera 11 to be positioned to provide an optimal perspective of the food waste being registered.

[0044]

[0051] Computer vision techniques can be used for the detection of ingredients of food-related products 2, classification of dishes and origin of waste streams (e.g. kitchen, restaurant). At the core of such computer vision techniques are neural networks and deep learning. This is a so-called semi-supervised machine learning approach. The term "supervised" typically means that the image recognition algorithm is trained to get progressively better at the task it needs to perform (detection). Training is typically done by giving the computer many examples where a human has manually labeled ingredients, dish types, waste stream types through a user interface such as a graphical user interface.

[0045]

[0052] There are many different types of image recognition techniques that can be applied. The most commonly used strategies are: (i) classification, to classify the entire image and assign a label; (ii) detection, to detect and label possibly multiple objects in the image; and / or (iii) segmentation, which is a fine-grained approach that allows for the assignment of labels to individual pixels of an image.

[0046]

[0053] For cooking and waste stream classification, the first two types of image recognition techniques, classification and detection, are best suited. For ingredient detection, the third strategy, the more powerful segmentation technique, is best suited. In segmentation techniques, pixel-by-pixel labeling can be used to calculate a ratio for ingredients occurring in the image. This ratio can be used to improve the weight estimate that can be assigned to each individual ingredient. The input images that can be used to train ingredient detection may require a lot of detail. This means that an ingredient name can be assigned to each pixel or group of pixels.

[0047]

[0054] Once trained, the model can be used to independently recognize components in new images captured by the camera 11 that are fed into the image recognition algorithm. Hence the term "semi-supervised" is applicable. As soon as the model is trained, it can be used to automatically recognize components in images without the need for any manual intervention.

[0048]

[0055] Additional domain knowledge (such as the physical location of the camera 11, housing 100 and / or scale 3 within a venue such as a restaurant, and / or the menu of that restaurant) can be used to improve accuracy by limiting the area in which the detection algorithm must operate. For example, physical location can be used to determine that only certain waste streams are recorded by a particular system 1, and menus can be used to limit the variety of ingredients that the system 1 may encounter.

[0049]

[0056] To improve the quality and accuracy of the detection algorithm, one or more of the cameras 11 may include a stereoscopic camera. A stereoscopic camera can obtain 3D depth images that provide more information about the volume of food being discarded and can help improve weight estimates. Compared to a single top-down camera, a top-down camera can experience occlusion issues where certain ingredients are not visible to the camera if covered by other materials, but a stereoscopic camera provides a better view using two slightly differently angled cameras. Computer vision techniques such as Siamese Neural Networks can use the stereoscopic images as input and can be used to better detect food-related products 2 when presented to the stereoscopic camera.

[0050]

[0057] A stereoscopic camera may be used to obtain volumetric information of the food-related product 2. The volumetric information, together with an identification of the food product itself, may provide an indication of the weight of the detected food product. The stereoscopic camera may then be used in place of the scale 3, in which case the system 1 need not include the scale 3.

[0051]

[0058] To improve the quality and accuracy of the detection algorithm, one or more of the cameras 11 may include a hyperspectral camera. A hyperspectral camera can acquire a spectrum for every pixel, which provides much more information than a standard RGB camera. This information can be used to detect, for example, fat, protein, and sugar levels, simplifying and improving the quality of ingredient detection.

[0052]

[0059] The weight of what ends up in the bin 31 may be registered with each food waste photograph, possibly depending on the proportion of ingredients detected in the image. This process may be performed in a short time frame, for example within a few seconds after the photograph is taken and the image and weight data may be transmitted together to the remote server 4.

[0053]

[0060] If the image recognition algorithm is unable to determine / detect the food-related product 2 from the image, the image may be sent to the remote server 4 or another remote server or cloud for re-detection within an updated detection model, resulting in a multiple choice of images that are sent to the display device 12 as a feedback screen. The end user may then select one or more images in the feedback screen to identify the product(s). If (part of) the image is not in the multiple choices, the user may be offered to provide further feedback, for example in the form of a selectable "explain" button to note the product(s) that were not detected. This feedback from the user may be added directly to the detection model of the image recognition algorithm.

[0054]

[0061] Feedback from the end user can be provided in a variety of ways. For example, the display device 12 may include a touch screen interface for providing feedback. Alternatively or additionally, a voice recognition interface can be installed on the housing 100 to allow the end user to interact with the system. Alternatively or additionally, one or more buttons on the housing 100 may allow the end user to provide feedback to the system.

[0055]

[0062] The system 1 can be used in a variety of use cases, including, but not limited to: (i) waste during cleaning by kitchen or restaurant staff, (ii) disposal during cleaning by customers in a kitchen or restaurant, (iii) assembly line detection, for example in a self-service restaurant, where food-related products 2 are presented to one or more cameras 11 without human intervention, (iv) detection of food-related products in tray carts, where multiple trays are collected before discarding leftovers and waste and washing the trays and cutlery, in self-service restaurants and health care facilities.

[0056]

[0063] Typically, the scale 3 and dustbin 31 are located below the camera 11, however, it is possible for the scale 3 and dustbin 31 to be located elsewhere. Preferably, the scale 3 and dustbin 31 are located near the camera 11 to facilitate handling of the waste from being held below the camera to disposing of the waste in the dustbin.

[0057]

[0064] System 1 may be used to encourage waste reduction by providing performance data, possibly anonymously, among neighboring and / or comparable restaurants for comparison.

[0058]

[0065] The system may alternatively or additionally detect other types of waste in addition to food related produce 2, examples of which include plastic, paper, cardboard and cutlery.

[0059]

[0066] The system 1 may be integrated with third party vendors that provide, for example, inventory management or point of sale solutions.

[0060]

[0067] FIG. 4 shows an exemplary block diagram of steps that may be performed by parts of the system 1 as shown in FIG. 1. Comparing FIG. 4 with FIG. 1, the smart camera may be similar to one or more cameras 11, the smart scale may be similar to the scale 3, the touch screen terminal may be similar to the display unit 12, and the cloud storage may be similar to the data storage 41. Starting with the smart camera, in FIG. 4, the camera may automatically take a picture of a container or plate in the field of view of the camera or depth sensor. Direct intermediate image feedback may be sent to the touch screen terminal, where the obtained image may be presented as a full color bitmap image. The taken picture may be input to a computer vision program, which may be executed on a local computing device 13 and / or in a remote server 4, such as a cloud. The computer vision program may apply an image recognition algorithm to the image to obtain a detection result. After the picture is taken, the waste may be dumped in a trash bin. Starting with the smart scale, the scale may detect a waste change of the waste that was then dumped in the trash bin. The waste change may indicate the weight of the waste that was last dumped in the trash bin. Direct intermediate weight feedback may be sent to the touch screen terminal, where the obtained weight information may be presented. The detection results can be matched with the weight information and a final registration feedback indicating the discarded waste can be presented on the touch screen terminal. The detection results and weight information can additionally or alternatively be stored in cloud storage from which statistics can be generated and actionable insights can be presented, for example, in a dashboard output on a website or in a dashboard application. The touch screen terminal can be used to obtain user-provided reinforcing learning feedback that can be fed to the computer vision program to improve the detectability of the waste.

[0061]

[0068] 5 shows an exemplary display unit 12 in a housing 100, which is a non-limiting alternative to the display unit and housing shown in FIG. 3. In this example, the camera 11 may be located at a higher position than the display 12. 100 may further include one or more light sources 15 for illuminating the food related product 2, allowing better detection by the camera 11. The other two circles shown on the top of the housing indicate the possibility of having further sensors, cameras and / or further light sources. These further sensors, cameras and / or light sources may be different in function to the camera 11 and light sources 15, e.g. a depth sensor or an infrared light source.

[0062]

[0069] The housing 100 may include a visual indicator 16 that indicates the height at which the food-related produce must be presented to the camera before being discarded in the trash bin 31. This allows the system to allow the user to optimally position the produce relative to the camera to detect what is to be discarded in the trash bin. The visual indicator 16 may be implemented in a variety of ways, for example as an illuminated LED strip. If a light-emitting visual indicator is used, the color of the light may change to indicate status. For example, once the produce is registered by the camera, the visual indicator 16 may change color to green, thereby indicating that the produce may be discarded in the trash bin.

[0063]

[0070] The housing 100 may include an audible indicator 17 to provide feedback to the user, for example to indicate that a product has been registered by the camera and therefore that the product may be disposed of in the trash.

[0064]

[0071] 6A and 6B show an exemplary scale 3 that may be used in the system of FIG. 1. The weight scale 3 of FIGS. 6A and 6B is designed to be as low as physically possible to allow more space for the dustbin 31 and more space between the dustbin 31 and the camera 11. Additionally, the scale 3 of FIGS. 6A and 6B includes at least one (e.g., two) sloping surfaces 33, i.e., sloping side walls, to allow the dustbin to roll onto and off the scale 3. The sloping surfaces 33 may form a single piece of material with the top of the scale, i.e., may be part of the moving parts of the scale 3 when measuring weight.

[0065]

[0072] One or more embodiments may be implemented as a computer program product for use with a computer system. The program(s) of the program product may define the functions of the embodiments (including the methods described herein) and may be contained in various computer-readable storage media. The computer-readable storage media may be non-transitory storage media. Exemplary computer-readable storage media include, but are not limited to, (i) non-writeable storage media that may store information permanently (e.g., a CD-ROM disk, a ROM chip, or any type of solid-state non-volatile semiconductor memory readable by a CD-ROM drive), and (ii) writeable storage media that may store information that may be changed, such as a hard disk drive or any type of solid-state random access semiconductor memory, flash memory.

Claims

1. A system (1) for detecting a food-related product (2) before it is discarded, comprising: One or more cameras (11); A display unit (12); a computing device (13) communicatively connected to the one or more cameras and the display unit; a scale (3) communicatively connected to the computing device, the scale configured to hold a trash can (31); the one or more cameras are configured to capture an image or video of the food related product when the food related product is within a field of view of the one or more cameras and before the food related product is disposed of in the bin; the scale is configured to obtain weight information of the food-related product when the food-related product is in the bin; The computing device comprises: obtaining information about said food-related products from said captured images or videos by applying an image recognition algorithm; receiving the weight information from the scale; A system configured to generate and output data on the display unit, the data being based on the information regarding the food-related product and the weight information.

2. The computing device is communicatively connected to a remote server (4), the computing device: transmitting the captured images or video to the remote server for applying the image recognition algorithm; The system of claim 1 , configured to receive information regarding the food-related product from the remote server.

3. 3. The system of claim 2, wherein the computing device is further configured to store one or more of the information about the food-related product, the weight information, the date, and a time stamp in a data storage (41) of the remote server.

4. 4. The system of claim 1, wherein the computing device is configured to present one or more questions on the display unit about one or more objects in the image or video if the image recognition algorithm is unable to identify one or more of the food related products from the captured image or video, the display unit comprising a user interface for receiving user input in response to the one or more questions, the responses being used by the image recognition algorithm to improve detection of the one or more objects.

5. The system of any one of claims 1 to 4, wherein the one or more cameras are configured to automatically capture the images or videos when the food-related product is in a substantially fixed position within a field of view for a dynamic minimum time required for successful detection.

6. 6. The system of claim 1, wherein the data comprises ratios of various food related products, the various food related products being detected by the image recognition algorithm, and the ratios being based on the weight information and the detected various food related products.

7. The system of any one of claims 1 to 6, wherein the one or more cameras comprise a stereoscopic camera for obtaining 3D information about the food-related product from the image or video.

8. 8. The system of claim 7, wherein the image recognition algorithm is configured to obtain volumetric information from the 3D information, and the computing device is configured to obtain a weight estimate of the food-related product based on the volumetric information, and the stereoscopic camera replaces the scale and the weight estimate is used instead of the weight information.

9. The system of any one of claims 1 to 8, wherein the one or more cameras comprise a hyperspectral imaging camera for obtaining substance information about the food-related product from the image or video.

10. The system of any one of claims 1 to 9, further comprising a depth sensor for detecting when the food related product is within the field of view of the one or more cameras.

11. The system of any one of claims 1 to 10, wherein the field of view lies in an area substantially around a downward line of sight from the one or more cameras.

12. 12. The system of claim 11, further comprising a housing (100), said housing comprising said display unit, said housing housing said one or more cameras, said housing comprising an exterior side (102) mounted at an angle (α) from a horizontal plane (103), said cameras being located on said exterior side within said housing such that a line of sight (101) is inclined vertically at said angle (α), said line of sight being perpendicular to said exterior side, said angle (α) being in the range of 15 to 45 degrees, preferably in the range of 15 to 30 degrees, more preferably in the range of 15 to 25 degrees.

13. The system of claim 12 , wherein the housing further comprises the computing device.

14. 14. The system of claim 12 or 13, wherein the housing further comprises a visual indicator (16) that indicates where the food related product is presented to the one or more cameras.

15. 15. The system of claim 14, wherein the visual indicator changes its color when the food-related product is registered by the one or more cameras.

16. The system of any one of claims 12 to 15, wherein the housing further comprises an audible indicator (17) for providing audible feedback.

17. 17. The system of claim 16, wherein the audible indicator emits a sound when the food-related product is registered by the one or more cameras.

18. The system according to any one of the preceding claims, wherein the scale (3) comprises at least one sloping sidewall (33) that allows the bin to be rolled onto and off the scale.

19. The system of claim 18 , wherein the angled sidewall forms an integral part with a top portion of the scale.

20. The system of any one of claims 12 to 17, wherein the housing (100) and the scale (3) are connected by a vertically aligned support structure to fix the housing at a vertical distance from the scale.

21. A housing comprising a display unit, the housing further comprising one or more cameras and a computing device, the housing configured for use in a system according to claim 12 or 13.

22. 1. A method for detecting a food-related product before it is discarded, the method comprising: capturing an image or video of the food related product using one or more cameras when the food related product is within a field of view of the one or more cameras and before the food related product is disposed of in a trash bin; obtaining weight information of the food-related product using a scale while the food-related product is in the bin, the scale being configured to hold the bin; obtaining, in a computing device, information relating to the food-related product from the captured image or video by applying an image recognition algorithm; generating and outputting data by said computing device on a display unit, said data being based on said information regarding said food-related product and said weight information.

23. transmitting the captured images or videos from the computing device to a remote server for applying the image recognition algorithm; 23. The method of claim 22, further comprising receiving the information regarding the food-related product from the remote server in the computing device.

24. 24. The method of claim 22 or 23, further comprising, if the image recognition algorithm is unable to identify one or more of the food-related products from the captured image or video, presenting one or more questions on the display unit about one or more objects in the image or video, and receiving user input from a user interface of the display unit in response to the one or more questions, wherein the response is used by the image recognition algorithm to improve detection of the one or more objects.

Citation Information

Patent Citations

  • Liver support system

    JP2004139278A

  • Meal information management system

    JP2005293395A

  • Print processor and print processing system

    JP2012093962A

  • Systems and methods for monitoring food waste

    JP2017520753A

  • Meal identification system and program therefor

    JP2018147415A