Empty container disturbance detection

Abstract

A computer-implemented method for detecting an empty container includes acquiring a first image of the container using an imaging system, and after acquiring the first image, causing a perturbing device to perturb contents of the container. The method further includes acquiring, after the perturbation, a second image of the container using the imaging system, and processing, using one or more computer processors, the first image and the second image to determine whether the container is empty based on whether there is a difference between the first image and the second image, the difference is a result of one or more objects inside the container being moved due to the disturbance.

Description

Technical Field

[0001] Embodiments of the present invention relate to methods, systems, and computer-readable media for detecting empty containers. Such methods, systems, and computer-readable media can be applied to systems that use robots to pick up items from containers, such as some automated storage and retrieval systems (AS / RS). Background Art

[0002] Automated storage and retrieval systems (AS / RS) can use containers (also known as totes) to hold various objects (e.g., working parts for an assembly line or inventory in a fulfillment center). Containers can be transported in logistics facilities using various conveyor systems. In automated logistics facilities, objects can be inserted into or removed from containers by machines (such as robots). When all objects are taken out of the container, the robot needs to stop picking up objects from the container. Therefore, the robot's control system may need to determine that the container is empty, and it is also possible to report an empty container to the AS / RS. If the robot's control system cannot accurately determine whether the container is empty, the robot may attempt to pick up non-existent objects from the container indefinitely, or may report inaccurate information to the AS / RS inventory tracking system. In addition, the AS / RS may also need to determine whether the container used to store objects is empty.

[0003] One way to determine if a container is empty is to weigh the container using a weighing scale and compare the current weight to the known weight of the container when the container is empty. However, the weight method may have some limitations. For example, the weighing scale may not be easily integrated with the conveyor system, the checking process may be slow, or the presence of lightweight objects may not be detected. In addition, the check may be specific to the weight used.

[0004] Machine vision systems using cameras can be attractive because they are fast and inexpensive. However, vision processing algorithms may not be robust enough. For example, comparing an image of a container to a reference image of an empty container (also called a ground truth image) may produce an erroneous determination due to image differences that are unrelated to objects inserted into and removed from the container. For example, a machine vision system may incorrectly detect dirt, scratches, or dents on the container and classify them as objects. Such systems may also be specific to a particular container type. Machine vision systems may also be sensitive to lighting conditions. Depth sensors may not detect small or flat objects that blend in with the base or walls of the container. Therefore, a fast and reliable method is needed to determine whether a container is empty. Summary of the invention

[0005] According to some embodiments, a computer-implemented method for detecting an empty container includes acquiring a first image of the container using an imaging system, and after acquiring the first image, causing a disturbance device to disturb the contents of the container. The method also includes acquiring a second image of the container using the imaging system after the disturbance, and processing the first image and the second image using one or more computer processors to determine whether the container is empty based on whether there is a difference between the first image and the second image as a result of one or more objects inside the container being moved due to the disturbance.

[0006] According to some embodiments, a system for detecting an empty container includes a disturbance device for disturbing the contents of the container, and an imaging system for acquiring a first image and a second image of the container. The first image is acquired before the disturbance, and the second image is acquired after the disturbance. The system also includes one or more computer processors configured to process the first image and the second image to determine whether the container is empty based on whether there is a difference between the first image and the second image as a result of one or more objects inside the container being moved due to the disturbance. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The subject matter of the present invention will be described in more detail below based on exemplary drawings. All features described and / or illustrated herein may be used alone or in combination in different combinations. The features and advantages of various embodiments will become apparent by reading the following detailed description with reference to the drawings, which are described as follows:

[0008] Figures 1A to 1D and FIG. 2A to FIG. 2C An image processing method for comparing two images of a container before and after being disturbed is illustrated in accordance with some embodiments.

[0009] Figure 3A and Figure 3B Two images of a container are shown before and after being disturbed, according to some embodiments.

[0010] Figure 4A and Figure 3B Two images of a container are shown before and after being disturbed, according to some embodiments.

[0011] Figure 5 is a flow chart illustrating a computer-implemented method for detecting empty containers according to some embodiments.

[0012] Figure 6 is a simplified block diagram of an empty container detection system according to some embodiments. DETAILED DESCRIPTION

[0013] Embodiments of the present invention provide a method for determining whether a container is empty using a machine vision system in combination with a disturbance device. The machine vision system may include one or more cameras and / or sensors. The camera may be a two-dimensional (2D) camera and / or a three-dimensional (3D) camera. The sensor may include a radar, a lidar, an ultrasonic sensor, etc. Immediately before and after the disturbance, the images captured by the camera / sensor are compared with each other to determine whether the container is empty. The method may provide many advantages, such as being fast and cheap, independent of lighting conditions and container type, applicable to a wide range of object sizes and object weights, and immune to electronic spoofing by stains or damage to the inside of the container.

[0014] According to some embodiments, a method for determining whether a container is empty may include the following steps. First, a first image of the container is acquired. Second, a disturbance is performed on the contents of the container. For example, the disturbance may be performed by shaking the container or tilting the container, moving a robot end effector along the interior of the container (as if stirring potential objects within the container), or by blowing compressed air into the container. Third, immediately after the container is disturbed, a second image of the container is acquired. If the disturbance is performed by a robot, the robot may be moved out of view before the second image is acquired. Fourth, the first image and the second image are compared using an image comparison algorithm to determine whether the container is empty.

[0015] According to some embodiments, if the second image is determined to be the same as the first image (e.g., using a threshold), it can be determined that the container does not contain any objects (e.g., the container is empty). The robot or AS / RS can take appropriate actions accordingly (e.g., remove the empty container from the conveyor, refill the empty container with a working part, etc.). On the other hand, if the second image is determined to differ from the first image by more than a threshold, it can be determined that the container contains one or more objects that have been displaced and / or reoriented due to the disturbance. It may be preferred that the threshold is general enough to ensure that the image comparison algorithm is unaware of the size, shape, color, and texture of the objects, as well as their distance from the camera and lighting levels. Since the first image and the second image are taken of the same container under the same environmental conditions (e.g., lighting conditions), any differences between the first image and the second image may be the result of objects inside the container being moved due to the disturbance.

[0016] In the following discussion, it may be assumed that the container has an open top. However, the methods described herein are applicable to other types of containers. According to various embodiments, various devices may be used to disturb the container. For example, such devices may include an end effector of a robot, a shaker, an air nozzle, or a combination thereof. In an exemplary embodiment, the disturbance may be performed by using compressed air. The robot end effector may be equipped with one or more air nozzles located near the fixture. For example, the air nozzle may be used to reposition the pick-up point position or orient an inappropriate object. In the case where the robot end effector is located inside the container, compressed air may be blown from the nozzle to move objects that may be in the container. Preferably, the airflow should not be too strong to knock objects out of the container, nor should it be too weak to push objects in the container. In some embodiments, the robot may be the same robot used to pick up objects from the container. In some other embodiments, the robot may be another robot that performs other tasks, or may be a dedicated robot. In some embodiments, the air nozzle may be independent of any robot.

[0017] Various image sensing devices can be used in a machine vision system. For example, the image sensing device may include a 2D camera, an RGBD camera, a 3D camera (e.g., a structured light camera, a radar, a lidar, etc.), an ultrasonic sensor, etc. The machine vision system may include a single image sensing device (e.g., an RGBD camera), or multiple image sensing devices. Different types of image sensing devices can be combined (e.g., an RGBD camera combined with an RGB camera, or an RGB camera combined with a lidar). In an exemplary embodiment, two cameras are used. In the event that an object close to the first camera is obscured by the inner wall of the container, the object can be captured by the second camera. Two cameras pointing at the container from relative vantage points tend to see everything in "crosstalk."

[0018] Various image processing algorithms can be used to compare the images of the container before and after the disturbance. In some exemplary embodiments, the mean square error (MSE) method and the structural similarity index measure (SSIM) method are used. The MSE method can focus on the absolute pixel error. The MSE between two grayscale images can be calculated as follows:

[0019]

[0020] Where I(i, j) is the intensity of pixel (i, j) of the first image, and K(i, j) is the intensity of pixel (i, j) of the second image. If the two images are the same, the MSE value is zero, and if the two images are different, the MSE value is greater than zero.

[0021] The SSIM method can focus on structural information, such as interdependence between pixels. The SSIM index is calculated on each window of the image. The metric between two windows x and y of common size N×N can be calculated as follows:

[0022]

[0023] where μ x is the mean value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y, c1=(k1L) 2 , c2=(k2L) 2 , L is the dynamic range of pixel values ​​(e.g., 2# bits per pixel l -1), k1 = 0.01 and k2 = 0.03 by default. If the two images are the same, the SSIM value is equal to unity ("1"), and if the two images are different, the SSIM value is less than 1.

[0024] As described above and below, embodiments of the present invention provide a method, system, and computer-readable medium for determining whether a container is empty using machine vision in conjunction with a perturbation device.

[0025] In a first aspect, the present invention provides a computer-implemented method for detecting an empty container. The method includes: acquiring a first image of the container using an imaging system; causing a disturbance device to disturb the contents of the container after acquiring the first image; acquiring a second image of the container using the imaging system after the disturbance; and processing the first image and the second image using one or more computer processors to determine whether the container is empty based on whether there is a difference between the first image and the second image as a result of one or more objects inside the container being moved due to the disturbance.

[0026] In a second aspect, the invention provides a method according to the first aspect, wherein the disturbing is performed by blowing air inside the container.

[0027] In a third aspect, the invention provides a method according to the first aspect, wherein the perturbation is performed by moving a robotic end effector along the interior of the container in a stirring motion.

[0028] In a fourth aspect, the invention provides a method according to the first aspect, wherein the perturbation is performed by shaking or tilting the container.

[0029] In a fifth aspect, the present invention provides a method according to the first aspect, wherein the first image and the second image are acquired under the same set of environmental conditions.

[0030] In a sixth aspect, the present invention provides a method according to the first aspect, wherein processing the first image and the second image includes comparing the first image and the second image by evaluating a value of a mean square error (MSE) function between the first image and the second image, and determining that the container is empty when it is determined that the value of the MSE function is less than a predefined threshold, and determining that the container is not empty when it is determined that the value of the MSE function is equal to or greater than the predefined threshold.

[0031] In a seventh aspect, the present invention provides a method according to the sixth aspect, wherein processing the first image and the second image further comprises, before evaluating the value of the MSE function, applying a blur function to the first image and the second image.

[0032] In an eighth aspect, the present invention provides a method according to the sixth aspect, wherein processing the first image and the second image further comprises, before evaluating the value of the MSE function, converting the first image and the second image into grayscale images.

[0033] In a ninth aspect, the present invention provides a method according to the first aspect, wherein processing the first image and the second image includes comparing the first image and the second image by evaluating a value of a structural similarity index measure (SSIM) function between the first image and the second image, determining that the container is empty when it is determined that the difference between the value of the SSIM function and the unit value is less than a predefined threshold, and determining that the container is not empty when it is determined that the difference between the value of the SSIM function and the unit value is equal to or greater than a predefined threshold amount.

[0034] In a tenth aspect, the present invention provides a method according to the ninth aspect, wherein processing the first image and the second image further comprises, before evaluating the value of the SSIM function, applying a blur function to the first image and the second image.

[0035] In an eleventh aspect, the present invention provides a method according to the ninth aspect, wherein processing the first image and the second image further comprises, before evaluating the value of the SSIM function, converting the first image and the second image into grayscale images.

[0036] In a twelfth aspect, the present invention provides a method according to the first aspect, further comprising, upon determining that the container is empty, reporting to an automated storage and retrieval system (AS / RS) that the container is empty.

[0037] In a thirteenth aspect, the present invention provides a system for detecting an empty container. The system includes a disturbance device for disturbing the contents of the container, and an imaging system for acquiring a first image and a second image of the container. The first image is acquired before the disturbance, and the second image is acquired after the disturbance. The system also includes one or more computer processors. The one or more computer processors are configured to process the first image and the second image to determine whether the container is empty based on whether there is a difference between the first image and the second image, the difference being a result of one or more objects inside the container being moved due to the disturbance.

[0038] In a fourteenth aspect, the present invention provides a system according to the thirteenth aspect, wherein the imaging system comprises one or more cameras, or one or more ultrasonic sensors, or one or more radars, or one or more lidars, or a combination thereof.

[0039] In a fifteenth aspect, the present invention provides a system according to the thirteenth aspect, wherein the imaging system comprises two RGBD cameras.

[0040] In a sixteenth aspect, the present invention provides a system according to the thirteenth aspect, wherein the agitation device comprises a nozzle for blowing air into the interior of the container.

[0041] In a seventeenth aspect, the present invention provides a system according to the thirteenth aspect, wherein the disturbance device includes a robotic end effector configured to be moved along the inside of the container in a stirring motion; and / or a shaker for shaking the container; and / or a tilting table for tilting the container.

[0042] In an eighteenth aspect, the present invention provides a system according to the thirteenth aspect, wherein processing the first image and the second image includes comparing the first image and the second image by evaluating a value of a mean square error (MSE) function between the first image and the second image, and when it is determined that the value of the MSE function is less than a predefined threshold, determining that the container is empty, and when it is determined that the value of the MSE function is equal to or greater than the predefined threshold, determining that the container is not empty.

[0043] In a nineteenth aspect, the present invention provides a system according to the thirteenth aspect, wherein processing the first image and the second image includes comparing the first image and the second image by evaluating a value of a structural similarity index measure (SSIM) function between the first image and the second image, and determining that the container is empty when it is determined that the difference between the value of the SSIM function and the unit value is less than a predefined threshold amount, and determining that the container is not empty when it is determined that the difference between the value of the SSIM function and the unit value is equal to or greater than the predefined threshold amount.

[0044] In a twentieth aspect, the present invention provides a tangible, non-transitory computer-readable medium having instructions thereon which, when executed by one or more hardware processors, alone or in combination, provide for the execution of the method according to the first aspect.

[0045] Figures 1A to 1D and FIG. 2A to FIG. 2C An image processing method according to some embodiments is illustrated. Figure 1A and Figure 1B The original image of the open top container 110 before and after the disturbance is shown. If the original image is a color image (e.g., an RGB image), the original image can be converted to a grayscale image. In some embodiments, a blur function such as a Gaussian filter can be applied to the grayscale image to produce Figure 1C and Figure 1D Blurred image shown. Gaussian blur can suppress artifacts due to false high-frequency information. The steps of converting to grayscale image and applying Gaussian filter are optional.

[0046] The MSE method or SSIM method can be used to Figure 1C and Figure 1D The blurred images shown are compared with each other. A predetermined threshold can be used to determine whether the two images are different. For example, in the SSIM method, if the SSIM value is less than 1 by an amount exceeding a predefined threshold, it can be determined that the two images are different. In the MSE method, if the MSE value is greater than zero by an amount exceeding a predefined threshold, it can be determined that the two images are different.

[0047] Figure 2A The regions identified as different in the two images are shown in white, and the remaining regions (using the SSIM method or the MSE method) are shown in black. For example, regions where the SSIM values ​​differ by an amount exceeding a threshold can be marked as "1", and the remaining regions can be marked as "0". The regions identified as different can be visualized by drawing bounding boxes in the original images, such as Figure 2B (before perturbation) and Figure 2C In this example, a small wheel (e.g., as small as 2 mm high) is shown before the disturbance (e.g., Figure 2B ) is located in the bounding box 220, and after the perturbation (as shown Figure 2C ) is moved into bounding box 230'.

[0048] exist Figures 1A to 1D and FIG. 2A to FIG. 2CIn an exemplary test of the example shown, the MSE value between the two images was 145.25 and the SSIM value was 0.92. In the test, the container was disturbed by blowing air inside the nozzle of the robot end effector. To ensure that the disturbance would affect objects anywhere in the container, the air was blown into the four corners of the container.

[0049] Figure 3A and Figure 3B The images of the container before and after the disturbance in another test are shown. Figure 3A and Figure 3B 310 / 310', 320 / 320' and 330 / 330' are shown in the figure to illustrate the different areas of the two images. As shown in the figure, the object in the bounding box 330 before the perturbation is moved to the bounding box 310' after the perturbation. After the perturbation, the object in the bounding box 320 is also slightly shifted in the bounding box 320'.

[0050] Figure 4A and Figure 3B The images of the container before and after the disturbance in another test are shown. Figure 4A and Figure 4B 4 shows a bounding box 410 / 410' to illustrate the different areas of the two images. As shown, the object in the bounding box 410 before the perturbation is rotated (e.g., about 10 degrees) in the bounding box 410' after the perturbation. This test illustrates that the method according to an embodiment of the present invention can be sensitive to slight rotations of the object, even without translation of the object.

[0051] Figure 5 is a flow chart illustrating a computer-implemented method 500 for detecting an empty container according to some embodiments. At 502, a first image of a container is acquired using an imaging system. At 504, after acquiring the first image, the contents of the container are disturbed using a disturbance device. At 506, after the disturbance, a second image of the container is acquired using the imaging system. At 508, the first image and the second image are processed using one or more computer processors to determine whether the container is empty based on whether there is a difference between the first image and the second image as a result of one or more objects inside the container being moved due to the disturbance.

[0052] Figure 6is a simplified block diagram of a system 600 for detecting empty containers according to some embodiments. The system may include a perturbation device 610 for perturbing the container and an imaging system 620 for acquiring a first image of the container before the container is perturbed and a second image of the container after the container is perturbed. According to various embodiments, the imaging system 620 may include one or more cameras, or one or more ultrasonic sensors, or one or more radars, or a combination thereof. The camera may be a two-dimensional camera or a three-dimensional camera, or a combination thereof. According to an embodiment, the imaging system includes two RGBD cameras.

[0053] The system 600 also includes one or more computer processors 630. As described above, the computer processor is configured to process the first image and the second image to determine whether the container is empty. The system 600 may also include a user interface 640. The user interface 640 may include a display screen (e.g., a touch sensing screen) for receiving input from a user and displaying output to the user. The system 600 may also include a communication device (one or more) 650, which may include a wired communication device and / or a wireless communication device. The perturbation device 610, the imaging system 620, the computer processor (one or more) 630, the user interface 640 may be coupled to each other in communication via wires, a bus, and / or the communication device 650.

[0054] Although the subject matter of the present invention has been described and illustrated in detail in the drawings and the foregoing description, such description and illustration should be regarded as illustrative or exemplary, rather than restrictive. Any statement describing the present invention herein should also be regarded as illustrative or exemplary, rather than restrictive, because the present invention is defined by the claims. It should be understood that changes and modifications may be made by those of ordinary skill in the art within the scope of the following claims, and these changes and modifications may include any combination of features from the different embodiments described above.

[0055] The terms used in the claims should be understood to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article "a" or "the" when introducing an element should not be interpreted as excluding multiple elements. Similarly, the description of "or" should be interpreted as inclusive, so that the description of "A or B" does not exclude "A and B", unless it is clear from the context or the previous description that only one of A and B is intended to be used. In addition, the description of "at least one of A, B and C" should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring that each of the listed elements A, B and C has at least one, regardless of whether A, B and C are related as categories. In addition, the description of "A, B and / or C" or "at least one of A, B or C" should be interpreted as including any single entity in the listed elements, such as A, any subset of the listed elements, such as A and B, or the entire list of elements A, B and C.

Claims

1. A computer-implemented method for detecting an empty container, the method comprising: acquiring a first image of the container using an imaging system; After acquiring the first image, causing a disturbance device to disturb the contents of the container; After the disturbance, acquiring a second image of the container using the imaging system; as well as The first image and the second image are processed using one or more computer processors to determine whether the container is empty based on whether there is a difference between the first image and the second image, the difference being a result of one or more objects inside the container being moved due to the disturbance.

2. The method according to claim 1, wherein the agitation is performed by blowing air inside the container.

3. The method of claim 1, wherein the perturbing is performed by moving a robotic end effector along the inside of the container in a stirring motion.

4. The method of claim 1, wherein the perturbing is performed by shaking or tilting the container. The method of claim 1 , wherein the first image and the second image are acquired under the same set of environmental conditions.

6. The method of claim 1, wherein processing the first image and the second image comprises: comparing the first image and the second image by evaluating a value of a mean square error (MSE) function between the first image and the second image; When determining that the value of the MSE function is less than a predefined threshold, determining that the container is empty; as well as When it is determined that the value of the MSE function is equal to or greater than the predefined threshold, it is determined that the container is not empty. 7 . The method of claim 6 , wherein processing the first image and the second image further comprises applying a blur function to the first image and the second image before evaluating the value of the MSE function.

8. The method of claim 6, wherein processing the first image and the second image further comprises, before evaluating the value of the MSE function, converting the first image and the second image into grayscale images.

9. The method of claim 1 , wherein processing the first image and the second image comprises: comparing the first image and the second image by evaluating a value of a structural similarity index measure SSIM function between the first image and the second image; determining that the container is empty upon determining that the difference between the value of the SSIM function and the unit value is less than a predefined threshold amount; as well as Upon determining that the difference between the value of the SSIM function and a unit value is equal to or greater than the predefined threshold amount, it is determined that the container is not empty.

10. The method of claim 9, wherein processing the first image and the second image further comprises applying a blur function to the first image and the second image before evaluating the value of the SSIM function.

11. The method of claim 9, wherein processing the first image and the second image further comprises, before evaluating the value of the SSIM function, converting the first image and the second image to grayscale images.

12. The method of claim 1, further comprising, upon determining that the container is empty, reporting to an automated storage and retrieval system (AS / RS) that the container is empty.

13. A system for detecting an empty container, the system comprising: a disturbance device for disturbing the contents of the container; an imaging system for acquiring a first image and a second image of the container, the first image being acquired before the disturbance and the second image being acquired after the disturbance; as well as One or more computer processors configured to process the first image and the second image to determine whether the container is empty based on whether there is a difference between the first image and the second image, the difference being a result of one or more objects inside the container being moved due to the disturbance.

14. The system of claim 13, wherein the imaging system comprises one or more cameras, or one or more ultrasonic sensors, or one or more radars, or one or more lidars, or a combination thereof.

15. The system of claim 13, wherein the imaging system comprises two RGBD cameras.

16. The system of claim 13, wherein the agitation device comprises a nozzle for blowing air into the interior of the container.

17. The system of claim 13, wherein the agitation device comprises a robotic end effector configured to be moved along the inside of the container in a stirring motion; and / or a shaker for shaking the container; and / or a tilting table for tilting the container.

18. The system of claim 13, wherein processing the first image and the second image comprises: comparing the first image and the second image by evaluating a value of a mean square error (MSE) function between the first image and the second image; When determining that the value of the MSE function is less than a predefined threshold, determining that the container is empty; as well as When it is determined that the value of the MSE function is equal to or greater than the predefined threshold, it is determined that the container is not empty.

19. The system of claim 13, wherein processing the first image and the second image comprises: comparing the first image and the second image by evaluating a value of a structural similarity index measure SSIM function between the first image and the second image; determining that the container is empty upon determining that the difference between the value of the SSIM function and the unit value is less than a predefined threshold amount; as well as Upon determining that the difference between the value of the SSIM function and a unit value is equal to or greater than the predefined threshold amount, it is determined that the container is not empty.

20. A tangible, non-transitory computer-readable medium having instructions thereon that, when executed by one or more hardware processors, alone or in combination, provide for the performance of the method of claim 1.

Images