Method for checking the completeness of a container containing a plurality of objects

Abstract

Method for checking the completeness of a container (2) provided with a plurality of objects (3) with the aid of a 3D camera (4) which provides a three-dimensional image (9) of the objects, and a multi-ROI (12), wherein the multi-ROI (12) comprises a plurality of ROIs (13) which have a shape which can be set by a user and are arranged within the multi-ROI (12) in a number of rows, columns and grid type which can be set by a user, with a learning mode and a temporally subsequent working mode, wherein the learning mode has the following steps: Capturing an image (9) of the objects (3) in the form of a three-dimensional pixel matrix by means of the 3D camera, wherein the pixel matrix contains distance values ​​of the captured objects (3) to the 3D camera (4) in one dimension and location values ​​in respective planes perpendicular thereto in the other two dimensions, Reproducing the captured image (9) as a two-dimensional image with the location values ​​in a first image area (8), Displaying input fields (10, 11, 14) for the shape or number of rows, columns and grid type of the ROIs (13), Displaying a two-dimensional multi-ROI (12) with a number of rows, columns and grid type of the ROIs (13) entered by the user and a shape of the ROIs (13) entered by the user in the first image area (8), Adjusting the size, position and rotation of the multi-ROI (12) in response to at least one user input, Adjusting the size of the ROIs (13) in response to at least one user input, Displaying height values ​​derived from the distance values ​​for each ROI (13) in a second image area (16) which is different from the first image area (8), Displaying a lower limit and / or an upper limit for the height values ​​in the second image area (16), Adjusting the lower limit and / or the upper limit for the altitude values ​​in response to at least one user input, and Switch to work mode, where the work mode has the following step: Displaying a state of a respective ROI (13), wherein the state of the respective ROI (13) is one of the states “overfill” and / or “underfill” and “good”, and wherein the “overfill” state is displayed when the height value of the respective ROI is above the upper limit, the “underfill” state is displayed when the height value of the respective ROI is below the lower limit, and the “good” state is displayed when the height value is not outside a predetermined limit.

Description

[0001] The invention relates to a method for checking the completeness of a container provided with a plurality of objects with the aid of a 3D camera which provides a three-dimensional image of the objects and preferably also of the container, and a multi-ROI, wherein the multi-ROI comprises a plurality of ROIs which have a shape which can be set by a user and are arranged within the multi-ROI in a number of rows, columns and grid type which can be set by a user, with a learning mode and a temporally subsequent working mode.

[0002] The present invention thus addresses the solution of completeness tasks. Typical applications include checking whether all bottles are in a crate, checking whether a milk carton is filled with all milk packages, or checking whether all yogurt cups are present within a package. Such a check is performed using a 3D camera, which provides a three-dimensional image of the container and the objects it contains. This image data can be evaluated in such a way that an automated process can be used to verify whether all the intended objects are actually present in the container.

[0003] EP 0 174 549 A1, for example, discloses a method for identifying bottles in a bottle crate. This method involves creating a three-dimensional image of the contents of the bottle crate using striped illumination and moving the bottle crate. Using stored 3D data, characteristic properties of the detected objects are identified.

[0004] DE 10 2008 025 658 A1 deals with a method for recording returned empties, in which the system is designed so that an optical object feature can be reliably detected and classified.

[0005] DE 43 05 328 A1 discloses a method for inspecting bottles in various bottle crates and focuses on detecting damage or foreign objects. For this purpose, translationally symmetrical grid patterns are applied. Deviations from symmetry are detected as damage to the objects or as foreign objects.

[0006] DE 10 2013 216 833 A1 discloses a 3D camera in which, in an initialization mode, detection areas are defined for a target / actual comparison. Based on a comparison of the captured depth image with a target depth image, status values ​​are determined and output, for example, to detect production deviations.

[0007] The method according to the invention is already described in principle in the subsequently published DE 10 2015 216 446 A1. In contrast to this subsequently published application, the present application additionally claims a selection of ROIs based on the existing grid type.

[0008] Such optical methods often work with the help of ROIs (Regions of Interest). An ROI is formed by a closed geometric element, such as a circle or square. The "Region of Interest" is then the area within the closed geometric element, e.g. within the circle or square. ROIs are used to carry out an optical analysis only in a specific recorded region, typically in the area in which an object is expected. Several ROIs can be combined to form a multi-ROI, so that a multi-ROI usually has a predetermined number of ROIs, which are arranged, for example, in a predetermined number of columns and rows.

[0009] When using a 3D camera, the captured image is typically available as a three-dimensional pixel matrix, where the pixel matrix contains distance values ​​of the captured objects to the 3D camera in one dimension and in the other two dimensions

[0010] It contains location values ​​in respective planes perpendicular to it. This allows each ROI within the multi-ROI to be assigned a distance value. If the expected object is actually present, the distance value between the object and the 3D camera will be smaller than if the object is missing. This allows thresholds to be defined for whether an object is detected as present or absent. Overall, this allows for checking whether the container is completely filled with objects.

[0011] The problem with such procedures known from practice is that setting these limit values ​​is sometimes complex and not very user-friendly.

[0012] Therefore, it is the object of the invention to provide such a method for checking completeness that is easy to set up.

[0013] This object is achieved by the subject matter of patent claim 1. Preferred developments of the invention are described in the subclaims.

[0014] According to the invention, a method is provided for checking the completeness of a container provided with a plurality of objects with the aid of a 3D camera which provides a three-dimensional image of the objects, and a multi-ROI, wherein the multi-ROI comprises a plurality of ROIs which have a shape which can be set by a user and are arranged within the multi-ROI in a number of rows, columns and grid type which can be set by a user, with a learning mode and a temporally subsequent working mode, wherein the learning mode comprises the following steps: Capturing an image of the objects in the form of a three-dimensional pixel matrix using the 3D camera, wherein the pixel matrix contains distance values ​​of the captured objects to the 3D camera in one dimension and location values ​​in respective planes perpendicular to it in the other two dimensions, Reproducing the captured image as a two-dimensional image with the location values ​​in a first image area, Display input fields for the shape or number of rows, columns and grid type of the ROIs, Displaying a two-dimensional multi-ROI with a user-specified number of rows, columns and grid type of the ROIs and a user-specified shape of the ROIs in the first image area, Adjusting the size, position and rotation of the multi-ROI based on at least one user input, Adjusting the size of the ROIs based on at least one user input, Displaying height values ​​derived from the distance values ​​for each ROI in a second image area that is different from the first image area, Displaying a lower limit and / or an upper limit for the height values ​​in the second image area, Adjusting the lower limit and / or the upper limit for the altitude values ​​in response to at least one user input, and Switch to work mode, where the work mode has the following step: Displaying a state of a respective ROI, wherein the state of the respective ROI is one of the states “overfill” and / or “underfill” and “good”, and wherein the “overfill” state is displayed when the height value of the respective ROI is above the upper limit, the “underfill” state is displayed when the height value of the respective ROI is below the lower limit, and the “good” state is displayed when the height value is not outside a specified limit.

[0015] It is therefore a key aspect of the invention that a display, preferably on a screen, takes place in two mutually different image areas. In the first image area, the captured image of the objects is reproduced as a two-dimensional image. In addition, the two-dimensional multi-ROI is displayed in the first image area, which can be adjusted by a user through appropriate inputs with regard to its size, position and rotation as well as with regard to the size of the ROIs. In the second image area, which is different from the first image area, height values ​​for each ROI derived from the captured distance values ​​are displayed, together with a lower limit and / or an upper limit for the height values, whereby these limits can be set by the user.This provides the user with a graphical user interface that allows the completeness check to be set up in a simple, fully visualized manner.

[0016] In the learning mode that follows the working mode, the state of each ROI is displayed, whereby the state of the respective ROI can assume at least two states, namely “good” on the one hand and “overfill” and / or “underfill” on the other. If only an upper limit or a lower limit is defined, the state of the respective ROI can assume exactly two states, namely “good” on the one hand and “overfill” or “underfill” on the other. If a lower limit and an upper limit are defined, the state of the respective ROI can assume three states: “good”, “overfill” and “underfill”. The “good” state is always displayed when the height value of the respective ROI does not lie outside a defined limit, i.e., where provided, not above the upper limit and not below the lower limit.

[0017] In principle, there are different ways to display the height values ​​derived from the distance values ​​for each ROI. For example, displaying them in the form of numerical values ​​is possible. However, according to a preferred embodiment of the invention, the height values ​​derived from the distance values ​​for each ROI are displayed in the second image area as lines with lengths corresponding to the respective height value. This makes it particularly easy and understandable for the user to visualize whether a particular object is located at its intended location in the container.

[0018] In this context, a numerical display that can be set by the user can also be provided for displaying the lower limit and / or the upper limit for the altitude values. According to a preferred development of the invention, the lower limit and / or the upper limit for the altitude values ​​are each displayed as boundary lines that preferably run perpendicular to the distances indicating the respective altitude values. In this way, the user of the method can directly recognize whether the respective altitude value lies in a corresponding ROI relative to the upper or lower limit. This makes setting the lower or upper limit as a switching point for detecting a specific state of the ROI particularly simple and reliable.

[0019] Finally, in this context, it is preferably provided that the boundary lines can be moved by the user using a cursor. This cursor can be moved, for example, using a mouse. Alternatively, a touch display can be provided so that the boundary lines can be moved using a finger gesture. Compared to entering numerical values, moving the boundary lines using a cursor or a finger gesture is significantly more intuitive, especially since this allows working directly relative to the displayed elevation values.

[0020] When displaying the captured image as a two-dimensional image with the location values ​​in the first image area, the display of distance data can generally be omitted. However, it is preferably provided that in learning mode, when displaying the captured image as a two-dimensional image with the location values ​​in the first image area, the distance data is displayed coded as colors, with different colors indicating different distances, e.g., the color "red" indicates a smaller distance than the color "blue."

[0021] In principle, the rows and columns of the multi-ROI do not all have to be the same width, so that different row widths and different column widths can be provided. However, according to a preferred development of the invention, it is provided that in learning mode the two-dimensional multi-ROI is automatically created on the basis of the number of rows and columns specified by the user such that all rows have the same row width and all columns have the same column width. According to a preferred development of the invention, it is further provided that the size, position and / or rotation of the multi-ROI and / or the size of the ROIs can be adjusted by the user using a cursor or a finger gesture. Furthermore, it is preferably provided that the shape of the ROIs is the same for all ROIs.

[0022] For the completeness check, it is generally sufficient if only one image of the objects is captured and displayed. However, according to a preferred embodiment of the invention, an image of the container is also captured and displayed. This is particularly helpful in learning mode because it facilitates the adjustment of the size, position, and rotation of the multi-ROI, namely by displaying the outer boundary of the container.

[0023] In working mode, the states of the respective ROIs can be displayed in different ways. Preferably, the states of the respective ROIs in working mode are displayed in the first image area coded as colors or grayscale values. This is thus a similar representation to that which can already be provided for the height values ​​in learning mode. Alternatively or additionally, according to a preferred development, the states of the respective ROIs are displayed in working mode in an image area different from the first image area, preferably in the form of a list. In particular, this list can be highlighted in color, so that, for example, the "Good" state is highlighted in green, while the "Overfill" and "Underfill" states are highlighted in red. This facilitates the rapid detection of inadmissible states and, at the same time, the determination of which object triggered this inadmissible state.Finally, in addition to the states “overfill”, “underfill” and “good”, an “invalid” state can also be provided, which is used if no reliable image data could be captured.

[0024] The invention is explained in more detail below with reference to the drawings using a preferred embodiment.

[0025] The drawings show Fig. 1 schematically shows the structure for carrying out a method for completeness testing according to a preferred embodiment of the invention, Fig. 2 a screen display for the completeness check method according to the preferred embodiment of the invention in a first method step, Fig. 3 a screen display for the completeness check method according to the preferred embodiment of the invention in a further method step, Fig. 4 a screen display for the completeness check method according to the preferred embodiment of the invention in a further method step, Fig. 5 a screen display for the completeness check method according to the preferred embodiment of the invention in a further method step, Fig. 6 a screen display for the completeness check method according to the preferred embodiment of the invention in a further method step and Fig. 7 a screen representation for the completeness check method according to the preferred embodiment of the invention in a further method step. Fig. 8 a screen display for the method with a honeycomb arrangement of the detection areas and a short, second honeycomb row. Fig. 9 a screen representation for the method with a honeycomb arrangement of the detection areas and a long, second honeycomb row. Fig. 10 a screen display for the method with a honeycomb arrangement of the detection areas and a second honeycomb row offset to the right. Fig. 11 a screen display for the method with a honeycomb arrangement of the detection areas and a second honeycomb row offset to the left. Fig. 12 a screen display for the method with a honeycomb arrangement of the detection areas and several selection fields for rows, columns and grid type

[0026] Out of Fig. Figure 1 schematically shows the setup for carrying out a completeness check method according to a preferred embodiment of the invention. Containers 2 are transported on a conveyor belt 1, each of which is to be filled with twelve objects 3 as required. The containers 2 are open cartons filled with objects 3 in the form of milk cartons.

[0027] With the help of a 3D camera 4, which is arranged above the conveyor belt 1, a container 2 provided with objects 3 can be optically captured. The 3D camera 4 delivers a three-dimensional image in the form of a three-dimensional pixel matrix, wherein the pixel matrix contains distance values ​​of the captured objects 3 from the 3D camera 4 in one dimension and location values ​​in respective planes perpendicular to it in the other two dimensions. This three-dimensional image captured by the 3D camera 4 is transmitted to a display device 5 in the form of a screen and can be displayed there, as explained in detail below. According to the preferred embodiment of the invention described here, a keyboard 6 and a pointing device 7, such as a mouse, are connected to the display device 5 in order to give a user the opportunity to make inputs.Alternatively, the display device 5 can be designed as a touch display, which makes a keyboard and a mouse unnecessary.

[0028] The sequence of a method according to the presently described preferred embodiment of the invention is as follows: First, the learning mode of the method is carried out. For this purpose, an image 9 of the objects 3 and the container 2 located below the 3D camera 4 is captured while the conveyor belt is switched off, i.e., while the container 2 is stationary. The captured image 9 of the objects 3 and the container 2 is then displayed in a first image area 8 on the display device 5, as shown in Fig. 2 can be seen. Fig. 2 also shows that the user of the present method is shown a field 10 for specifying the columns and a field 11 for specifying the rows. Based on the input of "4" for the columns and "3" for the rows, a multi-ROI 12 with a total of twelve ROIs 13, corresponding to four columns and three rows, is automatically created and displayed in the first image area 8. The column widths are automatically selected such that they are all the same, and the row widths are also automatically set so that they are all the same. This means that, according to the preferred embodiment of the invention described here, the multi-ROI is divided symmetrically into ROIs. In other words: in this step, the ROIs are optimally distributed across the multi-ROI.

[0029] In the Fig. 1 to 7, the method according to the invention is first shown using a regular structure, ie a structure in which the number of columns is the same for all rows. However, the method according to the invention can also be applied in the same way to a honeycomb structure according to Fig. 8 to 12. Furthermore, the method is also applicable to structures with a different number of columns per row, where the ROIs may also be irregularly distributed. Such arbitrary structures can then be captured, for example, by individually shifting the ROIs 13.

[0030] In the next step, the Fig. 3, the user of the method can select the shape of the ROIs. The corresponding buttons 14 offer the shapes of a square, a rectangle, a circle, and an ellipse. By selecting the circular shape using the mouse pointer 15, the ROIs 13 in the first image area 8 are given a circular shape.

[0031] In the next, from Fig. In the step of the method shown in Figure 4, the position and size of the multi-ROI 12 are now adjusted to the displayed image 9. This is done with the help of the mouse pointer 15, with which the multi-ROI 12 can be moved and its size changed by clicking on the multi-ROI 12.

[0032] In the next step, which in turn consists of Fig. 5, after the size and position of the multi-ROI, the rotation of the multi-ROI 12 is also adapted to the image 9. The edge of the multi-ROI 12 thus corresponds to the edge contour of the image 9, which is formed by the container 2. The individual ROIs 13 now lie exactly over the areas to be captured of the images of the objects 3 in the container 2. In addition, the mouse pointer 15 can be used to change not only the size of the multi-ROI 12 but also the sizes of the individual ROIs 13. The size of the individual ROIs 13 is preferably dimensioned such that they cover 80% of the images of the objects 3 to be captured.

[0033] Out of Fig. 6 now shows that in a second image area 16, different from the first image area 8, height values ​​derived from the distance values ​​are displayed for each ROI 13, specifically in the form of distances 17 corresponding to the height values. Perpendicular to these distances 17, an upper boundary line 18 for an upper limit for the height values ​​and a lower boundary line 19 for a lower limit for the height values ​​are displayed. These boundary lines 18, 19 can be moved up and down by the user using the mouse pointer 15. In this way, the switching points can be set to indicate different states of a respective ROI 13. If a respective height value of an ROI 13 lies on the upper or lower limit, or in between, this corresponds to the "Good" state. However, if the height value is above the upper limit, the "Overfill" state exists, and if the height value is below the lower limit, the "Underfill" state exists.In addition, an “invalid” state is possible, namely when no reliable altitude value has been recorded.

[0034] In the present case, which Fig. As can be seen in Figure 6, an object is missing in the leftmost column in the second row. For this reason, only a low height value is determined for the corresponding ROI, which is reflected in a correspondingly shorter distance 17' in the second image area 16. This distance 17' lies below the lower boundary lines 19, so the "underfill" condition is detected here.

[0035] As finally Fig. 7, this can be visualized in working mode by displaying the corresponding ROI 13' in the first image area 8 in a different color. Furthermore, a table display 20 is provided outside the first image area 8, indicating that the ROI 13' with the consecutive number "5" is in the "underfill" state.

[0036] Fig. Figure 8 shows a multi-ROI 12 for a honeycomb structure. The ROIs 13 are already shown circularly in image area 8. The shown structure has four rows, with the first and third rows each having 5 columns and the second and fourth rows each having 4 columns. Such a structure is particularly advantageous for round products in order to achieve a high packing density. To specify such a honeycomb structure, it is sufficient, for example, to specify the number of rows, the maximum number of columns, and preferably a property of the second row. Fig. In the case shown in Figure 8, the second row is characterized by being shorter than the first. The structure can then be specified as follows: rows 4, columns 5, grid type 'short'.

[0037] Fig. 9 shows a Fig. 8 complementary distribution of ROIs 13, where the second row is longer than the first. This structure can preferably be specified as follows: row 4, columns 5, grid type 'long'.

[0038] Fig. 10 and Fig. 11 show further embodiments in which all rows have the same number of columns, but in Fig. 10 the second line to the right and in Fig. 11 is offset to the left. The structures according to Fig. 10 and Fig. 11 receive the specification rows 4, columns 5, grid type 'Right' or grid type 'Left' accordingly.

[0039] Fig. 12 shows a display device analogous to the display device according to Fig. 2, in which, in addition to the fields for columns 10 and rows 11, an additional field 21 is provided for selecting the current structure or grid type. The grid type options, as already described above, are "short, long, right, left," and "regular" for cases where a honeycomb structure is not present.

[0040] The remaining elements correspond to those in Fig. 2 already presented, as well as the subsequent adjustment of the ROI analogous to the procedure for the regular structure according to the explanations on the Fig. 2 to 7.

[0041] The grid type options are, of course, not limited to the examples shown. It is also conceivable to store other arrangements in the selection menu. In particular, it is also conceivable to provide a "random" selection, which, for example, offers the possibility of individually positioning the ROIs in subsequent process steps. List of reference symbols 1 treadmill 2 containers 3 objects 4 3D camera 5 Display device 6 Keyboard 7 Pointing device 8 first image area 9 Image of the container with the objects 10 Field for columns of the multi-ROI 11 Field for rows of the multi-ROI 12 Multi-ROI 13 ROIs 14 Button for shape of ROIs 15 mouse pointers 16 second image area 17 routes 18 upper limit line 19 lower limit line 20 Table display 21 Field for grid type of the multi-ROI

Claims

[1] Method for checking the completeness of a container (2) provided with a plurality of objects (3) with the aid of a 3D camera (4) which provides a three-dimensional image (9) of the objects, and a multi-ROI (12), wherein the multi-ROI (12) comprises a plurality of ROIs (13) which have a shape which can be set by a user and are arranged within the multi-ROI (12) in a number of rows, columns and grid type which can be set by a user, with a learning mode and a temporally subsequent working mode, wherein the learning mode has the following steps: Capturing an image (9) of the objects (3) in the form of a three-dimensional pixel matrix by means of the 3D camera, wherein the pixel matrix contains distance values ​​of the captured objects (3) to the 3D camera (4) in one dimension and location values ​​in respective planes perpendicular thereto in the other two dimensions, Reproducing the captured image (9) as a two-dimensional image with the location values ​​in a first image area (8), Displaying input fields (10, 11, 14) for the shape or number of rows, columns and grid type of the ROIs (13), Displaying a two-dimensional multi-ROI (12) with a number of rows, columns and grid type of the ROIs (13) entered by the user and a shape of the ROIs (13) entered by the user in the first image area (8), Adjusting the size, position and rotation of the multi-ROI (12) in response to at least one user input, Adjusting the size of the ROIs (13) in response to at least one user input, Displaying height values ​​derived from the distance values ​​for each ROI (13) in a second image area (16) which is different from the first image area (8), Displaying a lower limit and / or an upper limit for the height values ​​in the second image area (16), Adjusting the lower limit and / or the upper limit for the altitude values ​​in response to at least one user input, and Switch to work mode, where the work mode has the following step: Displaying a state of a respective ROI (13), wherein the state of the respective ROI (13) is one of the states “overfill” and / or “underfill” and “good”, and wherein the “overfill” state is displayed when the height value of the respective ROI is above the upper limit, the “underfill” state is displayed when the height value of the respective ROI is below the lower limit, and the “good” state is displayed when the height value is not outside a predetermined limit. [2] Method according to claim 1, wherein the height values ​​derived from the distance values ​​are displayed for each ROI (13) in the second image area (16) as lines (17) with lengths corresponding to the respective height value. [3] Method according to claim 2, wherein the lower limit and the upper limit for the height values ​​are displayed as boundary lines (18, 19) running perpendicular to the distances. [4] Method according to claim 3, wherein the boundary lines (18, 19) can be moved by the user by means of a cursor or a finger gesture. [5] Method according to one of the preceding claims, wherein in the learning mode, when reproducing the captured image (9) as a two-dimensional image with the location values ​​in the first image area (8), the distance data are displayed coded as colors or gray values. [6] Method according to one of the preceding claims, wherein in the learning mode the two-dimensional multi-ROI (12) is automatically created on the basis of the number of rows and columns entered by the user such that all rows have the same row width and all columns have the same column width. [7] Method according to one of the preceding claims, wherein the shape of the ROIs (13) is the same for all ROIs (13). [8] Method according to one of the preceding claims, wherein in the working mode the states of the respective ROIs (13) in the first image area (8) are displayed coded as colors or gray values. [9] Method according to one of the preceding claims, wherein in the working mode the states of the respective ROIs (13) are displayed in an image area different from the first image area (8), preferably in the form of a list. [10] Method according to one of the preceding claims, wherein a time-of-flight camera is used as the 3D camera (4).

Images