Method and control unit for determining a change in position of a container

Abstract

The invention relates to a computer-implemented method (500) for determining a change in the position of a container (105), wherein the method (500) comprises a step of reading (510) a first image (130) representing the opened container (105) at a first point in time on a work surface (110), and reading a second image (135) representing the opened container (105) at a second point in time on the work surface (110) taken after the first point in time. Furthermore, the method (500) comprises a step of recognizing (520) a shape of a border (150) of an opening (120) of the container (105) in the first image (130) and a step of identifying (530) the shape of the border (150) of the opening (120) of the container (105) in the second image (135).Finally, the method (500) comprises a step of determining (540) the change in position of the container (105) using a detected rotation and / or translation of the shape of the border (150) in the first image (130) compared to the shape of the border (150) in the second image (135).

Description

State of the art

[0001] The present invention relates to a computer-implemented method and a control unit for determining a change in position of a container, as well as to a corresponding computer program product and a machine-readable storage medium.

[0002] In warehouse logistics, large shipping boxes (parcels) are often packed with many small items. If this packing process is to be monitored visually and automatically, it is necessary to always know the exact location of the parcel (parcel tracking). An open shipping box (parcel) can be reliably tracked using an active 3D camera (color and depth information per pixel per frame). The 3D camera is suspended centrally above the packing table and looks down onto the parcel. The parcel can be moved by an employee during the packing process, although it can be assumed that the parcel cannot be seen by the camera during the movement (obscured by the employee). In addition, the contents and the position of the flaps (parcel flaps) can change at will during these packing processes.If the corner patches are changed (for example, by changing the flap position or filling the parcel with items), the frequently used template matching approach fails because the initially stored patches no longer match those from the current image. Furthermore, the current approach is parameter-intensive and thus difficult to maintain and adapt to different conditions (sensor, measurement distance, etc.). Disclosure of the invention

[0003] Against this background, the approach presented here presents a computer-implemented method for determining a change in the position of a container, a control unit that uses this method, and finally a corresponding computer program product according to the main claims. Advantageous embodiments emerge from the respective subclaims and the following description.

[0004] The approach presented here provides a computer-implemented method for determining a change in the position of a container, the method comprising the following steps: - reading in a first image representing the opened container at a first time on a work surface and reading in a second image representing the opened container at a second time taken after the first time on a work surface; - detecting a shape of a border of an opening of the container in the first image; - identifying the shape of the border of the opening of the container in the second image; and - Determining the change in position of the container using a rotation and / or translation of the shape of the border in the first image compared to the shape of the border in the second image.

[0005] Preferably, the steps of the proposed method are carried out by means of a control unit or a computing unit or a computer.

[0006] In this case, a container can be understood as, for example, a carton or box that is open so that parts can be placed inside it. In this case, the container can be (unintentionally) moved or rotated when the parts are being placed in this container, so that it can slip on a work table or work surface during the entire placement process. A border around an opening in the container can be understood as, for example, a line or edge that forms a closed polygon and thus represents the upper end of the side walls of the container. The shape can be rectangular, round or generally multi-dimensional, depending on the shape of the container.This shape of the border can, for example, be recognized or identified in the recognition and / or identification step by determining a corresponding circumferential line in an image taken of the container lying or standing on the work surface, which line, for example, consists of image pixels or is drawn along corresponding pixels in the respective first or second image that depict a similar brightness and / or a similar height in relation to a camera. For this purpose, the camera can, for example, record a two- or even three-dimensional image of the container on the work surface, from which the corresponding shape of the border of the opening of the container can be recognized. In particular, identifying the shape can comprise recognizing or recognizing the shape that was recognized in the recognition step in the first image in the second image.The steps of recognition and identification can also be carried out in a similar way, although for the sake of clarity the procedure is usually only explained in more detail for the recognition step in the following description.

[0007] The approach presented here is based on the realization that even in the event of a slight movement or slippage of the container during packing with the corresponding parts, the shape of the container opening border often remains essentially the same. For this reason, it is advantageous to search for the corresponding shape of the opening border in the two images taken at different times. This shape usually remains the same or within a tolerance range. This allows the change in the container's position after the second time to be determined based on the shape of the container opening border detected in the two images.This also allows the packing process to be checked when parts are placed in the container, so that, for example, an automatic (for example, visual) check of completeness can be carried out if a list of parts is available that are to be packed in the corresponding container.

[0008] A favorable embodiment of the approach proposed here is one in which, in the recognition step, the border is recognized from the image using an edge detection algorithm, in particular, the border is recognized as a polygon and / or a rectangle. Such an embodiment offers the advantage of being able to quickly and cost-effectively detect the border of the container opening by using already available and technically mature approaches.

[0009] According to another embodiment of the approach presented here, in the step of recognizing the border, a polygon of measurement points can be recognized whose height above the work surface is the same within a tolerance range. For example, these measurement points can have the same height above the work surface within a tolerance range of ten percent. Such an embodiment offers the advantage of being able to utilize the most commonly used geometric dimensions of the container by using measurement points that have the same height within the tolerance range, where the opening is often arranged in a plane parallel to the work surface. This allows any measurement errors that may have occurred to be quickly and easily corrected or discarded for further calculations.

[0010] Another advantageous embodiment of the approach proposed here is one in which, in the recognition step, the shape of the border is recognized by evaluating at least one outer section of the container outside the opening, in particular wherein the outer section extends continuously away from the border of the opening. For example, such an outer section can be a flap of the container, for example of the carton or the box, which is folded outwards away from the opening and is, for example, continuously connected to the opening. In this way, the outer section will often form a surface which is, on the one hand, flat and / or, on the other hand, rises or falls evenly from an edge of the opening. This information can be used efficiently to identify the shape of the border of the opening of the container by determining the position and / or orientation of the outer section orthe outer sections, which, for example, border on different sides of the opening of the container, are evaluated.

[0011] According to a further embodiment of the approach proposed here, the shape of the border can be recognized in the recognition step using a detected distance of the outer section, which continuously increases or decreases with distance from the border, from an image sensor recording the first and second images. This allows, for example, the spatial position of this outer section(s) to be used again to identify or recognize the shape of the border of the container opening.

[0012] An embodiment of the approach proposed here is particularly advantageous in which, at least in the recognition step, the shape of the border is recognized by evaluating at least a second outer section of the container outside the opening, in particular wherein the second outer section extends continuously away from the border of the opening and / or wherein in a region between the outer section and the second outer section an image point has been detected which has a greater distance from an image sensor recording the relevant image to the work surface than a distance between the outer section and the image sensor. Such an embodiment offers the advantage of being able to further optimize the identification of the shape of the border by taking the second outer section or its spatial position into account.It can also be exploited that the two outer sections usually reveal an intermediate area through which the work surface on which the container is standing can be seen. If a jump in the depth of a three-dimensional image is detected in this intermediate area between two outer sections, this jump can be interpreted in such a way that the two outer sections are to be interpreted as corresponding flaps on the container opening, so that the shape of the border of the container opening can also be recognized or identified accordingly.

[0013] Another advantageous embodiment of the approach proposed here is one in which, in the detection step, the distance between the image sensor and the pixel is greater by more than one step value than half the distance between the image sensor and the outer section or half the distance between the outer distance and the work surface. Such an embodiment offers the advantage of allowing for reliable and, if possible, error-free detection or identification of the outer section(s) by taking the step value into account, which corresponds to half the height of the container, so that the shape of the border of the container opening can be clearly identified.

[0014] Furthermore, according to another embodiment of the approach proposed here, the shape recognized in the first image can be recognized as a closed line in the recognition step and / or the shape recognized in the second image in the identification step. This makes it possible to exploit the geometric relationship that the container opening is usually represented as a circumferential and thus closed line in an image, so that this relationship can be further used to improve the identification and / or recognition of the shape of the border of the container opening.

[0015] The change in position of the container can be identified particularly reliably if the shape of the border is identified in the identification step, and if the shape identified in the second image corresponds within a tolerance range to the shape recognized from the first image. For example, the tolerance range can be an area of ​​ten percent of the surface and / or a corridor twice or three times the width of the line recognized in the first image as the border of the container opening. Such an embodiment offers the advantage of being able to sufficiently take into account changes in the shape of the container opening to identify the change in position of the container, whereby these changes in the shape of the container opening can be caused, for example, by the insertion of bulky parts into the container.

[0016] A technically very simple embodiment of the approach proposed here is one in which, in the reading step, the first and / or second image is read by an optical image sensor positioned above the work surface, facing the work surface. This makes it possible to evaluate or determine the change in the position of the container using widely available and technically simple cameras as image sensors.

[0017] According to another embodiment of the approach presented here, in the step of determining a position change, a rotation angle and / or a displacement distance of the border detected from the first image relative to the border detected from the second image can be determined. Such an embodiment offers the advantage that the position change can be precisely identified, processed, or further evaluated using very few and easily mapped parameters.

[0018] The approach presented here further provides a control unit configured to perform or implement the steps of a variant of a method presented here in corresponding devices. This embodiment of the invention in the form of a control unit also allows the problem underlying the invention to be solved quickly and efficiently.

[0019] In this case, a control unit can be understood as an electrical device that processes sensor signals and outputs control and / or data signals depending on them. The control unit can have an interface that can be implemented in hardware and / or software. In a hardware implementation, the interfaces can, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the control unit. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In a software implementation, the interfaces can be software modules that are present, for example, on a microcontroller alongside other software modules.

[0020] The approach presented here further provides a control unit configured to perform or implement the steps of a variant of a method presented here in corresponding devices. This embodiment of the invention in the form of a control unit also allows the problem underlying the invention to be solved quickly and efficiently.

[0021] In this context, a control unit can be understood as an electrical device that processes sensor signals and outputs control and / or data signals depending on them. The control unit can have an interface that can be implemented in hardware and / or software. In a hardware implementation, the interfaces can, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the control unit. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In a software implementation, the interfaces can be software modules that are present, for example, on a microcontroller alongside other software modules.

[0022] Also advantageous is a computer program product with program code that can be stored on a machine-readable storage medium or a machine-readable carrier, such as a semiconductor memory, a hard disk memory, or an optical memory, and is used to carry out the method according to one of the embodiments described above when the program product is executed on a computer or a control unit. The machine-readable storage medium with the computer program product stored thereon is also advantageous.

[0023] The approach presented here is explained in more detail below using the attached drawings. They show: Fig. 1 shows a schematic application scenario of an embodiment of a presented control unit; Fig. 2 now shows a representation of how the container including the outer sections can be displayed in a first image or second image captured as a depth image; Fig. 3 a schematic representation of a model of a container; Fig. 4 a schematic representation of another model of a container; and Fig. 5 a flowchart of a method for determining a change in position of a container

[0024] In the following description of advantageous embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and having a similar effect, whereby a repeated description of these elements is omitted.

[0025] Fig. 1 shows a schematic application scenario of an embodiment of a control unit 100 presented here, which can be used to detect a change in position of a container 105. The container 105 can be designed as a box or carton that stands or lies on a work surface 110, which is used, for example, as a packing table, for example to sort parts or elements 115 through an opening 120 into the container 105, which can be done, for example, by a Fig. 1 not shown in a logistics center. In order to detect this change in position of the container 105, an image sensor 125, for example in the form of an optical camera, is arranged above this container 115 or above the work surface 110. This image sensor 125 can detect the container 105 standing on the work surface 110 and can thus look through the opening 120, for example onto the bottom of the container 105. In one embodiment, this image sensor 125 can be designed as a mono camera, or particularly advantageously as a stereo camera and / or as a camera with additional detection of a distance between parts of the container 105 and the image sensor, in order to advantageously provide height information in the images regarding the distance of the individual parts or components of the container 105 from this image sensor 125.

[0026] This image sensor 125 captures a first image 130 at a first point in time and a second image 135 at a second point in time following the first point in time. These images are each read into the control unit 100 via a read-in interface 140. The control unit 105 further comprises a determination unit 145 for recognizing a shape of a border 150 of the opening 120 of the container 105 from the first image 130. Furthermore, the control unit 105 has an identification unit 155 configured to identify the shape of the border 150 of the opening 120 of the container 105 from the second image 135.Using the shape of the border 150 of the opening 120 of the container 105 from the first image 130, determined by the determination unit 145, and the shape of the border 150 of the opening 120 of the container 105 from the second image 135, identified by the identification unit 155, the change in position of the container 105 can then be determined in a determination unit 160, and a change in position signal 165 representing this change in position of the container 105 can be output. This change in position signal 165 can then, for example, be used in a further, in the . Fig. 1, not shown, for example to track the filling of the container so that, for example, a number and / or position of the parts or elements 115 that are to be placed in the container 105 can be checked.

[0027] To improve the determination of the position of the container 105, outer sections 170 or their position or position relative to the border 150 of the opening 120 can also be taken into account. These outer sections 170 can be, for example, flaps or wings that are provided for closing the container 105, for example the carton, and that extend away from an interior of the opening 120. As will be explained in more detail below, the information that these outer sections 170, which, for example, as the first outer section 170a, the second outer section 170b, the third outer section 170c, and the fourth outer section 170d, each have different extension directions adjacent to an edge of the rectangular border 150 of the opening 120, can be utilized here.In this case, it can be evaluated, for example, that these outer sections 170 or at least one of the outer sections 170a, 170b, 170c and / or 170d is, on the one hand, itself flat or planar and, on the other hand, also has a continuously decreasing or continuously increasing distance from the border 150 in relation to the image sensor 125, which is caused, for example, by the fact that these outer sections 170 are bent or folded upwards or downwards away from the border 150 as flaps or wings.

[0028] Fig. 2 now shows a representation of how the container 105, including the outer sections 170, can be displayed in a first image 130 or second image 135 captured as a depth image. Height information or depth information is also included in this image, from which it becomes clear, for example, that the outer sections 170 of the container 105 are folded upward and are therefore displayed darker at an end opposite the opening 120 than at an end adjacent to the opening 120. From this representation of the first image 130 or second image 135, a point set of locations or points 200 can now be determined that lie at essentially the same height above the work surface 110 and that are connected, for example, as a circumferential polygon, so that it can be deduced from this that the border 150 of the opening 120 of the container 105 is formed by these points 200 or the formed point set.Through these points 200, the expected upper edge of the container 105 or the border 150 of the opening 120 can thus be “synthesized”.

[0029] If it is now recognized in the two images 130 and 135 acquired at different times that borders 150 and 150', respectively, can be formed from the point sets 200 acquired in the respective image, which are, for example, identical within a tolerance range, it can be concluded from this that this is the border 150 of the same container 105, which, however, has been rotated by the angle α and / or shifted by a corresponding displacement distance. In this way, by evaluating the first image 130 and the second image 135, the change in position of the container 105 can be determined using technically very simple means, which are often based on sophisticated image processing algorithms.

[0030] Fig. 3 shows a schematic sketch of a model of a container 105, which is now determined solely on the basis of a point cloud 300, as can be seen from the first image 130 or the second image 135. Here, the points of the border 150 of the opening 120 of the container can also be detected, for example, at the same, expected height d of the upper edge of the container 105 above the practical or work surface 110, wherein further scanning points 310 can be detected which represent the edges of the outer sections 170a, 170b, 170c and / or 170d and which in all probability have a distance from the border 150 which increases or decreases from the work surface 110, depending on whether these corresponding outer sections 170 are folded upwards or downwards.

[0031] Fig. Figure 4 shows a schematic representation of another model of a container 105, in which now, in addition to the Fig. 3, regions 400 are also shown, which represent regions between two of the outer sections 170a, 170b, 170c and / or 170d. In these regions, which lie between these outer sections 170, the Fig. 4, the image sensor 120 (not shown) can be used to look directly at the work surface 110, so that there is a jump in the height information from the first image 130 or the second image 135, by means of which the outer sections 170 can also be recognized as such. For example, this jump can be more than half of the Fig. 3, so that by evaluating such a jump value in the height information in the areas 400, the position of the outer sections 170 or, subsequently, the specific shape or position of the border 150 of the opening 120 of the container can be identified or recognized relatively precisely and reliably.

[0032] In summary, it should be noted that the approach presented here enables very robust detection of positional changes of the parcel or container 105 (e.g., flaps folded up or down) that frequently occur during the packing process. If the parcel tracking or the tracking of the container 105 is lost, the entire packing process does not need to be aborted and restarted. This eliminates high costs, as more parcels can be packed per hour, which has proven to be a key factor for cost-effectiveness in warehouse logistics. Furthermore, the approach presented here is less parameter-intensive and thus easier to configure for new applications (different sensor models, parcel sizes). This greatly increases the scalability of the approach presented here.

[0033] Furthermore, application-specific extensions can be easily incorporated, for example, a flap model. To further increase the robustness of the approach presented here (in particular, correct tracking during large displacements while the packer has covered the parcel or container 105 for the (image) sensor 125 with his body), additional flap points are generated in the model. These can be registered with the outer edge of the flaps. This exploits the fact that the flaps can only be rotated around the corresponding upper parcel edge. The lateral (short) edges of the flaps often move within the plane that runs perpendicularly in extension of the other parcel edge. This means that when projected onto the expected parcel height, with correct tracking, they coincide with the flap model points, as shown in the Fig. 3. In accordance with this insight, when registering the flap model, the vertical distance is also ignored (unlike with rim points) for the creation of, for example, ICP correspondences for use with the ICP algorithm ("closest points"). Interflap points can also be used. To robustly detect tracking problems, the measured depth is checked at so-called "inter-flap points."

[0034] The four inter-flap points used here are located between the flaps in the Fig. 4 designated areas 400. At these points or areas 400, the parcel or the parcel contents or the container 105 or the contents of the container 105 will not be visible in the depth image. Since the packing table or the work surface 110 should also be empty during the packing process, apart from the parcel or container 105, it can be expected that the measured depth here approximately corresponds to the depth of the packing table or the work surface 110. This can also be checked to determine successful tracking or tracing of the container 105 or its change in position.

[0035] Fig.5 shows a flowchart of a method 500 for determining a change in position of a container, wherein the method 500 comprises a step 510 of reading in a first image representing the opened container at a first point in time on a work surface and a step 520 of reading in a second image representing the opened container at a second point in time taken after the first point in time on a work surface. The method 500 further comprises a step 520 of recognizing a shape of a border of an opening of the container in the first image and a step 530 of identifying the shape of the border of the opening of the container in the second image. Finally, the method 500 comprises a step 540 of determining the change in position of the container using a rotation and / or translation of the shape of the border in the first image compared to the shape of the border in the second image.

[0036] The embodiments described and shown in the figures are selected only as examples. Different embodiments can be combined with each other completely or with regard to individual features. Furthermore, one embodiment can be supplemented by features of another embodiment.

[0037] Furthermore, the process steps presented here can be repeated and carried out in a different order than that described.

[0038] If an embodiment comprises an “and / or” link between a first feature and a second feature, this is to be read as meaning that the embodiment according to one embodiment has both the first feature and the second feature and according to another embodiment has either only the first feature or only the second feature.

Claims

[1] Computer-implemented method (500) for determining a change in position of a container (105), the method (500) comprising the following steps: - reading (510) a first image (130) representing the opened container (105) at a first time on a work surface (110) and reading a second image (135) representing the opened container (105) at a second time on the work surface (110) taken after the first time; - recognizing (520) a shape of a border (150) of an opening (120) of the container (105) in the first image (130); - identifying (530) the shape of the border (150) of the opening (120) of the container (105) in the second image (135); and - determining (540) the change in position of the container (105) using a detected rotation and / or translation of the shape of the border (150) in the first image (130) compared to the shape of the border (150) in the second image (135). [2] Method (500) according to claim 1, wherein in the step (520) of recognition, the border (150) is recognized using an algorithm for edge detection from the first and / or second image (130, 135), in particular wherein the border (150) is recognized as a polygon and / or as a rectangle. [3] Method (500) according to one of the preceding claims, wherein in the step (520) of recognizing the border (150) is recognized as a polygon of measuring points (200) whose height (d) above the working surface (110) is the same within a tolerance range. [4] Method (500) according to one of the preceding claims, wherein in the step (520) of recognition the shape of the border (150) is recognized by evaluating at least one outer portion (170, 170a, 170b, 170c, 170d) of the container (105) outside the opening (120), in particular wherein the outer portion (170, 170a, 170b, 170c, 170d) extends continuously away from the border (150) of the opening (120). [5] Method (500) according to claim 4, wherein in the step (520) of recognizing, the shape of the border (150) is recognized using a recognized distance of the outer portion (170, 170a, 170b, 170c, 170d) from an image sensor (125) recording the first (130) and second (135) image, which continuously increases or continuously decreases with distance from the border (150). [6] Method (500) according to claim 4 or 5, wherein at least in the step (520) of recognition, the shape of the border (150) is recognized by evaluating at least a second outer section (170b) of the container (105) outside the opening (120), in particular wherein the second outer section (170b) extends continuously away from the border (150) of the opening (120) and / or wherein in a region (400) between the outer section (170a) and the second outer section (170b) an image point has been detected which has a greater distance from an image sensor (125) recording the relevant image (130, 135) to the work surface (110) than a distance between the outer section (170, 170a, 170b, 170c, 170d) and the image sensor (125). [7] Method (500) according to claim 6, wherein in the step (520) of detecting, the distance between the image sensor (125) and the pixel is greater by more than one step value than half the distance between the image sensor (125) and the outer portion (170, 170a, 170b, 170c, 170d) or half the distance between the outer distance (170, 170a, 170b, 170c, 170d) and the work surface (110). [8] Method (500) according to one of the preceding claims, wherein in the step (520) of recognition the shape recognized in the first image (130) and / or in the step (530) of identification the shape of the border (150) recognized in the second image (135) is recognized as a closed line. [9] Method (500) according to one of the preceding claims, wherein in the step (530) of identifying the shape of the border (150) is identified if the shape of the border (150) identified in the second image (135) corresponds within a tolerance range to the shape of the border (150) recognized from the first image. [10] Method (500) according to one of the preceding claims, wherein in the step (510) of reading in, the first (130) and / or the second (135) image is read in by an optical image sensor (125) which is arranged above the work surface (110) with a view towards the work surface (110). [11] Method (500) according to one of the preceding claims, wherein in the step (540) of determining as a change in position a rotation angle (α) and / or a displacement distance of the border (150) recognized from the first image (130) in relation to the border (150) recognized from the second image (150) is determined. [12] Control unit (100) which is designed to carry out the steps (510, 520, 530, 540) of a method (500) according to one of claims 1 to 11 in corresponding devices (140, 145, 155, 160). [13] Computer program product with program code for carrying out and / or controlling the steps (510, 520, 530, 540) of the method (500) according to one of claims 1 to 11, when the program product is executed on a control unit (100). [14] Machine-readable storage medium with a computer program product for carrying out and / or controlling the steps of the method (500) according to one of claims 1 to 11, when the program product is executed on a computer or a control unit (100).

Images