Method and device for detecting at least one instance of an object during a workflow in a work environment and system for controlling a workflow in a work environment

Abstract

The invention relates to a method for detecting at least one instance of an object during a workflow in a work environment, comprising the steps (A) capturing an image of the work environment by a camera device, (B) transmitting the image to a control unit, (C) detecting a predetermined or predefinable source region for segmenting the instance in the image, whereby the instance is selected for segmentation by the control unit, and (D) segmenting the instance in the image starting from the detected source region located within the instance by the control unit and detecting the segmented instance.

Description

[0001] The invention relates to a method and a device for detecting at least one instance of an object during a workflow in a work environment.

[0002] In stationary applications and workflows where conditions continuously change over time, for example in the palletizing and depalletizing of goods or packages, stationary 2D and / or 3D sensor systems are often used to detect the goods or packages on belts and pallets, to segment individual instances of the goods or packages, and to determine the best gripping coordinates for a robot arm to grasp the instances.

[0003] Segmentation can be achieved using algorithms based on edge detection in the image data or on comparison with predefined object shapes (CAD matching). Neural networks trained on annotated data can also be used for image segmentation. The quality of the segmentation can be improved by providing preliminary information. This includes, for example, marking a region of interest, selecting positive and negative points to mark an area inside or outside an instance, or providing object dimensions. However, using positive and negative points as preliminary information necessitates manual marking of each individual instance by a specialist.This makes segmentation complex and inefficient, and therefore does not represent an option for automated instance segmentation tasks of a priori unknown objects.

[0004] A method according to the preamble of claim 1 is known from US 9,802,317 B1. Further methods are described in US 2020 / 0279389 A1 and in the article “AI-based recognition of dangerous goods labels and metric package features” by Brylka et al. (2021).

[0005] It is an object of the invention to provide an improved method and a device which enable simple, fast, efficient and cost-effective detection of an instance of an object during a workflow in a work environment.

[0006] The problem is solved in a first aspect of the invention by a method having the features of claim 1 and in particular by the fact that the method comprises the steps: - Taking a picture of the work environment using a camera device, - Transferring the image to a control and monitoring unit, - Detecting a predefined or predefinable source region for segmenting the instance in the image, whereby the instance is selected for segmentation by the control unit, with the detection of the source region in the image of the working environment being performed automatically by the control unit, - Segmenting the instance in the image starting from the detected source region located within the instance by the control unit and recognizing the segmented instance, whereby the segmentation of the instance in the image is carried out completely automatically and without interaction.

[0007] The method serves to detect at least one instance of an object during a workflow within a work environment. In this application, a work environment is defined as a three-dimensional area required during the workflow. The workflow can comprise a series of similar and / or different work steps. The work environment could, for example, be an area of ​​a room, a hall, or a warehouse in which a workflow is carried out where the conditions continuously change over time.

[0008] The workflow might involve, for example, palletizing or depalletizing goods or packages, bin picking, or loading and unloading from a conveyor belt. This requires identifying individual items, i.e., individual goods or packages, and recording their position and geometric dimensions. These items may differ in shape and / or size, but they can all be assigned to the same object, "goods" or "package."

[0009] According to the procedure, a first step involves capturing an image of the work environment. The camera device used for this purpose can be stationary and may include 2D and / or 3D sensor systems for monitoring the work environment. The image can generally contain multiple instances, whereby, depending on the arrangement and / or orientation of the instances, the respective predefined or predefined output region may or may not be recognizable or detectable in the image. However, at least one instance will generally be arranged in such a way that the control unit can detect the output region within that instance.

[0010] The source region can be a marker attached to the instance that can be detected and recognized in the instance's image by the control unit. The source region can be structured and, in particular, exhibit an inhomogeneous brightness distribution. The source region forms a positive point or marker for the instance that is currently selected and subsequently segmented. Simultaneously, for the current segmentation step, the source region forms a negative point or marker for all other possible instances in the image.

[0011] After the image of the work environment is transferred to the control unit, the control unit detects the source region of any instance in the image and selects it for the subsequent segmentation step. In other words, detecting the source region sets a positive point as preliminary information in / for the instance's image, which selects the instance for segmentation. Simultaneously, the detected source region acts as a negative point for all other instances in the image, thus excluding them from the current segmentation step. Once the instance has been selected, the control unit segments the instance in the image starting from the detected source region.The segmented image of the instance can now be used to determine parameters of the instance, such as position, orientation, or other geometric features or dimensions, and transfer them to a work device in order to perform a work operation on the instance.

[0012] The detection of the source region in the image of the work environment by the control unit is automatic. Instance segmentation can thus be fully automated and, in particular, requires no interaction from a person with specialized knowledge. For example, it is no longer necessary to manually place positive points as preliminary information in the instance image, such as by clicking on the corresponding position, in order to select the instance for segmentation. Segmentation of the instance in the image is therefore completely automatic and interaction-free. Consequently, the method is also suitable for automated instance segmentation tasks of a priori unknown objects.

[0013] The process can be continued with the step of detecting a source region in a further instance and subsequently segmenting it within the image. These two steps can, in principle, be repeated until all instances have been segmented whose position and orientation allow for the detection and recognition of the source region in the respective instance.

[0014] However, since the position and orientation of the instances can change during the workflow, the image is generally only used for segmenting a single instance or a few instances and then replaced by a more recent image. In this case, the process starts again with the step of capturing an image of the work environment, followed by transferring this image to the control unit and subsequently detecting an output region.

[0015] By detecting the source region, the method enables simple, fast, efficient and cost-effective detection of an instance of an object during a workflow in a work environment.

[0016] The method can be used in all logistics-related applications, such as order picking and decommissioning (bin picking, palletizing, depalletizing), and track & trace applications. More generally, the method is applicable to any application that relies on visual perception.

[0017] According to one embodiment of the invention, the source region in the image of the working environment comprises an area that is smaller than the area of ​​the instance in the image of the working environment, in particular wherein the area of ​​the source region is less than 80%, preferably less than 60%, of the area of ​​the instance. The source region can be significantly smaller than the instance within whose boundaries it is located and for whose segmentation it serves as a positive point or positive marker. For example, the source region can be a small marker attached to the instance that can be detected and recognized by the control unit. It is understood that the source region should not fall below a lower limit of its area in the image to ensure reliable detection by the control unit.

[0018] According to one embodiment of the invention, the source region and the instance in the image of the working environment differ in shape and / or brightness distribution and / or the histogram of brightness values ​​and / or the cumulative histogram of brightness values. The brightness value can, for example, be the intensity value of an image element or pixel of the image. In particular, the source region and the image of the instance outside the source region differ in the aforementioned features. The source region and the instance thus do not appear similar in the image. Consequently, the source region does not serve as a reference or template for segmenting the instance within the framework of "template matching" or "matched filter" processes, but can differ from the instance in terms of structure, shape, and brightness.In this way, the output region serves purely as a positive marker used to select the instance for segmentation, and not as a reference region for the segmentation itself. The output regions of different detected instances can themselves exhibit a self-similar structure, shape, or brightness distribution. The output regions can thus be understood as positive markers that belong to the same class but may differ in detail.

[0019] According to one embodiment of the invention, the source region is a coded marker, in particular a barcode or a QR code. Such coded markers are often applied as standard to identify and assign objects or their instances. The position and orientation of the marker can generally be determined with high accuracy. The method can now use this marker already present on the instance to select and segment the instance. Only the coded marker itself needs to be recognized and detected; reading and / or decoding the individual data encoded in the marker is not required for detecting the selection region. Markers already present and applied to the instances can thus be used for cost-effective, robust, and efficient segmentation of the instances.

[0020] According to one embodiment of the invention, the control unit comprises a neural network, wherein the neural network receives an object description of the output region, in particular text-based and / or audio-based, decodes and transmits the object description into a pictorial representation, and detects the output region in the image of the working environment for segmenting the instance based on the object description transmitted into a pictorial representation. The object description can also include components that are already pictorially represented and thus already have a pictorial form. The preliminary information provided by the output region is therefore not limited to geometric information such as positive and negative points, regions of interest, and in particular not to the positions of coded markers.Rather, the preliminary information can include a textual object description, which can be used as input for segmentation models that combine images and text using neural networks.

[0021] According to one embodiment of the invention, a predefined orientation and / or position of the detected source region on the instance is used as additional information for segmenting the instance. Markers are often placed on the instances at predefined positions and with a predefined orientation, for example, parallel to an edge or to multiple edges of the instance. This information can be used by the control unit to further improve and make the detection of the source region and the segmentation of the instance even more efficient.

[0022] According to one embodiment of the invention, the image is a two-dimensional or three-dimensional representation of the work environment. The camera device can be, for example, a 2D RGB camera, a 3D time-of-flight (ToF) camera, or a 3D stereo camera. The 3D cameras provide image data that includes additional depth information. This depth information is not necessary for the execution of the process, but after segmentation, it can enable the determination of more precise geometric parameters or features of the instance for the work device, for example, the determination of better gripping coordinates for a robot arm.

[0023] According to one embodiment of the invention, the method further comprises determining geometric features, in particular the position and / or extent and / or orientation, of the segmented instance, and in particular, the determined geometric features being transferred to a working device. The working device may, for example, be a robot to which gripping coordinates for gripping the instance are transmitted, for example, during bin picking or in the context of palletizing or loading or unloading a conveyor belt.

[0024] The problem is further solved in a second aspect of the invention by a device according to claim 9, and in particular by the device comprising a camera device and a control unit, wherein the camera device is configured to capture an image of the working environment and transmit it to the control unit, wherein the control unit is configured to automatically detect a predetermined or predefinable output region within the instance in the image of the working environment and thereby select an instance for segmenting the instance, and wherein the control unit is configured to recognize the instance in the image by means of segmentation, wherein the instance is segmented starting from the detected output region arranged within the instance, and wherein the segmentation of the instance in the image is carried out completely automatically and without interaction.

[0025] The device according to the invention and its embodiments are designed to carry out the method according to the invention or one of its embodiments. The descriptions of the method and its embodiments apply accordingly.

[0026] Accordingly, the device's control unit is designed to detect an output region of any instance in the image and select it for subsequent segmentation. The control unit is further designed to segment the instance in the image from the detected output region after selection. The device thus enables simple, fast, efficient, and cost-effective detection of an instance of an object during a workflow in a work environment.

[0027] According to one embodiment of the invention, the control unit is further configured to determine geometric features of the segmented instance, in particular its position and / or extent and / or orientation, and in particular, the geometric features are transmitted to a working device. The working device can, for example, be a robot to which gripping coordinates for grasping the instance are transmitted, for example, during bin picking or in the context of palletizing or loading or unloading a conveyor belt.

[0028] Furthermore, in a third aspect of the invention, the problem is solved by a system for controlling a workflow in a work environment, comprising a device according to claim 9 or 10, a work device, in particular a robot, configured to perform steps of the workflow, and a control device configured to calculate control parameters for the work device based on geometric features transmitted by the device and to transmit these parameters to the work device. The control device can be an integral part of the device or arranged separately from the device. The control parameters can, for example, include coordinates such as optimal gripping coordinates for grasping the instance by a robot arm.

[0029] According to one embodiment of the invention, the control device is configured to calculate the control parameters from geometric parameters transmitted by the device using a neural network. This enables a fast and precise calculation of the control parameters from the geometric features of the instance, which speeds up the workflow and improves its quality.

[0030] The invention is explained below only by way of example with reference to the figures. Fig. Figure 1 shows an embodiment of the system according to the invention in a schematic representation in a side view, Fig. Figure 2 shows a section of the work environment of the Fig. 1 in an overview.

[0031] Fig. 1 and Fig. Figure 2 shows a schematic embodiment of a system 10 according to the invention. The system 10 comprises a device 12 for detecting an instance of an object, a work device 14, shown here schematically as a robot, and a control device 16. The system 10 is located in a hall 18, which forms a work environment. On the floor 18a of the hall 18 is a pallet 20 loaded with packages of goods 22, which is to be unloaded. For this purpose, the stacked instances 22a to 22h of the packages of goods 22 are lifted one after the other from the pallet 20 by an arm 14a of the robot 14 and positioned on a conveyor belt 24.

[0032] The device 12 comprises a camera device 26 and a control unit 28. The camera device 26 is arranged above the pallet 20 and includes an image sensor 30 in which a plurality of image elements are arranged and which is configured to generate a two-dimensional image of the work environment 18. The image sensor 30 can, for example, be a 2D RGB camera. The camera device 26 is configured to record a continuous sequence of image data of an area 32 of the work environment 18 by means of the image sensor 30.

[0033] As in the Fig. As shown in the top view of the pallet 20 loaded with goods packages 22 in Figure 2, a QR code 34 is affixed to the top of each goods package 22 facing the camera device 26. The QR code 34 can be identical for the individual instances 22a to 22h of the goods packages 22 or can be individually designed and differ from instance to instance.

[0034] The control unit 28 is configured to select any instance of the goods packages 22, for example instance 22b, in an image of the work environment 18 transmitted by the camera device 26, by detecting the QR code 34 located within the area of ​​instance 22b as the starting region for segmentation. The control unit 28 is further configured, after selection, to segment instance 22c in the image starting from the detected starting region 34 and to determine geometric features such as the position, extent, and / or orientation of the segmented instance 22c.

[0035] The control device 16 is designed to calculate control parameters for the working device 14 based on geometric features of instance 22c transmitted by the device 12 and to transmit them to the working device 14.

[0036] One embodiment of the method for detecting at least one instance of an object during a workflow in a work environment is described using System 10. Fig. 1 and Fig. 2 explained.

[0037] The workflow involves removing individual items 22a to 22h from goods packages 22 from a pallet 20 and positioning the removed items on a conveyor belt 24. This is described in the Fig. At the early stage of the workflow shown in Figure 1, many instances 22a to 22h of goods packages 22 are still on pallet 20. During the unloading of pallet 20, the work environment 18, and in particular the pallet 20 loaded with goods packages 22, is monitored by the camera device 12. A sequence of image data of an area 32 of the work environment 18 is generated. These image data each represent a current image of the work environment 18.

[0038] At the start of the unloading of pallet 20, a current image of the work environment 18 is transmitted to the control unit 28, which analyzes the image and searches for output regions 34, each defined by a QR code 34. In this example, the control unit 28 will detect two possible output regions on instances 22a and 22b in the image. The control unit 28 then selects one of the two detected QR codes as the output region 34 for subsequent segmentation, thereby selecting that instance, for example, instance 22b, for the subsequent segmentation process. The output region 34 thus forms an automatically set positive point or positive marker for instance 22b, which is now currently selected and will subsequently be segmented.At the same time, the initial region 34 forms a negative point or negative marker for the second possible instance 22a in the image for the current segmentation process, thus excluding it from the current segmentation process.

[0039] After instance 22b is selected, the control unit 28 segments and thereby recognizes it in the image of the work environment 18 based on the detected QR code 34. Following the segmentation process, the control unit 28 determines geometric features such as the position, extent, and / or orientation of the segmented instance 22b and transmits this information to the control device 16. Based on the geometric features transmitted by the control unit 28, the control device 16 calculates control parameters and transmits them to the robot 14. These control parameters may include, for example, optimal gripping coordinates for grasping instance 22b on the pallet 20. The control device 16 may utilize a neural network to determine these control parameters, which can perform tasks such as calculating three-dimensional coordinates.The calculated control parameters can be transmitted to the host of the robot 14 via cable or wirelessly.

[0040] After instance 22b has been removed from pallet 20 and lifted onto conveyor belt 24, the process can be repeated until pallet 20 is completely unloaded. Instance 22a can be detected on the same image in which instance 22b was already detected. Alternatively, instance 22a can be detected on a new image taken after the image in which instance 22b was detected. Specifically, for the detection of further instances 22c to 22f, it is essential to transmit another new image to the control unit 28 for the detection of the QR codes affixed there.

[0041] This method enables the rapid, efficient, and cost-effective identification of instances of an object during a workflow. Instance segmentation can be fully automated, eliminating the need for human interaction. For example, it is no longer necessary to manually add positive markers to the instance image as preliminary information before selecting it for segmentation, such as by clicking on the corresponding position. Segmentation of the instance within the image is thus entirely automatic and interaction-free. This method is therefore also suitable for automated instance segmentation tasks involving unknown objects. Reference sign 10 System 12 Device for detecting an instance of an object 14 robots 14a Robot arm 16 Control device 18 Work environment / hall 18a Floor of the hall 20 pallets 22 goods package 22a to 22h Instances of the goods packages 24 Conveyor belt 26 Camera device 28 Control and control unit 30 image sensor 32 Area of ​​the work environment 34 Origin region / QR code

Claims

[1] Method for detecting at least one instance (22a to 22h) of an object (22) during a workflow in a work environment (18), comprising the steps - Taking a picture of the work environment (18) using a camera device (26), - Transferring the image to a control and monitoring unit (28), - Detecting a predefined or predefinable output region (34) for segmenting the instance (22a to 22h) in the image, whereby the instance (22a to 22h) is selected for segmentation by the control and regulation unit (28), - Segmenting the instance (22a to 22h) in the image starting from the detected output region (34) arranged within the instance (22a to 22h) by the control and regulation unit (28) and recognizing the segmented instance (22a to 22h), characterized by, that the detection of the output region (34) in the image of the work environment (18) is carried out automatically by the control and monitoring unit (28) and the segmentation of the instance (22a to 22h) in the image is carried out completely automatically and without interaction. [2] Method according to claim 1, wherein the starting region (34) in the image of the working environment (18) comprises an area that is smaller than the area of ​​the instance (22a to 22h) in the image of the working environment (18), in particular wherein the area of ​​the starting region (18) is less than 80%, preferably less than 60% of the area of ​​the instance (22a to 22h). [3] Method according to claim 1 or 2, wherein the output region (34) and the instance in the image of the working environment (18) differ in the shape and / or brightness distribution and / or the histogram of brightness values ​​and / or the cumulative histogram of brightness values. [4] Method according to any of the preceding claims, wherein the output region (34) is a coded mark, in particular a barcode or a QR code. [5] Method according to one of the preceding claims, wherein the control and control unit (28) comprises a neural network, wherein the neural network receives, in particular, a text-based and / or audio-based object description of the output region (34), decodes and transmits the object description into a pictorial representation, and detects the output region (34) in the image of the working environment (18) for segmenting the instance based on the object description transmitted into a pictorial representation. [6] Method according to any of the preceding claims, wherein a predetermined orientation and / or a predetermined position of the detected output region (34) at the instance (22a to 22h) is used as additional information for segmenting the instance. [7] Method according to any of the preceding claims, wherein the image is a two-dimensional or three-dimensional image of the working environment (18). [8] Method according to any of the preceding claims, further comprising determining geometric features, in particular position and / or extent and / or orientation, of the segmented instance (22a to 22h), in particular wherein the determined geometric features are transferred to a working device (14). [9] Device (12) for detecting at least one instance (22a to 22h) of an object during a work process in a work environment (18), wherein the device (12) comprises a camera device (26) and a control and monitoring unit (28), wherein the camera device (26) is designed to capture an image of the working environment (18) and transmit it to the control and monitoring unit (28), wherein the control and regulating unit (28) is configured to automatically detect a predefined or predefined output region (34) in the image of the working environment (18) and thereby select an instance (22a to 22h) for segmenting the instance (22a to 22h), and wherein the control and regulating unit (28) is configured to recognize the instance (22a to 22h) in the image by means of segmentation, wherein the instance (22a to 22h) is segmented starting from the detected output region (34) located within the instance, and wherein the segmentation of the instance (22a to 22h) in the image is carried out completely automatically and without interaction. [10] Device (12) according to claim 9, wherein the control and regulating unit (28) is further configured to determine geometric features of the segmented instance (22a to 22h), in particular of position and / or extent and / or orientation, in particular wherein the geometric features are transferred to a working device (14). [11] System (10) for controlling a workflow in a work environment (28), comprising a device (12) according to claim 9 or 10, a work device (14), in particular a robot, which is configured to perform steps of the workflow and a control device (16) which is configured to calculate control parameters for the work device (14) based on geometric features of an instance (22a to 22h) transmitted by the device (12) and to transmit them to the work device (14). [12] System according to claim 11, wherein the control device (16) is configured to calculate the control parameters from geometric features transmitted by the device (12) using a neural network.

Images