Method for at least partially automatically controlling a container handling device for handling containers, and container handling device

Abstract

The invention relates to a method for the at least partially automatic, preferably fully automatic, controlling of a container handling device (1) for handling containers (10), which has a transport apparatus (2) for guiding containers (10) along a predefined transport path, depending on an occupancy degree of at least one transport region (20) along the transport path, wherein by means of at least one image capture apparatus (4), image data (E1, E2) are generated which image containers (10) located in the transport region (20) and on the basis of which an image evaluation apparatus (6) performs an evaluation, in particular a real-time evaluation, for determining an occupancy degree variable which characterises the occupancy degree of the transport region (20). According to the invention, the image evaluation apparatus (6) performs a segmentation of the image data (E1, E2), at least in some sections, for determining the occupancy degree variable.

Description

[0001] Method for at least partially automatically controlling a container treatment device for treating containers and container treatment device

[0002] Description

[0003] The present invention relates to a method for at least partially automatically controlling a container treatment device for treating containers and to a container treatment device.

[0004] The present invention further relates to a method and a training data generation device for automatically generating a training data set for training, in particular for retraining, a container recognition model of machine learning of a container handling device in which containers can be guided or are transported along a predetermined transport path.

[0005] By means of an image capture device, image data can be generated or is generated which is characteristic of a transport area along the transport path. The transport area is preferably an area within which the containers located therein are to be recognized. The image data can be fed to the container recognition model as input variables for carrying out an evaluation, in particular a real-time evaluation, with regard to the containers located in the transport area and are in particular fed to it. The containers are preferably plastic containers (in particular PET containers), containers whose main component consists of pulp and / or glass containers and / or cans. The containers can be containers from the beverage and / or food and / or cosmetics industries.For example, these can be cans or bottles, such as glass bottles, pulp bottles and plastic bottles.

[0006] The containers (to be identified) can be fully formed containers and / or not yet fully formed containers, such as preforms. They can also be empty or already filled with a product. Furthermore, the containers to be identified can be equipped with closures or not.

[0007] With the help of neural networks, various objects can be recognized. The present invention particularly relates to the recognition of containers or parts of containers (lids, bottle mouths, etc.).

[0008] In order for the neural network to be able to recognize these containers, it must be trained. This requires training data. The training data consists primarily of labeled images.

[0009] The objects to be detected later can be marked with so-called "regions of interest" (ROIs). These can preferably be rectangles, circles, or even polygons. If multiple relevant objects are present in an image, multiple ROIs must also be created. This process is called labeling. During subsequent training, the algorithm learns to recognize these ROIs in other situations. In current technology, the labeling process is either completely manual or semi-automated.

[0010] With manual labeling, the images are annotated by a user by hand. There are various approaches to semi-automated labeling. Most approaches use a connected image sequence (video). The user labels part of the images by hand, and with the help of an algorithm, this annotation data is extrapolated or interpolated to the following images. It is also possible to use an existing neural network for labeling. With this variant, the user must decide whether the objects have been labeled correctly. Regardless of the variant, manual intervention by a user is always necessary. Experience from state-of-the-art methods shows that a trained neural network cannot recognize every container. Retraining for the respective situation will always be necessary. Therefore, labeled data is required for training.Manual labeling takes a very long time and must be done by a worker.

[0011] In the field of filling systems, the traditional method for assessing occupancy levels relies primarily on the use of sensors. These sensors are strategically positioned along transport and buffer segments and act as observers responsible for monitoring and regulating individual machine operations. They measure and collect important occupancy data, and these measurements enable precise control over machine performance, taking into account the speed of the transporters. Therefore, the more accurate the measurements, the more precise the control over machine performance and transporter speeds. Accurately measured occupancy levels can indicate whether a machine should reduce its output or a transporter should increase its speed to prevent a congestion.

[0012] The sensors currently used to assess the occupancy level in the filling systems are congestion sensors. However, these congestion sensors only detect whether a section is completely full. The occupancy level from the congestion sensor onwards is therefore 100%. If the congestion sensor is not occupied, the occupancy level is between 0 and <100%. However, a more precise occupancy level cannot be determined with such congestion sensors.

[0013] However, a significant disadvantage of this traditional approach is the inaccuracy of these sensors. This inaccuracy can be particularly problematic when dealing with transporters or buffer systems consisting of multiple lanes or with complex geometries. As a result, the data provided by these sensors often does not accurately represent actual occupancy levels, creating challenges in maintaining precise control over transport speeds / machine performance in the system.

[0014] The effects of inaccurate occupancy data go beyond mere data inconsistencies. They directly influence the performance of the machines and production lines within the filling system. Machines controlled based on incorrect occupancy information can operate inefficiently, which can lead to production bottlenecks and reduced throughput. Furthermore, due to this inaccurate data, congestion and disruptions in transport may occur more frequently, as the control of transport speeds / machine performance is based on this inaccurate data. Congestion in transport not only impairs production but also carries the risk of damage to the transported products.

[0015] From EP 2 132 129 B1 a method for monitoring, controlling and optimising filling plants is known in which finely graded occupancy levels are determined for each individual buffer section, wherein in order to determine these finely graded occupancy levels a recorded image is analysed to determine the number of bottles or cans that are on the buffer section at the time of recording, wherein each individual bottle or can is recognised as an object of the type, i.e. as such.

[0016] EP 300 523 1 B1 discloses a method for counting objects on a conveyor belt. This method involves tracking the position of an extracted characteristic of each object in the image data in frames by extracting the characteristic in at least one additional subsequent frame recorded by the same camera.

[0017] The present invention is based on the object of overcoming the disadvantages known from the prior art and of providing a precise, fast, and robust method for controlling a container handling device as a function of the occupancy level of a transport area, as well as for determining the occupancy level of the transport area, and a corresponding container handling device. A further object is to provide a method and a training data generation device for generating training data for training, in particular for retraining, a machine-learning container recognition model of a container handling device, which can mark or label the training images as far as possible without user intervention.A further object is to provide a precise, fast and robust method for detecting containers located in a transport area of ​​a container handling device and a corresponding container handling device.

[0018] The object is achieved according to the invention by the subject matter of the independent claims. Advantageous embodiments and further developments of the invention are the subject matter of the dependent claims. In a method according to the invention for the at least partially automatic, preferably fully automatic, control of a container treatment device for treating containers, the container treatment device has a transport device for guiding containers along a predetermined transport path. The container treatment device is controlled depending on the occupancy level of at least one (predetermined and / or predeterminable) transport area (of the transport device) along the transport path.

[0019] In this case (in particular during operation of the container treatment device), image data (preferably at a common recording time) are generated by means of at least one image capture device (preferably with a plurality of image capture devices), which image data depict containers located in the transport area and on the basis of which an image evaluation device, in particular a processor-based one, carries out an evaluation, in particular a real-time evaluation, to determine an occupancy level variable characteristic of the occupancy level of the transport area.

[0020] Preferably, the image data are generated in the working operation during continuous transport of the containers by the transport device, i.e. without the image data acquisition by the image capture device influencing and / or causing and / or correlating with an influence on the transport speed of the containers.

[0021] In particular, the occupancy rate for the containers located in the transport area is preferably for the number of containers located in the transport area and / or for the area share of the transport area occupied by the containers located in the transport area.

[0022] Preferably, the transport area is arranged in a detection area of ​​the at least one image capture device such that the entire (predetermined and / or predeterminable) transport area and / or containers located in the entire (predetermined and / or predeterminable) transport area can be detected by the at least one image capture device. Preferably, the image data generated (with the at least one image capture device) are characteristic of the containers located in the transport area (at the time of recording).

[0023] The image capture device is preferably an image capture device of the container treatment device. The image evaluation device is preferably an image evaluation device of the container treatment device. The image evaluation device preferably evaluates the image data to be evaluated using a (particularly computer-implemented) container evaluation model.

[0024] The image capture device (or the plurality of image capture devices) may be an image recording device such as a camera (preferably black and white and / or color), a CMOS sensor (CMOS abbreviation for Complementary metal-oxide-semiconductor), a CCD sensor, a 3D sensor, an X-ray-based image recording device, an optical element, a thermal imaging camera, a stereo camera, a LIDAR camera and the like, as well as combinations thereof.

[0025] The image data (generated by the image capture device and / or to be transmitted and / or supplied to the image evaluation device) are preferably two-dimensional (spatially resolved) image data. The image data that can be transmitted (or is transmitted) and / or can be supplied or is supplied to the image evaluation device for evaluation purposes preferably do not include any depth information (measured directly or immediately by the image capture device), i.e., in particular, no (depth and / or distance) measurement values ​​measured in the direction of the recording direction of the image capture device, which are characteristic of a distance between the image capture device and the objects depicted in the image data.In other words, the image capture device preferably does not generate (and / or capture) a measured value that is solely characteristic of a distance and / or a (relative) position of the image capture device with respect to an imaged object. Such a distance and / or position of the image capture device relative to the objects imaged by it could be determined from image data generated by image capture device(s) at different positions (e.g., via stereo images and / or a LIDAR camera).It is preferably possible, as described in more detail in a subsequent section within the framework of the method for automatically generating a training data set for training a container recognition model using machine learning for a container handling device (to which particular reference is made here), to calculate back to the spatial coordinates using a 2D image (captured by the image capture device) (if, for example, the diameter of the detected or transported container and / or the container to be treated is known or, alternatively, a reference surface or reference line with a known geometric extent is captured by the image capture device). Preferably, a color image or a color video sequence or a color image sequence (for determining and / or generating the image data) is captured by the image capture device.However, it is also conceivable that the recorded, generated, and / or determined image data is a grayscale image or gray-value images. In other words, it is conceivable that achromatic image data is transmitted to the image evaluation device or image evaluation model (or supplied as input variables). This advantageously allows a larger transport area of ​​the container handling device to be captured or represented and evaluated in the image data.

[0026] The image capture device (or the plurality of image capture devices) is preferably suitable for recording (static) (individual) images and / or moving images or image sequences (or video sequences) and is intended or used for this purpose.

[0027] The image data (generated by the image acquisition device) to be fed to the image evaluation device (or the image evaluation model) can be image sequences and / or individual images and / or image data recorded (essentially) at a single recording time or (essentially) simultaneously recorded or acquired.

[0028] It is conceivable that the image data (to be supplied and / or transmitted to the image evaluation device and / or the image evaluation model) (on the basis of which the image evaluation device performs the evaluation) are image data obtained from more than one individual image. For example, two (different) image acquisition devices can each capture an image (preferably substantially simultaneously). Preferably, an image composed of multiple images and / or image data generated from multiple images is supplied to the image evaluation device and / or the image evaluation model.

[0029] Additionally or alternatively, it is conceivable that the image data (on the basis of which the image evaluation device carries out the evaluation) are generated based on a plurality of individual images (recorded and / or captured by the at least one image capture device), in particular recorded and / or captured at different recording times. For example, individual frames of a video sequence (e.g. recorded by the at least one image capture device) could also be used as individual images. For example, a plurality of frames (e.g. recorded consecutively in time) can be averaged and / or added to generate the image data (on the basis of which the image evaluation device carries out the evaluation). The preferably plurality of image capture devices are preferably synchronized and / or can be synchronized with one another for recording or generating the image data.The multiple image capture devices preferably cover the entire transport area with their capture ranges. However, it is also conceivable for the multiple image capture devices to record the image data at different capture times and process the image data (e.g., depending on the transport speed of the transport device and / or the relative arrangement of the image capture devices) in such a way that image data is obtained that is characteristic of the occupancy level of the transport area at the same time and / or a location and / or orientation of the containers located in the transport area.

[0030] The (at least one) image capture device can preferably be arranged above the transport area. It is conceivable that the image capture device is arranged obliquely above the transport area and thus captures image data in particular from a capture direction that forms an angle other than 90° with a transport plane. The transport plane can, for example, be the contact surface provided by the transport device, such as a transport track, with the containers during transport of the containers. If the containers are transported (upright), for example, the transport plane could be the surface of the transport device that the container bases can contact during transport.

[0031] The image capture device can be arranged symmetrically and / or centrally with respect to the width of the transport area, with the width direction being, in particular, a direction perpendicular to the transport direction or the transport path. However, a lateral arrangement of the image capture device with respect to the (central) transport path and / or the transport area is also conceivable.

[0032] Preferably, the image capture device is arranged vertically above the transport area (at least in part and preferably completely). In particular, the capture direction of the image capture device forms a 90° angle with at least a portion of the transport area and preferably with the entire transport area.

[0033] Preferably, the container handling device (in particular each separate transport device) has precisely one image capture device for capturing image data relating to containers located in transport sections. Preferably, preprocessed image data is fed to the image evaluation device and / or the image evaluation model. Preprocessing steps of the image data preferably include cropping (for example to the transport area or to the image data points depicting the transport area), sharpening, and / or changing the brightness. Preferably, there is no change to the color range of the image data, in particular no conversion of a color image into a grayscale image. However, it would also be conceivable for the color values ​​of the image data to be changed or for the color range of the image data to be changed.

[0034] However, it is also conceivable that the image evaluation device carries out pre-processing of the image data and / or (e.g., pre-processing steps mentioned above) (individually or in combination).

[0035] According to the invention, the image evaluation device performs at least partial segmentation of the image data to determine the occupancy level. The image data is, in particular, the image data to be evaluated by the image evaluation device (and / or supplied to the image evaluation model, in particular as an input variable). However, it is also conceivable for the image evaluation device to perform preprocessing (for example, as described above) of the image data to be evaluated (by it) before segmenting the image data obtained by the preprocessing.

[0036] The image evaluation model is in particular a (preferably semantic) segmentation model to which image data can be supplied as input variables and by means of which a segmentation of the image data supplied as input variables can be carried out.

[0037] The determined occupancy level variable preferably serves as a control variable for the container treatment device, for example, for the (automatic) execution of a treatment function on at least one and / or multiple containers. It is also conceivable that the at least one occupancy level variable is used as a control variable for controlling and / or regulating the container flow and / or a container throughput through the container treatment device and / or a transport speed and / or a container feed and / or container discharge.In particular, performing segmentation is understood to mean a (computer-implemented) process of image evaluation and / or image processing, carried out in particular by the image evaluation device, in which the image data (to be evaluated by the image evaluation device and / or supplied to the image evaluation model, in particular the segmentation model, as input variables) are divided into segments (into meaningful image parts). The segments preferably comprise a (particularly preferably contiguous) set of data points which is characterized by a specific or predetermined and / or predeterminable property or attribute or which fulfills a predetermined and / or predeterminable specific relationship (e.g., gray values ​​and / or color values ​​lie in a predetermined color range).

[0038] Preferably, the segmentation or division of the image data into segments is carried out with respect to the containers (to be treated). Preferably, the segmentation of the image data or division of the image data is carried out depending on an assessment of whether the data (at least partially) depicts a container (or whether it does not (at least partially) depict a container).

[0039] Preferably, the segmentation of the image data and / or the division of the image data into segments with respect to the containers (to be treated) takes place depending on a predetermined and / or predeterminable detection area on the containers (to be treated).

[0040] The detection area on the containers (to be treated) is understood in particular to mean that the detection area can be a partial area of ​​the container (e.g., the top and / or bottom of the container) or an area of ​​an element that is arranged on the container (preferably in a rotationally fixed and / or translationally fixed manner) (such as a closure and / or a feature of the container, such as a label). This offers the advantage that an area that is particularly easily identifiable by optical image analysis and / or image processing (e.g., by comparatively high contrast and / or conspicuous color values ​​in relation to the other areas of the container and / or objects surrounding the containers and / or the transport area, such as a (reflective) railing) can be specified or selected as the detection area.

[0041] In a preferred method, the detection area (of the containers to be treated) is selected from a group comprising lids, closures, can tops, can bottoms, container tops, container bottoms, container side wall areas, a mouth area of ​​the container to be detected, a container feature (such as a label applied to the container, for example), a logo arranged on the container, a closure, a container set (entire sets), and the like, as well as combinations and sub-areas thereof. The detection area is preferably a partial area of ​​a container depicted (in a two-dimensional image). The detection area can be characteristic of an element arranged on the container (such as a label) and / or of a (sub-)area of ​​the container.

[0042] For example, in a preferred method, the detection area may be container lids, such as bottle caps.

[0043] Preferably, no individual containers are detected during segmentation. In particular, the object boundaries between adjacent containers are not determined during segmentation. In other words, the segments obtained by segmentation (e.g., in the image analysis model, in particular in the segmentation model as output variables) do not indicate object boundaries and / or the position and / or orientation of individual containers among several adjacent containers. In other words, a segment obtained by segmentation whose image data points depict a plurality of adjacent containers (at least in pairs) is not characteristic of one or the object boundaries running between the adjacent containers and / or of an alignment and / or orientation of the containers.

[0044] In particular, for example, it is not possible to distinguish solely on the basis of a segment obtained by segmentation, which combines image data points that depict at least one container (located on the transport area and captured by generating the image data), whether exactly one container is located on the transport area in a lying orientation on the transport area or whether two or more containers are instead located at its position in an upright orientation (and overall occupy approximately a comparable support area of ​​the transport area).

[0045] In a preferred method, the occupancy level variable is determined based on at least one segment obtained by the segmentation and preferably at least a plurality of segments obtained by the segmentation. This offers the advantage that a very fast and robust determination of the occupancy level variable is possible. The occupancy level variable is preferably determined based on the (determined) number of pixels or number of image data points of at least one segment and preferably of those segments which are assigned to container-imaging image data points. For example, this number or a variable derived therefrom can be compared with (at least) one comparison variable. The comparison can be carried out, for example, by forming a ratio and / or a difference. In addition, the comparison result obtained from the comparison can be compared with a (specified and / or specifiable) threshold value.

[0046] The (at least one) comparison value can be specified and / or can be specified (e.g. by an operator).

[0047] The (at least one) comparison variable can be determined or can be determined at least partially automatically and preferably fully automatically (by the container treatment device). It is conceivable that a setting mode of the container treatment device is provided for this purpose (described in more detail below), which preferably differs from an (intended) working mode of the container treatment device (with the highest possible production numbers or transport speeds) in which the containers are treated.

[0048] The (at least one) comparison variable is preferably stored in the image evaluation device. Preferably, the stored comparison variable is changeable, with the change particularly preferably being detectable by an operator and / or automatically (by the container handling device).

[0049] It is conceivable that the number of image data points and / or a measure of the area represented by the image data (e.g., relative to a plane within which the transport area extends) could be used as a comparison value, on the basis of which the segmentation is performed. This comparison value can serve as a reference value for the number of image data points (or measure of the area represented by the image data points) of the respective segments, thus enabling an objective evaluation.

[0050] In a further preferred method, the occupancy level is determined as a function of a transport area size, which is characteristic of an occupancy area and / or a maximum occupancy number of the transport area. This preferably enables a more precise evaluation of the image data because any background image data points, i.e., image data points that do not depict the transport area (or containers located therein) but rather an area surrounding the transport area, are not taken into account when evaluating the segments obtained by segmentation (or their segment sizes).

[0051] Preferably, the transport area size is chosen as (at least one) comparison value.

[0052] The maximum occupancy of the transport area refers in particular to the maximum number of containers that can be accommodated in the transport area.

[0053] A transport area size characteristic of a maximum occupancy of the transport area can, for example, also be understood as that area portion of the total area of ​​the transport area which is occupied by the containers at maximum occupancy, i.e., at the densest packing of the containers of a maximum number of containers that can be accommodated in the transport area. Such a transport area size could, for example, indicate or be characteristic of a number of image data points / pixels which each depict a container area of ​​one of the containers at maximum occupancy of the transport area (at the densest packing of the containers).

[0054] The occupancy area is preferably understood as a size representative of the area of ​​the transport area within which containers can be accommodated, transported, and / or guided, or a characteristic value thereof. Thus, the occupancy area can represent a measure of the area of ​​the actual transport area. However, it is also conceivable that the occupancy area represents a measure of the (total) area of ​​the transport area depicted (in the image data).

[0055] The comparison size and / or the transport area size can be independent of the type of container (to be handled). If, for example, a conveyor belt serves as the transport device, the total area of ​​the conveyor belt located in the transport area can be selected as the transport area size (or a corresponding size that is characteristic of this total area depicted in the image data). In particular, containers with a round cross-section (such as cans or bottles known from the beverage industry) do not completely fill this transport area when packed very tightly on the conveyor belt or in the transport area. Nevertheless, the total area of ​​the conveyor belt can already provide a good approximation for a size characteristic of a maximum occupancy rate of the transport area (especially for containers of known (cross-sectional) geometry).

[0056] The aforementioned threshold value can be dependent on the cross-sectional area occupied by the container (to be treated). For example, several threshold values ​​could be stored in the image analysis device (e.g., in the form of a database), each assigned to a different container type.

[0057] In this case, cross-sectional area and / or cross-sectional geometry is understood to mean, in particular, a corresponding cross-sectional size in relation to a cross-sectional plane which is perpendicular to a detection direction of the (at least one) image detection device.

[0058] The reference size and / or the transport area size may depend on the type of container (to be treated) (in particular, a cross-sectional area and / or a diameter). This may be the case, for example, if the transport area size is selected to be characteristic of a maximum occupancy, since a denser or less dense packing may be possible depending on the cross-sectional geometry.

[0059] The container handling device preferably has at least one human-machine interface (HMI) and / or an input device via which the transport area size can be (manually) entered (in particular by an operator of the container handling device). It is also conceivable for the container handling device to have a receiving device via which the transport area size can be transmitted to the container handling device (for example, via a wireless communication connection) (for example, from a (backend) server, in particular a cloud-based one (for example, from a manufacturer of the container handling device)).

[0060] It is also conceivable for an operator's terminal device, in particular a mobile one, to capture image data and / or position data depicting at least sections of the transport area, which preferably indicate a relative arrangement and / or orientation of the terminal device to the transport area and / or are characteristic thereof, and transmit them to the receiving device. Particularly preferably, the transport area can be (geometrically) measured by the terminal device, in particular a mobile one, and variables characteristic of a geometric extent of the transport area are preferably transmitted to the receiving device of the container handling device.

[0061] In a further preferred method, the transport area size is determined automatically (by the container handling device). This offers the advantage of a very user-friendly commissioning process, in which the operator does not have to perform any measurements and / or calculations, etc.

[0062] The container treatment device can preferably be operated in a setting mode, in which the transport area size is determined, in particular optically. The setting mode preferably differs from an (intended) working mode of the container treatment device (with the highest possible production numbers or transport speeds), in which the container treatment device is operated for (normal) production or treatment of the containers. For example, the transport speed of the transport device can differ at least in the section of the transport area.

[0063] In a further preferred method, in order to determine the transport area size, image data are generated or captured (and evaluated), in particular automatically, by means of the image capture device in a setting state of the transport device in which a maximum number of containers that can be accommodated therein are accommodated in the transport area.

[0064] The setting state is understood in particular to mean a state of the transport area or the transport device of the container treatment device in which there is a maximum occupancy of the transport area with (the containers to be treated) (i.e. in a state of the densest packing of the containers in the transport area, in which in particular the gaps between the containers are minimal).

[0065] Preferably, the transport area size is determined based on the captured and / or generated image data, which depict a setting state or a maximum occupancy of the transport area, i.e. preferably the quantity and / or number of image data points (e.g., pixels) that depict a part of one of the containers in the transport area. Preferably, the transport area size is determined based on the image data thus captured and / or generated by segmenting this image data. Preferably, a size (in particular as a transport area size) that is characteristic of the segment size (e.g., by determining the area and / or the number of data points) of the segment or of all those segments that depict a container and / or a detection area of ​​the container is determined.

[0066] This transport area size can then preferably be used as a comparison size with which, in a working operation of the container treatment device, the segment or segments determined therein by segmentation, which represent a container and / or a detection area of ​​the container, or a size derived therefrom, is compared.

[0067] It is also conceivable that several different setting states (within the same container type) are possible. This allows a slightly different arrangement of the containers, for example, offset in the transport direction. It is preferred that the image capture device captures and / or generates image data at least once for each setting state. Thus, a transport area size can be determined for each setting state, and the transport area size used in a working operation of the container handling device can be obtained by averaging these individual transport area sizes.

[0068] The use of the image capture device offers the advantage that this image capture device has the same configuration and / or arrangement, or can be configured and / or arranged in the same way, as in the operating mode of the container handling device, in which the image capture device generates and / or captures the image data for determining the occupancy level. Thus, these two images or image data can be directly compared with each other.

[0069] The setting operation of the transport device (or its execution) can preferably be triggered by an operator. The setting state can preferably be reached by the setting operation (and thus indirectly by triggering the setting operation). In other words, the setting state can be generated by executing the setting operation. For example, by executing the setting operation, containers can be transported into the transport area (by the transport device) until maximum occupancy is reached. Approaching and / or reaching a maximum occupancy of the transport area can be monitored by means of the image capture device and the image data thus captured by the transport area. In this way, image data can be captured with the image capture device at intervals in time (at different capture times), and a transport area size can be determined on the basis of this data.If this (continues to) increase compared to previous image data from earlier acquisition times, it can be assumed that maximum occupancy has not yet been reached. If, however, the transport area size remains constant, it can be assumed that maximum occupancy has been reached.

[0070] It is also conceivable that the setting operation is performed multiple times (and the transport area is emptied in the meantime and / or the containers located in the transport area are transported further), thus generating a setting state multiple times. It is conceivable that the transport area size is averaged and / or determined based on multiple setting states.

[0071] It is also conceivable that the transport area size is determined, for example, when the transport area is empty and / or independently of the occupancy of the transport area and on the basis of image data captured and / or generated here (by means of the image capture device) that depicts the transport area. For example, a boundary of the transport area could be detected (at least in sections) through object recognition (which could be carried out by the image evaluation device or an image recognition device external to the container handling device). Preferably, based on the detected boundary of the transport area, an area and / or a number of image data points (such as pixels) that lie within the boundary are determined. These sizes (or sizes derived therefrom and / or characteristic thereof) can be used, for example, as the transport area size.

[0072] The object recognition can, for example, detect and / or identify a side wall and / or a railing of the transport area and / or a side cover and / or an edge or a transition of the transport area to a surrounding area (e.g., the hall floor or railing). Preferably, a boundary (e.g., a lateral boundary) can be derived from this.

[0073] Additionally or alternatively, it is possible for a user to mark and / or specify at least one boundary (the entire boundary or border is also conceivable) of the transport area depicted in the image data, for example via the human-machine interface and / or the receiving device of the container handling device. Depending on the boundary, the transport area size can be, for example, a (area) size characteristic of the transport area and / or a number of image data points (depicting the transport area) corresponding to the area of ​​the transport area.

[0074] A (manual and / or automatic) geometric marking or identification of the arrangement of the transport area image in the image data offers the advantage that only image data points that depict a portion of the transport area are used when determining the occupancy level and / or the transport area size. This can ensure that image areas that are incorrectly identified as containers (e.g., due to reflections or mirroring on a metallic object such as a railing) but are located outside the transport area influence the determination of the occupancy level.

[0075] It is also conceivable that the transport area size is determined using a machine learning image analysis model that has been trained with a large number of image data and the associated transport area sizes.

[0076] In a further preferred method, the image data are evaluated to determine the occupancy level in such a way that a plurality of containers, in particular adjacent to one another, located in the transport area are recognized as a uniform container cluster, which does not indicate any differentiation between individual containers. This offers the advantage of faster image data evaluation because no individual containers or their orientation need to be distinguished. In particular, each container cluster in the transport area is assigned a (separate) segment obtained through segmentation.

[0077] In contrast to container recognition, in which each container is recognized and / or identified separately (and labeled and / or marked as such), individual containers within a uniform container cluster are no longer distinguishable after segmentation. In particular, the occupancy level is not determined using image recognition or an analysis of the image data, which recognizes all containers in the transport area in a distinguishable or individually identifiable form.

[0078] In a further preferred method, the occupancy level is determined independently of the position and / or orientation of the containers located in the transport area. This also offers the advantage of enabling a fast, robust, and meaningful determination of an evaluation result for the occupancy level.

[0079] In particular, a segment obtained by segmentation no longer contains any information about the number and / or orientation of the containers located in the transport area corresponding to the segment (which are represented by the image data points of this segment).

[0080] In a further preferred method, the segmentation is a semantic segmentation. In particular, the segmentation is not an instance segmentation. This also achieves a fast and robust determination of the occupancy level, which is of fundamental importance given the sometimes extremely high container transport speeds typical in the beverage industry.

[0081] In a further preferred method, a “container” class selected from several, preferably two, classes is assigned to an image data point by means of the segmentation if the image data point is assigned to a container and / or a container cluster, and / or a “background” class is assigned if the image data point is not assigned to a container and / or a container cluster.

[0082] It is conceivable that such assignment and / or allocation could be based on the detection area of ​​a container (such as a bottle cap). Thus, a container lying on the transport area could also be recognized by its bottle cap.

[0083] In a further preferred method, the occupancy level is determined on the basis of those image data points to which the class “container” was assigned during segmentation.

[0084] In a further preferred method, the segmentation is performed based on a predefined and / or predefinable color range. In particular, the segmentation can be performed based on a predefined color filter. The color range can (at least partially and preferably completely) include and / or consist of those color values ​​that the detection area of ​​the container and / or the container has. The color filter is preferably selected such that it identifies those image data points that (essentially) have the color values ​​of the detection area and / or the container. This advantageously allows for a quickly implemented and simultaneously efficient segmentation.

[0085] In a further preferred method, the segmentation is carried out on the basis of a machine learning segmentation model.

[0086] The machine learning segmentation model is preferably based on an (artificial) neural network. The neural network is preferably designed as a deep neural network (DNN), in which the parameterizable processing chain has a plurality of processing layers, and / or a convolutional neural network (CNN), and / or a recurrent neural network (RNN), and / or other DNN layer classes.

[0087] Preferably, the machine learning segmentation model is a segmentation model that has already been (fully) learned or trained.

[0088] Preferably, the machine learning segmentation model was / is trained using a training dataset comprising a plurality of image data (depicting the transport area with the containers located therein) (to be supplied as input variables for the segmentation model) and associated segmented data obtained after segmentation. The segmentation can be performed as described above.

[0089] The training data set can only comprise data relating to exactly one container type and / or to a given container handling device. However, it is also conceivable for the training data set to comprise image data that was captured and / or generated in various container handling devices (of the same and / or similar design) (as well as their respective segmented data). The use of image data across container handling devices offers the advantage that the segmentation model trained with it is more robust against structural changes (e.g., the course of the transport area) and / or changes in brightness and / or environmental changes. Preferably, the training data set to be generated serves or is used for (further) fine-tuning of the container recognition model.

[0090] It is conceivable that the container recognition model has already been trained with a general training data set, preferably independent of a specific or concrete container handling device.

[0091] In a further preferred method, at least one further transport area and preferably a plurality of transport areas are specified. Preferably, a (contiguous) transport section of the transport device is divided into the plurality of transport areas and is particularly preferably composed of these (in particular without overlap). The transport areas are arranged in particular along the transport path.

[0092] Preferably, image data for the at least one further transport area or for the respective transport areas of the plurality of transport areas are generated by means of at least one image capture device. A single image capture device can capture the image data. However, it is also conceivable for multiple image capture devices to collect the image data. It is also conceivable for exactly one image capture device to be assigned (uniquely) to each transport area.

[0093] Preferably, the generated image data depict the containers located in the respective (at least one further) transport area.

[0094] Preferably, the image evaluation device carries out an evaluation, in particular a real-time evaluation, on the basis of the respective image data in order to determine an occupancy level value with regard to the respective transport area, which is characteristic of the containers located in the respective (or in the at least one further) transport area.

[0095] In order to determine the respective occupancy level, the image evaluation device segments the image data at least in sections.

[0096] Preferably, the respective occupancy level variables are each determined using one of the (preferred) methods described above. It is conceivable that each occupancy level variable serves as a control variable for controlling a (separate) drive device of the transport device. In other words, each drive device of the transport device can be assigned a transport area, depending on whose (determined) occupancy level this drive device is controlled. However, it is also conceivable that a drive device is controlled depending on the (determined) occupancy levels of several transport areas.

[0097] Preferably, a transport section of the transport device is subdivided into a plurality of transport areas depending on the position of drive devices of the transport device and / or the position of areas of the transport device controlled by different drive devices.

[0098] Additionally or alternatively, the subdivision of a (particularly contiguous) transport section into a plurality of transport areas (for each of which, in particular, an occupancy level value is determined) can depend, at least in sections, on at least one geometric parameter that is characteristic of a (at least in section) geometric course of the transport section. The geometric parameter could, for example, be a parameter characteristic of a curvature and / or a width (of a section of the transport area) and / or a branching and / or a junction. Thus, in the case of strong curvatures (small radii of curvature) and / or comparatively large widths of the transport area in this section (compared to other sections of the transport area), a smaller subdivision into transport sections (in relation to a geometric extent along the main transport path) could be selected.Width refers in particular to the geometric extent of the respective section perpendicular to the main transport path.

[0099] In the proposed preferred method, the focus is on segmentation, in particular on the segmentation of, for example, bottle caps in filling systems, especially in the beverage industry. The preferred approach is to use a (particularly semantic) segmentation model such as a neural network or a color filter and to regulate the performance of the container handling device and / or individual machines in the system based on the number of detected bottle cap pixels in relation to the total number of pixels in a particular buffer or transport section (the transport range size). The system uses image capture devices and image processing computers for data acquisition. The particularly semantic segmentation model, which can be a neural network or a color filter, is used to identify bottle caps.The associated data processing systems apply advanced image analysis and segmentation techniques to identify and segment bottle caps based on pixel analysis.

[0100] The process preferably involves scanning buffer and transport segments in the filling system using cameras. However, the primary focus is on segmenting bottle caps based on pixel count. This segmentation process is carefully performed regardless of the position or orientation of the bottle caps. The system totals the number of detected pixels corresponding to bottle caps and compares this pixel count with the total number of pixels in the buffer or transport section. Based on this comparison, the system regulates the throughput capacity of each machine in the filling system.

[0101] This preferred approach, using advanced semantic segmentation models and pixel-based analysis, offers a more accurate representation of occupancy levels, improving machine performance and reducing the likelihood of transport jams. This not only leads to smoother production but also contributes to a more reliable and efficient filling process in the beverage industry.

[0102] Preferably, the container treatment device is selected from a group comprising a transport device (such as a conveyor belt) for transporting the containers, a buffer device for temporarily buffering containers, a pasteurization device (such as a tunnel pasteurizer), a heating device for heating a preform, a forming device for forming a plastic preform into a plastic bottle, a sterilization device for sterilizing a container, in particular a plastic preform, a manufacturing device for producing a glass bottle, a filling device for filling a container with a product, an inspection device for inspecting a plastic preform or a bottle, a labeling device for labeling a container, a closing device for closing a container, in particular a filled container, a control device,a packaging device, a direct printing device for printing on a container, a collating device for assembling a plurality of containers into a collection or bundle, and the like. The container stream is preferably a (particularly continuous) stream of successive or consecutive containers (on the transport path). The container stream can be guided or transported in single-lane or multi-lane regions (by means of the transport device), preferably within the entire container treatment device (as a mass flow).

[0103] The transport device can also be a mass conveyor for the transport of a large number of containers, preferably in multiple lanes and / or in a random order. The transport device can also be a buffer area for the buffering of a large number of containers, preferably in multiple lanes and / or in a random order.

[0104] The containers can be transported or guided standing or upright (by the transport device), preferably at least in sections and preferably along the entire transport area.

[0105] Preferably, the transport device is suitable and intended for at least partially guiding or transporting the plurality of containers, preferably along the entire transport area, of containers under dynamic pressure.

[0106] Preferably, the transport device is suitable and intended for transporting and / or guiding (at least within the transport area) at least 1 container per hour, preferably at least 5,000 (in particular to be recognized) containers per hour, preferably at least 20,000, preferably at least 100,000 (in particular to be recognized) containers, and particularly preferably at least 140,000 (in particular to be recognized) containers, and performs this within the operating mode of the container treatment device. Preferably, the transport device is suitable and intended for transporting and / or guiding (at least within the transport area) at most 150,000 (in particular to be recognized) containers per hour, and performs this within the operating mode of the container treatment device.

[0107] Preferably, the transport device is suitable and intended for transporting and / or guiding (at least within the single-lane transport area) at least 100,000 containers per hour and / or up to 150,000 containers per hour in a single-lane transport area, and performs this during the working operation of the container treatment device. The present invention is further directed to a method for at least partially automatically determining an occupancy level of (at least) one transport area in a container treatment device for treating containers with respect to containers located in the transport area.

[0108] The container treatment device comprises a transport device with which the containers (to be treated) can be guided and / or transported along a predetermined transport path (in particular towards the transport area).

[0109] The transport area is arranged along the transport path, wherein image data is generated by means of at least one image capture device, which image data depicts containers located in the transport area and on the basis of which an image evaluation device (in particular the container treatment device) carries out an evaluation, in particular a real-time evaluation, to determine an occupancy level variable which is characteristic of the containers located in the transport area.

[0110] According to the invention, the image evaluation device carries out at least a section-wise segmentation of the image data in order to determine the occupancy level.

[0111] The method can comprise all the method steps described above in connection with the method for at least partially automatically, preferably fully automatically, controlling a container treatment device, in particular for determining the occupancy level and / or for carrying out the segmentation, individually or in combination with one another.

[0112] The present invention is further directed to a container treatment device for treating containers with a transport device which is suitable and intended for transporting and / or guiding the containers along a predetermined transport path.

[0113] The container treatment device further comprises at least one image capture device for generating image data which depicts containers located in the transport area.

[0114] The container treatment device further comprises an image evaluation device which is suitable and intended to carry out an evaluation, in particular a real-time evaluation, on the basis of the image data in order to determine an occupancy level variable characteristic of the occupancy level of the transport area.

[0115] According to the invention, the image evaluation device for determining the occupancy level is suitable and intended to perform at least partial segmentation of the image data. Furthermore, the container treatment device comprises a control device for at least partially automatically, preferably fully automatically, controlling a container treatment device depending on the determined occupancy level.

[0116] In this case, the container treatment device can have all of the features described above in connection with the method for at least partially controlling a container treatment device and / or for at least partially automatically determining an occupancy level of a transport area in a container treatment device (in particular of the container treatment device) alone or in combination (and vice versa) and can be suitable and intended for carrying out all of the method steps described above in connection with the two methods (in particular also according to a preferred embodiment).

[0117] Conversely, the container treatment devices mentioned above in the context of the two methods can have features of the container treatment device described here individually or in combination with one another.

[0118] The present invention is further directed to a method, in particular a computer-implemented method for automatically generating a training data set for training, in particular for retraining, a (trainable) container recognition model of machine learning (in particular a (processor-based) real-time image evaluation device) of a container treatment device in which (in particular during operation of the container treatment device) containers can be guided and / or transported (by means of a transport device) along a predetermined transport path, preferably in the form of a container stream.

[0119] The container recognition model of machine learning can preferably be the image analysis model and / or segmentation model of machine learning described above. It is also conceivable that the method according to the invention, instead of generating a training data set for training, in particular for retraining, the container recognition model described above, is (only) directed at automatically labeling image data (whereby, based on the labeled image data, a training data set is generated, preferably in a subsequent step independent of or different from the method, for example by assigning generated annotation data to the respective image data).

[0120] In this case, image data can be generated (during the operating mode of the container treatment device) by means of (at least) one image capture device (and preferably with a plurality of image capture devices), which are characteristic of a (predetermined and / or predeterminable and / or in particular detectable by the at least one image capture device) transport area along the transport path (as well as containers located therein) and which can be fed to the container recognition model as input variables for carrying out an evaluation, in particular a real-time evaluation, with regard to the containers located in the transport area (for example, for determining an occupancy level of the transport area). The image capture device is preferably an image capture device of the container treatment device.Preferably, the transport area is an area within which the containers located therein are to be detected. In particular, the image data generated by (at least) one image capture device (and preferably by multiple image capture devices) depicts the containers located in the transport area.

[0121] In other words, (during the working operation of the container treatment device) images are taken with the at least one image capture device, in which the transport area (and containers located therein) are detected or depicted in the recorded images. These are preferably evaluated in real time by the real-time image evaluation device using the container recognition model. For example, the real-time image evaluation device can, using the container recognition model, determine as an evaluation result or as a recognition result at least one for the number of containers detected in the transport area and / or position (or location) of the (detected) container (or of each container detected in the transport area) and / or orientation (or of each container detected in the transport area) and / or type of container (or of each container detected in the transport area) and / or condition of the container (oreach container detected in the transport area) and / or the speed of the container (or each container detected in the transport area) and / or an occupancy level of the transport area or a section along the transport path with regard to containers located therein and / or distribution of the containers in a predetermined area within the transport area and / or make it available for output and / or transmission.

[0122] Preferably, the evaluation variable serves as a control variable for the container treatment device, for example, for the (automatic) execution of a treatment function on at least one and / or multiple containers. It is also conceivable for the at least one evaluation variable to be used as a control variable for controlling and / or regulating the container flow and / or a container throughput through the container treatment device and / or a transport speed and / or a container feed and / or container discharge.

[0123] The image capture device (or the plurality of image capture devices) may be an image recording device such as a camera (preferably black and white and / or color), a CMOS sensor (CMOS abbreviation for Complementary metal-oxide-semiconductor), a CCD sensor, a 3D sensor, an X-ray-based image recording device, an optical element, a thermal imaging camera, a stereo camera, a LIDAR camera and the like, as well as combinations thereof.

[0124] The image data (generated by the image capture device and / or to be supplied to the container recognition model) is preferably two-dimensional (spatially resolved) image data. The image data that can be or is supplied to the container recognition model for evaluation preferably does not include any depth information (measured immediately or directly by the image capture device), i.e., in particular, no (depth and / or distance) measured values ​​measured in the direction of the recording direction of the image capture device that are characteristic of a distance between the image capture device and the objects imaged in the image data. In other words, the image capture device preferably does not generate (and / or acquire) a measured value that is solely characteristic of a distance and / or a (relative) position of the image capture device with respect to an imaged object.Such a distance and / or position of the image capture device relative to the objects it images could be determined from image data generated by the image capture device(s) at different positions (e.g., via stereo images and / or a LIDAR camera). This makes it possible to calculate back to the spatial coordinates using a 2D image (captured by the image capture device) if the diameter of the detected or transported container and / or the container to be treated is known. Alternatively, a type of chessboard with known side lengths can be used as a back-calculation approach.For example, a predefined (preferably lying entirely in one plane) reference line or reference surface with a checkerboard-like configuration (preferably with at least two reference lines extending at least partially in different spatial directions and having predefined and / or measured arc lengths of the lines) could be specified as a reference quantity for determining three-dimensional spatial coordinates for a desired 2D image data point. Based on the reference surface depicted in the 2D image data or from the depicted reference lines and their known or predefined geometric dimensions, a distance from the image capture device and, in particular, a (3D) spatial coordinate can be determined for a 2D image data point of the 2D image.

[0125] Preferably, the image capture device records a color image or a color video sequence or a color image sequence (for determining and / or generating the image data). However, it is also conceivable that the recorded and / or generated and / or determined image data is a grayscale image or grayscale images. In other words, it is conceivable that achromatic image data is fed to the container recognition model as input variables. This advantageously allows a larger transport area of ​​the container handling device to be recorded or mapped and evaluated in the image data.

[0126] The image capture device (or the plurality of image capture devices) is preferably suitable for recording (static) (individual) images and / or moving images or image sequences (or video sequences) and is intended or used for this purpose.

[0127] The image data (generated by the image capture device and / or) to be supplied to the container recognition model may be image sequences and / or individual images and / or image data (essentially) recorded at a single recording time or (essentially) recorded or captured simultaneously.

[0128] It is conceivable that the image data (to be supplied to the container recognition model) is image data obtained from more than one individual image. For example, two (different) image capture devices can each capture an image (preferably essentially simultaneously). Preferably, an image composed of multiple images and / or image data generated from multiple images is supplied to the container recognition model.

[0129] Preferably, preprocessed image data is fed to the container recognition model. Preprocessing steps of the image data preferably include cropping, sharpening, and changing the brightness. Preferably, the color range of the image data is not changed, in particular, a color image is not converted to a grayscale image.

[0130] The machine learning container detection model is preferably based on an (artificial) neural network. The neural network is preferably designed as a deep neural network (DNN), in which the parameterizable processing chain has a plurality of processing layers, and / or a so-called convolutional neural network (CNN), and / or a recurrent neural network (RNN), and / or other DNN layer classes.

[0131] Preferably, the machine learning container recognition model is a previously trained container recognition model. In other words, the container recognition model to be trained, or more precisely, retrained, with the training data set to be generated is in a state after completion of a training process. Preferably, the training data set to be generated serves or is used for (further) fine-tuning of the container recognition model.

[0132] It is conceivable that the container recognition model has already been trained with a general training data set, preferably independent of a specific or concrete container handling device.

[0133] The method according to the invention comprises a provision and / or an evaluation of (predefined and / or predefinable) image data for generating the training data set.

[0134] The (predetermined and / or predeterminable) image data provided and / or to be evaluated for generating the training data set is preferably image data generated or acquired by (at least) one image capture device of a container handling device. Preferably, the container handling device is the container handling device whose container recognition model is to be trained or retrained with the training data set to be generated. Preferably, the (at least) one image capture device is the image capture device of the container handling device whose container recognition model is to be trained or retrained with the training data set to be generated.This offers the advantage that the (post-)training process is specifically tailored to the (finely adjustable) container handling device and, for example, specific conditions of the specific container handling device, such as optical properties of the image capture device or specific lighting conditions or reflection conditions or geometric conditions (e.g. of the container guidance) in the container handling device can be directly taken into account.

[0135] However, it is also conceivable that the data (specified and / or specifiable for generating the training data set) were or are generated or transmitted by a container handling device that is (at least partially and / or completely) identical (but different).

[0136] Preferably, the image data (provided and / or to be evaluated and / or specified and / or specifiable for generating the training data set) are recorded and / or generated and / or determined at a different location (with respect to the location where the training data set is generated), preferably directly in the vicinity of or on the container handling device whose container recognition model is to be retrained and / or trained with the training data to be generated (preferably during the (ongoing) operation of the container handling device). The different location is therefore located in particular outside the building and / or company premises in which the container handling device is arranged and is preferably at a distance from it. Preferably, the image data generated orThe image data determined (specified and / or specifiable for generating the training data set) are transmitted to an external storage device (with respect to the container treatment device), which is accessed in particular via the Internet (and / or via a, in particular at least partially wired and / or wireless, public and / or private network).

[0137] A sequence of image data could also be loaded into or transmitted to the (cloud-based) external storage device. Preferably, a training data set can be generated there based on the sequence of image data. Thus, a container recognition model or the container recognition model can be retrained there. Preferably, the method for generating a training data set comprises retrieving and / or receiving image data (which is specified and / or can be specified for generating a training data set) from the external storage device.

[0138] However, it is also conceivable that an image data set is received as image data using user data received as part of a user input.

[0139] The image data is preferably photorealistic. However, it is also conceivable that synthetic image data and / or augmented images (particularly based on photorealistic image data) are / are specified as image data. This offers the advantage that, for example, situations or conditions that do not or only rarely occur during normal operation of the container handling device (such as covered or overturned containers) can be recreated or simulated and used for training.

[0140] Preferably, the image data provided (for generating the training data set) and / or to be evaluated (specified and / or specifiable) and / or acquired with the image acquisition device depict a transport area along a transport path of containers to be transported in a container handling device (by a transport device), wherein at least one container and preferably a plurality of containers are located in the transport area. The containers are preferably containers of the type with respect to which an evaluation is to be performed in the machine learning container recognition model to be trained and / or retrained.

[0141] Preferably, at least some of the image data (particularly preferably all of the image data) depict the same transport area (in particular, the same container handling device). Preferably, the different image data (different image recordings) have different numbers and / or distributions of the containers within the transport area.

[0142] According to the invention, a subdivision of the transport area is and / or will be provided into at least one transport section, preferably at least two, and preferably a plurality of, preferably non-overlapping, transport sections. The individual transport sections each represent a section of the transport area that is contiguous along the transport path. For example, it is conceivable that the subdivision indicates a subdivision of the transport area into at least 5, preferably at least 10, preferably at least 13, and particularly preferably at least 20 different transport sections.

[0143] Preferably, such a subdivision is provided that the transport area can be composed of the plurality of transport sections. Preferably, each region of the transport area is contained in only one transport section. Preferably, the plurality of transport sections completely cover the transport area. It is also conceivable for the plurality of transport sections to cover only sections of the transport area, for example, a central region of the transport area.

[0144] According to the invention, image section data is determined and / or generated from the image data according to the subdivision of the transport area. In this case, the image section data is fed to a training data container recognition model as input variables for evaluating the image section data with respect to the containers located in the transport section (in particular for evaluating the image data to be evaluated to generate the training data set).

[0145] The image section data can be determined and / or generated from the image data according to the subdivision of the transport area by cropping (or trimming) and / or dividing and / or splitting and / or selecting the image data according to the provided subdivision of the transport area.

[0146] The image section data can result from processing the image data before and / or after a selection (cropping, trimming, dividing and / or splitting) of the image data to determine and / or generate the image section data by sharpening and / or changing the brightness.

[0147] In particular, the training data container recognition model is not the machine learning container recognition model (of the container handling device) to be trained and / or retrained with the training data set to be generated.

[0148] The training data container recognition model is preferably a (trained) machine learning model. The training data container recognition model is preferably suitable and intended to map image data supplied as input variables, here the image segment data supplied as input variables, to output variables that are characteristic of the containers located in the image data or image segment data supplied as input variables.

[0149] The machine learning model for training data container detection is preferably based on an (artificial) neural network. The neural network is preferably designed as a deep neural network (DNN), in which the parameterizable processing chain has a plurality of processing layers, and / or a so-called convolutional neural network (CNN), and / or a recurrent neural network (RNN), and / or other DNN layer classes.

[0150] Preferably, by applying the training data container recognition model to the image section data supplied as input variables, at least one (computer-implemented) computer vision method is used to evaluate the image section data, in which (computer-implemented) perception and / or detection tasks are carried out, for example (computer-implemented) 2D and / or 3D object recognition methods and / or (computer-implemented) methods for (preferably semantic) segmentation and / or (computer-implemented) object classification (“image classification”) and / or (computer-implemented) object localization and / or (computer-implemented) edge detection.

[0151] Preferably, by applying the training data container recognition model, the same type of computer vision method for evaluating the supplied image data can be carried out as by using the container recognition model for evaluating image data supplied thereto.

[0152] Preferably, (at least) one recognition result (as an evaluation result) is determined (using the training data container recognition model), preferably with respect to the respective supplied image section data and / or the image data on the basis of which the image section data was determined. The recognition result is preferably the same type of recognition result or evaluation variable that is determined using the container recognition model.

[0153] Thus, by means of the training data container recognition model or by using this training data container recognition model, at least one of the number of containers recognized in the transport section of the respective image section data and / or position (or location) of the (recognized) container (or of each container recognized in the transport section) and / or orientation (or of each container recognized in the transport section) and / or type of container (or of each container recognized in the transport section) and / or condition of the container (or of each container recognized in the transport section) and / or the speed of the container (oreach container detected in the v) and / or an occupancy level of the transport section or a section along the transport path with regard to containers located therein and / or distribution of the containers in a predetermined area within the transport section, determine a characteristic evaluation variable and / or make it available for output and / or transmission.

[0154] In a preferred method, all image segment data associated with the subdivision of the transport area are fed to the training data container recognition model as input variables. Preferably, the evaluation of the respective image segment data (using the training data container recognition model) is used to evaluate the image data (on the basis of which the image segment data was determined) with respect to the containers located in the transport area.

[0155] In this way, the individual evaluation results of the image segment data resulting from the subdivision of the image data are reassembled, and a reference to the (original) image data is established. The evaluation results can then be related to those areas of the image data to which the image segment data corresponds. This yields evaluation results related to the image data (covering the transport area). From the evaluation results, annotation data is advantageously determined, which is then assigned to the respective image data. In this way, a training data set is advantageously generated, which includes the respective image data as well as the annotation data (determined based on the image segment data determined according to the subdivision (via their evaluation results).

[0156] The (provided) subdivision can be applied to all image data. However, it is also conceivable that multiple subdivisions are provided, which are applied to different image data (which may, for example, depict a different transport area or the same transport area) to determine and / or generate image section data.

[0157] Analyzing image segment data instead of image data depicting the transport area offers the advantage of allowing the use of smaller training container recognition models. This advantageously leads to a faster and less complex generation of a training dataset (e.g., in terms of the provision or generation of the training container recognition model and / or the required data processing time).

[0158] A smaller training container detection model can be understood as a training container detection model with comparatively fewer parameters to be trained and / or retrained. For example, a (smaller) training container detection model can be used that, compared to a (larger) training container detection model with a set of, for example, 100 million parameters to be trained and / or retrained, (only) comprises a set with fewer than 10 million parameters to be trained and / or retrained, for example, only 7 million.

[0159] With such smaller training container recognition models, the evaluation results for the individual image data segments are obtained more quickly than with larger training container recognition models. This offers the advantage that, on the one hand, the training process can be completed more quickly and, on the other hand, even real-time evaluation of the supplied image data is possible (particularly with regard to the operating mode of the container handling device).

[0160] Preferably, a model is used as the training container recognition model to which only image data (or image section data) can be supplied as input variable or processed by this model, which does not exceed an image size of 1000x1000 pixels, preferably 900x900 pixels, preferably 750x750 pixels and particularly preferably 640x640 pixels.

[0161] Preferably, the subdivision is carried out in such a way that the image section data (supplied to the training container recognition model as input variable) does not exceed an image size of 1000x1000 pixels, preferably 900x900 pixels, preferably 750x750 pixels and particularly preferably 640x640 pixels.

[0162] In a preferred method, the subdivision of the transport area is specified and / or can be specified by an operator. The method can preferably comprise receiving and / or retrieving data characteristic of the (at least one) subdivision. The data characteristic of the (at least one) subdivision can preferably be entered by an operator via a human-machine interface and / or, after being received via an operator input, are stored on an (external) storage device and retrieved from there. The subdivision can preferably be changed by an operator.

[0163] In a further preferred method, the subdivision of the transport area is selected such that the resulting transport sections each have a capacity of a maximum of 300 containers, preferably a maximum of 250 containers, preferably a maximum of 200 containers. The containers are the containers with respect to which the image data are to be evaluated. This also advantageously ensures that training data container recognition models can be used to generate the training data set, whose high recognition accuracy is limited to supplied image data with approximately 300, 250, or 200 containers, respectively, and whose recognition accuracy decreases with very high numbers of containers to be recognized. This achieves a high quality of the training data to be generated.

[0164] In a further preferred method, the subdivision of the transport area is selected such that the maximum capacity of containers in the respective transport sections is between 10 and 300, preferably between 20 and 200. This advantageously ensures that, on the one hand, a rapid evaluation of the provided image data can be carried out and, on the other hand, the greatest possible recognition accuracy is achieved.

[0165] The (provided) (at least one) subdivision can be a regular subdivision. Regular is understood in particular to mean a regularity with respect to a main transport path. In particular, the extent or geometric size of the transport sections resulting from the subdivision, as seen along the main transport path, is essentially the same. The main transport path could be a mean and / or average transport path. It is also conceivable that the main transport path is a transport path that is mean from a geometric point of view (where the geometric center refers in particular to a width of the transport area, which is in particular perpendicular to the respective transport direction).

[0166] It is also conceivable, however, that the subdivision is irregular. Irregular subdivisions can offer the advantage of allowing for consideration of specific local geometrical features of the transport area, which can lead to comparatively easy and / or particularly difficult detection of the containers located in the respective transport sections. For example, detection can be difficult in tight curves in the transport area and / or transport sections in which the containers to be transported in these sections typically pile up and / or even become wedged or fall over. A comparatively finer subdivision is preferably selected here.

[0167] In a preferred method, the subdivision depends, at least in sections, on at least one geometric parameter which is characteristic of a (at least in section) geometric course of the transport area. The geometric parameter could, for example, be a parameter characteristic of a curvature and / or a width (of a section of the transport area) and / or a branching and / or an junction. Thus, in the case of strong curvatures (small radii of curvature) and / or comparatively large widths of the transport area in this section (compared to other sections of the transport area), a smaller subdivision into transport sections (with respect to a geometric extent along the main transport path) could be selected. Width is understood, in particular, to mean the geometric extent of the respective section perpendicular to the main transport path.

[0168] Preferably, the image section data is determined in such a way that the image data is not only divided into transport sections with respect to their arrangement and / or extension along a main transport path or the transport path, but is also additionally cropped in a direction perpendicular to the main transport path and / or the transport path in order to thereby determine the image section data. This offers the advantage that image data points that do not depict the transport area and are not essential for evaluating the image data with regard to the containers located in the transport area can be removed. This advantageously allows for faster and more precise recognition of the containers.

[0169] Preferably, the proportion of image section data points that depict a transport section of the total image section data points of the image section data (which are fed to the training data recognition model) is more than 30%, preferably more than 40%, preferably more than 50%, and particularly preferably more than 60%.

[0170] In a further preferred method, the subdivision of the transport area is carried out automatically. It is conceivable, for example, that the transport area depicted in the image data is automatically recognized (via image evaluation and / or recognition methods) and, depending on the (geometric) course of the transport area, a subdivision of the transport area into transport sections is (automatically) determined. It is also conceivable that the subdivision of the transport area is determined iteratively. It is conceivable, for example, that a recognition accuracy is determined for a given subdivision and, depending on the recognition accuracy, it is determined whether a finer subdivision can / could lead to an increase in the recognition accuracy. For this purpose, the determined recognition accuracy is preferably compared with a predetermined and / or predefinable threshold value.

[0171] Additionally or alternatively, it is also conceivable that the (determined) (geometric) course of the transport area and / or the main transport path in the section of the transport area is analyzed with regard to at least one geometric characteristic, such as a curvature behavior, and based on this, the subdivision is automatically determined.

[0172] In a preferred method, the image data (provided and / or specified for generating the training data set and / or recorded with the at least one image capture device) are available as color image data or as a color video sequence or color image sequence. Color image data is preferably supplied to the container recognition model and / or color image segment data is supplied to the training data recognition model.

[0173] Preferably, the image section data supplied to the training data recognition model have the same color spectrum as the corresponding image data.

[0174] In particular, no color filter is applied when determining the image segment data based on the image data. This offers the advantage of achieving greater recognition accuracy. In contrast to alternative methods, the use of a color filter can be omitted in this case because the use of a training data recognition model in conjunction with the individual processing of the image segment data resulting from the subdivision can be implemented sufficiently efficiently in terms of the resulting computational effort.

[0175] In a further preferred method, the color values ​​of the image segment data supplied to the training container recognition model and / or the image data supplied to the container recognition model extend substantially over the same color range as the respective or corresponding raw image data captured by the image capture device. In a further preferred method, the container recognition model and / or the test data container recognition model are used for, preferably semantic, segmentation of the image data or image segment data (to be evaluated).

[0176] In segmentation, particularly semantic segmentation, each pixel of the image data or image section data or data derived therefrom is assigned a class (for classifying an object), in this case “container” or “non-container” (class annotation).

[0177] The classes can, for example, be classes for classifying a container type (e.g. can | bottle | glass bottle | PET bottle and the like) and / or for classifying a respective container size (e.g. filling volume, container height, diameter) and / or for classifying a container's features (e.g. body label, neck label, closure type, closure color, and the like).

[0178] In a further advantageous method, an occupancy level of a transport section and / or the transport area is determined on the basis of the, preferably semantic, segmentation.

[0179] In a preferred method, annotation data and preferably training data comprising an assignment of the generated annotation data to the corresponding image data are generated (depending on the recognition result and / or depending on the performed, preferably semantic, segmentation).

[0180] Preferably, a preferably semantic segmentation (at least of the transport area) of the image data is performed based on the preferably semantic segmentation of the image section data (resulting from the subdivision of the transport area) performed by the training data recognition model and / or using the training data recognition model. In this case, all image data points that do not correspond to an image section data point can be assigned the class "non-container."

[0181] Preferably, based on the, preferably semantic, segmentation of the plurality of image section data and / or the image data (at least of the transport area of ​​the image data on the basis of which the image section data was determined), annotation data is generated for the image data (which are characteristic of the, preferably semantic, segmentation of the image data). Preferably, based on the, preferably semantic, segmentation performed, at least one occupancy level variable is determined, which is characteristic of an occupancy level of the transport area and / or a transport section of the image data. It is also conceivable that the occupancy level variable with respect to the transport area is determined as a function of the respective occupancy levels of the transport sections.

[0182] The pixel-based or data point-based determination of the occupancy rate offers the advantage of a very precise determination of the occupancy rate.

[0183] The occupancy rate size can be determined on a container type-specific basis.

[0184] Preferably, the position (and / or orientation) of a (preferably each) recognized container in the image data and / or the coordinates of a (preferably each) recognized container in a (predetermined) coordinate system (world coordinate system) of the (respective) container treatment device are determined (and assigned to the respective image data).

[0185] In a further preferred method, annotation data and preferably training data comprising an assignment of the generated annotation data to the image data are generated depending on the recognition result. The annotation data can include information on a (predefined) category, label, identification and / or marking of specific objects, localization and / or segmentation and / or video annotation (keypoints, polygons, bounding boxes for marking an object in the various frames). For example, it is conceivable that the annotation data include or are coordinates or position information of a recognized container in the image data. It is also conceivable that the annotation data include or indicate a marking of the recognized container, for example by means of a rectangle and / or a boundary line and / or a bounding box.

[0186] It is conceivable that for each image data to be labeled, a (text) file with annotation data is generated, which is then assigned to the image data. The assigned training dataset preferably comprises the image data and the annotation data assigned to it. Preferably, more than 100 different, preferably more than 1000 different images or image data are labeled in the manner described above, or annotation data is created for this purpose (and assigned to the respective image data). A training dataset is preferably generated from this.

[0187] Preferably, the container treatment device is selected from a group comprising a transport device (such as a conveyor belt) for transporting the containers, a buffer device for temporarily buffering containers, a pasteurization device (such as a tunnel pasteurizer), a heating device for heating a preform, a forming device for forming a plastic preform into a plastic bottle, a sterilization device for sterilizing a container, in particular a plastic preform, a manufacturing device for producing a glass bottle, a filling device for filling a container with a product, an inspection device for inspecting a plastic preform or a bottle, a labeling device for labeling a container, a closing device for closing a container, in particular a filled container, a control device,a packaging device, a direct printing device for printing a container, a compilation device for assembling a plurality of containers into a compilation or a bundle, and the like.

[0188] Preferably, the container flow is a (particularly continuous) flow of successive or consecutive containers (on the transport path). The container flow can be guided or transported in a single lane or multiple lanes (by means of the transport device) in certain areas and preferably within the entire container treatment device (as a mass flow).

[0189] The transport device can also be a mass conveyor for the transport of a large number of containers, preferably in multiple lanes and / or in a random order. The transport device can also be a buffer area for the buffering of a large number of containers, preferably in multiple lanes and / or in a random order.

[0190] The containers can be transported or guided in an upright or vertical position (by the transport device), preferably at least in sections and preferably along the entire transport area. The transport device is preferably suitable and intended for guiding or transporting the plurality of containers at least in sections, preferably along the entire transport area, by containers subjected to dynamic pressure.

[0191] Preferably, the transport device is suitable and intended for transporting and / or guiding (at least within the transport area) at least 1 container per hour, preferably at least 5,000 (in particular to be recognized) containers per hour, preferably at least 20,000, preferably at least 100,000 (in particular to be recognized) containers, and particularly preferably at least 140,000 (in particular to be recognized) containers, and performs this within the operating mode of the container treatment device. Preferably, the transport device is suitable and intended for transporting and / or guiding (at least within the transport area) at most 150,000 (in particular to be recognized) containers per hour, and performs this within the operating mode of the container treatment device.

[0192] Preferably, the transport device is suitable and intended for transporting and / or guiding (at least within the single-lane transport area) at least 100,000 containers per hour and / or up to 150,000 containers per hour in a single-lane transport area and carries out this within the working operation of the container treatment device.

[0193] The present invention is further directed to a method for training, in particular for retraining, a container recognition model of machine learning, preferably a real-time image evaluation device, of a container treatment device in which (in a working operation of the container treatment device) containers can be guided along a predetermined transport path, in particular in the form of a container stream.

[0194] In this case, image data can be generated (during operation) by means of an image capture device, which are characteristic of a (imaged) transport area (preferably of interest with regard to containers to be recognized) along the transport path. The image data can be fed to the container recognition model as input variables for performing an evaluation, in particular a real-time evaluation, with regard to the containers located in the transport area. According to the invention, the machine learning container recognition model is trained and / or retrained with training data generated according to one of the methods described above (in particular according to an embodiment described above).

[0195] Therefore, within the scope of the method according to the invention, it is also proposed that for training, preferably for retraining, the container recognition model, labeled image data or image data provided with annotation data (as a training data set) be used, the underlying image data being divided into smaller image sections to be evaluated by determining image section data according to a provided subdivision. These smaller image sections (image section data) are preferably evaluated using a trained training container recognition model using machine learning (with regard to the containers located in the transport area or transport sections). Based on the evaluation of the individual smaller image sections, annotation data (with regard to containers depicted in the image data, for example, an occupancy level of a transport area) of the image data is preferably generated.

[0196] The container recognition model and the container handling device can each be designed with all the features described above in connection with the method described above, individually or in combination with one another (and vice versa).

[0197] The container treatment device preferably records image data using one or more image capture devices to generate a training data set and transmits this data to an external storage device. The image data preferably represent the transport area of ​​the container treatment device. Image data is preferably generated (by the image capture device of the container treatment device) at different recording times and with different container distributions in the transport area and transmitted to the external storage device.

[0198] Preferably, the image data stored on the external storage device are retrieved (in particular by or triggered by a training data generation device). Preferably, a training data set is generated based on the retrieved image data (by the training data generation device) (using the method described above according to a preferred embodiment). The training data set is preferably generated remotely with respect to the container handling device. Preferably, the training data set generated (by the training data generation device) is retrieved from an external storage device (preferably by the container handling device).

[0199] Preferably, the data (to be processed), in particular the image data recorded by at least one image capture device, are fed to the container recognition model or the (artificial) neural network as input variables. Preferably, the container recognition model or the artificial neural network maps the input variables to output variables depending on a parameterizable processing chain, wherein the output variable is preferably a state of the respectively recognized container, a location of (each) recognized container, a speed of (each) recognized container, and / or a distribution and / or number of containers in the transport area.

[0200] It is also conceivable that the container recognition model determines as an output variable a position variable characteristic of a (current) location / position of the (each) recognized container and / or a variable characteristic of a recognized container type and / or an occupancy level of a transport area and / or a transport section and makes it available (for transmission and / or output).

[0201] Preferably, the container handling device and / or the real-time evaluation device determines, depending on the output variables of the container recognition model, a variable characteristic of a speed of one and preferably each container recognized (in the transport area) and / or a state of one and preferably each container recognized (in the transport area) and / or a distribution and / or number of the recognized containers in the transport area.

[0202] Preferably, the container recognition model based on machine learning or the artificial neural network is (re-)trained using the generated training data, whereby the training parameterizes the configurable processing chain. An iterative training process is preferably selected, which is repeated until a specified recognition accuracy is achieved.

[0203] The external storage device is preferably a (non-volatile) storage device, in particular a cloud-based storage device and / or an external server (including storage device), wherein the storage device is accessed in particular via the Internet (and / or via a public and / or private network, in particular a wired and / or wireless network at least in sections). An external server is understood to mean, in particular, a server external to a container handling device and / or real-time evaluation device, in particular a backend server.

[0204] The external server is, for example, a backend, in particular of a container handling device manufacturer or a service provider, which is configured to manage image data (in particular from a plurality of image capture devices and / or a plurality of container handling devices) and / or to configure container handling devices. The functions of the backend or the external server can be performed on (external) server farms. The (external) server can be a distributed system.

[0205] The present invention is further directed to a method for detecting containers located in a transport area (of a container treatment device) and / or for determining an occupancy level of a transport area of ​​a container treatment device.

[0206] According to the invention, a container recognition model of machine learning - preferably trained according to the method described above for training, in particular for retraining, a container recognition model of machine learning - is used to recognize and / or track containers located in the transport area of ​​a characteristic size and / or to determine at least one (occupancy) size characteristic of an occupancy level of the transport area and / or at least one transport section of the transport area.

[0207] It is conceivable that a subdivision of the transport area as described above is provided. Preferably, the transport area is divided into transport sections according to the subdivision, and at least one occupancy variable characteristic of an occupancy level in the respective transport section is determined.

[0208] Preferably, the container recognition model is suitable and determined for performing a (as described above), preferably semantic, segmentation (of image data). Preferably, the respective occupancy size is determined using the data point-by-data or pixel-by-pixel class assignment obtained by the, preferably semantic, segmentation (see, as described above, e.g., classes: "Container" | "Non-Container"). For example, to determine the respective occupancy level of the transport area or a transport section, a ratio can be formed between the proportion of data points or pixels with the class assignment "Container" and the total number of data points or pixels that represent the respective transport area or transport section.

[0209] The container treatment device preferably has a real-time image evaluation device, in particular a processor-based one, which is suitable and intended to carry out an evaluation, in particular a real-time evaluation, with regard to the containers located in the transport area by means of the container recognition model of machine learning, for which purpose the image data can be supplied to the container recognition model (and the real-time image evaluation device) as input variables.

[0210] It is therefore also proposed within the scope of the method according to the invention that a very precise container recognition model of machine learning, obtained by retraining as described above, is used to recognize the containers (and derive a recognition result) and / or determine the number and / or speed and / or distribution and / or occupancy level of the containers located in the transport area.

[0211] Preferably, the container recognition model and the container handling device can be equipped with all the features described above alone or in combination with each other (and vice versa).

[0212] It is also conceivable that the container recognition model determines as an output variable a position variable characteristic of a (current) location of the (each) recognized container and / or a variable characteristic of a recognized container type and makes it available (for transmission and / or output).

[0213] Preferably, the container handling device and / or the real-time evaluation device determines, depending on the output variables of the container recognition model, a variable characteristic of a speed of one and preferably each container detected (in the transport area) and / or a state of one and preferably each container detected (in the transport area) and / or position and / or a distribution and / or number and / or container type of the detected containers in the transport area and / or occupancy level of the transport area. Preferably, based on the recognition result and / or the determined number and / or distribution and / or position and / or container type, at least one detected container is tracked. For this purpose, image data recorded at different times is used and each evaluated (particularly in real time).Preferably, positions of the respectively detected container are determined based on image data recorded at different times. Furthermore, one or more speeds of the container are preferably determined based on the determined positions and the recording times and / or on the basis of a transport speed caused by the transport device. Preferably, an expected location area of ​​the container for at least one future point in time is determined based on the determined positions and / or speeds of the container, preferably by applying a Kalman filter.

[0214] Preferably, such tracking is applied to several and preferably all containers located in the transport area.

[0215] It is also conceivable that, based on the characteristic (occupancy) size determined for the occupancy level of the transport area and / or the detected containers, their positions and / or speeds, and / or their determined expected future locations, a virtual congestion counter is provided, in which a container congestion is determined or predicted with a predetermined probability (preferably a future occurrence). Preferably, in the case of a detected or predicted container congestion, a warning message is provided for transmission and / or output to a user.

[0216] It is also conceivable that, depending on the occupancy level or container distribution in the transport area, different warning levels are determined and corresponding messages are provided for transmission and / or output to a user.

[0217] The present invention is further directed to a preferably processor-based training data generation device for automatically generating a training data set for training, in particular for retraining, a container recognition model of machine learning of a container handling device in which containers can be guided along a predetermined transport path, wherein image data can be generated by means of an image capture device which are characteristic of a transport area along the transport path and which can be fed to the container recognition model as input variables for carrying out an evaluation, in particular a real-time evaluation, with regard to the containers located in the transport area.

[0218] The training data generation device (for generating the training data set) is suitable and intended for evaluating image data (in particular to determine a recognition result).

[0219] According to the invention, the training data generation device is suitable and intended to determine image section data from the image data on the basis of a provided subdivision of the transport area into at least one transport section, preferably at least two and preferably a plurality of, preferably non-overlapping, transport sections, and to supply the image section data to a training data container recognition model as input variables for evaluating the image section data with regard to the containers located in the transport section and to determine a recognition result.

[0220] The training data generation device can be suitable, intended and / or configured to carry out one or more of the method steps described above of the method for training, in particular for retraining, a container recognition model of machine learning (of a container handling device).

[0221] The present invention is further directed to a container treatment device for treating containers, comprising a transport device which is suitable and intended for transporting the containers along a predetermined transport path, comprising an image capture device by means of which image data can be generated which are characteristic of a transport area along the transport path.

[0222] The container handling device comprises a real-time image evaluation device, in particular a processor-based one, which is suitable and intended for performing an evaluation, in particular a real-time evaluation, with respect to the containers located in the transport area using a machine-learning container recognition model. For this purpose, the image data can be fed to the container recognition model (and the real-time image evaluation device) as input variables. The machine-learning container recognition model is preferably a container recognition model that has been trained and / or retrained with a training data set generated according to one of the methods described above.

[0223] Preferably, the containers are treated depending on the recognition result of the container recognition model or (at least) one output variable of the container recognition model or depending on the containers recognized by the container recognition model.

[0224] The container treatment device can have all the features described above in connection with a container treatment device alone or in combination (and vice versa) and can be suitable and intended for carrying out all the method steps described above in connection with the method for detecting containers located in a transport area of ​​a container treatment device.

[0225] The container recognition model may comprise one or more of the features described above individually or in combination.

[0226] The present invention is further directed to a computer program or computer program product comprising program means, in particular a program code, which represents or encodes at least some of the and preferably all method steps of the respective (above-described) inventive method (for generating a training data set and / or a method for training a container recognition model of a container handling device) and preferably one of the (above-described) preferred embodiments and is designed to be executed by a processor device.

[0227] The present invention is further directed to a data memory on which at least one embodiment of the computer program according to the invention or a preferred embodiment of the computer program is stored.

[0228] The present invention has been described with reference to containers to be recognized. The present invention is also applicable to objects to be recognized generally in machine image processing (based on machine learning image analysis models) for image recognition in the beverage and / or pharmaceutical sectors, particularly in container handling devices in the beverage and / or pharmaceutical sectors. The applicant reserves the right to also claim related subject matter, in particular the method for generating a training data set and the training data generation device.

[0229] Further advantages and embodiments can be seen from the attached drawings:

[0230] Fig. 1 is a schematic representation of an image recording from a container treatment device and an illustration of a preferred method for generating a training data set;

[0231] Fig. 2, 4 different input images;

[0232] Fig. 3, 5 the respective evaluation images obtained by evaluating the input images from Fig. 2, 4; and

[0233] Fig. 6 shows a roughly schematic structure of a container treatment device according to a preferred embodiment of the present invention.

[0234] Fig. 1 shows a schematic representation of an image recording BD1 of a container treatment device 1. The container treatment device 1 has a transport device 2 (here a mass conveyor in which the containers 10 (here cans) are transported in multiple rows and in a random manner). The reference numerals 16 and 18 denote lateral boundaries or guides of the transport device 2. By means of these, the area of ​​the transport device, for example a conveyor belt, can be defined.

[0235] It can be seen that the illustrated transport area of ​​the transport device 2 is divided into different transport sections, which are identified in Fig. 1 with the reference symbols B1, B2, B3, B4, ..., B9, B10.

[0236] An analysis of the image capture using a machine learning container recognition model can yield the respective numbers of containers within the respective transport sections B1, B2, ..., B10. Alternatively, or based on the number of containers, a respective occupancy level of the respective transport section can be specified and / or determined (which corresponds to the ratio between the containers currently transported in the transport section and the maximum occupancy level). In the image data BD1, 54 containers are currently being transported in transport section B1. With a maximum occupancy level of 119 containers (assumed here), this corresponds to an occupancy level of 45.38%.

[0237] In transport sections B2 - B5, no containers 10 are being transported at the time the image data BD1 is recorded. Therefore, the occupancy rate in these sections is 0%.

[0238] The transport sections B6 and B7 are only partially, but not completely, occupied with containers 10.

[0239] In transport sections B9 and B10, the maximum number of containers that can be accommodated in the respective transport sections is currently being transported in the image data BD1. In transport sections B9 and B10, this results in a (current) occupancy rate of 100%.

[0240] Fig. 1 also shows an illustration of a preferred method for generating a training data set. The transport area depicted in the recorded image data BD1 is divided into individual transport sections B1, ..., B10, and image data tailored to the individual transport sections (of the respective entire image BD1) is fed to an image analysis model (such as a segmentation model) and / or a container recognition model as input image data for analysis with respect to the containers 10 located in the transport sections.

[0241] The applicant has recognized in complex series of tests that if the image evaluation model or the recognition model is applied to a complete image BD1, the recognition accuracy is low; after cropping (here illustrated by, for example, the partial area B3), all objects (here containers or cans) are recognized.

[0242] This means that an image evaluation model or a container detection model with preprocessing such as (crop, sharpness and / or brightness) is preferably used as an autolabeler.

[0243] Fig. 2 and 4 each show different input image data (input images) E1, E2, which in turn depict a container treatment device 1 with a transport device 2 and containers 10 (here cans) transported by the latter and which are evaluated by means of a, in particular processor-based, image evaluation device (preferably with an image evaluation model or a recognition model of machine learning).

[0244] The reference numerals 16 and 18 in turn indicate a lateral boundary of the receiving area 14 of the transport device 2 in which containers 10 can be received for transporting them.

[0245] The image analysis device (e.g., the machine learning recognition model) preferably performs (semantic) segmentation. During (semantic) segmentation, a class is assigned to each pixel of an (input) image or each data point of the input image data. In this case, the class used is "container" (or "can"). All image data or pixels that belong to a (recognized) container (or can) are assigned the class "container." These image data points or pixels are shown in white in the analysis images A1, A2 in Figs. 3 and 5 and are labeled with the reference symbol P10. The class "container" (or correspondingly the class "can") is to be understood here in particular as a class "image points belonging to (any) container (of the container mass flow)," because the classified points cannot be assigned to individual containers or individual cans.Even if several "container pixels", i.e. data points of the "container" class are detected, the evaluation result does not include the information as to whether there are one or more containers (or whether there are one or more cans) or whether, for example, two adjacent data points of the "container" class belong to the same container or depict (adjacent) areas of the same container.

[0246] All other image data points that do not belong to objects recognized as containers (or cans) are assigned a pixel value different from the class “container” (or “can”), for example a black colored pixel data point, a “background pixel value” PH.

[0247] In Figs. 3 and 5, all image data points that the machine learning recognition model has identified as not belonging to a container (or can) are displayed as black pixels.

[0248] These figures result in a binary representation consisting of white pixels (container) and black pixels (non-container). Preferably, such an evaluation can be used to quickly determine the occupancy level (of a given transport area) of a transport device, for example, by determining the white image portions P10 of the respective evaluation images A1, A2.

[0249] Fig. 6 shows a roughly schematic structure of a container treatment device 1 for treating containers 10 according to a preferred embodiment of the present invention. Reference numeral 2 denotes a transport device that transports the containers 10 to a treatment device 12 for treating the containers 10.

[0250] Reference numeral 4 denotes an image capture device, in this case a camera, which—here from above—captures a transport area designated by reference numeral 20. In particular, image data is generated that depicts the containers 10 located in the transport area 20.

[0251] The captured image data (possibly after preprocessing) are transmitted to an image evaluation device 6. The image evaluation device 6 performs an evaluation based on the image data in order to determine an occupancy level value that is characteristic of the occupancy level of the transport area 20.

[0252] The image evaluation device 6 determines the occupancy level based on at least a segmented image data. Preferably, a control variable for controlling the container handling device 1 is determined based on the occupancy level.

[0253] Reference numeral 8 denotes a control device, which the container treatment device 1 preferably has and which serves to control the container treatment device depending on the control variable, for example for the (automatic) execution of a treatment function on at least one and / or several containers. It is also conceivable that the at least one control variable is used to control and / or regulate the container flow and / or a container throughput through the container treatment device and / or a transport speed and / or a container feed and / or container discharge.

[0254] The applicant reserves the right to claim all features disclosed in the application documents as essential to the invention, provided they are novel, individually or in combination, over the prior art. It is further noted that the individual figures also describe features that may be advantageous in and of themselves. The skilled person will immediately recognize that a particular feature described in a figure may be advantageous even without adopting further features from that figure. Furthermore, the skilled person will recognize that advantages may also arise from a combination of several features shown in individual or different figures.

[0255] List of reference symbols

[0256] 1 container treatment device

[0257] 2 Transport device 4 Image capture device

[0258] 6 Image evaluation device

[0259] 8 Control device

[0260] 10 containers

[0261] 12 Container handling facility 20 Transport area

[0262] B1, B2, B10 transport sections

[0263] BD1, BD2, BD3 image data

[0264] TB1 partial image

[0265] E1, E2, E3 Input images PH Background pixels

[0266] P10 pixels assigned the class “container”

Claims

Patent claims 1. A method for the at least partially automatic, preferably fully automatic, control of a container treatment device (1) for treating containers (10), which device has a transport device (2) for guiding containers (10) along a predetermined transport path, as a function of an occupancy level of at least one transport area (20) along the transport path, wherein image data (E1, E2) are generated by means of at least one image capture device (4), which image data depict containers (10) located in the transport area (20) and on the basis of which image data an image evaluation device (6) carries out an evaluation, in particular a real-time evaluation, in order to determine an occupancy level variable characteristic of the occupancy level of the transport area (20), characterized in that the image evaluation device (6) carries out an at least section-wise segmentation of the image data (E1, E2) in order to determine the occupancy level variable.

2. Method according to claim 1, characterized in that the occupancy level is determined on the basis of at least one segment obtained by the segmentation and preferably at least a plurality of segments obtained by the segmentation.

3. Method according to one of the preceding claims, characterized in that the occupancy level is determined as a function of a transport area size which is characteristic of an occupancy area and / or of a maximum occupancy number of the transport area.

4. Method according to the preceding claim, characterized in that the transport area size is determined automatically.

5. Method according to one of the two preceding claims, characterized in that, in order to determine the transport area size, image data are generated (and evaluated) by means of the image capture device (4) in a setting state of the transport device (2), which can preferably be reached by an operator by triggering a setting operation of the transport device (2), in which a maximum number of containers (10) that can be accommodated therein are accommodated in the transport area (20), in particular automatically.

6. Method according to one of the preceding claims, characterized in that, in order to determine the occupancy level, the image data are evaluated in such a way that a plurality of adjacent containers (10) located in the transport area (20) is recognized as a uniform container cluster which does not indicate any differentiation between individual containers (10).

7. Method according to one of the preceding claims, characterized in that the occupancy level is determined independently of a position and / or orientation of the containers (10) located in the transport area (20).

8. Method according to one of the preceding claims, characterized in that the segmentation is a semantic segmentation.

9. Method according to one of the preceding claims, characterized in that by means of the segmentation, a class “container” (P10) selected from several, preferably two, classes is assigned to an image data point when the image data point is assigned to a container and / or a container cluster, and a class “Background” (PH) is assigned if the image data point is not assigned to a container and / or a container cluster.

10. Method according to the preceding claim, characterized in that the occupancy level is determined on the basis of those image data points (P10) to which the class “container” (P10) was assigned during segmentation.

11. Method according to one of the preceding claims, characterized in that the segmentation is carried out on the basis of a predetermined and / or predeterminable color range.

12. Method according to one of the preceding claims, characterized in that the segmentation is carried out on the basis of a machine learning segmentation model.

13. Method according to one of the preceding claims, characterized in that at least one further transport area is predetermined (B7) and image data are generated by means of at least one image capture device (4), which image containers (10) located in the at least one further transport area (20) and on the basis of which an image evaluation device (6) carries out an evaluation, in particular a real-time evaluation, to determine an occupancy level variable which is characteristic of the containers (10) located in the at least one further transport area (20), wherein the image evaluation device (6) carries out an at least section-wise segmentation of the image data to determine the occupancy level variable.

14. Method for at least partially automatically determining an occupancy level of a transport area in a container treatment device (1) for treating containers (10) with respect to containers located in the transport area (20). Containers (10), wherein the container treatment device (1) has a transport device (2) with which the containers (10) can be guided along a predetermined transport path, wherein the transport area (20) is arranged along the transport path, wherein by means of at least one image capture device (4) image data (E1, E2) are generated which depict containers (10) located in the transport area (20) and on the basis of which image data an image evaluation device (6) carries out an evaluation, in particular a real-time evaluation, to determine an occupancy level variable which is characteristic of the containers (10) located in the transport area (20), characterized in that the image evaluation device (6) carries out an at least section-wise segmentation of the image data (E1, E2) to determine the occupancy level variable.

15. Container treatment device (1) for treating containers (10) with a transport device (2) which is suitable and intended for transporting the containers (10) along a predetermined transport path, with at least one image acquisition device (4) for generating image data (E1, E2) which depict containers (10) located in the transport area (20), and with an image evaluation device (6) which is suitable and intended for determining an occupancy level variable characteristic of the occupancy level of the transport area (20) and is intended for carrying out an evaluation, in particular a real-time evaluation, on the basis of the image data (E1, E2), characterized in that the image evaluation device (6) for determining the occupancy level variable is suitable and intended for carrying out an at least section-wise segmentation of the image data (E1, E2),and that the container treatment device (1) has a control device (8) for at least partially automatically, preferably fully automatically, controlling a container treatment device (1) depending on the determined occupancy level.,

Images