METHOD FOR COMPUTER-AID EMPTY DETECTION OF A TRANSPORT CONTAINER AND DEVICE FOR COMPUTER-AID EMPTY DETECTION OF A TRANSPORT CONTAINER

Description

[0001] The invention relates to a method for computer-aided empty detection of a transport container and a device for computer-aided empty detection of a transport container.

[0002] In general, transport containers can be used to move items, such as goods in production or sales. It may be necessary to identify whether and, if so, what is inside the transport container, for example, when registering goods at a checkout (this can also be referred to as "bottom of basket" detection - BoB). This helps reduce costs that arise when unregistered goods pass through the checkout undetected (this can also be referred to as loss prevention).

[0003] Traditionally, computer-aided pattern recognition methods are used to identify objects arranged in the transport container. Distinctive patterns of the objects are recognized and compared with a database containing the patterns of known objects, e.g., goods.

[0004] Matching the patterns of items contained in the transport container with the database, and considering the database's size, can be quite resource-intensive, especially when a large number of items and their respective patterns are stored in the database and / or when similar items need to be identified. For example, it may be necessary to store a separate record for each item according to different views of the item if, for instance, it cannot be guaranteed that the item is always in the same orientation. Similar items may require a larger number of features to be compared to enable reliable identification.

[0005] As the database grows in size—that is, with an increasing number of records and / or an increasing number of entries per record—the required storage capacity and computing power, or the required data analysis speed, can increase to ensure timely object recognition, e.g., in real time. Furthermore, the effort required to maintain the records (data maintenance) and thus the personnel needed can increase with the database's size to prevent uncontrolled growth due to outdated records. Therefore, these computer-aided pattern recognition systems can incur significant costs in terms of acquisition and / or maintenance, especially if, for example, one system is required for each checkout.Furthermore, the pattern recognition process is very complex when dealing with a very high number of objects to be distinguished, and the susceptibility to errors in pattern recognition also increases with an increasing number of similar patterns.

[0006] Furthermore, conventional detection of the transport container contents can be limited and / or inaccurate, for example, with flat objects (which, for instance, have a small cross-section in one orientation); objects with little or no highlighting and / or color, e.g., with only low information, color, and / or texture content (for example, a nearly homogeneous layout such as black, white, gray, etc.); and / or transparent objects. Additionally, objects not stored in the database are regularly not detected or are detected incorrectly.

[0007] US 2008 / 226129 Al describes a method and device for shopping cart verification which increases the level of suspicion for a suspicious transaction when an image comparison reveals differences between the video image of the shopping cart recorded during the transaction and the video image of an empty shopping cart.

[0008] The invention provides a method and a device for computer-aided empty detection of a transport container according to the independent claims.

[0009] According to various embodiments, a method for computer-aided empty detection of a transport container (i.e., intuitively, for detecting whether the transport container is empty, in other words, free of any objects contained within it) and a device for computer-aided empty detection of the transport container are provided, which exhibit lower complexity, thereby reducing computational and storage requirements. This allows for the use of simpler equipment, thus reducing acquisition costs. Furthermore, the size of the database can be reduced, thereby decreasing the effort required for maintaining and updating the data records, which can lead to cost savings.

[0010] In various embodiments, a method for computer-aided empty detection of a transport container and a device for computer-aided empty detection of the transport container are provided, which determine pattern deviations of one or more patterns taken from one or more images of the transport container (possibly filled with one or more objects) taken by means of, for example, a camera, from one or more stored patterns of the transport container in an empty state (i.e., without objects in the transport container).

[0011] The pattern deviations are caused, for example, by objects in the transport container; that is, they arise or increase when the transport container is compared (e.g., in a previous training procedure) to one or more reference transport containers, which is / are empty, and based on the pattern deviations, a decision is made as to whether the transport container under investigation is empty or not.

[0012] According to various embodiments, it is no longer necessary to use a database in which every possible recognizable object is stored; instead, the database can be reduced in size to the transport container(s) used. Thus, the size of the database no longer increases with the number and / or variety of objects that could be transported using the container and that need to be recognized.

[0013] According to various embodiments, depth information is also acquired using three-dimensional (3D) image acquisition. This depth information can be used, for example, to determine whether an object is located in different areas of the transport container, such as on a lower and / or upper level. This allows for differentiation of the object's location within the transport container. For instance, the depth information can be acquired through one of the areas, enabling a clear visual assessment from above of whether something is located in a lower or lower section of the transport container. This reduces the number of required image acquisition sensors, as the entire transport container can be examined from a single perspective.

[0014] According to various embodiments, a system for the computer-aided analysis of objects in a (e.g., open) transport container (which can also be referred to as a transport medium) is provided. The system makes it possible to determine the state of the transport container (e.g., empty or not empty) at a specific time and / or location, e.g., automatically.

[0015] According to various embodiments, a method for computer-aided empty container detection can include the following: capturing image data of an area of ​​the transport container; determining a contour pattern (e.g., by means of feature recognition) that represents the transport container (e.g., one caused by the transport container) using the image data; determining a deviation magnitude that represents a deviation of the contour pattern from at least one reference pattern, wherein the at least one reference pattern represents an empty transport container; outputting a signal when the deviation magnitude meets a predetermined criterion.

[0016] According to various embodiments, the method may further include: capturing (e.g., additional) image data of an additional area of ​​the transport container; determining an additional contour pattern representing the additional area (e.g., caused by the additional area) based on the (e.g., additional) image data; wherein a reference pattern of the at least one reference pattern includes the additional contour pattern.

[0017] According to various embodiments, the method can further include: saving the additional contour pattern to a data storage device.

[0018] According to various embodiments, at least one reference pattern can be part of a database or form the database itself.

[0019] According to various embodiments, the at least one reference pattern or the database can have several reference patterns (e.g. two or more than two, e.g. three, four, five, six, seven, eight, nine, ten or more than ten, e.g. twenty, fifty, e.g. one hundred or more than one hundred).

[0020] According to various embodiments, the method may further include: reading a reference pattern of at least one reference pattern from a data storage device (e.g., from the database).

[0021] According to various embodiments, the transport container for capturing the image data can be arranged in an image acquisition area, wherein the image acquisition area defines an image background (figuratively, the transport container); wherein a reference pattern of the at least one reference pattern has a contour pattern that represents an image background (e.g., when no or an empty transport container is arranged in the image acquisition area). Figuratively, the transport container can be arranged between an image acquisition system with which the image data is captured and the image background.

[0022] According to various embodiments, the deviation magnitude can represent a contrast deviation of the contour pattern from the at least one reference pattern.

[0023] According to various embodiments, the deviation magnitude can represent a contour deviation of the contour pattern from the at least one reference pattern.

[0024] According to various embodiments, the deviation magnitude can represent an area coverage deviation of the contour pattern from the at least one reference pattern.

[0025] According to various embodiments, the method can further include: detecting when the transport container is located in an image acquisition area in which the image data is acquired; wherein the image data is acquired when it has been detected that the transport container is located in the image acquisition area (e.g. during a search phase).

[0026] The search phase can include the following: capturing reflections (e.g. in the IR range); and comparing (e.g. feature-based) the reflections to the reference pattern (e.g. with the features from the reference image of the transport medium).

[0027] According to various embodiments, the method may further include: detecting whether the transport container is arranged in an image acquisition area in which the image data is acquired; outputting a further signal (which may also be referred to as a not-detected signal) if no transport container is arranged in the image acquisition area.

[0028] According to various embodiments, the method may further include: detecting whether the transport container is one of several container types; and outputting a further signal (which may also be referred to as a non-detected signal) if no container type of several container types has been detected.

[0029] According to various embodiments, the specified criterion can represent an empty transport container and the signal can be an empty-detected signal.

[0030] According to various embodiments, the specified criterion can represent a non-empty transport container and the signal can have a non-empty detected signal.

[0031] According to various embodiments, the specified criterion can represent an error range in which empty detection is unreliable, and the signal can contain or represent an error signal.

[0032] According to various embodiments, the output of the signal can include issuing an input prompt, whereby a cash register system process enters a waiting state until an input is received in response to the input prompt.

[0033] According to various embodiments, the output of the signal can include issuing an input prompt, the method further comprising: updating the at least one reference pattern based on the contour pattern when an input to the prompt represents an empty state of the transport container.

[0034] According to various embodiments, the output of the signal can include issuing an input prompt, the method further comprising: forming an additional reference pattern based on the contour pattern when an input to the prompt represents an empty state of the transport container; and adding the additional reference pattern to the at least one reference pattern (e.g., to the database).

[0035] According to various embodiments, the acquisition of image data from the transport container can include: acquiring image data from an image acquisition area in which the transport container is located; identifying a portion of the image data from the image acquisition area (which may also be referred to as the "region of interest" - ROI or analysis area) that contains the image data of the area of ​​the transport container.

[0036] According to various embodiments, a portion of the image data can be determined using one or more markers on the transport container. For example, one or more markers can be used to enable the delimitation of the region of interest (ROI) by means of adapted recognition algorithms.

[0037] According to various embodiments, the transport container can have one or more markers (e.g., grayscale markers (e.g., ArUco) and / or reflective markers).

[0038] According to various embodiments, the acquisition of image data from the transport container can comprise the following: acquiring image data from an image acquisition area in which the transport container is located; wherein determining the contour pattern comprises the following: identifying a portion of the image data from the image acquisition area that represents an image background defined by the image acquisition area; comparing this portion of the image data with reference image data, where the reference image data represents an empty image acquisition area (this can also be referred to as image-background comparison). Intuitively, the contour pattern can be determined by identifying which part of the image background is obscured by the transport container (e.g., a lattice structure).

[0039] According to various embodiments, determining the contour pattern can involve: identifying a portion of the image data (e.g., an area of ​​interest) that has a predefined contour density, and then determining the contour pattern using that portion of the image data that has the predefined contour density. For example, a feature-rich region of the transport container can be used for empty container detection.

[0040] According to various embodiments, the image data acquisition can comprise the following: sequential acquisition of a multitude of image data; and identification of image data from the multitude of image data that represent the transport container and / or that exhibit a predefined contour density. For example, a series of images can be captured, with the images showing the transport container being used for empty detection. This facilitates empty detection when the transport container does not remain within the image acquisition area and / or is not always positioned in the same way within the image acquisition area.

[0041] According to various embodiments, the acquisition of image data of the transport container can include the following: acquiring image data of an image acquisition area in which the transport container is arranged; determining a spatial location (position and / or orientation) of the transport container relative to the image acquisition area; determining a part of the image data (e.g., an area of ​​interest) of the image acquisition area, which contains the image data of the area, based on the spatial location (position and / or orientation).

[0042] According to various embodiments, the method may further include: recognizing a container type of the transport container, wherein the transport container is a container type from several container types; and wherein the reference pattern represents an empty transport container of container type.

[0043] According to various embodiments, the method may further include: selecting the reference pattern from a plurality of reference patterns, each of which is assigned to one of the several container types.

[0044] According to various embodiments, the method may further include: recognizing a container type of the transport container, wherein the transport container is one container type from several container types; wherein the criterion represents the container type.

[0045] According to various embodiments, the method may further include: selecting the criterion from a plurality of criteria, each of which is assigned to one of the several container types.

[0046] According to various embodiments, the method can further include: determining color information (e.g., of the transport container) based on the image data; wherein the deviation magnitude further represents a deviation of the color information from reference color information, the reference color information representing an empty transport container.

[0047] According to various embodiments, determining the deviation magnitude can involve weighting the deviation of the color information and the deviation of the contour pattern according to a predetermined weight characteristic.

[0048] According to various embodiments, the method may further include: determining topographic information (e.g., of the transport container) based on the image data; wherein the deviation magnitude further represents a deviation of the topographic information from reference topographic information, where the reference topographic information represents an empty transport container.

[0049] According to various embodiments, determining the deviation magnitude can involve weighting the deviation of the topography information and the deviation of the contour pattern according to a predetermined weight characteristic.

[0050] According to various embodiments, the method may further include: determining depth information based on the image data; wherein the deviation magnitude further represents a deviation of the depth information from reference depth information, the reference depth information representing an empty transport container.

[0051] According to various embodiments, determining the deviation magnitude can involve weighting the deviation of the depth information and the deviation of the contour pattern according to a predetermined weight characteristic.

[0052] According to various embodiments, determining the depth information can involve capturing image information of an additional area of ​​the transport container through the area and / or through a surface element of the transport container (which is, for example, arranged between the first area and the second area).

[0053] According to various embodiments, the method can further comprise: identifying a first part of the image data of the transport container, which represents a first surface element of the transport container; and identifying a second part of the image data of the transport container, which represents a second surface element of the transport container, wherein the first surface element and the second surface element are at an angle to each other and / or are spaced apart; wherein the second part of the image data is captured through the first surface element; and wherein the contour pattern represents the first surface element and / or the second surface element. Intuitively, according to various embodiments, it can be distinguished which area of ​​the transport container is examined, e.g., by recognizing the surface element that separates the areas from each other, and / or e.g.by identifying the surface elements that define the areas.

[0054] According to various embodiments, the contour pattern can represent at least one of the following: a contour pattern of the image background (e.g., a contour pattern of the floor structure and / or the image background structure); a grid structure of the transport container and / or a surface element of the transport container; one or more struts of the transport container and / or a surface element of the transport container; an advertising medium of the transport container and / or a surface element of the transport container; a disturbance caused by an object when the object is arranged in the transport container; and / or an image background defined by an image capture area in which the image data is captured.

[0055] Depending on the design, the transport container can be a shopping cart (e.g. a compact shopping cart) or a shopping basket.

[0056] According to various embodiments, the transport container can have a first area (e.g. a first transport area) and a second area (e.g. a second transport area), wherein the first area and the second area are separated from each other by at least one surface element of the transport container.

[0057] According to various embodiments, a surface element can have at least one of the following: a grid structure; one or more struts; and / or an advertising medium.

[0058] According to various embodiments, the transport container can have a chassis. The chassis can have at least three, e.g., at least four, wheels (e.g., self-steering Castor wheels).

[0059] According to various embodiments, the chassis can provide two translational degrees of freedom and optionally one rotational degree of freedom, along which the transport container is movable.

[0060] According to various embodiments, the first translational degree of freedom and the second translational degree of freedom of the two translational degrees of freedom can be different from each other, e.g., perpendicular to each other. The two translational degrees of freedom can, for example, run along a surface defined by the substrate, e.g., horizontally.

[0061] According to various embodiments, the transport container can be made of plastic and / or a metal, e.g. aluminum and / or steel.

[0062] According to various embodiments, the second area can be arranged between the first area and the chassis.

[0063] According to various embodiments, the first area can have a pivotable surface element which is arranged in such a way that at least two transport containers can be pushed into one another.

[0064] Depending on the specific design, the transport container can have a handle and / or a grip strip. The handle can, for example, be pivotally mounted.

[0065] According to various embodiments, the transport container can have an opening on one side and a surface element (e.g., a base or side wall) on a second side opposite the first, wherein the cross-sectional area of ​​the surface element is smaller than the parallel cross-sectional area of ​​the opening, so that the surface element fits into the opening. Several transport containers can be designed to be nested inside one another.

[0066] According to various embodiments, a device for computer-aided empty container detection can comprise the following: an optical image acquisition system for acquiring image data; a data storage device for storing at least one reference pattern and / or in which at least one reference pattern is stored, wherein the reference pattern represents an empty transport container; a processor configured to perform the following procedure: acquiring image data of an area of ​​the transport container; determining a contour pattern representing the transport container (e.g.,which is caused by the transport container), using the image data; determining a deviation magnitude that represents a deviation of the contour pattern from at least one reference pattern, where the at least one reference pattern represents an empty transport container; outputting a signal when the deviation magnitude meets a predefined criterion.

[0067] According to various embodiments, the optical image acquisition system can define an image acquisition area in which image data is acquired; wherein the transport container for acquiring the image data is or is arranged in the image acquisition area, wherein the image acquisition area defines an image background of the transport container; and wherein a reference pattern of the at least one reference pattern has a contour pattern of the image background (e.g., when no and / or an empty transport container is arranged in the image acquisition area).

[0068] The processor may also be configured to perform one of the procedures described herein, e.g., as described above.

[0069] According to various embodiments, the image acquisition system can have at least two (optical) image acquisition sensors (e.g., provided by one or two cameras) for capturing depth information (e.g., stereoscopically). To determine the depth information, the image data captured by two image acquisition sensors can be superimposed, e.g., taking into account the relative spatial position and / or orientation of the two image acquisition sensors to each other.

[0070] Alternatively or additionally, the image acquisition system can include at least one plenoptic camera (which can also be called a light field camera) for capturing depth information.

[0071] Alternatively or additionally, the image acquisition system may include a projector configured to project an optical pattern into the image acquisition area, and an image acquisition sensor configured to capture the optical pattern for the purpose of capturing depth information.

[0072] A camera can have an (optical) image acquisition sensor and at least one lens arrangement associated with the image acquisition sensor. The lens arrangement of a plenoptic camera can comprise a grid of several microlenses.

[0073] An image acquisition sensor (also called an image sensor or optical sensor) can have several photoelectrically active areas (also called pixels) which generate and / or modify an electrical signal in response to electromagnetic radiation (e.g., light, such as visible light). The image acquisition sensor can, for example, be a CCD sensor (charge-coupled device sensor) and / or an active pixel sensor (also called a CMOS sensor), or be composed of these components.

[0074] According to various embodiments, an image acquisition sensor can be configured to be wavelength-sensitive (e.g., for capturing color information and / or for capturing a pattern projected into the image acquisition area).

[0075] According to various embodiments, the processor can be configured to determine the magnitude of the deviation by measuring the contrast deviation of the contour pattern.

[0076] According to various embodiments, the processor can be configured to determine the magnitude of the deviation, specifically the area coverage deviation of the contour pattern.

[0077] According to various embodiments, the processor can be configured to determine the magnitude of the deviation, specifically to determine a contour pattern deviation.

[0078] Depending on the specific design, the data storage device can be non-volatile. For example, the data storage device can consist of or be comprised of a hard drive and / or at least one semiconductor memory (such as read-only memory, random access memory, and / or flash memory). The read-only memory can, for example, be a erasable programmable read-only memory (which may also be referred to as EPROM). The random access memory can be non-volatile random access memory (which may also be referred to as NVRAM).

[0079] According to various embodiments, the device may further include a (e.g., digital) point-of-sale system configured to perform one or more point-of-sale processes. A point-of-sale process may, for example, be a calculation process, an inventory process, and / or a registration process.

[0080] The point-of-sale (POS) system may include at least one of the following: a screen (which may also be referred to as the primary screen, e.g., a touchscreen), a printer (e.g., for printing an invoice and / or a label), a scanner (e.g., a barcode scanner) for registering items, a cash drawer, a (e.g., programmable) POS keyboard (which may also be part of the touchscreen), an electronic payment terminal (which may also be referred to as an EC terminal, "EC" - electronic cash, e.g., for reading a debit card and / or a credit card), and / or an additional screen (which may also be referred to as a secondary screen, e.g., a customer monitor), a signal output (which may also be part of the screen), and an operator station (where a user operating the POS system may be located).

[0081] The primary screen can display information representing the current state of a checkout process. The secondary screen can display some of the information shown on the primary screen.

[0082] According to various embodiments, the signal can be output via signal output, e.g. by displaying a prompt via the screen and / or e.g. by displaying an acoustic signal via acoustic signal output.

[0083] The prompt can be displayed in such a way that continuing the registration process is only possible if an entry has been made in response to the prompt.

[0084] According to various embodiments, a database (which may also be referred to as a reference database) containing one or more data records can be stored in the data storage device. Each data record can contain a reference pattern. Furthermore, each data record can contain at least one of the following pieces of information: reference depth information; reference topography information; reference color information; a spatial location (e.g., position and / or orientation of a transport container) associated with the reference pattern; a weight characteristic; and / or a transport container type associated with the reference pattern.

[0085] Alternatively, detection and / or acquisition can be performed using an algorithm that is invariant under translation and / or rotation. For example, an algorithm for the rapid and robust detection of image features, a so-called "robust feature recognition algorithm" (SURF - Speeded Up Robust Features algorithm), can be used. The algorithm used for feature-based detection can, for example, be rotationally invariant, so that the transport container can be detected regardless of its orientation. In this case, information about the spatial position of the transport container can be dispensed with.

[0086] According to various embodiments, the image acquisition system can be oriented in such a way that the image acquisition area (or the image background) has a substrate (e.g. a hall floor).

[0087] According to various embodiments, a method for computer-aided empty container detection can comprise the following: capturing image data of an image acquisition area in which a transport container is arranged, wherein the image acquisition area defines an image background; determining a contour pattern representing the transport container and / or the image background using the image data; determining a deviation magnitude representing at least one deviation of the contour pattern from at least one reference pattern, wherein the at least one reference pattern represents an empty transport container; and outputting a signal if the deviation magnitude meets a predefined criterion. Intuitively, it can be determined whether a contour of the image background and / or the transport container is obscured and / or disrupted by an object.

[0088] The topographic information can, for example, represent a three-dimensional profile (e.g., an area in space). Based on the depth information, a spatial location (e.g., position, orientation, and / or distance to a reference point) in space can be determined. Based on the color information, a color histogram and / or color spectrum can be determined. The deviation magnitude can, for example, represent a deviation of the three-dimensional profile, the spatial location, the color spectrum, and / or the color histogram from corresponding reference data.

[0089] According to various embodiments, the criterion can represent a threshold value. For example, the "empty detected" signal can be output if the deviation is less than the threshold value. Conversely, the "non-empty detected" signal can be output if the deviation is greater than the threshold value.

[0090] Exemplary embodiments of the invention are shown in the figures and are explained in more detail below.

[0091] They show Figure 1: A method according to various embodiments in a schematic flowchart; Figure 2: A device according to various embodiments in a schematic view; Figure 3: A device according to various embodiments in a schematic perspective view; Figure 4: A transport container according to various embodiments in a schematic side view or cross-sectional view; Figure 5: A transport container according to various embodiments in a schematic side view or cross-sectional view; Figure 6: A method according to various embodiments in a schematic flowchart; Figure 7: A method according to various embodiments in a schematic flowchart; Figure 8: A method according to various embodiments in a schematic flowchart; Figure 9: A device according to various embodiments in a schematic perspective view;Figure 10: A device according to various embodiments in a schematic perspective view; Figure 11: An image acquisition system according to various embodiments in a schematic perspective view; Figure 12: A method according to various embodiments in a schematic view; Figure 13: A method according to various embodiments in a schematic view; Figure 14: A method according to various embodiments in a schematic view; Figure 15: A method according to various embodiments in a schematic view; Figure 16: A method according to various embodiments in a schematic view; Figure 17: A method according to various embodiments in a schematic view; Figure 18: A method according to various embodiments in a schematic view; Figure 19: A method according to various embodiments in a schematic view;Figure 20 shows a method according to different embodiments in a schematic view; and Figures 21 to 37 each show a method according to different embodiments in a schematic view.

[0092] The following detailed description refers to the accompanying drawings, which form part thereof and illustrate specific embodiments in which the invention can be implemented. In this context, directional terminology such as "top," "bottom," "front," "back," "anterior," "rear," etc., is used with reference to the orientation of the described figure(s). Since components of embodiments can be positioned in a number of different orientations, the directional terminology serves only for illustration and is in no way limiting. It is understood that other embodiments may be used and structural or logical modifications may be made without deviating from the scope of protection of the present invention.It is understood that the features of the various exemplary embodiments described herein can be combined with one another, unless specifically stated otherwise. The following detailed description is therefore not to be interpreted in a limiting sense, and the scope of protection of the present invention is defined by the appended claims.

[0093] Within the scope of this description, the terms "connected," "attached," and "coupled" are used to describe both direct and indirect connections, direct or indirect links, and direct or indirect couplings. In the figures, identical or similar elements are labeled with identical reference symbols where appropriate.

[0094] According to various embodiments, the transport container can include or be formed from a shopping cart, a shopping basket or another cart (and / or basket).

[0095] According to various embodiments, it is possible to detect remaining items inside the transport container, e.g. on a lower shelf of the shopping cart.

[0096] According to various embodiments, compared to conventional pattern recognition, it can be improved and / or made possible to detect whether or not one of the following objects (e.g. goods, which can also be referred to as articles) is arranged in the transport container (which cannot be reliably determined, for example, via color histograms or depth information): a flat object (which, for example, limits 3D detection), a barely or not at all highlighted and / or colored object (e.g., with only low information, color, and / or texture content), a monochrome and / or homogeneous object (for example, an almost homogeneous layout such as black, white, gray, etc.), a transparent object.

[0097] According to various embodiments, the recognition of the object can be based on the fact that the object disturbs (or interrupts) the contour pattern (e.g. a grid contour) of the transport container and / or the image background and / or creates a contrast to the transport container and / or the image background (e.g. surface and / or floor).

[0098] According to various embodiments, a computer-aided detection system for objects in an open transport container is provided. This allows the status of the transport container (empty, not empty, and / or full) to be determined at a specific time and / or location, e.g., automatically.

[0099] According to various embodiments, an image acquisition system (e.g., Intel RealSense F200, Intel R200 and / or Intel SR300) can have a 2D image acquisition sensor and / or a 3D image acquisition sensor.

[0100] The processor can be provided by means of an electronic (programmable) data processing system. The electronic data processing system can also include the data storage. For example, the electronic data processing system can include or be comprised of a microcomputer, e.g., a PC system (personal computer system) or a digital point-of-sale system (

[0101] The processor can be connected to the image acquisition system via a USB interface (Universal Serial Bus interface), e.g. via USB 3.0.

[0102] According to various embodiments, the data processing system can have at least one co-processor (e.g., provided by a co-processor card or external graphics card), such as a graphics processing unit (GPU). The co-processor allows for the acceleration of computationally intensive processes by offloading them to the co-processor.

[0103] Fig.1 A process 100 is illustrated according to various embodiments in a schematic flowchart.

[0104] According to various embodiments, the method 100 can, as described in 101, include: capturing image data of an area of ​​the transport container. The method 100 can, as described in 103, further include: determining a contour pattern representing the transport container (e.g., one created by the transport container) using the image data. The method 100 can, as described in 105, further include: determining a deviation quantity representing a deviation of the contour pattern from at least one reference pattern, wherein the at least one reference pattern represents an empty transport container. The method 100 can, as described in 107, further include: outputting a signal when the deviation quantity meets a predetermined criterion.

[0105] Fig.2 A schematic view illustrates a device 200 according to various embodiments.

[0106] The device 200 can comprise an optical image acquisition system 202, a data storage device 204, and a processor 206. The processor 206 can be coupled to the optical image acquisition system 202 and the data storage device 204, e.g., by means of a data line (i.e., so that data can be transferred between them).

[0107] For example, the processor 206 can be configured to acquire image data using the optical image acquisition system 202. For example, the processor 206 can control the image acquisition system 202.

[0108] For example, the image data can be stored in a data storage device 202d (which can also be referred to as a buffer 202d) of the image acquisition system 202. The image data can be acquired using an optical sensor 202s (which can also be referred to as an image acquisition sensor 202s). At least after processing the image data using the processor 206, it can be read from the buffer 202d.

[0109] The optical sensor 202s can have a photoelectrically active surface 202o which is directed towards an image acquisition area 208, e.g. in the direction 208b (can also be referred to as image acquisition direction 208b).

[0110] Fig.3 Figure 300 illustrates a device 300 according to various embodiments in a schematic perspective view.

[0111] According to various embodiments, a transport container 400, e.g. a shopping cart, can be arranged in the image acquisition area 208, as shown in Fig.3 This is illustrated. The transport container 400 can have a chassis 312 with several rotatably mounted wheels 312r and a transport frame 322. The transport frame 322 can be movably mounted on the chassis 312. The transport frame 322 can provide at least one transport area 322a.

[0112] The transport container 400 can have a handle 302g, by means of which the transport container 400 can be guided, e.g., by a person. The transport container 400 can stand on a base 208u and / or be moved. The base 208u can limit the image capture area 208. The base 208u can, for example, define the image background of the image capture area 208.

[0113] For example, the image acquisition direction 208b of the image acquisition sensor 202s can be directed towards the surface 208u. The image acquisition direction 208b can optionally be configured to change, for example, by at least one angle 202w (e.g., a solid angle). This allows the image acquisition direction 208b to be adjusted, for example, based on the image data and / or by detecting the position of the transport container within the image acquisition area 208. To illustrate, the image acquisition direction 208b can, for example, be tracked to follow a moving transport container 400 and / or adjusted to a transport container 400 that is not precisely positioned.

[0114] Alternatively or additionally, the angle 202w can define an image capture angle 202w (which can also be referred to as viewing angle). The image capture angle 202w can be understood as the angle in spatial space that is limited by the edges of the image format. For the usual rectangular image format, the image capture angle 202w can refer to the value corresponding to the image diagonal. Horizontal and vertical image capture angles 202w can be smaller than the value corresponding to the image diagonal.

[0115] The image capture system 202 can be held or mounted on a support 320, e.g., a cash register counter 320. Optionally, several elements of the cash register system can be arranged on the cash register counter 320, e.g., a screen, printer, scanner, and / or a payment terminal.

[0116] According to various embodiments, the transport container 400 can be searched for in the image acquisition area 208 (which can also be referred to as the camera's field of view).

[0117] To capture image data (and detect empty containers based on that data) of the transport container 400, a direct line of sight between the image capture system 202 and the transport container 400 can be provided or established. In other words, optical detection of objects in the transport container 400 can take place.

[0118] Fig.4 Figure 4 illustrates a transport container 400 according to various embodiments in a schematic side view or cross-sectional view, e.g. a transport container 400 of a first transport container type.

[0119] The transport frame 322 can have several transport areas 322a, 322b (a first transport area 322a and a second transport area 322b), each of which can be equipped to hold an object (which is to be transported). For example, each transport area 322a, 322b can be provided and / or delimited by means of a support surface element 402a, 402b (e.g. a floor element).

[0120] The transport frame 322 can have several surface elements, e.g., one or more support surface elements 402a, 402b (e.g., base elements 402a, 402b), one or more side surface elements 412a, 412b (e.g., side elements 412a, 412b). Each transport area 322a, 322b can be bounded by one or more surface elements, e.g., by at least one support surface element 402a, 402b and optionally by several (up to four) side surface elements 412a, 412b. At least one first side surface element 412b can be pivotally mounted 401, e.g., between an open state and a closed state. In an open state, an opening 402o can be formed, which is configured (size and position) such that a second side surface element 412b fits into the opening 402o. Thus, several transport containers 400 can be pushed into one another, e.g. in the direction 403.

[0121] For example, the cross-sectional area (e.g. cut perpendicular to direction 403) of the first transport area 322a can decrease in direction 403, which points from the first side surface element 412a to the second side surface element 412b.

[0122] In at least one direction, each transport area 322a, 322b can be open 502o (i.e., have an opening 502o), so that, for example, an object can pass through the opening 502o into the transport area 322a, 322b.

[0123] According to various embodiments, the transport container 400 can have a surface element (e.g., a support surface element 402a, 402b and / or a side surface element 412a, 412b) which has several through-openings (e.g., a grid or a grate). For example, the transport container 400 can have an intermediate floor surface 402a that is not completely closed in order to enable detection in the second transport area 322b (illustratively, the lower transport area 322b) from a predetermined perspective (e.g., from above), in which the intermediate floor surface 402a is arranged between the image acquisition system 202 and the second transport area 322b.

[0124] According to various embodiments, a first support surface element 402a (which can also be referred to as an intermediate floor surface 402a) and / or a second support surface element 402b can have multiple through-openings. In other words, the first support surface element 402a and / or the second support surface element 402b can be opaque. This allows it to be determined through the first support surface element 402a whether an object is located in the second transport area 322b, for example, using depth information. For instance, the depth information can be used to determine that certain features are associated with the first support surface element 402a and can be discarded to form the contour pattern if, for example, only the second transport area 322b is to be examined.

[0125] Alternatively or additionally, this allows contours of the second surface element 402b and / or the image background 602h to be used to form the contour pattern through the first surface element 402a. Similarly, this allows contours of the image background to be used to form the contour pattern through the second surface element 402b. This can facilitate or simplify the detection of empty spaces (more intuitively, since the contour pattern has more features).

[0126] Alternatively or additionally, the side surface elements 412a, 412b (e.g., those facing the image acquisition system 202) can have multiple through-openings. This allows contours of the first support surface element 402b, the second support surface element 402b, and / or the image background to be used to form the contour pattern through a side surface element 412a, 412b. For example, the depth information can be used to determine that certain features are assigned to a side surface element 412a, 412b, and these features can be discarded for forming the contour pattern.

[0127] For example, a (non-opaque) surface element 402a, 402b, 412a, 412b may have or be formed from a (e.g., metallic or non-metallic) lattice structure. Alternatively or additionally, a (non-opaque) surface element 402a, 402b, 412a, 412b may have or be formed from a continuous (e.g., transparent or semi-transparent) surface or a combination of a continuous surface and a lattice structure (e.g., transparent and / or semi-transparent).

[0128] Several transport container types can have a chassis 312 and a transport frame 322. For example, two transport container types can differ in at least one of the following: the extent of the first area 322a and / or the second area 322b, or the presence of the first area 322a and / or the second area 322b.

[0129] Fig.5 Figure 4 illustrates a transport container 400 according to various embodiments in a schematic side view or cross-sectional view, e.g. a transport container 400 of a second transport container type.

[0130] The first type of transport container may differ from the second type of transport container at least in the presence of a chassis 312.

[0131] The transport container 400 can have an open transport area 322a (with an opening 502o). The transport area 322a can be bounded by a support surface element 402a and several side surface elements 412a, 412b.

[0132] According to various embodiments, the cross-sectional area (e.g., cut transversely to direction 403) of the first transport area 322a can decrease in direction 403, which points from the opening 502o to the support surface element 402a. This makes it possible to nest several transport containers 400 inside one another, e.g., along direction 403, e.g., for stacking the several transport containers 400.

[0133] The transport container 400 can have a handle 502, which can optionally be mounted in a swiveling position.

[0134] Analogous to the above description, contours of the image background can optionally be detected through the support surface element 402a and used to create the contour pattern, for example, if it has multiple openings. This can facilitate or make it easier to detect empty areas.

[0135] Analogous to the above description, optionally, contours of the image background and / or the support surface element 402a can be detected through a side surface element 412a, 412b and used to form the contour pattern, e.g., if this has several openings. This can facilitate or make it easier to detect empty spaces.

[0136] Several transport container types may not have a chassis 312. For example, two transport container types may differ in at least one dimension of area 322a (e.g., height and / or width).

[0137] Fig.6 A process 600 is illustrated according to various embodiments in a schematic flowchart.

[0138] According to various embodiments, image data 602 of an area of ​​the transport container 400 can be acquired 602e. Using the image data 602, a contour pattern 604a, 604b, 604c can be determined which represents the transport container 400.

[0139] A first contour pattern 604a can, for example, represent an image background 602h of the image acquisition area (e.g., the surface 208u), which is disturbed 612h due to the transport container 400. Visually, a structure 400s (e.g., a grid) of the transport container 400 can obscure part of the image background 602h, so that its contour is disturbed 612, e.g., interrupted 612s.

[0140] Alternatively, a second contour pattern 604b can, for example, represent the structure 400s (e.g. a grid 400s) of the transport container 400.

[0141] According to various embodiments, a third contour pattern 604c can represent the structure 400s (e.g. a grid) of the transport container 400 and the image background 602h of the image acquisition area 208.

[0142] Furthermore, the first contour pattern 604a, the second contour pattern 604b and / or the third contour pattern 604c can be used to determine the deviation size.

[0143] Contour pattern 604 (e.g., contour pattern 604a, contour pattern 604b, and / or contour pattern 604c) can be used, for example, as a reference pattern when transport container 400 is empty. A reference contour pattern can be taught to the system in a practical way.

[0144] The optical features of the transport container 400 (such as a shopping cart) or feature-rich regions of it, and / or the background / image, can be initially trained. The recognized features can be stored in a reference database.

[0145] Fig.7 A process 700 is illustrated according to various embodiments in a schematic flowchart.

[0146] According to various embodiments, a determined contour pattern 604 (e.g., contour pattern 604a, contour pattern 604b, and / or contour pattern 604c) can have a contour representing an object 702 when it is arranged in the transport container 400. Visually, the contour of the transport container 400 and / or the image background can be disturbed by the object 702, e.g., (at least partially) obscured. For example, the contour pattern 604 can have one or more disturbances 702u (e.g., interruptions) caused by the object 702.

[0147] Contour pattern 604 can be compared to a reference contour pattern 704 (which may also be referred to as reference pattern 704). Reference pattern 704 can represent the image background 602h of the image acquisition area and / or the structure 400s of the transport container 400 when the transport container 400 is empty.

[0148] Based on contour pattern 604 and reference pattern 704, a deviation quantity 706 can be determined, which represents a deviation of contour pattern 604 from reference pattern 704. Reference pattern 704 can have a contour pattern of the image acquisition area in which no or an empty transport container 400 is located (e.g., an empty transport container 400 in front of the image background 602h).

[0149] The deviation quantity 706 can represent a contrast deviation 706a, a color deviation 706a, a contour gradient deviation 706b, and / or an area coverage deviation 706c. The contrast deviation 706a can clearly indicate the degree of deviation in contrast 706k between the reference pattern 704 and the contour pattern 604. The contour gradient deviation 706b can clearly indicate the degree of interruption in the gradient of a contour between the reference pattern 704 and the contour pattern 604. The area coverage deviation 706c can clearly indicate the degree of deviation in an area that is covered by the contours of the reference pattern 704 and the contour pattern 604, respectively.

[0150] The contrast deviation 706a can be determined pixel by pixel, for example by comparing the values ​​706w (e.g. contrast values ​​706w and / or brightness values ​​706w) of corresponding pixels of the reference pattern 704 and the contour pattern 604.

[0151] Alternatively or in addition to the contrast deviation 706a, a color deviation 706a can be determined, for example pixel by pixel, e.g. by comparing the values ​​706w (e.g. color values ​​706w and / or brightness values ​​706w) of corresponding pixels of the reference pattern 704 and the contour pattern 604.

[0152] For contour deviation 706b, for example, a contour 604k of contour pattern 604 can be assigned a reference contour 704k of reference pattern 704, and the (e.g., spatial and / or pixel-wise) deviation 706 of contour 604k from the reference contour 704k can be determined. The deviation 706 of contour 604k from the reference contour 704k can be determined, for example, in a spatial space and / or a pixel space spanned by at least two coordinates 706x, 706y (spatial coordinates and / or pixel coordinates).

[0153] The area coverage deviation 706c can be determined, for example, by comparing the area coverage 706f (e.g., number of occupied pixels and / or occupied proportion of the pattern) of the reference pattern 704 and the contour pattern 604.

[0154] The deviation quantity 706 can have a numerical value (e.g., an area coverage deviation 706c, a contrast deviation 706a and / or a color deviation), a vector (e.g., a contour deviation, a depth deviation, spatial coordinates and / or pixel coordinates) and / or a matrix (e.g., a topography deviation).

[0155] Fig.8 A process 800 is illustrated according to various embodiments in a schematic flowchart.

[0156] According to various embodiments, the deviation quantity 706 can be compared with a predefined criterion 806. Furthermore, a signal can be output if the deviation quantity 706 satisfies the criterion 806. The signal can, for example, be a visual signal (e.g., a colored signal and / or a geometric signal), an input prompt, and / or an acoustic signal, or be composed of these.

[0157] The specified criterion 806 can have a first criterion 806a (which can also be called the empty criterion 806a), representing an empty transport container. The signal can then have a first signal 802a (which can also be called the empty-detected signal 802a), representing an empty detection. For example, the first signal 802a can be output if the deviation value 706 is less than a first threshold value represented by the first criterion 806a.

[0158] The specified criterion 806 can have a second criterion 806b (which can also be referred to as error criterion 806b), representing an error range. The signal can then have a second signal 802b (which can also be referred to as error signal 802b), representing error detection. For example, the second signal 802b can be output if the deviation value 706 is greater than the first threshold and is not greater than a second threshold represented by a third criterion 806c.

[0159] The specified criterion 806 can include a third criterion 806c (which can also be referred to as the non-empty criterion 806c), representing a non-empty transport container. The signal can then include a third signal 802c (which can also be referred to as the non-empty detected signal 802c), representing a non-empty detection. Intuitively, the third signal 802c can be output if the deviation value 706 is greater than the second threshold value represented by the third criterion 806c.

[0160] To illustrate, the error range can lie between the empty criterion 806a and the non-empty criterion 806c and represent a range in which a clear decision as to whether the transport container 400 is empty or not cannot be reliably made.

[0161] In the event of a defect detection, the procedure 100 can start again from the beginning. For example, a different area of ​​the transport container 400 and / or a different analysis area can be used to form the contour pattern 604.

[0162] Fig.9 A device 900 according to various embodiments is illustrated in a schematic perspective view.

[0163] The cash register system can have a primary screen 802, a barcode scanner 804, a secondary screen 808, and an EC terminal 816. Screen 802 can be configured to output a signal. For example, the screen can display a colored signal, a geometric signal, and / or a prompt representing the status of the transport container 400 (located here outside the image capture area 208) (e.g., empty or not empty) and / or the status of the image capture area 208 (e.g., with or without transport container 400).

[0164] The device 900 can include an image acquisition system 202, which has several optical sensors 202s. Each sensor 202s can define a detection area 208s, which together define the image acquisition area 208. For example, the detection areas 208s of at least two sensors 202s of the image acquisition system 202 can overlap. The illustrated mounting positions can be understood as exemplary. The two sensors 202s can alternatively be mounted such that their detection areas are separated from each other or adjacent to each other.

[0165] The substrate 208u can have a contour which represents, for example, the floor covering (e.g., its tiles and / or its joints).

[0166] According to various embodiments, the image acquisition area 208 can be used to search for the transport container 400. It can then be determined whether the transport container 400 is one of several (known) container types, e.g., the first container type or the second container type. Alternatively or additionally, it can be determined whether a transport container 400 is located in the image acquisition area 208.

[0167] Furthermore, a signal (which can also be referred to as a "not detected" signal) can be output if no container type from among the multiple container types has been detected and / or no transport container 400 has been detected in the image acquisition area 208. In the event that no container type from among the multiple container types and / or no transport container 400 has been detected in the image acquisition area 208, an error detection may occur. In this case, the "not detected" signal can be an error signal.

[0168] In the event of an error detection, the procedure 100 can start again from the beginning. For example, image data from another area of ​​the image acquisition area 208 can be used to create the contour pattern 604.

[0169] According to various embodiments, the signal (e.g., the error signal) can include an input prompt, whereby a cash register system process enters a waiting state until an input is received in response to the prompt. This allows an operator (e.g., the cashier) to be prompted to create a state (e.g., by moving the transport container 400) in which the transport container 400 and / or its transport container type can be recognized. This further reduces the number of unregistered items.

[0170] Once the transport container and / or its type has been identified, contour pattern 604 can be determined. Alternatively or additionally, a reference pattern representing the transport container type can be selected. This allows for a more precise determination of the deviation magnitude and / or reduces the number of error detections. Visually, the reference pattern can be adapted to the identified transport container type, so that the reference pattern of the identified transport container type is used to determine the deviation magnitude.

[0171] Optionally, the predefined criterion can be adapted based on the detected transport container type. For example, a first criterion can be used for the first transport container type and a second criterion for the second transport container type. This allows for more accurate empty container detection and / or reduces the number of false positives.

[0172] The primary screen 802 can be configured to display information representing the item(s) registered by the barcode scanner 804. For example, the registered items can be listed and displayed on the primary screen 802 along with associated information (e.g., registration information), such as item number information, item name information, and / or price information.

[0173] Fig.10 Figure 1000 illustrates a device 1000 according to various embodiments in a schematic perspective view.

[0174] According to various embodiments, at least one optical sensor 202s can be attached to a holder 808h (e.g., a rod) which holds the secondary screen 808. The secondary screen 808 can be configured to display some of the information shown by the primary screen 802, e.g., article name information and / or price information.

[0175] Fig.11 An image acquisition system 202, according to various embodiments, is illustrated in a schematic perspective view.

[0176] The image acquisition system 202 can have multiple (e.g., one or more than two) image acquisition sensors 202s. The image acquisition system 202 can have a mounting structure to which it can be attached. Furthermore, the image acquisition system 202 can have a data line 202k by means of which it can be connected, for example, to the processor 206.

[0177] The image acquisition system 202 can be configured to capture image data representing two-dimensional (2D) information, e.g., in a spatial context. Optionally, the image acquisition system 202 can be configured to capture image data representing three-dimensional (3D) information (also referred to as depth information), e.g., in a spatial context.

[0178] The image acquisition system 202 can optionally include a laser projector 202p, which is configured to project an optical pattern into the image acquisition area 208, e.g., using structured laser light (i.e., light structured in the form of the pattern). In other words, the laser projector 202p can emit light according to the pattern (e.g., in the form of a grid) into the image acquisition area 208. The emitted light can be outside the visible light spectrum, e.g., in an infrared (IR) spectral range (more than approximately 780 nm). Alternatively or additionally, other spectral ranges can be used, e.g., also in the visible light range (approximately 380 nm to approximately 780 nm). If light from external light sources is emitted into the image acquisition area, spectral ranges can be used that are, for example, outside the range of light emitted by the external light sources.At least one image acquisition sensor 202s can be configured to capture the spectral range of the laser projector 202p. Thus, deformations of the pattern caused by topography in the image acquisition area and / or an object within the image acquisition area can be detected by the image acquisition sensor 202s (i.e., topographic information can be determined). In other words, the image acquisition sensor 202s can acquire 3D image data. This 3D image data can then be further processed by the processor. Based on the optical pattern projected into the image acquisition area 208, depth information can be obtained alternatively or additionally, for example, by determining the distance of the image acquisition system 202 to contours, objects, and / or structures within the image acquisition area, and / or by evaluating the topographic information.

[0179] Alternatively or additionally, the image acquisition system 202 can be configured to acquire stereoscopic image data using at least two image acquisition sensors 202s. The image data can then consist of two corresponding image data parts, which are captured simultaneously from different angles.

[0180] The image data can be processed using the processor, and thus, for example, the distance of the image acquisition system 202 to contours, objects and / or structures in the image acquisition area can be measured (i.e., depth information can be determined).

[0181] Fig.12 A method 1200 is illustrated in a schematic view according to various embodiments.

[0182] In a 2D mode, it can first be detected whether a transport container 400 is located within the image acquisition area 208 (i.e., whether a transport container 400 is present). For example, detection can be achieved by capturing reflections (on objects), e.g., in the IR range. It can be determined whether the reflections originate from a transport container, e.g., from its structure and / or from markers. This can enable faster detection. The reflections can optionally be compared with the reference pattern. For this purpose, the reference pattern can contain reference reflection data.

[0183] The transport container 400 can be identified based on the image data 602. For example, pattern recognition and / or contour recognition can be performed based on the image data 602 and compared with the database. Contour recognition can, for example, simply determine whether or not a contour is present.

[0184] Alternatively or additionally, the contour pattern 604 (superimposed on the image data in the representation) can be determined to identify whether a transport container 400 is located in the image acquisition area 208.

[0185] Furthermore, in 2D mode, for example, if it has been detected that a transport container 400 is located in the image acquisition area 208, it can be determined, based on the contour pattern 604, whether an object is located in the transport container 400. If a deviation of the contour pattern 604 from the reference pattern 704 meets the criterion (e.g., all elements of the contour pattern 604 match the elements of the reference pattern 704), an empty-detected signal 802a can be output, for example, on the primary screen 802. The empty-detected signal 802a can, for example, display information representing that the transport container 400 has been detected as empty, e.g., text and / or a color (e.g., green).

[0186] Optionally, in 2D mode, a deviation of the color information 1204 can be used to determine the magnitude of the deviation. For example, it can be determined whether (and if so, how many) pixels of the image data 602 have a color value that deviates from the reference color information. If, for example, the transport container 400 only contains shades of gray, it can be determined whether the image data 602 contains color values.

[0187] Alternatively or additionally, a 3D mode can be used to determine whether (and if so, how many) pixels of the image data 602 have a depth value (e.g., a distance to the image acquisition system 202) that deviates from the reference depth information. Optionally, a topography representing the transport container 400 (or, if applicable, its contents) can be determined based on the image data 602.

[0188] For example, a first transport area 322a can first be checked in 2D mode and optionally afterwards a second transport area 322b located behind it in 3D mode (i.e. the first transport area 322a is located between the second transport area 322b and the image acquisition system 202).

[0189] Fig.13 A method 1300 according to various embodiments is illustrated in a schematic view.

[0190] In 2D image mode, it can be determined if the transport container 400 is not positioned within the image capture area 208 (e.g., it has not been placed in front of the cash register system and / or is at least partially obscured). For example, the contour pattern 604 may have fewer features than specified by the criterion. Alternatively or additionally, the contour pattern 604 may have features that represent an edge of the transport container 400. This allows for a clear determination of whether the transport container 400 is not completely within the image capture area 208 and / or is at least partially obscured (e.g., by a person).

[0191] If no transport container 400 is detected in the image acquisition area 208 and / or it is detected that it is not completely positioned in the image acquisition area 208 and / or is at least partially obscured, then an error signal 802b can be output, e.g., an error signal 802b (e.g., the "Not Detected" signal). The error signal 802b can, for example, be displayed on the primary screen 802. The error signal 802b can, for example, display information representing that no transport container 400 was detected, e.g., text and / or a color (e.g., yellow).

[0192] Optionally, color information 1204 (in 2D mode) and / or depth information 1206 (in 3D mode) can be used to detect whether a transport container 400 is located in the image acquisition area 208. For example, the contour pattern 604 may not be clearly detected if the image acquisition area 208 is too brightly illuminated. In this case, for example, if the detection of the transport container 400 based on the contour pattern and / or the color information 1204 or the depth information 1206 leads to different results, the error signal 802b may contain information representing the cause of the error.

[0193] Optionally, if the transport container type was not recognized, the error signal 802b can contain information representing the cause of the error "transport container type not recognized".

[0194] If the transport container has 400 areas that are prone to error detection (e.g., advertising areas or highly reflective surfaces), these areas (error areas) can be hidden (i.e., not used to determine the contour pattern 604). Alternatively or additionally, the error signal 802b can contain information representing a corresponding cause of the error (e.g., "error area detected").

[0195] Fig.14 A method 1400 according to various embodiments is illustrated in a schematic view.

[0196] In the event that the transport container 400 is detected in the image acquisition area 208 (e.g., in 2D mode), but it cannot be determined whether it is empty or not (i.e., whether an object is located in the transport container 400), an error signal 802b can be output. For example, error signal 802b can be output if no contour pattern could be determined because the image data is noisy, underexposed, and / or overexposed. For example, in 2D mode, transparent objects, thin objects, and / or small objects can interfere with the contour pattern, even if only partially, to such an extent that the determination of the contour pattern is impaired.

[0197] Optionally, color information 1204 (in 2D mode) and / or depth information 1206 (in 3D mode) can also be used to detect whether the transport container 400 is present. However, it is also possible that the color information 1204 cannot be clearly assigned, for example, if the object has low saturation (e.g., a white color), high reflectivity, or is at least partially obscured by part of the transport container 400.

[0198] The error signal 802b can, for example, represent information indicating the cause of the error "transport container type detected" and / or "no decision possible", e.g., a text and / or a color (e.g., orange).

[0199] Fig.15 A method 1500 is illustrated according to various embodiments in a schematic view.

[0200] According to various embodiments, it can be recognized on the basis of the contour pattern 604 (shown here superimposed with the image data 602) when an object 702 is arranged in the transport container 400, e.g. in the first transport area 322a.

[0201] If an item 702 (e.g., a product) is detected in the transport container 400 in 2D mode, a "Not Empty" signal 802c can be output, for example, on the primary screen 802. The "Not Empty" signal 802c can, for example, display information representing that the transport container 400 was detected as not empty, such as text and / or a color (e.g., red). Alternatively or additionally, the "Not Empty" signal 802c can include a prompt 1402. The prompt 1402 can, for example, request confirmation that the "Not Empty" signal 802c is correct (i.e., that no false alarm was triggered). If the non-empty detected signal 802c is incorrect (i.e., a false alarm was triggered), i.e., it was incorrectly detected that an item is located in the transport container 400, the criterion can optionally be adjusted, e.g.such that a similar contour pattern 604, like the one that leads to the false alarm, will trigger an error signal in the future (e.g., that the detected contour pattern falls within the error range).

[0202] Optionally, based on the color information 1204, it can be detected and / or confirmed whether the object 702 is located in the transport container 400, e.g., in the first transport area 322a. For example, contour and color recognition can be triggered simultaneously.

[0203] Fig.16 A method 1600 is illustrated in a schematic view according to various embodiments.

[0204] According to various embodiments, it can be recognized, based on the contour pattern 604, when an object 702 is arranged in the transport container 400, e.g., in the first transport area 322a. For example, the object 702 can form part of the first support surface element 402a (compare Fig.3 and Fig.4 ), of chassis 312 (compare Fig.3 ) and / or the image background. The covering and / or the area covered can be determined using contour pattern 604.

[0205] If the object 702 is transparent, the color information 1204, for example, may have too small a deviation to meet the criterion.

[0206] Optionally, based on the depth information 1206, it can be detected and / or confirmed that the object 702 is located in the transport container 400, e.g., in the first transport area 322a. For example, contour and topography recognition can be triggered simultaneously.

[0207] Fig.17 A method 1700 is illustrated according to various embodiments in a schematic view.

[0208] If an object 702 is arranged in a second area of ​​the transport container 400, which is, for example, at least partially obscured by the first area 322a, a deviation caused by object 702 can be detected based on the contour pattern 604. For example, object 702 may obscure part of the second support surface element 402b (see Figure 604). Fig.3 ), of the chassis 312 and / or of the image background 602h.

[0209] Optionally, based on the color information 1204 and / or the depth information 1206, it can be detected and / or confirmed that the object 702 is located in the transport container 400, e.g., in the second transport area 322b. For example, contour detection and color detection and / or depth detection can be triggered simultaneously.

[0210] To detect when the transport container 400 is empty, a contour pattern 604 and optionally color information 1204 and / or depth information 1206 can be used, according to various embodiments. A database can contain reference image data and / or reference color information or reference depth information representing an empty transport container 400. Therefore, an item database is not required for detecting when the transport container 400 is empty.

[0211] Fig.18 A method 1800 is illustrated in a schematic view according to various embodiments.

[0212] According to various embodiments, the image data can represent a first part 1802 of the first transport area 322a (can also be referred to as the first sub-area 1802).

[0213] According to various embodiments, additional image data of an additional area of ​​the transport container can be acquired. The additional image data of the additional area can represent a second part 1804 of the first transport area 322a (which can also be referred to as the second sub-area 1804). The image data and the additional image data can be acquired at once (e.g., simultaneously and / or using the same optical sensor), sequentially, and / or using different optical sensors.

[0214] Based on the additional image data, an additional contour pattern 614 (second contour pattern 614) can be determined, which can be used as reference pattern 704. The second sub-area 1804 can be used as a reference to determine whether an object is located in the first sub-area 1802.

[0215] Alternatively or additionally, the additional image data of the additional area can represent the second transport area 322b. Based on the additional image data, a further additional contour pattern 624 (third contour pattern 624) can be determined. The image data and the additional image data can be acquired all at once (e.g., simultaneously and / or using the same optical sensor), sequentially, and / or using different optical sensors. The deviation magnitude can then, for example, represent a disturbance 702u of the third contour pattern 624 caused by the object 702.

[0216] If the surface elements that define a transport area 322a, 322b (e.g., in the case of a basket, the inner surfaces of the basket, if present, such as larger grids), e.g., the support surface element 402a, 402b (in the case of a basket, a basket shelf, e.g., with struts and / or a continuous insert) are arranged in the image acquisition area 208 (e.g., the camera's field of view), the presence of an object 702 in the transport container 400, e.g., resting on the support surface element 402a, 402b, can be determined. For this purpose, a deviation caused by the visible surface of the object from characteristic properties of a previously learned contour pattern 704 (i.e., the stored reference pattern 704) and / or a currently determined contour pattern 614 (i.e., the additional contour pattern 614) can be determined. The deviation can be represented by the magnitude of the deviation.

[0217] The deviation magnitude can be visualized as a measure of the absence 702u of the expected contour (e.g., lattice structure or lattice surface) and / or its disturbance 702u in the examined area of ​​the transport container 400, caused by the object 702. For example, the expected contour pattern (e.g., lattice structure or lattice surface) may be present in the second sub-area 1804 (e.g., around the object 702) and / or in another part of the image data.

[0218] Alternatively or additionally, the deviation magnitude can be a measure of contrast transitions and / or contrast jumps, e.g., between adjacent areas and / or surfaces (e.g., between the support surface element 402a, 402b and the object 702). This can be done on the basis of a learned (e.g., stored) reference pattern 704 and / or on the basis of two determined contour patterns 604, 614, 624, e.g., by relating the determined (e.g., contiguous and / or adjacent) contour patterns 604, 614, 624 to each other or to the reference pattern 704.

[0219] Alternatively or additionally, a contour pattern 1814 can be determined, which represents the substrate 208u (for example, tiled floor with grout lines, applied design elements such as advertising logos on the floor, up to cyclically acquired soiling, etc.). The deviation magnitude can then be a measure of the absence 702u of the expected contour of the substrate 208u and / or of its disruption 702u caused by the object 702, e.g., deviating from the reference pattern 704 and / or a determined contour pattern 604, 614, 624 (visually expected contours / textures).

[0220] According to various embodiments, objects that are not and / or are difficult to detect using 3D and / or color recognition (e.g., bottle crates with narrow bottles and thus too small a surface area for 3D projection correspondence analysis, colorless and / or colorless objects) can be detected in the second transport area 322b (illustratively below the basket on the lower shelf) by using the method described herein (e.g., to compare the examined area to the neighborhood).

[0221] According to various embodiments, a disturbance (which can also be referred to as a parasitic disturbance) that is not caused by an object 702 (e.g., reflections on metal, discoloration of the transport container 400 due to rust, deformation of the transport container 400, etc.) can be detected. The parasitic disturbance (which can also be referred to as a negative case) can thus be distinguished from the presence of an object 702 in the transport container 400 (which can also be referred to as a positive case). This allows false alarms to be avoided or reduced.

[0222] Depending on the specific configuration, color information can optionally be provided or made available via color recognition. For example, the deviation magnitude can represent a deviation of the color information in the image data from reference image data, which represents the transport container 400 and / or the image background 602h (e.g., the substrate 208u) when the transport container 400 is empty. This can be illustrated by contour difference detection and color difference detection (deviating features from the reference data set). The color information of the substrate 208u in the image acquisition area 208 can be initially trained and stored in the database.

[0223] For example, if a change compared to a reference color spectrum is detected in the analysis area, this can be used as an indicator that the transport container 400 is not empty.

[0224] According to various embodiments, the deviation from the reference color information in the color space can be dynamically evaluated. This reduces false alarms caused by ambient light. In other words, the illumination state of the image acquisition area 208 can be taken into account to determine the magnitude of the deviation, e.g., based on the additional contour pattern 614.

[0225] Depending on the specific embodiment, 3D information (depth information) can optionally be provided or generated by means of depth detection. The position of the area (e.g., an adjustable area between the substrate 208u, above the second support surface element 402b, and the first support surface element 402a) in which the depth information is to be determined can be initially trained (this can be intuitively defined as the zone to be examined). For example, this can be done based on the detected transport container type. The position of the zone being examined (e.g., its boundaries and / or threshold) can be stored in the database, e.g., assigned to the respective transport container type.

[0226] If a deviation of the depth information from reference depth information is detected in the area to be examined (e.g., the second transport area 322b), this can be used as an indicator that the transport container 400 is not empty.

[0227] According to various embodiments, a combination variance can optionally be provided by means of a weight characteristic.

[0228] Each deviation, represented by the deviation magnitude, can be used to detect the non-empty state (transport container 400 is not empty). Every active sensor can contribute to this detection. The weighting characteristic allows the data from the individual sensors to be weighted. This enables an intelligent (e.g., learnable) combination to improve the quality of the evaluation. For example, the weighting characteristic can be adjusted if a false alarm is detected. Then, based on the cause of the error, the data associated with that cause (contour pattern, depth information, and / or color information) can be weighted less. Alternatively or additionally, the weighting characteristic can depend on the type of transport container.The weight characteristics and / or their dependencies can be stored in the database.

[0229] According to various embodiments, the method for computer-aided empty detection of the transport container 400 and the device for computer-aided empty detection of the transport container 400 can be retrofitted cost-effectively.

[0230] According to various embodiments, the complex process of training target objects (recognizable objects 702) can be dispensed with. In other words, an object database can be dispensed with. Instead, a transport container database can be used.

[0231] According to various embodiments, a simple (e.g. subsequent) mounting of an image capture sensor 202s to an existing peripheral, e.g. brackets of the elements of the cash register system (illustratively the cash register furniture), can be provided or made available.

[0232] According to various embodiments, the image capture direction 208b can be directed towards the surface 208u. This reduces the intrusion into the privacy of individuals (e.g., customers), as the image capture sensor 202s is mounted at a non-critical viewing angle.

[0233] According to various embodiments, the acquisition of the additional image data and the image data can be performed simultaneously. Alternatively or in addition to the additional contour pattern 614, the reference pattern 704 can be read from a data storage device.

[0234] The image data used can be provided with a sufficiently large dimension to provide a clearly defined signal-to-noise ratio and / or a sufficient threshold value. Alternatively, the image data used can be provided with a sufficiently small dimension to minimize computational effort.

[0235] Fig.19 A method from 1900, according to various embodiments, is illustrated in a schematic view.

[0236] According to various embodiments, the contour pattern 614 of the transport container 400 and / or (e.g., feature-intensive areas) of the image background 602h (e.g., the ground structure of the substrate 602h, such as structure and / or texture) can first be trained, e.g., features, contours, and / or patterns (e.g., by means of pattern recognition) of the contour pattern 614, and optionally depth information and / or color information of the transport container 400 and / or the image background 602h. This data can be stored in a database (reference database).

[0237] Image information of an empty transport container 400 can be acquired using the image acquisition system 202 (e.g., against the background 208u). Based on this image information, a reference pattern 704, 614 can be determined. The reference pattern 704, 614 can be stored using a data storage device 204.

[0238] Fig.20 A method 2000 is illustrated according to various embodiments in a schematic view.

[0239] According to various embodiments, the transport container 400 and / or its spatial position within the image acquisition area 208 (position determination) can be detected based on the image data 602. Alternatively or additionally, the transport container 400 and / or its spatial position within the image acquisition area 208 can be detected using an acoustic sensor (e.g., using sound reflection), an optoelectronic sensor (e.g., a light barrier), and / or a radio frequency identification (RFID) sensor (e.g., identification using electromagnetic waves). For example, the transport container 400 can have a radio frequency identification (RFID) tag (which can also be referred to as an RFID transponder) and / or optical reflectors (optical markers).

[0240] According to various embodiments, reflections, e.g., their distinctive shape, can be captured (in the form of reflection data), e.g., in 2D image space. The reflections can be excited, for example, by emitting light with a defined wavelength range (e.g., IR, e.g., emitted by an IR projector) and / or at a defined angle into the image acquisition area 208. The reflections can be caused by a metallic lattice structure of the transport container and / or the optical reflectors. Based on a comparison of the captured reflection data (e.g., by means of feature comparison) with the reference data, it can be determined whether a transport container is located in the image acquisition area 208. This can lead to a significantly optimized response time.

[0241] Optionally, an analysis area 602b (ROI) can be determined based on the spatial location. Analysis area 602b can be used to describe the area of ​​the image data that will be processed further, e.g., by processor 206.

[0242] According to various embodiments, the processor 206 can be used to determine the state of the contour pattern 604, e.g. whether the contours of the grid structure of the transport container 400 are complete or are interrupted or obscured by objects.

[0243] Optionally, based on the image data 602 or the analysis area 602b, a color and / or texture deviation can be determined which is caused by an object 702 in the transport container 400 (illustratively by existing objects in the basket or in the lower area of ​​the transport medium), compared to the reference pattern 704, which represents, for example, the floor structure 208u located under the transport container 400 and / or other parts of the transport container 400.

[0244] Optionally, deviation data can be combined with data from the acoustic sensor, the optoelectronic sensor, and / or the radio tag sensor. This can reduce the tendency for false alarms (in other words, optimize decision reliability).

[0245] Optionally, depth information can be determined based on image data 602 or analysis area 602b. For example, analysis area 602b can be adjusted based on the depth information, e.g., so that it overlaps with the second transport area 322b.

[0246] According to various embodiments, a signal can be output, e.g. to the operating personnel and / or by means of a screen 802, which represents a result of the empty detection.

[0247] According to various embodiments, a combinatorial processing for the empty detection of the transport container 400 can be carried out, which has the following features: It can be detected whether a determined contour pattern, representing a transport container 400 to be analyzed, exhibits a disturbance 702u caused by an object 702 in the transport container 400. In other words, a deviation from a contour pattern known to the system (which, for example, represents an exposed grid structure) can be detected. Optionally, a color difference analysis can be performed against the reference image data, e.g., against image background color and feature information, e.g., limited to the analysis area 602b. Optionally, it can be detected whether an object 702 is located in an (e.g., scalable and / or adjustable) analysis area 602b that causes an area of ​​the transport container 400 to be at least partially obscured; it can be detected whether the transport container 400 is in the empty state; a signal (message signal) can be output (e.g.,A signal is generated if a state other than the empty state (e.g., if an item 702 is placed in transport container 400) and / or no transport container 400 is detected; optionally, time-varying objects (e.g., with changing surfaces, such as advertising material holders) can be trained and / or hidden to ensure reliable empty detection of transport container 400 (e.g., even though its contour pattern differs from the reference pattern). The training process can include determining a contour pattern that represents the variable object and updating a reference pattern and / or the database to account for the variable object.

[0248] Fig.21 A process 2100 is illustrated according to various embodiments in a schematic flowchart.

[0249] Optionally, procedure 2100 in 2101 (which can also be referred to as initial phase 2101) can include: Initiating an image acquisition system.

[0250] Furthermore, procedure 2100 in 2103 (which may also be referred to as transport container search phase 2103) may include: searching for a transport container in an image acquisition area of ​​the image acquisition system.

[0251] Furthermore, the method 2100 in 2105 (which can also be referred to as empty detection phase 2105, detection phase 2105, or article detection phase 2105) can include: Detecting whether the transport container is empty. The empty detection phase 2105 can, according to various embodiments, include: Determining a deviation value; and Determining whether the deviation value meets a predetermined criterion.

[0252] Furthermore, procedure 2100 in 2107 (which can also be referred to as evaluation phase 2107) can include: Determining whether an empty detection was successful.

[0253] Furthermore, the procedure 2100 in 2109 (which can also be referred to as signal output phase 2109) can include: outputting a signal that represents a result of the empty detection.

[0254] Fig.22 illustrates an initial phase 2101 according to various embodiments in a schematic flowchart.

[0255] The initial phase 2101 in 2201 (which can also be referred to as program start 2101) can optionally include: starting a program that is configured to carry out a procedure according to various embodiments. For example, a processor can be put into an operational state. The processor can be configured to carry out the procedure according to various embodiments, e.g., by executing the program.

[0256] Furthermore, the initial phase 2101 in 2203 (which can also be referred to as camera initialization 2203) can include: Initiating an image acquisition system, e.g., an image acquisition sensor or multiple image acquisition sensors of the image acquisition system. For example, the image acquisition system (e.g., the single image acquisition sensor or multiple image acquisition sensors) can be put into an operational state. Alternatively or additionally, at least one other (e.g., non-optical) sensor can be initiated, e.g., at least one acoustic sensor, at least one optoelectronic sensor, and / or at least one radio tag sensor.

[0257] Furthermore, the initial phase 2101 in 2205 can include: providing (e.g., loading and / or creating) one or more reference patterns. The reference pattern can, for example, be loaded (i.e., called) from a reference database.

[0258] Furthermore, the initial phase 2101 in 2207 can include: providing (e.g., loading and / or creating) reference data. The reference data can include or be derived from one or more reference patterns, reference topography information, reference depth information, reference color information, and / or reference depth information. The reference data can be generated for the sensors.

[0259] Fig.23 illustrates a transport container search phase 2103 according to different embodiments in a schematic flowchart.

[0260] The transport container search phase 2103 can include in 2301: Providing (e.g., loading and / or creating) image data (which can also be referred to as a camera frame). The image data can represent a single image. The image data can be captured for loading.

[0261] Furthermore, the transport container search phase 2103 in 2303 (which can also be referred to as empty detection 2303 of the image acquisition area) can include: Detecting whether a transport container is located in the image acquisition area. To detect the transport container in the image acquisition area, 3D data (e.g., topographic data and / or depth information) can be acquired within the image acquisition area. For example, this can be achieved by emitting (e.g., projecting) an optical pattern into the image acquisition area and acquiring image data of the image acquisition area that represents the optical pattern. The acquisition of the 3D data (e.g., topographic data and / or depth information) can be based on the optical pattern (which can then also be referred to as IR search phase 2303). The optical pattern can be visibly influenced by a transport container in the image acquisition area.To capture 3D data, a change in the optical pattern caused by the transport container when it is positioned within the image acquisition area can be detected. At least one image acquisition sensor of the image acquisition system can be configured to capture the spectral range of the optical pattern.

[0262] Furthermore, the transport container search phase 2103 in 2305 can include: Deciding whether a transport container has been detected in the image acquisition area. If no transport container has been detected in the image acquisition area (2305b, decision = no 2305b), the process can continue to 2301 by providing further image data. If a transport container has been detected in the image acquisition area (2305a, decision = yes 2305a), the process can continue to 2307. The decision as to whether a transport container has been detected in the image acquisition area can, for example, be based on the 3D data. For example, the 3D data can represent whether and / or how strongly the optical pattern is altered when the transport container is positioned in the image acquisition area.The transport container search phase 2103 can be visualized as a loop in which image data is captured until it is detected that a transport container is being brought into the image capture area and / or is already positioned within it.

[0263] To determine whether a transport container has been detected within the image acquisition area, a 3D deviation metric can be calculated. This metric represents the deviation of the 3D data from reference 3D data, where the reference 3D data represents an empty image acquisition area. Visually, the 3D data can be compared with the reference 3D data. If the deviation meets a predefined criterion, it can be determined that a transport container has been detected within the image acquisition area (decision = yes 2305a). If the deviation does not meet the predefined criterion, it can be determined that no transport container has been detected within the image acquisition area (decision = no 2305b).

[0264] The transport container search phase 2103 can optionally include in 2307: Creating a large number of image data points using additional image data (which can also be referred to as another camera frame). The additional image data points can represent at least one further individual image. The large number of image data points can clearly represent an image sequence (e.g., in a temporal and / or spatial order). To create the large number of image data points, the additional image data points can be acquired after it has been detected that a transport container is being brought into the image acquisition area and / or is positioned within the image acquisition area. This allows for the clear acquisition of a large number of image data points, which, for example, represent the transport container in different positions.

[0265] If a large number of image data sets are available, the transport container search phase 2103 in 2309 can include: Selecting the image data sets that represent the transport container and / or that have a predefined contour density. For example, this can be done by identifying (visually selecting) the image data sets from the large number of image data sets that represent the transport container and / or that have a predefined contour density. To select the image data sets, already processed image data sets from the large number of image data sets can be discarded (i.e., not considered).

[0266] Furthermore, the transport container search phase 2103 in 2311 can include: detection of the transport container using a contour pattern. This can be achieved by determining a contour pattern representing the transport container using the image data. In 2311, for example, the transport container type and / or its location (orientation and / or position within the image acquisition area) can be detected.

[0267] Furthermore, the transport container search phase 2103 in 2313 may show: Decide whether the transport container has been detected.

[0268] Fig.24 A process 2400 is illustrated according to various embodiments in a schematic flowchart.

[0269] The procedure 2400 can include the transport container search phase 2103 and the empty detection phase 2105.

[0270] Furthermore, procedure 2400 in 2401 can show: whether further image data is available.

[0271] If it was decided 2313 that a transport container was detected in the image acquisition area 2103a (decision=yes 2103a), the empty detection phase 2105 can be continued.

[0272] If it was decided 2313 that no transport container was detected 2103b (decision=no 2103b), the decision 2401 can be made as to whether further image data is available (which, for example, has not yet been analyzed).

[0273] If it was decided in step 2401 that further image data is available (decision = yes 2401a), then in step 2309, other image data can be selected from the multitude of available image data. If it was decided in step 2401 that no further image data is available (decision = no 2401b), the transport container search phase can continue in step 2103; for example, in step 2301, other image data can be provided (e.g., captured and / or loaded).

[0274] Fig.25 A process 2500 is illustrated according to various embodiments in a schematic flowchart.

[0275] The procedure 2500 can include the transport container search phase 2503 and the empty detection phase 2505.

[0276] Optionally, the procedure can include step 2500 in step 2501: Determining an analysis area. A portion of the image data can be selected for use in the empty detection phase 2105 (detecting the empty transport container). This reduces the effort required.

[0277] Fig.26 Illustrates a void detection phase 2105 according to various embodiments in a schematic flowchart.

[0278] The empty detection phase 2105 in 2601 can include: Determining a deviation magnitude based on a contour pattern. This can be done by determining the contour pattern representing the transport container using the image data (or part of the image data); and by determining the deviation magnitude representing a deviation of the contour pattern from at least one reference pattern, where the at least one reference pattern represents an empty transport container.

[0279] Optionally, the empty detection phase 2105 in 2603 can include: Determining the deviation magnitude based on color information. This can involve determining the color information representing the transport container using the image data (or part of the image data); and determining the deviation magnitude representing a deviation of the color information from reference color information, where the reference color information represents an empty transport container.

[0280] Optionally, the empty detection phase 2105 in 2605 can include: Determining the deviation magnitude based on depth information. This can be done by determining the depth information representing the transport container using the image data (or part of the image data); and by determining the deviation magnitude representing a deviation of the depth information from reference depth information, where the reference depth information represents an empty transport container.

[0281] Optionally, the empty container detection phase 2105 in 2607 can include: Determining the deviation magnitude based on topographic information. This can be done by determining the topographic information representing the transport container using the image data (or a portion thereof); and by determining the deviation magnitude representing a deviation of the topographic information from reference topographic information, where the reference topographic information represents an empty transport container. The topographic information can be determined, for example, using the optical pattern.

[0282] The in Fig.26 The illustrated sequence can be performed sequentially (this can also be referred to as a single-threaded process). For example, after each determination of the deviation quantity 2601, 2603, 2605, 2607, it can be determined whether a clear decision (e.g., whether the transport container is empty or not) can be reliably made (compare error range detection). The reliability of the deviation quantity can also be determined.

[0283] For example, the deviation magnitude can be refined (its reliability increased) by incorporating additional information (color information, depth information, and / or topography information) to determine it. For instance, further determination of deviation magnitudes 2603, 2605, and 2607 can be performed if it has been determined that a clear decision cannot be reliably made. Visually, the deviation magnitude data can be successively superimposed until sufficient decision reliability is achieved.

[0284] Fig.27 illustrates a void detection phase 2105 according to various embodiments in a schematic flowchart analogous to Fig.26 .

[0285] The in Fig.27 The illustrated sequence can occur at least partially in parallel (this can also be referred to as multithreading). For example, at least two (e.g., three or four) pieces of information for determining the deviation values ​​2601, 2603, 2605, 2607 can be processed simultaneously.

[0286] Fig.28 An evaluation phase 2107 is illustrated according to various embodiments in a schematic flowchart.

[0287] Evaluation phase 2107 can include in 2801: Deciding whether an object (e.g., an item) has been detected in the transport container. If an object has been detected in the transport container 2801a (decision = yes 2801a), a corresponding signal 2109 can be output (e.g., a "not empty" signal). If no object has been detected in the transport container 2801b (decision = no 2801b), a corresponding signal 2109 can optionally be output (e.g., an "empty" signal).

[0288] If no object was detected in the transport container 2801b (decision=no 2801b), optionally in 2803 of the evaluation phase 2107, it can be determined whether the decision was not possible using the image data, for example because the deviation size was not reliable enough and / or because the image data was not sufficiently representative.

[0289] If it was determined that a decision based on the image data used was not possible (2803a, decision = yes), a corresponding signal can be output (2109, e.g., an error signal). If a decision was possible using the image data (2803b, decision = no), a final decision can optionally be made in the evaluation phase (2107, 2805) as to whether no object (e.g., an article) was detected in the transport container.

[0290] If it has been definitively decided that no object was detected in the transport container (2805a) (decision = yes 2805a), the transport container search phase (2103) can be restarted. For example, it can continue with step 2301, for instance, by providing new image data of the transport container. Visually, the loop 2301, 2303, 2305 can then be executed until the next transport container is detected.

[0291] If it has not been definitively decided that no object was detected in the transport container (2805b) (decision = no, 2805b), a decision can be made in 2401 as to whether further image data is available (which, for example, has not yet been analyzed). The decision that no object is located in the transport container can be intuitively verified using further image data from the multitude of available image data.

[0292] According to various embodiments, the final decision 2805 can be made when a predetermined number of decisions based on one and the same set of image data agree. This reduces the risk of error detection.

[0293] If further image data is available (2401a, decision=yes 2401a), other image data can be selected from the multitude of available images (2309). If no further image data is available (2401a, decision=no 2401b), the transport container search phase (2103) can be restarted, e.g., it can continue from step 2301. Then, for example, new image data can be captured from the same transport container.

[0294] Fig.29 illustrates a signal output phase 2109 according to different embodiments in a schematic flowchart.

[0295] Signal output phase 2109 can include in 2901: Issuing an input prompt and / or transferring a POS system process to a wait state (which can also be referred to as program stop and / or waiting for user input). For example, the program can be stopped. Signal output phase 2109 can also include in 2903: Determining whether an input has been received in response to the input prompt.

[0296] If an input is received at the prompt 2903a (decision=yes 2903a), the transport container search phase 2103 can be restarted, e.g., it can continue with 2301. If no input is received at the prompt 2903b (decision=no 2903b), the prompt and / or the wait state can be maintained.

[0297] Fig.30 illustrates a method 3000 according to various embodiments in a schematic flowchart, analogous to the Fig.21 bis Fig.29 .

[0298] Fig.31 A process 3100 is illustrated according to various embodiments in a schematic flowchart.

[0299] The procedure 3100 can include in 3101: Processing the image data (can also be referred to as process phase 3101).

[0300] Procedure 3100 may optionally include in 3103: Forming and / or adapting a reference pattern (which may also be referred to as reference pattern learning phase 3103), e.g. if no (e.g. suitable) reference pattern is available and / or there are too few reference patterns available.

[0301] Procedure 3100 may optionally include in 3105: Forming and / or adapting a criterion (may also be referred to as criterion learning phase 3105).

[0302] Procedure 3100 may further include in 3107: Determining a deviation quantity (may also be referred to as detection phase 3107).

[0303] Procedure 3100 may also include in 3109: Output of a result (which may also be referred to as output phase 3109), e.g. output of the deviation magnitude.

[0304] Fig.32 illustrates a process phase 3101 according to different embodiments in a schematic flowchart.

[0305] Process phase 3101 may include in 3201: Acquisition of image data.

[0306] Process phase 3101 in 3203 may further include: Determining a contour pattern using the image data. For example, the contour pattern may be generated using a Gaussian difference algorithm (also known as difference of Gaussian, DoG). The Gaussian difference algorithm may involve generating a blurred version of the image data and subtracting this blurred version from the image data. Generating the blurred version may be done using a Gaussian kernel, for example, by convolving the image data using the Gaussian kernel. Alternatively or additionally, generating the blurred version may be done using a blur filter.

[0307] Process phase 3101 in 3205 may also include: Requesting a reference sample.

[0308] Process phase 3101 in 3207 may further include: Deciding whether a reference pattern exists. If a reference pattern exists (3207a, decision = yes), the process can proceed to 3209. If no reference pattern exists (3207b, decision = yes), the process can proceed to 3103, for example, by creating a reference pattern.

[0309] Process phase 3101 in 3209 may optionally include: Creating an analysis area for each deviation of the contour pattern from the reference pattern. For this purpose, one or more deviations of the contour pattern from the reference pattern can be determined.

[0310] Fig.33 Illustrates a reference pattern learning phase 3103 according to various embodiments in a schematic flowchart.

[0311] The reference pattern learning phase 3103 can include in 3301: Providing (e.g., creating) grayscale image data. For this purpose, the image data can be converted to grayscale (if it is not already in grayscale).

[0312] The reference pattern learning phase 3103 can optionally include in 3303: Determining a mean variation of the gray image data. The mean variation can represent a mean variation of the gray values ​​in the gray image data. To determine the mean variation, a mean deviation and / or a standard deviation can be calculated.

[0313] The reference pattern learning phase 3103 may optionally include in 3305: forming a grayscale matrix using the grayscale image data.

[0314] The reference pattern learning phase 3103 can optionally include in 3307: Determining a fluctuation of the grayscale matrix, e.g., a mean fluctuation and / or a fluctuation of neighboring entries of the grayscale matrix. The fluctuation can represent a contrast, a homogeneity, and / or an energy of the grayscale matrix.

[0315] The reference pattern learning phase 3103 may further include in 3309: forming a reference pattern based on the gray image data, e.g. using the fluctuation (according to the gray value matrix and / or according to the gray image data).

[0316] The reference pattern learning phase 3103 can optionally include in 3311: saving the reference pattern and optionally the fluctuation, e.g., in a reference database. Alternatively or additionally, the reference pattern can be processed directly.

[0317] Alternatively or additionally, the reference pattern learning phase 3103 can be carried out if it has been definitively decided that no item is arranged in the transport container, and / or if an input represents that no item is arranged in the transport container.

[0318] Fig.34 Illustrates a criterion learning phase 3105 according to different embodiments in a schematic flowchart.

[0319] The criterion learning phase 3105 can be found in 3401: generating grayscale image data of the analysis area. For this purpose, the analysis area of ​​the image data can be converted into grayscale (if it is not already in grayscale).

[0320] The criterion learning phase 3105 in 3403 may further include: Determining a mean variation in the grayscale image data of the analysis area. The mean variation can represent a mean variation in the grayscale values ​​of the grayscale image data. To determine the mean variation, a mean deviation and / or a standard deviation can be calculated.

[0321] Alternatively or additionally, criterion learning phase 3105 in 3405 may include: creating a grayscale matrix using the grayscale image data of the analysis area. In that case, criterion learning phase 3105 in 3407 may include: determining a variation in the grayscale matrix of the analysis area, e.g., a mean variation and / or a variation of neighboring entries in the grayscale matrix. The variation may represent contrast, homogeneity, and / or energy of the grayscale matrix.

[0322] Fig.35 A process 3500 is illustrated according to various embodiments in a schematic flowchart.

[0323] Procedure 3500 can include process phase 3101 and output phase 3109. Optionally, procedure 3500 can include criterion learning phase 3105.

[0324] Method 3500 may further include in 3501: detecting whether there is a deviation of the contour pattern from the reference pattern, e.g. a further deviation if at least one deviation has already been detected.

[0325] Detecting whether a contour pattern deviates from the reference pattern can be performed after process phase 3101, for example, after creating an analysis area 3209 for each deviation of the contour pattern from the reference pattern. In this way, an analysis area can be assigned to each deviation iteratively.

[0326] The result, whether there is a deviation of the contour pattern from the reference pattern, can be output in 3109.

[0327] If a deviation of the contour pattern from the reference pattern is detected 3605a (decision=yes 3605a), the criterion learning phase 3105 can optionally be carried out, e.g. continuing with the creation 3401 of grey image data of the analysis area.

[0328] Fig.36 Illustrates a recognition phase 3107 according to various embodiments in a schematic flowchart.

[0329] Recognition phase 3107 can include in 3601: comparing the contour pattern with the reference pattern. For this purpose, the contour pattern can be compared with the reference database, e.g., to select the corresponding contour pattern. The deviation magnitude can be determined based on this comparison.

[0330] The detection phase 3107 can further exhibit in 3603: comparing the deviation magnitude with the criterion.

[0331] The detection phase 3607 in 3603 can further include: Determining whether an object in the transport container has been detected. If the deviation value meets the criterion, it can be decided that an object in the transport container has been detected. If the deviation value does not meet the criterion, it can be decided that no object in the transport container has been detected.

[0332] If an item is detected in the transport container (3605a, decision = yes 3605a), the process can proceed to step 3109, meaning the result that an item was detected in the transport container can be output (3109). If no item is detected in the transport container (3605b, decision = yes 3605b), the process can proceed to step 3501. Then, the next deviation and / or the next analysis area can be processed.

[0333] Once all deviations and / or analysis areas of the image data have been processed, the process can continue with step 2401.

[0334] Fig.37 illustrates a method 3700 according to various embodiments in a schematic flowchart, analogous to the Fig.31 bis Fig.36 .

[0335] The output result 3109 can be further processed, e.g. to output the signal.

Claims

1. Method (100) for the computer-aided recognition of a transport container (400) being empty, the method (100) comprising: • capturing (101) image data of a region of the transport container (400); • determining (103) a contour pattern by means of feature recognition, which contour pattern represents a lattice structure of the transport container (400), using the image data; • determining (105) a deviation variable, which represents as pattern deviation an area occupation deviation, a contrast deviation and / or a contour profile deviation of the contour pattern from at least one reference pattern, wherein the at least one reference pattern represents an empty transport container (400); • determining depth information on the basis of the image data; wherein the deviation variable furthermore represents a deviation of the depth information from reference depth information, wherein the reference depth information represents an empty transport container (400); • outputting (107) a signal if the deviation variable satisfies a predefined criterion.

2. Method (100) according to Claim 1, furthermore comprising: • capturing image data of an additional region of the transport container (400); • determining an additional contour pattern, which represents the additional region, on the basis of the image data; • wherein a reference pattern of the at least one reference pattern comprises the additional contour pattern.

3. Method (100) according to Claim 1 or 2, • wherein the transport container (400) is arranged in an image capture region for capturing (101) the image data, wherein the image capture region defines an image background of the transport container (400); and • wherein a reference pattern of the at least one reference pattern comprises a contour pattern representing the image background.

4. Method (100) according to any of Claims 1 to 3, wherein the predefined criterion represents an empty transport container (400) and the signal comprises a recognized-as-empty signal.

5. Method (100) according to any of Claims 1 to 4, wherein outputting the signal comprises outputting an input request, and wherein a checkout system process transitions into a standby state until an input is carried out in response to the input request.

6. Method (100) according to any of Claims 1 to 5, furthermore comprising: determining colour information on the basis of the image data; wherein the deviation variable furthermore represents a deviation of the colour information from reference colour information, wherein the reference colour information represents an empty transport container (400).

7. Method (100) according to any of Claims 1 to 6, furthermore comprising: determining topography information on the basis of the image data; wherein the deviation variable furthermore represents a deviation of the topography information from reference topography information, wherein the reference topography information represents an empty transport container (400) .

8. Method (100) according to any of Claims 1 to 7, wherein outputting the signal comprises outputting an input request, the method furthermore comprising: updating the at least one reference pattern on the basis of the contour pattern if an input in response to the input request represents an empty state of the transport container.

9. Method (100) according to any of Claims 1 to 8, wherein determining the depth information comprises capturing (101) image information of an additional region of the transport container (400) through the region.

10. Method (100) according to any of Claims 1 to 9, wherein the contour pattern represents at least one of the following: • a disturbance that is brought about by an object if the object is arranged in the transport container (400); and / or • an image background defined by an image capture region in which capturing (101) the image data is carried out.

11. Device for the computer-aided recognition of a transport container (400) being empty, the device comprising: an optical image capture system (202) for capturing image data; a data storage medium (204) for storing at least one reference pattern and / or in which at least one reference pattern is stored, wherein the reference pattern represents an empty transport container (400); a processor (206), configured to carry out the following method (100): • capturing (101) image data of a transport container (400) by means of the image capture system; • determining (103) a contour pattern by means of feature recognition, which contour pattern represents a lattice structure of the transport container (400), using the image data; • determining (105) a deviation variable, which represents as pattern deviation an area occupation deviation, a contrast deviation and / or a contour profile deviation of the contour pattern from at least one reference pattern, wherein the at least one reference pattern represents an empty transport container (400); • determining depth information on the basis of the image data; wherein the deviation variable furthermore represents a deviation of the depth information from reference depth information, wherein the reference depth information represents an empty transport container (400); and • outputting (107) a signal if the deviation variable satisfies a predefined criterion.