METHOD, CONTROL DEVICE AND REGISTRATION TERMINAL FOR COMPUTER-AID EMPTY DETECTION OF A TRANSPORT CONTAINER

Description

[0001] Various embodiments relate to a method for computer-aided empty detection of a transport container, a control device and a registration terminal.

[0002] In general, transport containers can be used to move items, such as goods in production or sales. It may be necessary to identify whether and, if so, what is inside the transport container, for example, when registering goods at a checkout (this can also be referred to as "bottom of basket" detection - BoB). This helps reduce costs that arise when unregistered goods pass through the checkout undetected (this can also be referred to as loss prevention).

[0003] Compared to non-computerized methods, such as the use of ceiling mirrors, ID numbers on the chassis of the shopping cart, or video playback on the cashier's monitor, computerized methods require less attention from the cashier and therefore place less strain on their performance, are more reliable, and less prone to errors.

[0004] Traditionally, computer-aided pattern recognition methods are used to identify items arranged in the transport container. EP 3 211 611 A1 serves as an example. For instance, to detect non-empty shopping carts, items on the cart's shelf are identified via object recognition or image comparison with empty shopping carts. To identify items, distinctive patterns are recognized and compared with a database containing the patterns of known items.

[0005] These computer-aided methods have in common that, due to the image processing or pattern recognition used, they require a great deal of computing power, tend to be prone to errors, and must be adapted to the product range or the type of vehicle, i.e., they increase personnel costs elsewhere.

[0006] The resulting problem is solved by the subject matter of the attached claims.

[0007] In contrast, according to various embodiments, a method for computer-aided empty detection of a transport container, a control device or a registration terminal with this for carrying out the method are provided, which require less computing power, reduce the susceptibility to errors, are more robust against disturbances and require fewer adjustments.

[0008] A method for computer-aided empty container detection is presented, illustrating various embodiments. This method reduces the complexity of the image data to be considered, for example, images of the container (possibly filled with one or more objects) captured by a camera. This results in less computing power being required and the use of leaner algorithms.

[0009] Visually, segments of the image data are filtered out (and, for example, discarded) before further processing, reducing the complexity of the image data to be considered. Filtering can involve discarding those segments of the image data that, for instance, do not belong to the transport container itself but to its background (also referred to as the image background), thus excluding them from the empty container detection. Determining these excluded segments is achieved using depth information (also known as depth segmentation). For example, the image data is sorted pixel by pixel according to its depth values. This ensures that those segments of the image data that, for instance, depict spatial areas below the shopping cart are filtered out.This mechanism, for example, relies solely on depth information without considering the image content and is therefore dependent only on spatial conditions. This achieves a high degree of robustness and reliability in depth segmentation.

[0010] They show Figure 1 shows a method according to various embodiments in a schematic flowchart; Figure 2 shows a registration terminal according to various embodiments in a schematic assembly diagram; Figure 3 shows a registration terminal according to various embodiments in a schematic communication diagram; Figure 4 shows a registration terminal according to various embodiments in a schematic side view; Figures 5, 6 and 7 each show a method according to various embodiments in a schematic flowchart; Figures 8 and 10 each show a classifier according to various embodiments in a schematic assembly diagram; and Figure 9 shows a registration terminal in the method according to various embodiments in a schematic side view.

[0011] The following detailed description refers to the accompanying drawings, which form part thereof and illustrate specific embodiments in which the invention can be implemented. In this context, directional terminology such as "top," "bottom," "front," "back," "anterior," "rear," etc., is used with reference to the orientation of the described figure(s). Since components of embodiments can be positioned in a number of different orientations, the directional terminology serves only for illustration and is in no way limiting. It is understood that other embodiments may be used and structural or logical modifications may be made without deviating from the scope of protection of the present invention.It is understood that the features of the various exemplary embodiments described herein can be combined with one another, unless specifically stated otherwise. The following detailed description is therefore not to be interpreted in a limiting sense, and the scope of protection of the present invention is defined by the appended claims.

[0012] Within the scope of this description, the terms "connected," "connected," and "coupled" are used to describe both direct and indirect connections (e.g., resistive and / or electrically conductive, such as an electrically conductive connection), direct or indirect connections, and direct or indirect couplings. In the figures, identical or similar elements are designated with identical reference numerals where appropriate.

[0013] Within the scope of this description, the terms "connected," "attached," and "coupled" are used to describe both direct and indirect connections (e.g., resistive and / or electrically conductive, such as a communication-capable connection), direct or indirect connections, and direct or indirect couplings. In the figures, identical or similar elements are designated with identical reference symbols where appropriate.

[0014] The term "control device" can be understood as any type of logic-implementing entity that may, for example, have circuitry and / or a processor capable of executing software stored in a memory medium, firmware, or a combination thereof, and issuing instructions based on that software. The control device can be configured, for example, using code segments (e.g., software) to control the operation of a system (e.g., its operating point), such as a machine or plant, or its components.

[0015] The term "processor" can be understood as any type of entity that allows the processing of data or signals. The data or signals can be processed, for example, according to at least one (i.e., one or more than one) specific function performed by the processor. A processor can be an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA), an integrated circuit, or any combination thereof. Any other type of implementation of the respective functions, which are described in more detail below, can also be understood as a processor or logic circuit.It is understood that one or more of the process steps described in detail herein can be executed (e.g., implemented) by a processor through one or more specific functions performed by the processor. The processor can therefore be configured to perform one of the procedures described herein or its components for information processing.

[0016] Depending on the specific design, a data storage device (more generally referred to as a storage medium) can be a non-volatile data storage device. The data storage device can, for example, consist of or be composed of a hard drive and / or at least one semiconductor memory (such as read-only memory, random access memory, and / or flash memory). The read-only memory can, for example, be a erasable programmable read-only memory (which may also be referred to as EPROM). The random access memory can be a non-volatile random access memory (which may also be referred to as NVRAM – "non-volatile random access memory").For example, the data store can contain one or more of the following: a database (which can also be called a reference database), a processing algorithm, a criterion, or code segments that implement one or more processing algorithms (also referred to simply as an algorithm). The database can contain one or more records, each of which associates a product identifier with payment information.

[0017] This text refers, among other things, to an artificial neural network (kNN). The described concepts can be applied analogously if, alternatively or additionally to the kNN, another trainable algorithm is used, such as a support vector machine or a long short-term memory (LSTM). The kNN can have a large number of nodes (figuratively, artificial neurons) and a connection network (the mapping of connections to nodes). In a kNN, the processes for information acquisition and processing are modeled analogously to biological neurons. This is achieved through a number of layers of hidden neurons, dependent on the specific circumstances, and the activation functions that transmit the signals.

[0018] The topology of the algorithm describes the structure of the interconnection network, i.e., how many nodes are distributed across how many layers, and how these layers are interconnected. For example, multiple nodes can be arranged in successive layers, the first layer of which forms the algorithm's input and the last layer its output. The last layer whose output is visible outside the interconnection network is called the output layer. The preceding layers are accordingly referred to as hidden layers. Using a graph, the artificial neurons can be represented as nodes and their connections as edges. In this case, the algorithm is exemplified as a directed graph with typed nodes.

[0019] After the algorithm is constructed, it is first trained (also called the training phase), during which it is adapted to the desired behavior (figuratively speaking, it "learns"). For this purpose, data (the training data) is fed into the algorithm, based on which it learns to replicate the desired behavior. An accurate output from the algorithm reinforces what it has learned (i.e., a specific signal path through the network), whereas an inaccurate output weakens the signal path. In this way, the paths through the network that best describe the desired behavior gradually emerge.

[0020] During the training phase, one or more of the following operations can occur: modifying the connection network (e.g., creating or deleting connections); changing the weight of nodes; modifying the properties (e.g., thresholds) of the nodes; modifying the number of nodes; modifying the activation, propagation, and / or output function.

[0021] During algorithm training (e.g., kNN), the training data is selected according to the desired input parameters. In one example, the incoming training data consists of image data of the detection area, for which the respective target variable (e.g., class or objects to be detected) is known. The training data can be synchronized and / or related to each other, for example, via timestamps or their origin. It should be noted that both the parameters in the algorithm's input vector and the parameters in its output vector are highly application-dependent and must be selected accordingly.

[0022] The following refers to image data and its processing. The image data can be a digital representation of reality (e.g., the detection area) at the point in time when the image data is acquired. Reality can be represented, for example, by a lens that projects light onto the surface of an image acquisition sensor (e.g., a Bayer sensor). Image acquisition can involve reading the image acquisition sensor while light is projected onto its surface. The resulting image data can initially be in the so-called raw data format (also known as RAW), which contains the pixel-by-pixel readings from the image acquisition sensor and / or can be processed as such. Optionally, the image data can be converted into another image format during processing, e.g.,The image can be converted into a raster graphic (different from RAW as a raster graphic) or a vector graphic, so that further processing takes place in this image format, or it can be converted between these formats as needed. The conversion process can optionally include interpolating the measurements from the image acquisition sensor (e.g., using demosaicing), for example, to obtain complete multi-color information for each pixel or to reduce storage space or processing power. The image data can optionally be compressed (e.g., to reduce storage space or processing power) or uncompressed (e.g., to avoid distortion). The respective image format can also define the color space according to which the color information is specified.

[0023] The simplest case is a binary color space, where a black-and-white value is stored for each pixel. In a slightly more complex color space (also known as a grayscale color space), intermediate shades between black and white are stored (also known as grayscale values). The color space can also be defined by several primary colors (e.g., two or more), such as red, green, and blue. If the measured values ​​are to contain multicolored information, a wavelength-sensitive image acquisition sensor can be used. Its measured values ​​can be encoded according to a color space. The color information, or the underlying color space, can therefore be multicolored (also known as polychrome) or monochromatic (also known as single-color).The monochrome color information can, for example, consist only of grayscale values ​​(then also referred to as grayscale information) or black and white values ​​(then also referred to as black and white information), which represent the intensity of the detected radiation at the wavelength or wavelength range at which the monochrome sensor is sensitive. For visual display of the image data on a screen, it is converted into the image format specified by the graphics card's image memory. For ease of understanding, the image data described herein is presented as such a visual display. Generally, the image data, e.g., stored on a storage medium, can exist as a file (also referred to as a digital image or image file) in the respective image format.

[0024] Image data can also include depth information (also known as 3D information), either as an alternative or in addition to color information. A 3D camera can be used to capture this depth information, as will be described in more detail later. The 3D camera's measurements can contain information about the topography of the depicted reality (also known as depth information), for example, pixel by pixel. This depth information can specify the distance of each pixel of the camera from a location in space that is mapped to that pixel.

[0025] Depth information can vividly add a third spatial dimension (referred to here as depth) to the two spatial dimensions represented by the image data. For example, an object can be represented by the image data as a projection onto the two-dimensional surface of the image sensor (also referred to as 2D image data). The depth information then provides this third spatial dimension. For example, the depth information can assign values ​​(also referred to as depth values) to the image data segment by segment (i.e., for each segment of the image data), indicating its depth. Alternatively, the depth information can assign depth values ​​to the image data pixel by pixel (i.e., for each pixel of the image data), indicating its depth.

[0026] The image data is segmented using depth information, for example, by dividing the image data into segments (e.g., pixels) of two types, depending on whether the depth values ​​assigned to the segments meet a criterion. For example, the depth values ​​of segments of the first type might meet the criterion, while the depth values ​​of segments of the second type might not. Optionally, more than two types can be used, for example, if more than one criterion is applied. The boundaries between the image data segments are defined by those depth values ​​that most closely match the criterion.

[0027] According to various embodiments, depth information is acquired using three-dimensional (3D) image acquisition. This depth information can be used, for example, to determine whether an object is located above a reference plane. Thus, it is possible to distinguish the spatial distance of an object from the reference plane. For instance, depth information can be acquired across the entire acquisition area, allowing for a clear, top-down view of the spatial position of one or more objects relative to the reference plane.

[0028] According to various embodiments, the image acquisition device can provide image data of the detection area from multiple optical perspectives (e.g., provided by means of multiple lenses), which represent depth information of the detection area (e.g., stereoscopically). To determine the depth information, the image data acquired from different perspectives (e.g., by means of multiple lenses) can be superimposed, e.g., taking into account the relative spatial position (position and / or orientation) of the lenses to one another. A camera can have an (optical) image acquisition sensor and at least one lens (also referred to as a lens array) associated with the image acquisition sensor. The lens array of a plenoptic camera can also include a grid of multiple microlenses. For example, the image acquisition device (e.g.,RealSense F200, INTEL R200, Intel RealSense D415, Intel RealSense D435 and / or Intel SR300) have an RGB image capture sensor and / or a 3D image capture sensor.

[0029] An image acquisition sensor (also called an image sensor or optical sensor) can have one or more photoelectrically active areas (which can also be called pixels) that generate and / or modify an electrical signal in response to electromagnetic radiation (e.g., light, such as visible light). The image acquisition sensor can, for example, be a CCD sensor (charge-coupled device sensor) and / or an active pixel sensor (which can also be called a CMOS sensor), or be composed of these. Optionally, an image acquisition sensor can be wavelength-sensitive (e.g., for capturing color information), for example, by means of several color filters (e.g., in a grid configuration), and thus distinguish between different wavelengths.

[0030] Depth information can be quantified, for example, by specifying the depth as a value (also called a depth value), coded, or provided directly using image data, for example, by mapping image data captured simultaneously from different perspectives (e.g., separately or superimposed). The multiple simultaneously captured perspectives can, for instance, be superimposed to quantify the depth information. Each depth value can then correspond, for example, to a deviation between the multiple simultaneously captured perspectives.

[0031] By using 3D-based segmentation of the image data and classification with a kNN (k-nearest neighbors) algorithm, empty field detection can be simplified. This allows for the detection of fraud and the analysis of the entire process for deviations from normal behavior. The entire registration process (also known as the checkout process) can then be optionally assessed for suspiciousness, and / or an employee can be notified.

[0032] Due to the preprocessing of image data through depth-based segmentation and targeted optimization of the kNN architecture for the given problem, very high recognition rates can be achieved compared to conventional mechanisms. Alternatively or additionally, this preprocessing of image data through depth-based segmentation (also known as depth segmentation) and targeted optimization of the kNN architecture for the given problem can enable more cost-effective hardware compared to conventional mechanisms. In other words, less computing power is required to achieve high recognition rates.

[0033] Depending on the specific design, a registration terminal can be configured to register the products a customer wishes to purchase, for example, by scanning the products with a scanner (e.g., a barcode scanner). Furthermore, the registration terminal can include a (e.g., digital) point-of-sale (POS) system (e.g., a self-checkout or a cashier workstation) configured to process payments. The payment process can, for example, involve the customer paying for the purchased products. The POS system can include at least one of the following: a screen (e.g., a touchscreen), a printer (e.g., for printing an invoice and / or a label), a (e.g., programmable) POS keyboard (which can also be integrated into the touchscreen), and a payment terminal for accepting payment (e.g., cash or a debit card).The payment terminal can be, for example, an electronic payment terminal (also known as an EC terminal or electronic payment terminal, "EC" - electronic cash, e.g., for reading a debit card and / or a credit card). For example, the registration terminal can have a (e.g., digital) point-of-sale system configured to perform one or more POS processes, such as a registration session. A registration session can, for example, include a calculation process, an inventory process, and / or a registration process.

[0034] In a self-service checkout terminal, the cash register system and the scanner can be located on the same side (e.g., of a column) of the terminal, allowing them to be operated from a single position. In a non-self-service checkout terminal (also known as a cashier-operated checkout terminal), the cash register system and the scanner can be operated from different sides.

[0035] The following refers to products (e.g., goods, which can also be called articles) as objects. The description can also apply analogously to other objects that can be transported in a transport container.

[0036] According to various embodiments, it is possible to detect remaining objects inside the transport container, e.g., on a lower shelf of a shopping cart. In contrast to conventional pattern recognition, it is easier to determine whether or not an object (e.g., goods, which can also be referred to as an item) is located in the transport container. This detection can be performed optically by analyzing image data. The complexity of the image data can be reduced, according to various embodiments, by filtering out the image components that only show the surroundings through comparison with depth information. This reduces the complexity of the data to be analyzed and minimizes the occurrence of interference (the amount of data, for example, can remain the same).

[0037] Fig.1 A process 100 is illustrated according to various embodiments in a schematic flowchart.

[0038] Method 100 comprises, in 101, identifying one or more than one segment of image data of the transport container using depth information associated with the image data (also referred to as depth segmentation 101); in 103, assigning the image data to one of several classes (also referred to as classification 103), of which a first class represents that the transport container is empty, and a second class represents that the transport container is not empty; and in 105, outputting a signal which represents that the filtered image data are assigned to the first class and / or the second class.

[0039] Each segment of the image data (also called a data segment) contains one or more pixels.

[0040] Deep segmentation can, for example, involve identifying a large number of data segments (which are separated from each other), e.g., at least 5 data segments, e.g., at least 10 data segments, e.g., at least 50 data segments.

[0041] One or more identified data segments can be disregarded during classification (also referred to here as filtering out), for example by discarding, masking, modifying, etc. In other words, filtering out can involve deep segmentation and selective classification, which means that the segment identified through deep segmentation is not considered during classification.

[0042] Examples that implement filtering show that the data segment(s) identified by deep segmentation: is removed (i.e., discarded) from the image information before it is fed to classification, is masked (e.g., hidden before classification) before it is fed to classification, is replaced (e.g., with pixels of uniform color information) before it is fed to classification, is marked as not to be considered before it is fed to classification.

[0043] Of course, another selective mechanism can also be used that prevents the information of the segment identified through deep segmentation from being considered during classification. The following refers to specific implementations of this filtering (e.g., replacement and masking). The above can be applied analogously to other selective mechanisms that prevent the information of the segment identified through deep segmentation from being considered during classification.

[0044] Depth segmentation 101 can effectively ensure that only the portion of the image data representing the interior of the transport container is subjected to classification. In other words, the monitored image area can be limited to a region of interest (ROI) within which the interior of the transport container is located.

[0045] Classification 103 can be performed using an algorithm that is robust against changes in the type of transport container (also referred to as container type). However, it may be necessary to train fundamentally different container types, e.g., those differing in material (e.g., metal or plastic), separately, as will be described in more detail later.

[0046] Procedure 100 can optionally be executed in response to the detection of an event representing the presence of a transport container within the spatial area (also referred to as the detection area) from which the image data is being acquired. The event can be detected, for example, by means of motion detection. This motion detection can be based on the image data and / or a motion sensor. The event can also mark the start of a recording session, as will be described in more detail later.

[0047] Alternatively, the process can be repeated cyclically 100 times (e.g. permanently), e.g. regardless of whether a transport container is present.

[0048] In a simple implementation, depth information can be determined using stereoscopic infrared image data. In other words, depth information can be determined by detecting infrared radiation. This infrared radiation can have a wavelength in the range of approximately 700 nm (nanometers) to approximately 1400 nm (e.g., 1000 nm), for example, approximately 900 nm. Alternatively or additionally, an optical pattern (e.g., a dot pattern) can be emitted into the detection area, and the depth information is then determined based on this pattern.

[0049] Infrared image data can therefore be monochrome. However, polychrome image data (i.e., containing color information), e.g., red-yellow-green information, can also be used.

[0050] Depth Segmentation 101 can be applied, for example, to stereoscopic infrared image data, specifically to one of the multiple perspectives of the stereoscopic infrared image data. More generally, Depth Segmentation 101 can be applied to the same sensor data, or at least a portion thereof, on which the depth information is also based. This simplifies the necessary image acquisition device. In the case of the optical pattern, for example, there might only be one perspective to which Depth Segmentation 101 is applied.

[0051] If the image data is assigned the second class, optionally, 2D image data in the visible spectral range (e.g., in a range from approximately 380 nm to approximately 780 nm) can be acquired, which can then be output as a video signal or a still image. A cashier (or another employee) can then use this data to verify the result of classification 103 and, if necessary, acknowledge it (e.g., confirm or reject).

[0052] By excluding one or more identified data segments during classification (103), the information content supplied for classification is reduced (compared to supplying the entire image data). This allows for the use of a less complex classification algorithm (103), e.g., requiring fewer nodes and / or less computing power.

[0053] For example, image information in the depth range can be cropped from a selected distance, so that image information beyond a predefined (e.g., saved) depth value is selectively excluded from the classification. For classification purposes, a segmented (grayscale) image is used, for example, without the feet of a user, the wheels of the container, etc., or other objects / structures below the lower shelf of the transport container. This prevents image content located on the surface or below the container's interior from causing misclassifications. For example, misclassifications caused by different textures on the surface beneath the shopping cart can also be avoided.

[0054] Fig.2 A schematic diagram illustrates a registration terminal 200 according to various embodiments. The registration terminal 200 can include an image acquisition device 102, a product acquisition device 104, and a control device 106. The control device 106 can be communicatively coupled 161 to the image acquisition device 102 and the product acquisition device 104, for example, by means of a fieldbus communication network 161 or another data bus 161. A fieldbus communication network can be used for applications in automation technology. In the area of ​​point-of-sale systems, another data transmission method 161 can also be used.

[0055] The image acquisition device 102 can be configured to supply the control device 106 with image data of the acquisition area (e.g., in raw data format or a pre-processed version of the raw data format), e.g., pixel-based image data (also referred to as raster graphics). The image acquisition device 102 can, for example, be arranged above the acquisition area. The image acquisition device 102 can, for example, have one or more cameras, as will be described in more detail later.

[0056] The product detection device 104 can be configured to supply the control device 106 with a product identifier it has detected. The product identifier can, for example, be uniquely assigned to a product or its type. The product identifier can be determined, for example, based on an optical feature (also referred to as an identifier) ​​of the product being detected. The identifier (e.g., a pattern) can have a visual code representing the product identifier, such as a binary code or similar (e.g., ArUco Marker corresponding recognition algorithms, as these are designed for fast position detection on moving objects). For example, the identifier can have a barcode or another machine-readable code.

[0057] The product detection device 104 enables the identification of individual product identifiers (also referred to as identifier detection). The area in which the product detection device 104 can detect the product can, for example, be a product detection zone. Identifier detection can involve presenting the product to be detected to the product detection device 104. This presentation can involve placing the product to be detected in the product detection zone and orienting its identifier towards the product detection device 104.

[0058] For example, the product detection device 104 may include an optical detection device, an RFID (radio frequency identification) scanning device, or the like. The optical detection device may, for example, include a barcode scanning device or an image scanning device. The barcode scanning device may include appropriate sensors for implementing scanning functionality, such as one or more infrared sensors, one or more cameras, and the like. The product detection device 104 may, for example, be configured to capture and process the machine-readable code in order to extract the product identifier.

[0059] The product detection device 104 and the image detection device 102 can have a corresponding infrastructure (e.g., including a processor, storage medium, and / or bus system) or the like, which implements a measurement chain. The measurement chain can be configured to control the corresponding sensors (e.g., camera, scanner, etc.), process their measured value as an input, and, based on this, provide the image data or product identifier as an output.

[0060] The product detection device 104, the image detection device 102, and the control device 106 do not necessarily require dedicated infrastructure. For example, their information processing functions can also be provided as components of the same software (also referred to as an application) that is executed by one or more processors of the registration terminal 200. Of course, multiple applications and / or multiple processors (e.g., a group thereof) can also be used to provide the information processing functions of the product detection device 104, the image detection device 102, and the control device 106.

[0061] The control device 106 can be configured to implement the method 100, as described below by way of example.

[0062] Fig.3 A registration terminal 200 according to various embodiments 300 is illustrated in a schematic communication diagram.

[0063] The image acquisition device 102 can be configured to supply the control device 106 with the image data 202b and / or depth information of the acquisition area 201a (e.g., in raw data format or a pre-processed version of the raw data format), e.g., continuously as a sequence. The product acquisition device 104 can be configured to supply the control device 106 with a captured product identifier 202a 201b.

[0064] The control device 106 can be configured to determine 1009 payment information 204 based on the product identifier 202a (also referred to as payment information determination). The payment information 204 can clearly represent the price charged for the corresponding product with the product identifier 202a. For example, the recorded product identifier 202a can be compared with a database for this purpose.

[0065] For example, the control device 106 can be configured to start a registration session 202, e.g., in response to a detected event (also referred to as a session start event) that indicates a self-service registration is to take place. Examples of the session start event might include a user standing in front of the registration terminal 200 and / or making a corresponding entry at it, a product being presented to the product capture device 104, and / or a previous registration session being terminated.

[0066] Similarly, the control device 106 can be configured to terminate the registration session 202, for example, in response to a detected event (also referred to as a session end event) that indicates the self-service registration should be settled. Examples of session end events include a user making a corresponding entry at the registration terminal 200, a bank card or other payment method being scanned by the registration terminal 200, and / or a predefined period of time elapsed since the last product was scanned.

[0067] To end the registration session 202, the control device 106 can be configured to determine billing information 224 and display it by means of a display device of the registration terminal 200. The information determined during a registration session 202

[0068] Payment information 204, for example, can be aggregated, and the result of this aggregation added to billing information 224. Billing information 224 can clearly show the total amount payable for the registered products. Billing information 224 can optionally include further information, such as the tax rate, a list of the recorded products, an itemized breakdown of the recorded products, or similar information.

[0069] To end the registration session 202, a payment process can also be initiated, by means of which the amount due according to billing information 224 can be settled. Payment can be made, for example, using the payment terminal (not shown).

[0070] The control device 106 can be configured to perform the procedure 100 based on the image data 202b. For this purpose, the control device can identify one or more segments 702 (also referred to as data segments) of the image data 202b 101 and perform the classification 103 of the image data without the identified data segment(s). The classification 103 allows the class 706 of the image data to be determined. The control device 106 can also be configured to output the signal 302 105.

[0071] The image data class 706 can be, for example, the first class (also known as the empty class or the "empty" class) or the second class (also known as the non-empty class or the "non-empty" class). If no transport container was located in the detection area, the class can be a third class (also known as the nothing class or the "nothing" class), representing that the image data does not represent a transport container or that no transport container was present in the detection area.

[0072] The procedure 100 can of course also be carried out on a random sample basis, e.g. only for a part of the registration sessions 202.

[0073] Signal 302 can, for example, be configured to intervene in the registration session 202 (or another process of the registration terminal 200) 107 and / or to control one or more components of the registration terminal 200, e.g. to output perceptible information representing the result of the classification 103 and / or to request acknowledgment of the result of the classification 103.

[0074] Fig.4 Figure 400 illustrates a registration terminal 200 according to various embodiments in a schematic side view.

[0075] In general, the registration terminal 200 can have a support structure 352 by means of which various components of the registration terminal 200 are supported, for example, a storage device 302a, the image capture device 102, the product capture device 104, the control device (not shown), etc. The support structure 352 can, for example, have a frame and a housing attached to it, the housing containing the sensitive components of the registration terminal 200. The support structure 352 can, for example, have a base by which the support structure 352 rests on a substrate 902 and a vertically extended section 354 (also referred to as a column or "pole mount") which supports the elevated components, e.g., the image capture device 102.The subsurface 902 can be used as a reference plane 902 for depth information, for example representing depth zero, from which positions above the subsurface 902 have negative depth values.

[0076] The storage device 302a can, for example, include one or more conveyor belts and / or the product detection device 104 (e.g., a scanner). More generally, the storage device 302a can provide one or more storage levels (e.g., a checkout counter) on which products can be placed and / or moved across.

[0077] The image acquisition device 102 can be configured to acquire image data of the detection area 102e. For this purpose, the image acquisition device 102 can have one or more cameras. For example, the image acquisition device 102 can be configured to generate image data that includes depth information of the detection area 102e.

[0078] To determine depth information, the image acquisition device 102 can, for example, include one or more 3D cameras (also referred to as 3-dimensional cameras). A 3D camera can generally be configured to capture image data that contains (and is thus associated with) the depth information, or to provide the depth information separately from the image data but associated with it. The association can be configured such that each depth value of the depth information is assigned to a segment (e.g., pixel) of the image information.

[0079] Examples of 3D cameras include: a plenoptic camera (also known as a light field camera), a stereoscopic camera (also called a stereo camera), a camera with a triangulation system, a time-of-flight (TOF) camera, and a camera with an interference system. A stereo camera is a particularly cost-effective 3D camera that is easier to implement. It is also more robust against reflective surfaces and does not require a laser, thus reducing the health risk for people in public spaces. A so-called passive stereo camera therefore does not necessarily require a laser. In contrast, an active stereo camera (e.g., Intel RealSense D415) can incorporate an infrared laser projector to emit a dot pattern (a so-called IR-active stereo camera).More generally speaking, the active stereo camera can be set up to emit structured light that does not necessarily have to be point-like and / or IR light.

[0080] The TOF camera can, for example, be configured to illuminate the detection area 102e using a light pulse and to record the time (the so-called transit time) for each pixel, the time it takes for the light pulse to return. However, a different type of signal (e.g., sound) can also be used to measure the spatial distribution of the signal's transit time (e.g., an ultrasonic transit-time method). This allows for the use of an even simpler camera or the provision of a higher image resolution compared to a light pulse.

[0081] Alternatively or additionally, another mechanism can be used to provide the depth information. For example, the image acquisition device 102 can be configured to use variable focus (autofocus) to determine the depth information. The focus can be directed, for example, at the bottom of the transport container, so that the distance between the image acquisition device 102 and the bottom of the transport container can be determined as depth information based on the focus position. Alternatively or additionally, the depth information can be determined based on edge contrast measurement and / or phase comparison.

[0082] Each storage device 302a can be configured to hold one or more products. For example, a storage device 302a can include a shelf, a hook for bags, and / or a table. Optionally, each storage device 302a can include a scale configured to measure the weight of the products placed on it. For example, the storage device 302a can have three product handling zones: a storage location for products not yet scanned, the product capture zone (e.g., scan zone), i.e., the area used during scanning, and a storage location for scanned products.

[0083] Optionally, the registration terminal 200 can have an information output device 124, as will be described in more detail later. The information output device 124 can, for example, be configured to output human-perceived information (e.g., audible or visible), for example, by means of a display device on the information output device 124. The information can, for example, contain a prompt and / or assistance for the user. The output of the information and / or the content of the information can, for example, be controlled by signal 302.

[0084] During a registration session 202, a transport container 402 can be positioned within or moved through the detection area, e.g., standing on the surface 902. The transport container 402 can be empty, or it can contain one or more products 404, meaning it is not empty. This difference, i.e., whether the transport container 402 is empty or not, can be determined by means of empty detection.

[0085] Fig.5 The method 100 is illustrated according to various embodiments in a schematic flowchart 500, e.g. implemented by means of the control device 106.

[0086] In 101, one or more data segments can be filtered out from the image data 202b, representing the stationary (invariant) background of the acquisition area 102e (also known as depth segmentation or, more intuitively, masking) and which should not be considered during classification 103. More generally, depth segmentation 101 can be performed using the depth information.

[0087] The depth information specifies a depth pixel by pixel, e.g., a distance from the image acquisition device 102 or another reference object 902 (e.g., the reference plane 902). For example, those data segments that exhibit a visually large depth can be filtered out 101. Alternatively or additionally, those data components that visually exhibit a shallow depth can be protected from filtering out 101. Depth segmentation is less sensitive to the prevailing lighting conditions, reflections, or other disturbances in the acquisition area 102e. More generally, depth segmentation can superimpose the depth information and the color information of the image data, so that those data components exhibiting a large depth are filtered out 101.

[0088] For depth segmentation, a filter mask (also simply called a mask) can be used. A filter mask can, for example, contain a graphics filter and a segmentation mask. The segmentation mask can specify, for example, pixel by pixel, which data segments (e.g., pixels) of the image data 202b the graphics filter should be applied to, or which data segments should be protected from the application of the graphics filter. For example, light areas of the segmentation mask can indicate that the graphics filter should be applied, and dark areas can indicate that the graphics filter should not be applied, or vice versa. Gradations between light and dark can, of course, also be used to achieve partial application of the graphics filter and / or a transition.The application of a graphics filter can, for example, modify the color information of the image data, such as darkening it (e.g., using a black filter), replacing it (e.g., using a replace filter), or removing it (e.g., using a remove filter). The mask can have the same pixel-related dimensions (also called pixel dimensions) as the color information (e.g., infrared information).

[0089] The segmentation mask can, for example, be applied as an intermediate step to a corresponding graphics filter, thus enabling the selective application of the graphics filter to the image data 202b. As a result of depth segmentation 101, filtered image data 212b (also referred to as additional image data 212b) is obtained, which represents the moving objects 402, 404 isolated according to the depth information within the detection area 102e, e.g., without their background. Depth segmentation 701 can, for example, mask and / or discard (e.g., not use in classification) one or more image data segments 702 (also referred to as segments or data segments) of the image data that represent the background of the detection area 102e.

[0090] Depth Segmentation 101, for example, can map the depth information onto the mask according to which the image data is masked or segmented. This mapping can be done, for example, using a function that does not necessarily have to be continuous. A discontinuous function can, for instance, provide a sharp transition between applying the graphics filter and preventing its application.

[0091] In 500, depth segmentation 101 can be performed, for example, by processing the image data 202b. Alternatively or additionally, depth segmentation 101 can also be performed focally, for example, by using a shallow depth of field or large depth of field blur when acquiring the image data. This ensures that the background of the acquisition area 102e is rendered blurred when the focus is on the object 404, 402. In this case, the image data 202b may already contain one or more isolated objects. For focal depth segmentation, an image acquisition device that provides a shallow depth of field (e.g., of a few centimeters) can be used. A shallow depth of field can be provided, for example, with a large image sensor (e.g., more than 2.5 cm) and / or a small aperture (e.g., less than f / 2.0).More generally, the hyperfocal distance of the image acquisition device 102 can be greater than its distance to the object or the reference plane. For example, the image sensor may have a diagonal extent greater than the depth of field. The following refers to the processed 212b image data. The description can be applied analogously to optically isolated image data.

[0092] According to various embodiments, the depth information can be used to depth-segment the movable objects arranged above the reference plane 902. The depth information can, in turn, be determined by a stereoscopic camera. This camera determines the distance (figuratively, depth) of specific image areas via the disparity of the different perspectives of the stereoscopic camera. The depth of field of the stereoscopic camera can be used, but is not necessarily required.

[0093] In the context of a stereo camera and the determination of depth information, disparity (also called deviation) refers to a shift in position that the same object occupies in the image from two different perspectives. The focal points of the two lenses corresponding to these perspectives are separated by a distance b. If both lenses have a focal length f, the distance r of the object from the lenses (also called depth) is given by r = b · f / d, where d denotes the disparity. Thus, the depth information r of an object can be determined based on the calculated disparity of the object in the stereo image. A disparity map of a stereo image is therefore equivalent to a depth map, which indicates the spatial distribution of the depth information. The disparity map can be generated, for example, using a census transformation.

[0094] In general, depth segmentation can be performed completely or partially. Depth segmentation can result in a reduction of the contrast and / or image sharpness of those data components of the image data that represent the background.

[0095] For depth segmentation, the image data 202b is divided into data segments (also referred to as segmentation) with a stored threshold value as the criterion for subdivision. One or more first data segments 702 (shown hatched) are identified, each with a depth value greater than the threshold value (i.e., representing the background of the detection area 102e), thus fulfilling the criterion. For example, the first data segment 702 could represent the environment. Furthermore, a second data segment 704 (also referred to as an object data segment) could be identified, each with a depth value less than the threshold value. For example, the second data segment 704 could represent one or more moving objects (e.g., the transport container 402 and / or the product 404 within it) in the detection area 102e.The first data segment 702 (also known as the environment data segment) can be filtered out by, for example, removing, redacting, or marking it.

[0096] In addition to depth segmentation, depth segmentation can optionally be performed using the image data from the empty acquisition area (also known as comparison filtering). For example, those data components that match the image data of the empty acquisition area can be filtered out.

[0097] In addition to depth segmentation and / or comparison filtering, depth segmentation can optionally be performed using the color information of the image data (also known as color filtering), for example, RGB information (red-yellow-blue information). For instance, data components exhibiting a specific color (such as the background color) can be filtered out. Color filtering can be performed channel by channel, e.g., individually for the red, blue, and green channels. Naturally, a color space other than RGB can also be used.

[0098] However, comparison filters and / or color filters can be susceptible to the current lighting conditions and / or reflections and should therefore only be used to supplement depth segmentation. Alternatively or additionally, unlike depth segmentation, it may be necessary to regularly adjust the comparison filter to the situation and / or adjust the color filter at least once per location.

[0099] Preferably, only depth segmentation (optionally together with color filtering) can be performed. Alternatively, a combination of several filtering types 701 can be used, e.g., depth segmentation together with color filtering and / or comparison filtering. In general, combining different filtering types 101 allows for obtaining more information and / or lower sensitivity to interference.

[0100] As a result of the depth segmentation 101, those data components can remain to be considered during classification 103 which represent a moving object 402, 404 in the detection area 102e that differs from the stationary (invariant) background of the detection area 102e. This makes it possible to optimize the amount of data supplied to classification 103 and to reduce sources of error.

[0101] In section 103, the filtered image data 212b can be classified (also referred to as classification), i.e., assigned to a class 706 from several classes (also referred to as image class). Classification can be performed based on the filtered image data 212b, for example, using a classifier or similar tool applied to the filtered image data 212b. Depending on whether an empty or non-empty transport container is detected based on the filtered image data 212b, the corresponding class 706 can be assigned.

[0102] The multiple image classes can, for example, include an empty class (also referred to as the "empty" class), which indicates that no moving object 404 was detected in the transport container 402 based on the filtered image data 212b (i.e., the transport container 402 was empty). The multiple image classes can include a non-empty class (also referred to as the "non-empty" class), which indicates that one or more moving objects 404 were detected in the transport container 402. The multiple image classes can optionally include a third class (also referred to as the "nothing" class), which represents that the detection area is empty or that no transport container 402 was detected.

[0103] Classification 103 can be performed according to various embodiments using a trained processing algorithm (also referred to as a classification algorithm or, more simply, a classifier). A classifier described herein can, for example, be provided by means of an artificial neural network (kNN), as described in more detail herein. Optionally, in addition to the kNN, the classifier can also be implemented using object detection, edge detection, transformation, size detection, and / or color detection.

[0104] Examples of training data for the classification algorithm include: image data of the empty detection area, image data of the detection area with an empty transport container, and image data of the detection area with a non-empty transport container. Examples of target variables for training the classification algorithm include multiple image classes.

[0105] Alternative or additional possible examples of training data (also referred to as input vectors) can contain depth information, so that, intuitively, it does not necessarily have to be used only for depth segmentation. Similarly, the depth information can optionally also be taken into account during classification.

[0106] Alternatively or additionally, possible examples of target quantities might show that the "non-empty" class is further subdivided to compare the appearance of the items with one or more reference appearances. For example, it can be determined during classification how much is contained in the non-empty transport container 402. A measure of this could be, for example, the area of ​​the transport container designated as non-empty or a quantity representing this area.

[0107] For example, the class "non-empty" can be subdivided, for instance, into a subclass "non-empty but harmless" and a subclass "non-empty and harmful." The subclass "non-empty but harmless" can be determined, for example, if the appearance of the object(s) in the transport container cannot be assigned to any item that is stored in the database (e.g., its pattern and / or product identifier). Examples of such objects include wallets, receipts, and shopping bags. This makes it possible to reduce false alarms. The subclass "non-empty but harmless" can optionally be mapped to the class "empty," so that, for example, no signal or the signal of the class "empty" is issued.

[0108] Examples of the processing algorithm can include various forms of machine learning for image classification, such as... the so-called "Histogram of oriented gradients" (HOG) in combination with a support vector machine (SVM); a pre-trained kNN with "standard architecture" (e.g. GoogLeNet), which is further adapted to the given problem by "transfer learning" and fine-tuning.

[0109] However, the latter entails higher hardware requirements and / or higher error rates for the given problem, since the pre-training database is very different from the input vectors considered here.

[0110] When the filtered (and thus reduced) image data 212b is fed to the classification algorithm, a less complex algorithm and / or a less powerful processor can be used to execute the classification algorithm. In essence, the filtered image data 212b contains only those data parts that represent objects disjoint from the background, or are empty. This optimizes the amount of data to be processed and the classes are more clearly separated from each other.

[0111] In 105, signal 302 can be output based on a result 706 of the classification 103.

[0112] In a simple implementation, the classification algorithm can, for example, only distinguish between an empty transport container and non-empty transport containers. The class "nothing" can be automatically determined, for instance, if the filtered image data 212b is more than 99% filtered (e.g., blacked out). The class "nothing" can also be automatically determined, for example, if no transport container is present in the detection area 102e. For example, the 2D image data can be used to determine whether a transport container 402 is located in the detection area 102e (i.e., whether a transport container 402 is present).

[0113] For example, the detection of the presence of transport container 402 can trigger the execution of procedure 100.

[0114] For example, the presence of transport container 402 can be detected by capturing reflections (on objects), e.g., in the IR range. It can be determined whether the reflections originate from a transport container, e.g., from its structure and / or markers. Alternatively or additionally, color information (from the 2D image data) and / or depth information can be used to detect the presence of transport container 402. Other mechanisms can also be used to detect the presence of the transport container, such as a scale embedded in the substrate 902, a light barrier, a motion detector, etc.

[0115] According to various embodiments, a combination of one or more 3D cameras and / or one or more RGB cameras (as 2D cameras) can be positioned above the checkout counter. A calibration algorithm can train the processing algorithm using the visible 3D geometry of the detection area 102e in front of the registration terminal (when empty) and approximates this 3D geometry using planes of varying relative distances. Based on the topology of the detection area 102e thus determined, an object located above the reference plane 902 can be effectively isolated from the background. This mechanism is more effective than purely RGB image-based segmentation or pattern recognition and is virtually independent of environmental influences such as light, shadows, and reflections. The resulting image data 212b can therefore also be processed virtually independently of these influences.the location of the registration terminal. This enables robust classification of the image data without the need for complex training of the algorithms for each individual location.

[0116] Calibration to the reference plane 902, and thus subsequent object isolation, can be performed independently of the classification of the preprocessed image data. For example, a classifier trained on a specific geometry after isolation can also be applied to a different (e.g., similar) geometry without retraining. As long as the geometry remains comparable, i.e., approximately the same distances between the relevant objects and the camera, the algorithm can be trained on one registration terminal 200 (or even in a subsequently reconstructed geometry) and used on another registration terminal 200. Calibration to the given reference plane ensures that the image data is not only independent of the environment but also virtually independent of the specific geometry of the registration terminal 200.

[0117] Fig.6 The method 100 is illustrated according to various embodiments in a schematic flowchart 600. The method can, for example, be implemented by means of the control device 106.

[0118] The image data 202b acquired by means of the image acquisition device 102 can comprise color image data 914 and depth image data 912. The color image data 914 can comprise a pixel matrix with (monochrome or polychrome) color information, wherein the color information specifies the color value of each pixel. The depth image data 912 can comprise a pixel matrix with depth information, wherein the depth information specifies the depth value of each pixel.

[0119] The depth image data 912 and the color image data 914 can be acquired using the same camera or using separate cameras (e.g., independently of each other). For example, a 3D camera can determine the depth image data 912 and an RGB camera can determine the color image data 914.

[0120] Depth information can be provided, for example, by means of a stereo camera as a 3D camera. A stereo camera can have several (e.g., two) lenses 102o arranged side by side, which are directed towards the detection area 102e. The multiple lenses 102o can be configured to image the detection area 102e onto one or more image acquisition sensors of the image acquisition device 102. The image acquisition sensor can, for example, be infrared-sensitive, i.e., detect infrared radiation.

[0121] The stereo camera captures vivid image data that represents multiple perspectives of the detection area 102e simultaneously (also known as stereoscopic image data), i.e., that represents the detection area 102e viewed from different directions and / or from different locations. The exposure control and / or focus adjustment of the lenses can be linked for this purpose.

[0122] Depth information can be determined based on the stereoscopic image data of the detection area 102e. For example, the different perspectives of the stereoscopic image data can be superimposed and the depth information derived from this. The description given for the stereoscopic image detection device 102 can also apply analogously to a differently configured image detection device 102 that implements a different mechanism to provide the depth information.

[0123] In general, the depth information 912 can contain information that places several segments of the image data in a spatial relationship to each other and / or to a reference object 902, 102. Each corresponding image data segment can be assigned a depth value that represents the spatial relationship. For example, individual pixels (i.e., the depth information 912 can be pixel-by-pixel) or multiple pixels (the so-called pixel group) can be used as image data segments. The reference object 902, 102 can, for example, include or be formed from the image acquisition device 102 and / or the reference plane 902. For example, the depth information 912 can specify a pixel-by-pixel distance from the reference object 902, 102 as a depth value.

[0124] The depth information 912 can be mapped to the color information. For this purpose, the depth information can be mapped to a filter mask 916 905. The filter mask 916 can, for example, be provided as a vector graphic or raster graphic, corresponding to the image data 202b. Its application to the color information can, for example, color the image data segments outside the interest layer 901 black. Whether an image data segment lies within the interest layer 901, and is therefore considered during classification, or lies outside the interest layer 901, and is therefore not considered during classification, can be determined by checking whether a depth value assigned to the segment fulfills a stored criterion.

[0125] For example, those image data segments 101 are filtered out and thus not considered during classification 103 if their depth information fulfills a predefined (e.g., stored) criterion. This criterion can be intuitively fulfilled if the depth information represents a small distance from the reference object 902, 102, e.g., less than 20 centimeters, e.g., less than 10 centimeters. As a result of filtering 101, those image data segments 704 may remain that represent an object 404, 402 within a layer 901 (also referred to as the interest layer 901) of the acquisition area 102e. The interest layer 901 can be spatially separated from the reference object 902, 102, e.g., by a non-interest layer 903 that borders the reference object 902, 102.

[0126] The interest layer 901 can, for example, be arranged such that one or more than one bottom (e.g., bottom wall and / or bottom grid) of the transport container 402 and / or an interior space of the transport container 402 is located in the interest layer 901. The boundaries of the interest layer 901 can, for example, be used to form (e.g., delimit) the criterion.

[0127] The criterion can be met, for example, if the depth information (e.g., its value) represents a distance from the reference object 902 of less than approximately 20 centimeters (e.g., 10 centimeters) and / or more than approximately 1.5 meters (e.g., 2 meters).

[0128] Fig.7 The method 100 is illustrated according to various embodiments in a schematic flowchart 700, for example implemented by means of the control device 106.

[0129] Procedure 100 in 1301 may include the calibration of any processing algorithm. Calibration can be used, for example, to create or update the criterion.

[0130] Calibration 1301 may, for example, only be necessary once, e.g., at system startup and / or after mounting the recording terminal 200, and subsequently be omitted. Alternatively or additionally, calibration 1301 can be repeated, e.g., during idle phases of the recording terminal 200. Calibration 1301 may involve capturing initial image data of the empty capture area 102e (e.g., its reference plane) using the image capture device 102. The initial image data may contain depth information and / or color information (e.g., both captured simultaneously) (also referred to as reference depth information and reference color information, respectively). The reference plane 902 may, for example, be planar and therefore have little or no topography (also referred to as 3D structure). The initial image data of the empty capture area 102e can be saved and / or used as training data for the processing algorithm (e.g.,the object recognition algorithm and / or the classification algorithm).

[0131] Calibration of the 1301 can be performed using the calibration algorithm, which, for example, determines or updates the criterion based on the reference depth information or reference color information. The criterion can, for example, specify a threshold value that is greater than any value in the reference depth information, such as the sum of the reference depth information and a tolerance value. The tolerance value could be, for example, 20 centimeters or less, or 10 centimeters or less. The criterion can be met, for example, if the depth value exceeds the threshold value.

[0132] Method 100 can be described in 1303 as using the image acquisition device 102 to acquire second image data of the acquisition area 102e (e.g., its reference plane), in which, for example, a transport container and optionally a product (also referred to as an article) are arranged within the transport container. The second image data 202b can contain depth information and / or color information (e.g., both acquired simultaneously). The depth information can, for example, represent the topography above the reference plane 902 by means of a light / dark gradient.

[0133] Method 100 can include in 101 applying depth segmentation to the second image data 202b. As a result of the depth segmentation 101, third image data 212b (also referred to as segmented image data 212b) can be provided. The third image data 212b can contain one or more object data segments from the second image data 202b, which represent one or more moving objects (e.g., the transport container and optionally an object, e.g., an item, within it) in the detection area 102e (or its zone), and / or contain fewer background data segments than the first image data 202b.

[0134] Depth segmentation 101 can, for example, filter out the color information pixel by pixel according to the depth information. For instance, in this way, every pixel of the image data whose assigned depth value meets the criterion can be filtered out. The filtering can, for example, blacken those pixels of the image data whose assigned depth information values ​​essentially correspond to the reference plane 902 (i.e., those pixels that are not, for example, intuitively located above the reference plane). As a result of the masking, only the pixels of one or more object data segments can be non-black, and the remaining pixels can be black.

[0135] Procedure 100 can include in 103 the classification of the third image data 212b (i.e., the segmented image data). The classification 103 can be performed, for example, using a classifier 1307, such as a kNN. As a result of the classification 103, the third image data can be assigned a class 706 from several (e.g., two or more, e.g., exactly three) classes. The several classes can, for example, include: the class "empty" and the class "non-empty", and optionally the class "nothing".

[0136] Fig.8 Figure 1 illustrates the classifier 1307 according to various embodiments in a schematic diagram 800, for example, configured according to method 100 and / or implemented by means of the control device 106. The classifier 1307 can be configured as a convolutional neural network. Of course, a differently configured classifier 1307 that provides the corresponding functions can also be used.

[0137] The Classifier 1307 can have a multitude of layers, for example, at least one input layer 1402, at least one output layer 1404 (classification layer, also called the final layer), and one or more processing layers 1406, 1408, between these. The input layer 1402 can, for example, define a resolution for the incoming image data (here, for example, 120 x 106 pixels) and / or a color space for the color information of the incoming image data (here, for example, RGB). Alternatively or in addition to RGB (red-green-blue color space), any other color space suitable for the Classifier 1307 can be used, for example, another technical-physical color space such as CMYK (cyan-magenta-yellow-black color space), a perceptual color space (which describes, for example, characteristics such as brightness, saturation, and hue), or a grayscale color space.In the case of the grayscale color space, unlike the network architecture shown, only one channel can be used, e.g., instead of three channels in the case of RGB. Of course, a different pixel dimension can also be used for the incoming image data, e.g., 224 x 224 pixels.

[0138] When a convolutional neural network is used as a classifier 1307, the one or more processing layers can have at least one convolutional layer 1406 and at least one intermediate layer 1408 (fully connected layer). Optionally, a pooling layer (not shown) can be placed between the at least one convolutional layer 1406 and the at least one intermediate layer 1408 (also called a fully connected layer).

[0139] The incoming image data can be multidimensional, for example, represented by a corresponding matrix (e.g., the pixels of a grayscale or color image). The matrix (also called a tensor when there are more than three dimensions) can, for example, reference a first spatial coordinate (e.g., x-coordinate) in a first dimension, a second spatial coordinate (e.g., y-coordinate) in a second dimension, color information (e.g., a point in color space) in one or more third dimensions, and depth information (e.g., z-coordinate) in a fourth dimension. Optionally, infrared images can also be referenced using additional channels, thus enabling the processing of multispectral images.The coordinate of the depth information can, for example, be referenced along a direction that runs along the direction of image data acquisition and / or perpendicular to the reference plane 902.

[0140] Depending on the dimension of the image data, the nodes of each convolution layer 1406 can also be arranged multidimensionally. Optionally, the depth segmentation can include removing the dimension of the depth information to form the filtered image data. Alternatively or additionally, the depth segmentation can include replacing those entries of the tensor's color information where the depth information exceeds a threshold with zeros.

[0141] The activity of each node in a convolution layer 1406 can be computed using a discrete convolution. The discrete convolution of each node can be transformed into an output of the node by an activation function. For example, a ReLU function can be used as the activation function. In the context of artificial neural networks or training methods, a ReLU function (also called a rectifier function, rectifier function, or simply ReLU) is an activation function that is essentially defined as the positive part of its argument.

[0142] Several parameters control the size of the output (also called output volume) of a convolution layer 1406, for example the parameters filter count ("depth"), step size ("stride") and padding ("padding" or "zero padding").

[0143] The filter count corresponds to the number of filters (here, for example, 18 or 12 filters) used, with each filter being trained to search for something different (e.g., specific structures) in the input data of convolution layer 1406. The size of a filter can be specified in pixels (here, for example, 7 x 7 pixels or 5 x 5 pixels). For instance, if the image data is fed to the first convolution layer 1406 in raw data format (also called raw image data), different nodes along the depth dimension can be activated in the presence of differently oriented edges or color patches. The step size specifies how far a filter is shifted. If the step size is 1, then a filter is shifted by one pixel. If the step size is 2 or 3, then the filter is shifted by 2 or 3 pixels at a time, respectively. This results in spatially smaller power volumes.In the process from the input to the output of the classifier 1307, the step size of the convolution layers 1406 can decrease (here, for example, from 3 to 1). Using zero padding, input data can be filled with zeros at the edges. This allows control over the spatial size of the output data of a convolution layer 1406.

[0144] The at least one (i.e., one or more than one) termination layer 1404 can terminate the network according to the architecture of the multilayer perceptron.

[0145] This final layer 1404, for example, has three nodes for three classes ("nothing", "empty", "non-empty"). The upstream intermediate layer 1408 is a so-called hidden layer. This intermediate layer 1408 allows for the mapping of different decision paths between the final folding layer 1406 and the actual classification 1404. This highly redundant, so-called one-out-of-n encoding ("one-hot encoding") has the advantage that implicit assumptions about the similarities of classes do not necessarily have to be made.

[0146] Visually, block 1406, composed of folding layers, can provide the feature extraction function, and the one or more fully connected intermediate layers 1408 downstream of this block can provide the decision-making function, for example, based on the previously extracted features. These features, or rather their presence in different regions, could also be passed directly to the final layer 1404, but this limits the possible decision paths and may be insufficient depending on the prevailing boundary conditions.

[0147] A similarly sized neural network as classifier 1307, which solves more general problems, has two or more such intermediate layers 1408 for decision-making.

[0148] Since an intermediate layer 1408 assumes the most parameters, it is prone to overfitting. To counteract this overfitting, a so-called dropout rate p can be specified. For this purpose, during each training phase, individual nodes are removed from the network with a probability of 1-p (here, p=0.5 for example) or retained with a probability p, leaving a reduced network. In the training phase, only this reduced network is trained on the data from that training phase. After the training phase, the removed nodes are added back to the network with their original weights.

[0149] The output layer 1404 (also called the classification layer) can transform the output of at least one intermediate layer 1408 into the corresponding classes. For example, output layer 1404 can output a K-dimensional vector σ (here, K is 3 for example) and use a softmax function as the activation function. The so-called softmax function (also called softmax or normalized exponential function) is a generalization of the logistic function that transforms a K-dimensional vector with real components into a K-dimensional vector σ, also a vector of real components, in the range (0, 1), where the components sum to 1.

[0150] The number of nodes (here, for example, 64) of an intermediate layer 1408 corresponds, for example, to the number of (object) classes that the following final layer 1404 is to distinguish and to the number of possible structural arrangements that the previous layer 1406 extracts.

[0151] Examples of training data for classifier 1307 may include: image data of the empty detection area, image data of the detection area with an empty transport container, image data of the detection area with a transport container and a harmless object inside, and image data of the detection area with a transport container and an item (i.e., a harmful object) inside. Examples of target variables for training classifier 1307 may include multiple image classes.

[0152] Alternatively or additionally, possible examples of training data may show that the depth information is also used to train the classifier 1307, so that it does not necessarily have to be used only for depth segmentation.

[0153] Alternatively or additionally, possible examples of target variables for training classifier 1307 might involve further subdividing the "non-empty" class to compare the appearance of the items with one or more reference appearances. Put simply, this could mean ensuring, for example, that the item (object) in the shipping container is indeed an item for which fraudulent activity could occur (i.e., one that is not the customer's property), and therefore potentially problematic.

[0154] Examples of the classifier 1307 can exhibit various forms of machine learning for image classification, such as... the so-called "Histogram of oriented gradients" (HOG) in combination with a support vector machine (SVM); a pre-trained kNN with "standard architecture" (e.g. GoogLeNet), which is further adapted to the given problem by "transfer learning" and fine-tuning.

[0155] However, the latter entails higher hardware requirements and / or higher error rates for the given problem, since the pre-training database is very different from the input vectors considered here.

[0156] Fig.9 A registration terminal 200 in the method 100 according to various embodiments is illustrated in a schematic side view 900.

[0157] According to various implementations, the use of the 3D image (i.e., depth image) serves to limit the monitored image area and increases the robustness of empty detection. Furthermore, data protection-relevant information is reduced to a minimum, making the visual identification of individuals and thus a potential breach of personal data unlikely.

[0158] The 3D camera 102 can be attached to the PoleMount system 354 of the checkout counter 200 in such a way that its perspective is from a slightly elevated angle and / or directed at least partially downwards. The perspective (visual angle) of the 3D camera 102 allows for an image of the basket base (general base of the transport container 402), the interior of the transport container 402, and the lower storage area of ​​the transport container 402 (e.g., a shopping cart). The camera 102 can provide, for example, output, 3D images of the transport container 402. Using the depth information of the 3D images, the background, i.e., the floor 902, can be hidden. Environmental influences and color variations of the transport container 402 or the items inside it do not affect the 3D image.What remains is a prototypical silhouette of a Transport Container 402, which is essentially comparable to that of the majority of Transport Container 402 available on the market. An algorithm 1307, based on a convolutional neural network, classifies the depth profiles thus recorded into three different classes: . "nothing" (situations in which everything except a shopping cart or a partial shopping cart is visible); "empty" (situations in which a shopping cart is fully visible that contains no item); "non-empty" (situations in which a shopping cart is fully visible that contains at least one item).

[0159] The universal 3D grid structure of a basket (e.g., as a hand basket or part of a shopping cart) enables the use of a highly optimized kNN algorithm, which, unlike conventional object recognition approaches, allows for very robust yet very fast image classification. Therefore, these algorithms require no additional computer hardware and can be operated on the existing control device 106 of a point of sale 200 (POS). The control device 106 can, for example, be provided by a PC.

[0160] The classification of the captured image data (e.g., from a video) can, for example, result in the following classes in sequence: "nothing" -> "not empty" -> "nothing". Such a sequence then triggers, for example, a silent alarm as signal 302 for cashier 952, allowing them to specifically inspect a transport container 402, which has been identified as not empty, for unscanned items.

[0161] The control device 106 can, for example, provide the alarm in such a way that the registration terminal prompts the user to acknowledge it after the inspection of the transport container 402. Alternatively or additionally, a video (e.g., showing a sequence of color images) of the directly detected area can be output as signal 302. This video is displayed on the information output device 124 (e.g., a monitor) as an alternative or additional to the silent alarm. The video can, for example, show the event of the transport container 402 passing by, e.g., in an endless loop. Objects not relevant for empty container detection, such as receipts or the customer's personal belongings, can be easily identified by the cashier 952 without significantly interrupting the process.

[0162] The method 100 described herein can be applied, as shown in 900, to a manned registration terminal (i.e., one operated by a cashier) and / or analogously to a self-service registration terminal.

[0163] In the case of a self-service registration terminal, signal 302 can, for example, trigger an alarm to an employee, identifying the self-service registration terminal from which it originated. Alternatively or additionally, upon detection of a non-empty transport container (e.g., a carrying basket), signal 302 can place the registration session 202 into a waiting state at a self-service registration terminal. Alternatively or additionally, upon detection of an empty transport container (e.g., a carrying basket), signal 302 can be used to resume the registration session 202, for example, by enabling the initiation or execution of the payment process.

[0164] Conventional mechanisms for detecting empty containers include, for example: a mirror on the ceiling or on the opposite checkout counter, a continuous (or light-triggered) camera stream displayed on the cashier's monitor, or ID numbers on the shopping cart chassis that must be entered to authorize payment, thus forcing the cashier to inspect the cart. These mechanisms require the cashier's constant attention, which can naturally lapse. In contrast, the cashier's attention for the method 100 described herein can be focused solely on the case of a transport container 402 being detected as not empty and its inspection. This is further facilitated by a playback of the recorded image data of the passing transport container 402.

[0165] Conventional empty-tray detection mechanisms include, for example, a so-called DataLogic Lanehawk, which performs a one-to-one identification of items on the lower shelf of the cart using a camera and object recognition. Such one-to-one identification of items using object recognition is highly error-prone. Furthermore, constant variations in the product range or packaging of items require retraining of the algorithm. This one-to-one identification also only covers the lower shelf of the cart. In contrast, Method 100, according to its various embodiments, is independent of the item's appearance and also covers one or more shelves (e.g., the bottom) of the transport container.

[0166] Conventional mechanisms for empty cart detection include, for example, a so-called StopLift system, which uses a ceiling-mounted security camera to capture color images and compares them to reference images of empty shopping carts to detect those that are not empty. The use of conventional color images for object recognition is subject to high error rates. Sources of error include varying cart designs, lighting conditions, and backgrounds (such as floor surfaces or dirt). Furthermore, the use of ceiling-mounted cameras results in suboptimal perspective and resolution. In contrast, Method 100, according to its various embodiments, enables a very low error rate (e.g., <5%) regardless of the location and type of shopping cart.

[0167] Conventional empty cart detection mechanisms include, for example, CaddyCheck, which uses self-learning algorithms to detect shopping carts, requiring training for each POS type. In contrast, lengthy training procedures lasting several weeks at the respective point of use (i.e., the checkout workstation) are not necessarily required for Method 100 according to its various implementations.

[0168] According to various embodiments, the transport container can be a shopping cart, a shopping basket, or another cart (and / or basket), a carrying basket, or similar item. Alternatively or additionally, the transport container (e.g., its base) can have or be formed from a grid, or at least partially (e.g., at least its base) be opaque. According to various embodiments, the transport container, e.g., the shopping cart, can have a chassis. The chassis can have at least three, e.g., at least four, wheels.

[0169] According to various embodiments, the transport container can be made of or formed from plastic and / or metal, e.g. aluminum and / or steel.

[0170] According to various embodiments, the transport container can have a handle strip or a swiveling handle with a handle strip.

[0171] Fig.10 Figure 1000 illustrates the classifier 1307 according to various embodiments, for example, set up according to method 100 and / or implemented by means of the control device 106. The explanations regarding Fig.8 The same applies here.

[0172] Claimed is a registration terminal (e.g., comprising a cashier workstation or a self-service registration terminal) comprising: an image capture device configured to capture image data; a product capture device for capturing a product identifier of a product when the product is presented to the product capture device; a control device according to Example 22, further configured to determine payment information based on the product identifier, wherein the image capture device (e.g., its capture direction) is, for example, at least partially directed downwards (i.e., its capture direction has at least one directional component parallel to the direction of gravity).

Claims

1. Method (100) for the computer-aided detection of an empty transport container (402), the method (100) comprising: • determining (101) one or more than one segment of image data using depth information assigned to the image data; wherein the depth information has depth values assigned pixel-by-pixel to the image data and the image data are segmented using the depth information, wherein a depth value of the one or more than one segment is greater than a stored threshold value, wherein the one or more than one segment has one or more pixels; • assigning (103) the image data to one of a plurality of classes, a first class of which represents the transport container (402) being empty, and a second class of which represents the transport container (402) not being empty, wherein the one or more than one segment is not taken into account during assignment; • outputting (105) a signal which represents the image data being assigned to the second class; • wherein the one or more than one segment represents a background of the transport container (402).

2. Method (100) according to Claim 1, wherein the one or more than one segment is determined if depth information assigned to the segment satisfies a stored criterion.

3. Method (100) according to one of Claims 1 to 2, wherein the one or more than one segment of the image data is filtered out in such a way that it is not taken into account during assignment.

4. Method (100) according to one of Claims 1 to 3, wherein a contour of the transport container (402) is released by means of the filtering out.

5. Method (100) according to one of Claims 1 to 4, wherein the assignment (103) is carried out by means of a classifier (1307) configured as a convolutional neural network, wherein the classifier (1307) has an input layer (1402) and an output layer (1404) and a processing layer (1406, 1408) between the input layer (1402) and the output layer (1404), wherein the image data supplied to the assigned (103) enter the input layer (1402) and are represented by means of a matrix, wherein the matrix has a first spatial coordinate in a first dimension, a second spatial coordinate in a second dimension, colour information in a third dimension, and the depth information in a fourth dimension.

6. Method (100) according to one of Claims 1 to 5, wherein the signal has an instruction to change a registration session to a waiting state.

7. Method (100) according to Claim 6, the registration session comprising: • determining payment information based on a product identifier; • initiating a payment process based on the payment information.

8. Method (100) according to one of Claims 1 to 7, wherein the signal has an instruction to display colour information of the image data relating to the transport container (402).

9. Method (100) according to one of Claims 1 to 8, wherein the signal has an instruction to request a user input.

10. Method (100) according to one of Claims 1 to 9, wherein the signal has an instruction to output an alarm.

11. Method (100) according to one of Claims 1 to 10, wherein the image data have stereoscopic image data and the depth information is determined on the basis of the stereoscopic image data.

12. Method (100) according to one of Claims 1 to 11, wherein the image data are assigned to one of a plurality classes by means of an artificial neural network.

13. Control device (106) which is configured to carry out the method (100) according to one of Claims 1 to 12.

14. Registration terminal (200), comprising: • a control device (106) according to Claim 13, • an image capture device (102) configured to capture the image data; • a product capture device (104) for capturing a product identifier of a product when the product is presented to the product capture device; and • wherein the control device (106) is further configured to determine payment information based on the product identifier.

15. Non-volatile storage medium comprising code segments which, when executed by a processor, cause the processor to carry out the method according to one of Claims 1 to 12.