SELF-SERVICE TERMINAL

Description

[0001] Several examples of implementation involve a self-service terminal.

[0002] At a self-service terminal, such as a self-checkout, a user (e.g., a customer) can access various services, such as paying for a purchase, without interacting with another person. This may require identifying the items the user intends to purchase. Traditionally, the barcode of an item can be scanned to determine its type. For items without barcodes, such as fresh produce (e.g., fruits, vegetables, bread, etc.), the user may need to manually select the item type and, if the price is based on weight, weigh it. Both the operator of such a self-service terminal (e.g., a store owner) and the users (e.g., customers) may wish to expedite the payment process.

[0003] Document EP3450934A1 discloses a similar self-service terminal.

[0004] Several embodiments of such a self-service terminal provide a faster payment process. This is achieved, for example, by using a scale and at least one imaging sensor of the self-service terminal to determine whether one or more items, and if so, which items, have been removed by a user. It may be sufficient for the user to remove the items from the scale one after the other and, for example, pack them. This eliminates the need to scan items and / or manually select items without barcodes, thus significantly speeding up the payment process. Furthermore, this allows the use of a self-service terminal that does not require an additional scale (e.g., a verification scale) and / or a scanner unit for scanning barcodes, making the self-service terminal smaller and / or less expensive.

[0005] They show Figures 1A to 1: Your self-service terminal according to various embodiments; Figures 2A to 2F: Aspects of a processing scheme for determining goods removed from a platform of a scale of the self-service terminal according to various embodiments; Figure 3: A flowchart for removing goods from the platform of the scale according to various embodiments; Figure 4: An exemplary display on a display device of the self-service terminal according to various embodiments; Figure 5: A processing scheme for determining goods removed from the platform of the scale according to various embodiments. Figur 6 an exemplary image showing the removal of goods by a user; Figures 7A and 7B each show a processing scheme for verifying a determined type of goods or goods removed according to different embodiments; and Figures 8 and 9 each show a method for determining whether and, if so, which goods or goods have been removed from a scale of a self-service terminal, according to different embodiments.

[0006] The following detailed description refers to the accompanying drawings, which form part thereof and in which specific embodiments in which the invention can be implemented are shown for illustration purposes.

[0007] Various embodiments relate to a self-service terminal (e.g., a self-service checkout) which enables a faster payment process (compared to conventional payment processes) by using a scale and at least one imaging sensor of the self-service terminal to determine whether one or more goods, and if so, which goods, have been taken by a user, thus eliminating the need to scan goods and / or manually select barcode-less goods (e.g., fresh produce).

[0008] FIG. 1A bis FIG.1D Figure 1 shows a self-service terminal 100 according to various embodiments. The self-service terminal 100 can, for example, be a self-service checkout.

[0009] The self-service terminal 100 can be configured to allow a self-service payment transaction by a user of the self-service terminal (e.g. a customer of the self-service checkout).

[0010] The self-service terminal 100 can have a scale 102. The scale 102 can have a storage area 104 on which goods 118 can be arranged. For example, in FIG.1C As shown, the storage area 104 can be configured such that the goods 118 can be arranged directly on the storage area 104. In some embodiments, these goods can be fresh produce. Fresh produce may, for example, not have a barcode. In other embodiments, these goods may be fresh and / or non-fresh produce. Non-fresh produce may, for example, have a barcode. The scale 102 can be configured to record weight data indicating the total weight of the goods arranged on the storage area 104. The scale 102 can be configured to record the weight data (and, for example, provide it to the one or more processors 110) as soon as a fluctuation of a recorded weight value for a predefined period is less than or equal to a predefined fluctuation threshold (e.g., the weight value is essentially stable for the predefined period).

[0011] In various embodiments, the shelf 104 of the scale 102 can be configured such that a shopping basket 116 can be placed on the shelf 104. In this case, the goods 118 placed on the shelf 104 can be contained in the shopping basket 116 (see, for example, Figure 1). FIG.1D Alternatively, the goods 188 can be located directly on the shelf 104 of the scale 102 (e.g., lying on it) (see, for example, FIG.1C ).

[0012] The self-service terminal 100 can have at least one imaging sensor 106. The at least one imaging sensor 106 can be configured to capture images of the storage area 104 of the scale 102 (see, for example, FIG.1A The at least one imaging sensor 106 can be configured to capture images of an environment above the storage surface 104 of the scale 102 (see, for example, FIG.1D In this way, for example, an image can be captured when a user is taking an item from the storage area 104 (e.g., the image can show the item in the user's hand). Similarly, an image can be captured in this way as a top view of the goods 118 located in the shopping basket 116.

[0013] The self-service terminal 100 can have a display device 112. The display device 112 can be oriented towards the user of the self-service terminal 100. The display device 112 can, for example, have a screen. Optionally, the display device 112 can also provide an operating function and therefore be a display and operating device. An example of a display and operating device is a touchscreen, which (as a user interface) can enable operation of the self-service terminal 100.

[0014] The self-service terminal 100 can have one or more processors 110. The one or more processors 110 can be any type of logic-implementing entity, as described herein. The one or more processors 110 can be configured to provide a graphical display and / or user interface on the display device 112. The display and / or user interface can assist the user in a self-service process at the self-service terminal 100. In some aspects, software (e.g., a point-of-sale program) for operating the self-service terminal 100 can be implemented by the one or more processors 110. In other aspects, the software can be implemented on a server (e.g., a terminal server) connected to the self-service terminal 100 via a wired and / or wireless network (e.g., via cloud processing).The one or more processors 110 can provide a corresponding display and / or user interface on the display device 112 (e.g., according to instructions received from the server) (e.g., the one or more processors can be part of a thin client).

[0015] The self-service terminal 100 can have a housing 108. The one or more processors 110 can be arranged in the housing 108 (e.g. as part of a controller).

[0016] The self-service terminal 100 may include a storage device. The one or more processors 110 may be configured to use the storage device when processing data. The storage device may include at least one memory. A memory may be, for example, volatile memory, such as DRAM (dynamic random-access memory), or non-volatile memory, such as PROM (programmable read-only memory), EPROM (erasable PROM), EEPROM (electrically erasable PROM), or flash memory, such as a floating-gate memory device, a charge-swapping memory device, MRAM (magnetoresistive random-access memory), or PCRAM (phase-change random-access memory).

[0017] It is understood that the self-service terminal 100 may have additional components. For example, the self-service terminal 100 may have a shelf 114 on which the user (e.g., customer) can place a bag, basket, etc., to pack the goods 118 from the shelf 104 of the scale 102. Furthermore, the self-service terminal 100 may have other peripheral devices. A peripheral device can be any type of device that the self-service terminal 100 can use, for example, to support a user's self-service process and / or to provide security for the self-service terminal 100. For example, a peripheral device may be one of the following: a printer (e.g., a receipt printer), a mobile, handheld, and / or stationary scanner (e.g., a barcode scanner), another camera, a card reader, an operating device (e.g., another operating device), a verification device (e.g., a digital scanner), or a digital scanner.(e.g., using facial recognition, fingerprint recognition, etc.), another scale, an antenna unit (which, for example, enables the detection and / or invalidation of radio frequency, electromagnetic, and / or acousto-magnetic tags and / or radio frequency identification (RFID) tags (e.g., for an Electronic Article Surveillance System (EAS))), etc. It is understood that the self-service terminal 100 described herein may do without another scale (e.g., a verification scale) and / or a scanner unit (e.g., a mobile, handheld, and / or a stationary (e.g., barcode) scanner) and therefore, in some embodiments, does not include such a unit.

[0018] FIG.2A bis FIG.2F Each shows aspects of a scheme according to different embodiments for identifying one or more goods that are or have been taken from the storage area 104 of the scale 102.

[0019] According to various embodiments, the self-service terminal 100 can be configured such that for goods arranged on the shelf 104 (with a number N i of goods), a corresponding image 26 can be captured by means of the at least one imaging sensor 106, and that a total weight 28 of these goods can be recorded by means of the scale 102. "N" can be any integer greater than or equal to 0. It is understood that N = 0 means that no goods are (or are no longer) arranged on the shelf 104. The self-service terminal 100 can be configured such that a respective total weight 28 and a respective corresponding image 26 are recorded each time a weight change detected by the scale 102 exceeds a detection threshold.

[0020] In some embodiments (see for example FIG.2A The scale 102 can be configured to detect a weight change 22 (in some aspects also referred to as a change in a detected weight value). In response to the detection of the weight change 22, the scale 102 can be configured to provide the total weight 28 of the goods N i located on the storage area 104 to the one or more processors 110. The scale 102 can be configured to provide a trigger signal 24 to the at least one imaging sensor 106 in response to the detection of the weight change 22. The at least one imaging sensor 106 can be configured to capture the image 26 showing the goods N i on the storage area 104 in response to the trigger signal 24. The at least one imaging sensor 106 can be configured to provide the image 26 to the one or more processors 110.

[0021] In other embodiments (see for example FIG.2B The scale 102 can be configured to provide the total weight 28 of the goods N i located on the shelf 104 to the one or more processors 110, and the one or more processors 110 can be configured to determine, using a previously provided total weight 21 of goods N i-1 previously located on the shelf 104, whether (e.g., that) the weight change 22 has occurred. The one or more processors 110 can be configured, in response to the detection of the weight change 22, to provide the trigger signal 24 to the at least one imaging sensor 106. The at least one imaging sensor 106 can then, in response to the trigger signal 24, capture the image 26 showing the goods N i on the shelf 104.

[0022] In this way, the scale 102 can be configured to record respective weight data as soon as a change in the total weight of goods arranged on the storage surface 104 is greater than or equal to the (predefined) recording threshold. In response to the recording of respective weight data 28, the one or more processors 110 or the scale 102 can then provide a respective trigger signal 24 to the at least one imaging sensor 106, so that it records an image 26 of the storage surface 104 corresponding to the respective weight data 28.

[0023] As in FIG.2C As shown, in each iteration, i, in which M goods (where "M" can be any integer greater than or equal to one; i.e., M goods can be one or more goods) are removed from the storage area 104, a corresponding total weight 28 of the goods arranged on the storage area 104 and a corresponding image 26 showing the goods arranged on the storage area 104 can be recorded. Consequently, in a subsequent iteration, i+1, N i - M remain on the storage area 104.

[0024] According to various embodiments, the self-service terminal 100 can be set up to follow this scheme until no goods remain on the storage surface of the scale 102 (i.e., N i = 0).

[0025] Therefore, in one iteration, i-1, the one or more processors 110 can receive a first total weight, G i-1, of N i-1 goods from the scale 102 and a corresponding first image, B i-1, showing the N i-1 goods on the shelf 104, from at least one imaging sensor 106. In a (directly subsequent) iteration, i, (e.g., after the user removes M goods), the one or more processors 110 can then receive a second total weight, G i, of N i goods from the scale 102 and a corresponding second image, B i, showing the N i goods on the shelf 104, from at least one imaging sensor 106.

[0026] The one or more processors 110 can be configured to determine, from the first image, B i-1, of the at least one imaging sensor 106, which N i-1 goods are arranged on the storage surface 104. The one or more processors 110 can be configured to determine, from the second image, B i, of the at least one imaging sensor 106, which one or more (M) goods have been removed from the storage surface 104. FIG.2D bis FIG.2F Figure 200 shows a processing scheme for this purpose according to various embodiments. As described, the one or more processors 110 can be configured to determine, from an image of the at least one imaging sensor 106, which goods are arranged on the storage surface 104. This can be understood to mean that the one or more processors 110 can be configured to determine, for each item shown in the image, what type of goods it is. A product can be characterized by its type. For example, in the case of vegetables, the type of goods can indicate whether the product is a red bell pepper, a green bell pepper, a cucumber, a carrot, etc. In the case of non-perishable goods, the type of goods can be a precise product identifier (such as a barcode).

[0027] With reference to FIG.2D The one or more processors 110 can be configured (in 204) to determine, from the first image, B i-1, for each item, n i-1, of the N i-1 items on the storage area 104, a respective item type, W(n i-1). The one or more processors 110 can be configured (in 202) to determine, from the second image, B i, for each item, ni, of the N i items on the storage area 104, a respective item type, W(ni). The one or more processors 110 can be configured to determine the respective item type W(n) for the N items shown in an image (with 1 ≤ n ≤ N) using a computer vision-based approach. For example, the one or more processors 110 can be configured to implement an algorithm for semantic image segmentation.In semantic image segmentation, the one or more processors 110 can divide an image into several image segments, with each of the multiple image segments being assigned to an object (e.g., a product) (consequently, each pixel can be assigned to exactly one object). In this way, objects, such as products on the storage area 104, can be recognized. The one or more processors 110 can also classify the pixels of each image segment with respect to a product type. In this way, products can be recognized and assigned a respective product type. Consequently, the one or more processors 110 can also determine position data (e.g., using the pixel positions) that indicates the position of the respective products on the storage area 104 of the scale 102. The storage area 104 itself can also be an object to be classified.

[0028] The one or more processors 110 can be configured to determine, using the product types of the N i-1 goods on the storage area 104 determined (in 204) and the product types of the N i goods on the storage area 104 determined (in 202), which product or products (e.g., products of which product type(s)) have been removed (e.g., taken away) from the storage area 104. The one or more processors 110 can be configured to determine (in 206) whether more than (exactly) one product has been removed from the storage area 104 of the scale 102. If the one or more processors 110 determine (in 206) that exactly one product has been removed from the storage area 104, the processing scheme 200 can be applied to the one described in FIG.2E The processing scheme 200A shown is followed. If the one or more processors 110 (in 206) determine that several items have been removed from the storage area 104, the processing scheme 200 can be followed as shown. FIG.2F Follow the processing scheme shown in 200B.

[0029] According to various embodiments, the one or more processors 110 can be configured to receive product data 212 (e.g., to retrieve it from the storage device). The product data 212 can specify a weight-based price (e.g., a price per kilogram) or a unit price for each product (e.g., each product that can be purchased in the associated store). This can be achieved by having the product data 212 specify a weight-based price (e.g., a price per kilogram) or a unit price for each type of product (e.g., each type of product that can be purchased in the associated store). In some embodiments, the product data 212 can also specify a corresponding target weight for each product or type of product for which a unit price is specified.

[0030] With reference to FIG.2E The one or more processors 110 (in 208) can determine the product type W(M) of the (exactly one) withdrawn item, M (e.g., using semantic image segmentation, as described herein). The one or more processors 110 can be configured to determine whether the product data 212 for the product type W(M) determined (in 208) of the withdrawn item, M, specifies a weight-based price or a unit price. If the product data 212 specifies a unit price for the product type W(M), the one or more processors 110 (in 214) can determine the unit price as the price of the withdrawn item, M. The one or more processors 110 can be set up if the goods data 212 for the goods type W(M) specifies a weight-dependent price, (in 210) to determine a difference between the first total weight, G i-1 , of the N i-1 goods and the second total weight, G i , of the N i goods as the weight of the withdrawn goods, M.In this case (in 214), the one or more processors 110 can determine the price of the withdrawn goods, M, as the product of the weight-dependent price and the weight of the withdrawn goods, M.

[0031] With reference to FIG.2F The one or more processors 110 (in 218) can determine a respective product type W(m) for each product, m, of the multiple products (with a number M) taken (e.g., by means of semantic image segmentation, as described herein). The one or more processors 110 can be configured to determine (in 220) whether all products of the M taken have the same product type W(m). According to various embodiments (as also in FIG.2F (as shown), the one or more processors 110 can be configured to provide instructions 222 to one or more output devices if (in 220) it is determined that at least one of the multiple (M) goods has a different type of good than the other goods. The self-service terminal 100 can have the one or more output devices. An output device can be any type of output device capable of providing information to the user of the self-service terminal 100. The information can be provided by a respective output device of the one or more output devices, for example, acoustically (e.g., by means of a loudspeaker) and / or visually (e.g., by means of a screen).Instructions 222 can instruct the user (by means of one or more output devices) to place the multiple items back onto the tray 104 of the scale 102 and to remove only items of the same type simultaneously from the tray 104 of the scale 102. If, for example, several different items are removed, these items cannot be assigned a respective partial weight of the total weight of the items removed (determined in 224).

[0032] In some embodiments, the one or more processors 110 can be configured to determine, using the product data 212, whether each type of product W(m) of the multiple (M) withdrawn goods has a unit price assigned to it. In this case, the one or more processors 110 can then determine the respective unit price as the price of the corresponding product. The one or more processors 110 can also be configured to determine, using the product data 212, whether each type of product W(m) of the multiple (M) withdrawn goods has the same weight-based price assigned to it. In this case, the one or more processors 110 cannot determine an individual price for each of the multiple withdrawn goods, but can determine a common price for the multiple withdrawn goods based on the product of the (same) weight-based price and the total weight of the M withdrawn goods determined (in 224).In these configurations, the one or more processors 110 can be set up to provide the instructions 222 if the goods data 212 do not specify a unit price for all types of goods withdrawn and / or if the goods data 212 do not specify the same weight-dependent price for all types of goods withdrawn.

[0033] With reference to FIG.2F The one or more processors 110 can be configured (in 220) to determine, using the goods data 212, whether the goods data 212 specifies a unit price or a weight-based price for this goods type W(m). If the goods data 212 specifies a unit price for the (same) goods type W(m), the one or more processors 110 (in 226) can determine the price of the multiple goods, M, withdrawn from the product of the unit price and the number, M, (in some aspects also referred to as the quantity of goods) withdrawn.The one or more processors 110 can be configured, if the goods data 212 specifies a weight-dependent price for the (same) goods type W(m), to determine (in 224) a difference between the first total weight, G i-1, of the N i-1 goods and the second total weight, G i, of the N i goods as the weight of the several (M) withdrawn goods. In this case, the one or more processors 110 can determine (in 2226) the price of the several withdrawn goods as the product of the weight-dependent price and the weight of the several withdrawn goods.

[0034] As with reference to FIG.2C As described, the processing scheme 200 can be carried out iteratively until no more goods are arranged on the storage area 104. FIG.3 Figure 300 illustrates an exemplary flowchart for removing goods from the storage area. The user can place a variety of goods (in this example, 2 lemons, a green bell pepper, a red apple, and a banana) onto the storage area 104 of the scale 102 (in Figure 302), for example, to pay for the goods. Therefore, in the first iteration, Ni = 5 goods are on the storage area 104. The one or more processors 110 can determine the respective type of goods (i.e., lemon, green bell pepper, red apple, banana) using the corresponding image, Bi. In this example, the user removes exactly one item (M = 1) (in Figure 304). The one or more processors 110 can determine, using the weight data, that at least one item has been removed and, using the image, Bi+1, can determine that the red apple has been removed.The one or more processors 110 can therefore determine, according to the processing scheme 200A, that the red apple has a weight-dependent price and can determine the weight of the red apple from the difference between the weight data, Gi, of the first iteration and the weight data, Gi+1, of the second iteration, and can determine the price as the product of the weight-dependent price and the weight of the red apple. According to various embodiments, the one or more processors 110 can be configured to provide corresponding instructions to the display device 112, which instruct the display device 112 to display the determined price for the red apple. In this example, the user (in 306) removes several (namely two) items (M = 2).The one or more processors 110 can determine, using the weight data, that at least one item has been removed and, using the image, B i+2, can determine that two lemons have been removed. The one or more processors 110 can then, according to the processing scheme 200B, determine that the two items are of the same type (lemon) and can then determine from the item data 212 that lemons have a unit price, and can calculate the price for the two lemons as the product of the quantity (i.e., 2) and the unit price for lemons. The one or more processors 110 can then provide appropriate instructions to the display device 112, which instruct the display device 112 to display the calculated price for the two lemons. In this example, the user (in 308) then removes exactly one item (M = 1).The one or more processors 110 can determine, using the weight data, that at least one item has been removed and, using the image, B i+3, can determine that the banana was removed. Therefore, according to the processing scheme 200A, the one or more processors 110 can determine that the banana has a weight-based price and can calculate the weight of the banana from the difference between the weight data, G i+2, of the third iteration and the weight data, G i+3, of the fourth iteration. They can then calculate the price of the banana as the product of the weight-based price and the weight of the banana. The one or more processors 110 can subsequently provide appropriate instructions to the display device 112, which instruct the display device 112 to display the determined price for the banana. In this example, the user (in 310) then removes exactly one item (M = 1).The one or more processors 110 can determine, using the weight data, that at least one item has been removed and, using the image, B i+4, can determine that the green bell pepper has been removed. The one or more processors 110 can therefore determine, according to the processing scheme 200A, that the green bell pepper has a weight-dependent price and can determine the weight of the green bell pepper from the difference between the weight data, G i+3, of the fourth iteration and the weight data, G i+4, of the fifth iteration. They can then determine the price of the green bell pepper as the product of the weight-dependent price and the weight of the green bell pepper. The one or more processors 110 can subsequently provide appropriate instructions to the display device 112, which instruct the display device 112 to display the determined price for the green bell pepper. The one or more processors 110 can then, as in . FIG.2C shown, determine that no more goods are arranged on the shelf.

[0035] According to various embodiments, the one or more processors 110 can be configured to determine a total price as the sum of the determined prices. The one or more processors 110 can be configured to provide the display device 112 with the corresponding information for displaying the total price. FIG.4 shows an example display on the display device 122 for the in FIG.3 The example shown illustrates this. The self-service terminal 100 can include a payment device (e.g., a card reader). According to various embodiments, the one or more processors 110 can be configured to provide instructions for paying the total price to the payment device.

[0036] According to various embodiments, the one or more processors 110 can provide corresponding instructions so that the display device 112 can show the types of goods determined (in 204) (e.g., initially placed on the storage area 104), and / or the respective price determined (in 214 or 226), and / or the unit price or weight-dependent price specified by the goods data 212 for the types of goods, and / or the determined total price. Optionally, the display device 112 can show information for one or more types of goods indicating that the price for the respective type of goods has not yet been determined.

[0037] FIG.5 Figure 500 shows a processing scheme similar to processing scheme 200 according to various embodiments. As described herein, the at least one imaging sensor 106 can be configured to capture an image, Bi, of an area above the storage surface 104 in each iteration, i. In this way, when the user removes an item (M = 1) from the storage surface 104, the at least one imaging sensor 106 can capture an image showing the item in the user's hand (an exemplary image is shown in Figure 1). FIG.6 (shown, with a lemon in the user's hand). The user can take the item directly from the shelf or from the shopping basket. The one or more processors 110 can be configured to determine from the respective image, B i , (in 502) which item is in the user's hand (i.e., which type of item W(M) item M is). Semantic image segmentation can be used to determine which image segment shows a hand, so that a corresponding adjacent image segment showing an item can be recognized as the item in the hand. As described with reference to processing scheme 200A, the one or more processors 110 can use the product data 212 to determine whether the product type W(M) has a unit price or a weight-based price.The one or more processors 110 can then (in 506) determine the unit price as the price of the good M in the case of the unit price and can determine the weight of the withdrawn good from the difference of the weight data in the case of the weight-dependent price (in 504) and determine the price (in 506) as the product of weight and weight-dependent price.

[0038] The processing schemes described herein can clearly demonstrate an inverse weighing of goods.

[0039] According to various embodiments, the one or more processors 110 can be configured to determine, already in an initial iteration (i.e., when the goods are placed on the storage area 104 by the user), whether the product data 212 specifies a unit price for one or more of these goods. In this case, the price for these goods can be determined in the initial iteration using the unit price. Optionally, this price can therefore be displayed on the display device 112 in the initial iteration.

[0040] According to various embodiments, the one or more processors 110 can be configured to verify a determined type of goods.

[0041] FIG.7A Figure 700A shows a first processing scheme for verifying the identified type of goods for a withdrawn item according to various embodiments. According to these embodiments, at least one imaging sensor 106 can be configured to capture both the image above the storage surface 104, which may show the item M in the user's hand, and the image of the storage surface itself. Therefore, the one or more processors 110 can identify a first type of goods 208 according to processing scheme 200A and a second type of goods 502 according to processing scheme 500 for the item M = 1 withdrawn. The one or more processors 110 can be configured (in 702) to compare the first type of goods 208 with the second type of goods 502 and can (in 706) verify the first type of goods 208 as the type of goods for item M if the first type of goods 208 corresponds to the second type of goods 502.Otherwise, the one or more processors 110 (in 704) cannot verify the type of goods. In this case, the one or more processors 110 can, for example, provide instructions to the one or more output devices instructing the user to place the goods back on the storage surface (e.g., in the shopping cart, if present).

[0042] FIG.7B Figure 700B shows a second processing scheme for verifying a determined product type of a withdrawn product according to various embodiments. As described herein, the product data 212 can specify an associated target weight for each product type for which a unit price is given. As described herein, the one or more processors 110 can determine, for determining the price of one or more products, whether the product data 212 specifies a unit price for each product type. If the product data 212 specifies a unit price for product type ("Yes" in 708), the one or more processors 110 (in 710) can determine whether the weight G(M) of the one withdrawn product (210 or 504) or of the several withdrawn products (224) substantially corresponds to the target weight.It is understood that in the case of multiple items of the same type (i.e., processing scheme 200B), an expected weight for these multiple items can be determined as the product of the target weight for the (same) type of item and the number of items, M. If exactly one item is removed, the expected weight can correspond to the target weight. The expected weight can substantially match the weight G(M) of the single item removed (210 or 504) or of the multiple items removed (224) if the difference between the expected weight and the weight G(M) is less than or equal to a predefined verification threshold. If this is the case, the determined type of item (in 712) can be verified. Otherwise, the one or more processors 110 can be configured not to verify the type of item.In this case, the one or more processors 110 can, for example, provide instructions to the one or more output devices, instructing the user to place the item(s) back on the storage surface (e.g., in the shopping basket, if available).

[0043] As described herein, the self-service terminal 100 may optionally include a scanner unit (e.g., a barcode scanner). According to a third processing scheme for verifying the identified product type of a dispensed item, the one or more processors 110 may be configured, when a barcode of the dispensed item is captured by the scanner unit, to receive information representing the captured barcode from the scanner unit. The one or more processors 110 may be configured to use this information to identify the product type of the dispensed item and then compare it with the product type identified according to processing scheme 200A (or 200B) or processing scheme 500 to verify the product type if there is a match.

[0044] It is understood that a combination of the first verification processing scheme 700A and / or the second verification processing scheme 700B and / or the third verification processing scheme can be used.

[0045] If the one or more processors 110, as described herein, provide instructions directing the user to place one or more items back onto the storage area 104, the respective processing scheme may stop until the item(s) have been placed back onto the storage area 104.

[0046] FIG.8 shows a flowchart of a method 800 for determining whether and if so, which goods or goods have been taken from a scale of a self-service terminal, according to various embodiments.

[0047] Method 800 may include determining, from at least one first image of a storage area of ​​the scale, which goods are arranged on the storage area (in 802).

[0048] Method 800 may include determining, using first weight data indicating a first total weight of the goods located on the shelf, and second weight data indicating a second total weight of goods located on the shelf, whether one or more goods have been removed from the shelf (in 804).

[0049] Method 800 may include determining from at least one second image of the storage area which one or more items have been taken from the storage area (in 806).

[0050] The respective determination (in 802, 804 and / or 806) can, for example, be carried out as in relation to FIG.2A bis FIG.2F described below.

[0051] FIG.9 shows a flowchart of a method 900 for determining whether and if so, which goods or goods have been taken from a scale of a self-service terminal, according to various embodiments.

[0052] Method 900 may involve determining, using first weight data indicating a first total weight of goods located on a platform of a scale, and second weight data indicating a second total weight of goods located on the platform, whether any goods have been removed from the platform (in 902).

[0053] Method 900 may include determining, from at least one image of an environment above the storage area, whether a product, and if so, which product, is in the hand of a user (in 904).

[0054] Method 900 may involve identifying the goods that are in the user’s hand as the goods removed from the storage area (in 906).

[0055] The respective determination (in 902, 904 and / or 906) can, for example, be carried out as in relation to FIG.5 described below.

[0056] This document describes various aspects of the self-service terminal 100 and procedures 800 and 900. It is understood that aspects described in relation to the self-service terminal 100 can also be applied to the procedures (e.g., they can be executed as procedures).

[0057] The following examples describe various aspects of the self-service terminal 100 and the procedures 800, 900.

[0058] Example 1 is a self-service terminal comprising: a scale with a shelf on which goods (e.g., fresh produce such as fruit, vegetables, bread, etc.) can be arranged, wherein the scale is configured to record weight data indicating a total weight of goods arranged on the shelf; at least one imaging sensor configured to record images of the shelf of the scale; one or more processors configured to: determine from at least one initial image of the at least one imaging sensor which goods are arranged on the shelf; receive initial weight data from the scale, wherein the initial weight data indicates a first total weight of the goods located on the shelf; receive second weight data from the scale, wherein the second weight data indicates a second total weight of goods located on the shelf;to determine, using the first weight data and the second weight data, whether one or more items have been removed from the storage area; and (if it is determined that one or more items have been removed from the storage area) to determine from at least one second image from the at least one imaging sensor which one or more items have been removed from the storage area.

[0059] Example 2 is set up according to Example 1, where the second total weight indicates a total weight of the goods that are arranged on the shelf when at least one second image is captured.

[0060] Example 3 is a self-service terminal according to Example 1 or 2, wherein the one or more processors are set up to determine, when it is determined from at least one second image that (exactly) one item has been removed from the storage area, whether the product data, which specifies a weight-dependent price or a unit price for each item, specifies a weight-dependent price or a unit price for the removed item.

[0061] Example 4 is set up according to Example 3, wherein the one or more processors are set up when it is determined that the goods data for the withdrawn goods indicate a weight-based price: to determine a difference between the first total weight and the second total weight as the weight of the withdrawn goods; and to determine a price of the withdrawn goods using the weight-based price and the weight of the withdrawn goods.

[0062] Example 5 is a self-service terminal according to Example 3 or 4, wherein one or more processors are set up to determine, when it is determined that the goods data for the withdrawn goods indicate a unit price, the unit price as the price of the withdrawn goods.

[0063] Example 6 is set up according to Example 5, wherein the goods data further specifies a target weight for each item for which a unit price is given; and wherein the one or more processors are set up, when it is determined that the goods data specifies a unit price for the withdrawn item: to determine a difference between the first total weight and the second total weight as the weight of the withdrawn item; to determine whether a difference between the target weight specified by the goods data and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold; and, if it is determined that the difference is less than or equal to the predefined verification threshold, to verify the withdrawn item determined from the at least one second image and to determine the unit price as the price of the withdrawn item.

[0064] Example 7 is a self-service terminal according to one of Examples 1 to 6, wherein the one or more processors are set up to determine, when it is determined from at least one second image that several goods have been removed from the storage area: whether each of the several goods belongs to the same type of goods; and when it is determined that each of the several goods belongs to the same type of goods, whether the goods data, which specify a weight-based price or a unit price for each type of goods, specify a weight-based price or a unit price for the type of goods of the several goods.

[0065] Example 8 is set up according to Example 7, wherein the one or more processors are set up when it is determined that the goods data for the type of goods of the multiple goods specify a weight-dependent price: to determine a difference between the first total weight and the second total weight as the weight of the multiple goods; and to determine a price of the multiple goods using the weight-dependent price and the weight of the multiple goods.

[0066] Example 9 is a self-service terminal according to Example 7 or 8, wherein one or more processors are set up when it is determined that the product data for the product type of the multiple products specifies a unit price, to determine the price of the multiple products using the unit price and a quantity of the multiple products.

[0067] Example 10 is set up according to Example 9, wherein the goods data further specifies, for each type of goods for which a unit price is given, a respective target weight for one item of that type; and wherein the one or more processors are set up, when it is determined that the goods data specifies the unit price for the type of goods of the multiple goods: to determine a difference between the first total weight and the second total weight as the weight of the multiple goods; to determine an expected weight of the multiple goods using the number of items of the multiple goods and the target weight specified by the goods data for one item of the type of goods of the multiple goods; to determine whether a difference between the expected weight and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold;and if it is determined that the difference is less than or equal to the predefined verification threshold, to verify the product type as the product type of the multiple goods and to determine the price of the multiple goods using the unit price specified for this product type in the product data and the quantity of the multiple goods.

[0068] Example 11 is a self-service terminal according to one of Examples 7 to 10, wherein the one or more processors are configured: when it is determined that at least one item of the multiple items has a different type of goods than the other items of the multiple items, to provide instructions to an output device (e.g. of the self-service terminal) that instruct a user of the self-service terminal to place the multiple items back onto the platform of the scale and to remove only items of the same type from the platform of the scale at the same time.

[0069] Example 12 is configured according to one of Examples 1 to 11, wherein the one or more processors are configured: when they receive the first weight data from the scale, to provide a first trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to capture the at least one first image of the storage surface, and when they receive the second weight data from the scale, to provide a second trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to capture the at least one second image of the storage surface.

[0070] Example 13 is a self-service terminal according to any one of Examples 1 to 11, wherein the scale is configured: when it acquires the first weight data, to provide a first trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to acquire the at least one first image of the storage surface, and when it acquires the first weight data, to provide a first trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to acquire the at least one second image of the storage surface.

[0071] Example 14 is set up according to one of Examples 1 to 13, wherein the scale is set up to provide the recorded weight data to the one or more processors as soon as a fluctuation of a recorded weight value for a predefined period is less than or equal to a predefined fluctuation threshold (e.g. the weight value is essentially stable for the predefined period).

[0072] Example 15 is a self-service terminal according to one of Examples 1 to 14, wherein the scale is configured to record respective weight data and provide it to the one or more processors as soon as a change in the total weight of goods arranged on the shelf is greater than or equal to a predefined recording threshold; wherein the scale or the one or more processors are configured, in response to the recording of respective weight data, to provide a respective trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to record an image of the shelf corresponding to the respective weight data;and wherein the one or more processors are configured to determine, using the weight data recorded and the weight data previously recorded, whether one or more items have been removed from the storage area and, using the corresponding images, to determine which one or more items have been removed from the storage area.

[0073] Example 16 is set up according to Example 15, wherein the self-service terminal is set up to determine the one or more goods removed until no goods remain on the scale's shelf.

[0074] Example 17 is set up according to Example 16, wherein the one or more processors are set up to determine, based on the goods identified from the at least one first image, a respective price for the one or more goods taken (e.g. according to one of Examples 4 to 10) and to determine a total price for the goods identified from the at least one first image as the sum of the determined prices.

[0075] In Example 18, the self-service terminal according to Example 17 may optionally further include: a payment device; wherein one or more processors are configured to provide instructions to the payment device for paying the total price.

[0076] In Example 19, the self-service terminal according to one of Examples 1 to 18 may optionally further include: a display device which is configured to display each type of goods which includes at least one item of the goods identified from the at least one first image (e.g. before one or more items are removed from the platform of the scale).

[0077] Example 20 is set up according to Example 19, wherein the display device is set up to further display, for each type of goods: if in combination with one of Examples 4 to 10, the price determined for the one or more goods; and / or a number of goods that exhibit the respective type of goods in the goods determined from the at least one first image; and / or if in combination with one of Examples 3 to 10, the unit price or weight-based price assigned to the respective type of goods; and / or if in combination with Example 17, the total price determined.

[0078] Example 21 is set up according to one of Examples 1 to 20, wherein the scale is set up in such a way that a shopping basket in which the goods can be arranged can be placed on the shelf.

[0079] Example 22 is a self-service terminal according to one of Examples 1 to 21, wherein the at least one imaging sensor includes a camera and / or a video camera.

[0080] Example 23 is a self-service terminal comprising: a scale with a shelf on which goods can be arranged, wherein the scale is configured to record weight data indicating a total weight of goods arranged on the shelf; at least one imaging sensor configured to record images of the area around the self-service terminal above the shelf of the scale; one or more processors configured to: receive initial weight data from the scale, wherein the initial weight data indicates an initial total weight of goods located on the shelf; receive second weight data from the scale, wherein the second weight data indicates a second total weight of goods located on the shelf; and, using the initial weight data and the second weight data, determine whether an item has been removed from the shelf.If it is determined that (exactly) one item has been removed from the shelf, it is necessary to determine from at least one image from at least one imaging sensor (e.g., using semantic image segmentation) whether an item, and if so, which item, is in the hand of a user of the self-service terminal; and to identify the item in the hand of the user of the self-service terminal as the item removed from the shelf.

[0081] Example 24 is set up according to Example 23, wherein the scale is set up in such a way that a shopping basket can be placed on the shelf, in which the goods placed on the shelf can be located.

[0082] Example 25 is a self-service terminal according to Example 23 or 24, wherein the one or more processors are set up to determine whether the goods data, which specify a weight-dependent price or a unit price for each item, specify a weight-dependent price or a unit price for the item taken.

[0083] Example 26 is set up according to Example 25, wherein the one or more processors are set up when it is determined that the goods data for the withdrawn goods indicate a weight-based price: to determine a difference between the first total weight and the second total weight as the weight of the withdrawn goods; and to determine a price of the withdrawn goods using the weight-based price and the weight of the withdrawn goods.

[0084] Example 27 is a self-service terminal according to Example 25 or 26, wherein one or more processors are set up to determine, when it is determined that the goods data for the withdrawn goods indicate a unit price, the unit price as the price of the withdrawn goods.

[0085] Example 28 is set up according to Example 27, wherein the goods data further specifies, for each item for which a unit price is given, a respective target weight of such item; and wherein the one or more processors are set up, when it is determined that the goods data specifies a unit price for the item taken: to determine a difference between the first total weight and the second total weight as the weight of the item taken; to determine whether a difference between the target weight specified by the goods data and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold; and, if it is determined that the difference is less than or equal to the predefined verification threshold, to verify the item taken from the at least one image and to determine the unit price as the price of the item taken.

[0086] Example 29 is a self-service terminal according to one of Examples 23 to 28, wherein the one or more processors are configured: when they receive the first weight data from the scale, to provide a first trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to capture the at least one first image of the storage surface, and when they receive the second weight data from the scale, to provide a second trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to capture the at least one second image of the storage surface.

[0087] Example 30 is set up according to one of Examples 23 to 28, wherein the scale is set up: when it acquires the first weight data, to provide a first trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to acquire the at least one first image of the storage surface, and when it acquires the first weight data, to provide a first trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to acquire the at least one second image of the storage surface.

[0088] Example 31 is a self-service terminal according to one of Examples 23 to 30, wherein the scale is configured to provide the recorded weight data to the one or more processors as soon as a fluctuation of a recorded weight value for a predefined period is less than or equal to a predefined fluctuation threshold (e.g. the weight value is essentially stable for the predefined period).

[0089] Example 32 is configured according to one of Examples 23 to 31, wherein the scale is configured to acquire respective weight data and provide it to the one or more processors as soon as a change in the total weight of goods arranged on the storage surface is greater than or equal to a predefined acquisition threshold; wherein the scale or the one or more processors are configured, in response to the acquisition of respective weight data, to provide a respective trigger signal to the at least one imaging sensor, which causes the at least one imaging sensor to acquire an image of the storage surface corresponding to the respective weight data;and wherein the one or more processors are set up to determine, using the weight data recorded and the weight data recorded previously, whether an item has been removed from the storage area and, using the corresponding images, to determine which item was removed.

[0090] Example 33 is set up according to Example 32, wherein the self-service terminal is set up to determine the one or more goods removed until no goods remain on the scale's storage surface.

[0091] Example 34 is configured according to Examples 24 and 33, wherein the one or more processors are configured to: determine from the at least one image or another image from the at least one imaging sensor whether the goods located on the shelf are lying directly on the shelf or are arranged in the shopping basket; and if it is determined that the goods are arranged in the shopping basket, determine that no goods remain on the shelf of the scale when the total weight recorded by the scale is substantially equal to a weight value predefined for the shopping basket.

[0092] Example 35 is a self-service terminal according to Example 33 or 34, wherein one or more processors are set up to determine, from a multitude of goods, the respective price of each item removed and to determine a total price for the multitude of goods as the sum of the determined prices.

[0093] In Example 36, the self-service terminal according to Example 35 may optionally further include: a payment device; wherein one or more processors are configured to provide instructions to the payment device for paying the total price.

[0094] In Example 37, the self-service terminal according to one of Examples 23 to 36 may optionally also include: a display device which is set up to indicate a type of goods of the goods taken.

[0095] Example 38 is set up according to Example 37, wherein the display device is set up to display: if in combination with one of Examples 26 to 28, the price determined for the withdrawn goods; and / or if in combination with Example 27 or 28, the assigned unit price or weight-based price; and / or if in combination with Example 35, the determined total price.

[0096] Example 39 is set up according to one of Examples 23 to 38, wherein the at least one imaging sensor includes a camera and / or a video camera.

[0097] Example 40 is a self-service terminal according to one of Examples 23 to 39, wherein the at least one imaging sensor is further configured to capture images of the storage surface of the scale; wherein the one or more processors are configured to determine, from at least one first storage surface image received by the at least one imaging sensor and captured before the at least one image, which one or more first goods are located on the storage surface (e.g.to determine, from at least one second image of the storage area (received by the at least one imaging sensor and captured after the first image), which one or more second items the second image of the storage area shows; to determine, using the one or more first items and the one or more second items, which item was removed from the storage area; and, if the removed item determined using the one or more first items and the one or more second items corresponds to the removed item determined from the at least one image, to verify the removed item.

[0098] Example 41 is set up according to one of Examples 23 to 40, wherein the goods identified from the at least one image have a first goods type; wherein the self-service terminal has a scanner unit for capturing barcodes; wherein the one or more processors are configured: when a barcode of the withdrawn goods is captured by the scanner unit, to receive information representing the captured barcode from the scanner unit and to determine a second goods type of the withdrawn goods on the basis of the captured barcode; and to verify the first goods type as the goods type of the withdrawn goods if the first goods type corresponds to the second goods type.

[0099] Example 42 is set up according to one of Examples 23 to 40, wherein the self-service terminal does not have a scanner unit.

[0100] Example 43 is set up according to one of Examples 23 to 42, wherein the self-service terminal does not have any other scale (e.g., no verification scale).

[0101] Example 44 is a method comprising: determining, from at least one first image of a scale's tray, which goods are arranged on the tray; determining, using first weight data indicating a first total weight of the goods located on the tray, and second weight data indicating a second total weight of goods located on the tray, whether one or more goods have been removed from the tray; and determining, from at least one second image of the tray, which one or more goods have been removed from the tray.

[0102] Example 45 is set up according to Example 44, wherein the second total weight indicates a total weight of the goods that are arranged on the shelf when at least one second image is detected.

[0103] Example 46 is a method according to Example 44 or 45, further comprising: if it is determined from at least one second image that (exactly) one item has been removed from the storage area, determining whether item data which specifies a weight-dependent price or a unit price for each item specifies a weight-dependent price or a unit price for the removed item.

[0104] In Example 47, the procedure according to Example 46 may optionally further include: if it is determined that the goods data for the withdrawn goods indicate a weight-dependent price: determining a difference between the first total weight and the second total weight as the weight of the withdrawn goods; and determining a price of the withdrawn goods using the weight-dependent price and the weight of the withdrawn goods.

[0105] In Example 48, the procedure according to Example 46 or 47 may optionally further include: if it is determined that the goods data for the withdrawn goods indicate a unit price, determining the unit price as the price of the withdrawn goods.

[0106] Example 49 is set up according to Example 48, wherein the goods data further specifies, for each item for which a unit price is given, a respective target weight of such item; and wherein, if it is determined that the goods data specifies a unit price for the item taken, the procedure further comprises: determining a difference between the first total weight and the second total weight as the weight of the item taken; determining whether a difference between the target weight specified by the goods data and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold; and if it is determined that the difference is less than or equal to the predefined verification threshold, verifying the item taken, as determined from the at least one second image, and determining the unit price as the price of the item taken.

[0107] In Example 50, the procedure according to one of Examples 44 to 49 may optionally further include: if it is determined from the at least one second image that several goods have been removed from the storage area: determine whether each of the several goods belongs to the same type of goods; and if it is determined that each of the several goods belongs to the same type of goods, determine whether the goods data, which specify a weight-based price or a unit price for each type of goods, specify a weight-based price or a unit price for the type of goods of the several goods.

[0108] In Example 51, the procedure according to Example 50 may optionally further include: if it is determined that the goods data for the type of goods of the multiple goods indicate a weight-dependent price: determining a difference between the first total weight and the second total weight as the weight of the multiple goods; and determining a price of the multiple goods using the weight-dependent price and the weight of the multiple goods.

[0109] In Example 52, the procedure according to Example 50 or 51 may optionally further include: if it is determined that the goods data for the type of goods of the multiple goods specify a unit price, determining the price of the multiple goods using the unit price and a quantity of the multiple goods.

[0110] Example 53 is set up according to Example 52, wherein the goods data further specifies, for each type of goods for which a unit price is given, a respective target weight for one item of that type; and wherein, if it is determined that the goods data specifies the unit price for the type of goods of the multiple goods, the procedure further comprises: determining a difference between the first total weight and the second total weight as the weight of the multiple goods; determining an expected weight of the multiple goods using the number of items of the multiple goods and the target weight specified by the goods data for one item of the type of goods of the multiple goods; determining whether a difference between the expected weight and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold;and if it is determined that the difference is less than or equal to the predefined verification threshold, verify the product type as the product type of the multiple goods and determine the price of the multiple goods using the unit price specified for this product type in the product data and the quantity of the multiple goods.

[0111] In Example 54, the procedure according to one of Examples 50 to 53 may optionally further include: if it is determined that at least one of the several goods has a different type of goods than the other goods of the several goods, providing instructions that instruct a user to place the several goods back onto the platform of the scale and to remove only goods of the same type from the platform of the scale at the same time.

[0112] In Example 55, the method according to one of Examples 44 to 54 may optionally further include: capturing respective weight data as soon as a change in the total weight of goods arranged on the storage area is greater than or equal to a predefined capture threshold; in response to the capture of respective weight data, capturing an image of the storage area corresponding to the respective weight data; and determining, using the respective captured weight data and the respective previously captured weight data, whether one or more goods have been removed from the storage area, and determining, using the respective corresponding images, which one or more goods have been removed from the storage area.

[0113] Example 56 is set up according to Example 55, whereby the respective weight data are recorded until no goods remain on the scale's platform.

[0114] In Example 57, the procedure according to Example 56 may optionally further include: determining, based on the goods identified from the at least one first image, a respective price of each of the one or more goods taken, and determining a total price for the goods identified from the at least one first image as the sum of the determined prices.

[0115] Example 58 is a method comprising: determining, using first weight data indicating a first total weight of goods located on a platform of a scale, and second weight data indicating a second total weight of goods located on the platform, whether any goods have been removed from the platform; determining, from at least one image of an environment above the platform, whether any goods, and if so, which goods, are in a user's hand; and determining the goods located in the user's hand as being the goods removed from the platform.

[0116] In Example 59, the procedure according to Example 58 may optionally further include: Determining whether product data, which specifies a weight-dependent price or a unit price for each product, specifies a weight-dependent price or a unit price for the product withdrawn.

[0117] In Example 60, the procedure according to Example 59 may optionally further include: if it is determined that the goods data for the withdrawn goods indicate a weight-dependent price: determining a difference between the first total weight and the second total weight as the weight of the withdrawn goods; and determining a price of the withdrawn goods using the weight-dependent price and the weight of the withdrawn goods.

[0118] In Example 61, the procedure according to Example 59 or 60 may optionally further include: if it is determined that the goods data for the withdrawn goods indicate a unit price, determining the unit price as the price of the withdrawn goods.

[0119] Example 62 is set up according to Example 61, wherein the goods data further specifies, for each item for which a unit price is given, a respective target weight of such item; and wherein, if it is determined that the goods data specifies a unit price for the item taken, the procedure further comprises: determining a difference between the first total weight and the second total weight as the weight of the item taken; determining whether a difference between the target weight specified by the goods data and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold; and if it is determined that the difference is less than or equal to the predefined verification threshold, verifying the item taken, as determined from the at least one image, and determining the unit price as the price of the item taken.

[0120] In Example 63, the procedure according to one of Examples 58 to 62 may optionally further include: recording respective weight data as soon as a change in the total weight of goods arranged on the storage area is greater than or equal to a predefined recording threshold; in response to the recording of respective weight data, recording an image of the storage area corresponding to the respective weight data; and determining, using the respective recorded weight data and the respective previously recorded weight data, whether an item has been removed from the storage area, and determining, using the respective associated images, which item has been removed.

[0121] Example 64 is set up according to Example 63, whereby the respective weight data are recorded until no goods remain on the platform of the scale.

[0122] Example 65 is a method according to Examples 58 and 64, further comprising: determining, from at least one image or another image from at least one imaging sensor, whether the goods located on the shelf are lying directly on the shelf or are arranged in the shopping basket; and, if it is determined that the goods are arranged in the shopping basket, determining that no goods remain on the shelf of the scale when the total weight recorded by the scale is substantially equal to a weight value predefined for the shopping basket.

[0123] In Example 66, the method according to one of Examples 58 to 65 may optionally further include: determining, from at least one first image of the storage area captured before the at least one image, which shows the first image of one or more of the first goods located on the storage area (e.g., in the shopping basket); determining, from at least one second image of the storage area captured after the at least one image, which shows the second image of one or more of the second goods; determining, using the one or more first goods and the one or more second goods, which goods were removed from the storage area; and verifying the removed goods if the removed goods determined using the one or more first goods and the one or more second goods correspond to the removed goods determined from the at least one image.

[0124] Example 67 is set up according to one of Examples 58 to 66, wherein the goods identified from the at least one image have a first type of goods; wherein the method further comprises: capturing a barcode of the extracted goods and identifying a second type of goods of the extracted goods using the captured barcode; and verifying the first type of goods as the type of goods of the extracted goods if the first type of goods corresponds to the second type of goods.

[0125] Example 68 is a computer-readable medium (e.g., a computer program product, a non-volatile storage medium, a non-transitory storage medium, a non-volatile storage medium) which stores instructions which, when executed by a processor, cause the processor to control a self-service terminal to carry out the procedure according to any one of Examples 44 to 67.

[0126] The term "processor," as used herein, can be understood as any type of entity that permits the processing of data or signals. The data or signals can, for example, be processed according to at least one (i.e., one or more than one) specific function performed by the processor. A processor may be an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA), an integrated circuit, or any combination thereof. Any other type of implementation of the respective functions, which are described in more detail below, may also be understood as a processor or logic circuit.It is understood that one or more of the process steps described in detail herein can be executed (e.g., implemented) by a processor through one or more specific functions performed by the processor. The processor can therefore be configured to perform one of the procedures described herein or its components for information processing.

[0127] It is understood that, as described herein, when a device provides data to another device or a device receives data from another device, this data can be transferred to the respective device directly or indirectly. In the case of indirect transfer, the data can, for example, be stored in the storage device and a device can read the data from the storage device.

[0128] Several aspects described herein relate to captured images. These images can be digital images. Digital images can be any type of digital image data and can be a digital representation of reality (e.g., the detection area) at the point in time when the digital image data is captured. The representation of reality can be achieved, for example, by means of a lens that projects light onto the surface of an imaging sensor (e.g., a Bayer sensor). Capturing the image data can involve reading the imaging sensor while light is projected onto its surface. The resulting image data can initially be in the so-called raw data format (also referred to as RAW), which contains the pixel-by-pixel readings from the imaging sensor and / or can be processed as such. Optionally, the image data can be converted into another image format during processing, e.g.,The image can be converted into a raster graphic (different from RAW as a raster graphic) or a vector graphic, so that further processing takes place in this image format, or it can be converted between these formats as needed. The conversion process can optionally include interpolating the measurements from the imaging sensor (e.g., using demosaicing), for example, to obtain complete multi-color information for each pixel of a multi-pixel image or to reduce storage space or processing power. The image data can optionally be compressed (e.g., to reduce storage space or processing power) or uncompressed (e.g., to avoid distortion). The respective image format can also define the color space according to which the color information is specified.

[0129] The simplest case is a binary color space, where a black-and-white value is stored for each pixel. In a slightly more complex color space (also known as a grayscale color space), intermediate shades between black and white are stored (also known as grayscale values). The color space can also be defined by several primary colors (e.g., two or more), such as red, green, and blue. If the measured values ​​are to contain multicolored information, a wavelength-sensitive imaging sensor can be used. Its measured values ​​can be encoded according to a color space. The color information, or the underlying color space, can therefore be multicolored (also known as polychrome) or monochromatic (also known as single-color).The monochrome color information can, for example, consist only of grayscale values ​​(then also referred to as grayscale information) or black and white values ​​(then also referred to as black and white information), which represent the intensity of the detected radiation at the wavelength or wavelength range at which the monochrome sensor is sensitive. For visual display of the image data on a screen, it is converted into the image format specified by the graphics card's image memory. For ease of understanding, the image data described herein is presented as such a visual display. Generally, the image data, e.g., stored on a storage medium, can exist as a file (also referred to as a digital image or image file) in the respective image format.

[0130] Depth information (also referred to as 3D information) can also be assigned to the image data (e.g., as an alternative or in addition to color information). A 3D camera can be used as an imaging sensor to capture this depth information. The measurements from the 3D camera can contain information about the topography of the depicted reality (also referred to as depth information), e.g., pixel by pixel. For example, the depth information can indicate the distance of one or more pixels in the camera from a location in space represented by that pixel. In other words, the distance to an object, or the distances between multiple objects, can be determined using the depth information. The depth information can vividly add a third spatial dimension (referred to here as depth) to the two spatial dimensions depicted by the image data.For example, an object can be represented by the image data as a projection onto the two-dimensional surface of the image sensor (also referred to as 2D image data). Depth information provides the third spatial dimension. For example, the depth information can include values ​​(also referred to as depth values) assigned to the image data segment by segment (i.e., for each segment of the image data), indicating its depth. Alternatively, the depth information can include depth values ​​assigned to the image data pixel by pixel (i.e., for each pixel of the image data), indicating its depth.

[0131] The at least one imaging sensor 106 described herein can be any type of suitable camera and / or video camera. An imaging sensor (also referred to as an image sensor) is of the optical sensor type and can have one or more photoelectrically active areas (which can also be referred to as pixels) that generate and / or modify an electrical signal, for example, in response to electromagnetic radiation (e.g., light, such as visible light). The imaging sensor can, for example, be a CCD sensor (charge-coupled device sensor) and / or an active pixel sensor (which can also be referred to as a CMOS sensor), or be composed of these. Optionally, an imaging sensor can be configured to be wavelength-sensitive (e.g., to detect color information), for example, by means of several color filters (e.g., in grating form), and thus distinguish between different wavelengths.

Claims

1. Self-service terminal (100), comprising: • scales (102) with a set-down surface (104), on which products can be arranged, the scales (102) being configured to capture weight data indicating a total weight of products arranged on the set-down surface (104); • at least one imaging sensor (106) which is configured to capture images of the set-down surface (104) of the scales (102); • one or more processors (110) which are configured: ∘ to determine which products are arranged on the set-down surface (104) from at least one first image from the at least one imaging sensor (106); ∘ to receive first weight data from the scales (102), the first weight data indicating a first total weight of the products located on the set-down surface (104); ∘ to receive second weight data from the scales (102), the second weight data indicating a second total weight of products located on the set-down surface (104); characterized in that ∘ the first weight data and the second weight data are used to determine whether one or more products have been removed from the set-down surface (104); and ∘ at least one second image from the at least one imaging sensor (106) is used to determine which one or more products have been removed from the set-down surface (104).

2. Self-service terminal (100) according to Claim 1, wherein, if it is determined from the at least one second image that a product has been removed from the set-down surface (104), the one or more processors (110) are configured to determine whether product data (212), which respectively indicate a weight-dependent price or a unit price for each product, indicate a weight-dependent price or a unit price for the product that has been removed; and wherein the one or more processors (110) are optionally also configured: • if it is determined that the product data (212) indicate a weight-dependent price for the product that has been removed, to determine a difference between the first total weight and the second total weight as the weight of the product that has been removed, and to determine a price of the product that has been removed using the weight-dependent price and the weight of the product that has been removed; and / or • if it is determined that the product data (212) indicate a unit price for the product that has been removed, to determine the unit price as the price of the product that has been removed.

3. Self-service terminal (100) according to Claim 1 or 2, wherein, if it is determined from the at least one second image that a plurality of products have been removed from the set-down surface (104), the one or more processors (110) are configured to determine whether each product of the plurality of products belongs to the same product type; and, if it is determined that each product of the plurality of products belongs to the same product type, to determine whether the product data (212), which respectively indicate a weight-dependent price or a unit price for each product type, indicate a weight-dependent price or a unit price for the product type of the plurality of products; and wherein the one or more processors (110) are optionally also configured: • if it is determined that the product data (212) indicate a weight-dependent price for the product type of the plurality of products, to determine a difference between the first total weight and the second total weight as the weight of the plurality of products, and to determine a price of the plurality of products using the weight-dependent price and the weight of the plurality of products; and / or • if it is determined that the product data (212) for the product type of the plurality of products indicate a unit price, to determine the price of the plurality of products using the unit price and a number of products of the plurality of products.

4. Self-service terminal (100) according to Claim 3, wherein, if it is determined that at least one product of the plurality of products has a product type different from the other products of the plurality of products, the one or more processors (110) are configured to provide instructions to an output device which instruct a user of the self-service terminal (100) to place the plurality of products back onto the set-down surface (104) of the scales (102) and to remove only products of the same product type from the set-down surface (104) of the scales (102) at the same time; and wherein the self-service terminal (100) optionally further comprises the output device.

5. Self-service terminal (100) according to one of Claims 1 to 4, • wherein the one or more processors (110) are configured: when they receive the first weight data from the scales (102), to provide the at least one imaging sensor (106) with a first trigger signal which causes the at least one imaging sensor (106) to capture the at least one first image of the set-down surface (104), and, when they receive the second weight data from the scales (102), to provide the at least one imaging sensor (106) with a second trigger signal which causes the at least one imaging sensor (106) to capture the at least one second image of the set-down surface (104); or • wherein the scales (102) are configured: when they capture the first weight data, to provide the at least one imaging sensor (106) with a first trigger signal which causes the at least one imaging sensor (106) to capture the at least one first image of the set-down surface (104), and, when they capture the second weight data, to provide the at least one imaging sensor (106) with a second trigger signal which causes the at least one imaging sensor (106) to capture the at least one second image of the set-down surface (104).

6. Self-service terminal (100) according to one of Claims 1 to 5, • wherein the scales (102) are configured to capture respective weight data and to provide said data to the one or more processors (110) as soon as a change in the total weight of products arranged on the set-down surface (104) is greater than or equal to a predefined capture threshold value; • wherein the scales (102) are configured or the one or more processors (110) are configured, in response to the capture of respective weight data, to provide the at least one imaging sensor (106) with a respective trigger signal which causes the at least one imaging sensor (106) to capture an image of the set-down surface (104) associated with the respective weight data; and • wherein the one or more processors (110) are configured to use respectively captured weight data and the respectively previously captured weight data to determine whether one or more products have been removed from the set-down surface (104) and to use the respectively associated images to determine which one or more products have been removed from the set-down surface (104) in each case.

7. Self-service terminal (100) according to Claim 6, wherein the self-service terminal (100) is configured to determine the one or more products that have been removed in each case until no more products remain on the set-down surface (104) of the scales (102); and wherein the one or more processors (110) are configured, on the basis of the products determined from the at least one first image, to determine, for the one or more products that have been removed in each case, a respective price of the one or more products according to Claim 2 or 3 and to determine a total price for the products determined from the at least one first image as the sum of the determined prices.

8. Self-service terminal (100) according to one of Claims 1 to 7, further comprising: a display device configured to display each product type comprising at least one product of the products determined from the at least one first image.

9. Self-service terminal (100) according to Claim 8, wherein the display device is configured to further display for each product type: • if in combination with Claim 2 or 3, the price determined for the one or more products; and / or • a number of products having the respective product type in the products determined from the at least one first image; and / or • if in combination with Claim 2 or 3, the unit price or weight-dependent price assigned to the respective product type; and / or • if in combination with Claim 7, the total price determined.

10. Self-service terminal (100), comprising: • scales (102) with a set-down surface (104), on which products can be arranged, the scales (102) being configured to capture weight data indicating a total weight of products arranged on the set-down surface (104); • at least one imaging sensor (106) which is configured to capture images of an environment of the self-service terminal (100) above the set-down surface (104) of the scales (102); • one or more processors (110) which are configured: ∘ to receive first weight data from the scales (102), the first weight data indicating a first total weight of products located on the set-down surface (104); ∘ to receive second weight data from the scales (102), the second weight data indicating a second total weight of products located on the set-down surface (104); ∘ to use the first weight data and the second weight data to determine whether a product has been removed from the set-down surface (104); characterized in that ∘ if it is determined that a product has been removed from the set-down surface (104), to use at least one image from the at least one imaging sensor (106) to determine whether a product and, if so, which product is in a hand of a user of the self-service terminal (100); and ∘ to identify the product in the hand of the user of the self-service terminal (100) as the product that has been removed from the set-down surface (104).

11. Self-service terminal (100) according to Claim 10, wherein the scales (102) are configured such that a shopping basket can be arranged on the set-down surface (104), in which the products arranged on the set-down surface (104) can be located.

12. Self-service terminal (100) according to Claim 10 or 11, wherein the one or more processors (110) are configured to determine whether product data (212), which respectively indicate a weight-dependent price or a unit price for each product, indicate a weight-dependent price or a unit price for the product that has been removed; and wherein the one or more processors (110) are optionally further configured: • if it is determined that the product data (212) indicate a weight-dependent price for the product that has been removed, to determine a difference between the first total weight and the second total weight as the weight of the product that has been removed, and to determine a price of the product that has been removed using the weight-dependent price and the weight of the product that has been removed; and / or • if it is determined that the product data (212) indicate a unit price for the product that has been removed, to determine the unit price as the price of the product that has been removed.

13. Self-service terminal (100) according to Claim 12, wherein the product data (212) also indicate, for each product for which a unit price is indicated, a respective target weight of such a product; and wherein, if it is determined that the product data (212) indicate a unit price for the product that has been removed, the one or more processors (110) are configured: • to determine a difference between the first total weight and the second total weight as the weight of the product that has been removed; • to determine whether a difference between the target weight indicated by the product data (212) and the weight determined from the difference between the first total weight and the second total weight is less than or equal to a predefined verification threshold value; and • if it is determined that the difference is less than or equal to the predefined verification threshold value, to verify the removed product determined from the at least one image and to determine the unit price as the price of the product that has been removed.

14. Self-service terminal (100) according to one of Claims 10 to 13, wherein the at least one imaging sensor (106) is further configured to capture set-down surface images of the set-down surface (104) of the scales (102); wherein the one or more processors (110) are configured: • to determine, from at least one first set-down surface image received from the at least one imaging sensor (106) and captured before the at least one image, which one or more first products of the products located on the set-down surface (104) are shown by the first set-down surface image; • to determine, from at least one second set-down surface image received from the at least one imaging sensor (106) and captured after the at least one image, which one or more second products are shown by the second set-down surface image; • to use the one or more first products and the one or more second products to determine which product has been removed from the set-down surface (104); and • if the removed product determined using the one or more first products and the one or more second products corresponds to the removed product determined from the at least one image, to verify the product that has been removed.

15. Self-service terminal (100) according to one of Claims 10 to 14, wherein the self-service terminal (100) does not have a scanner unit; and / or wherein the self-service terminal (100) has no other scales.