System and method for verifying a product placed in a shopping cart and / or shopping basket

Abstract

The invention relates to a method (400, 500) for automatically constructing an image database and / or for automatically training a model for verifying a product placed in a shopping cart and / or shopping basket, said method having the steps of: receiving (410) the target weight of a scanned product; receiving (420) the measured weight of the placed product; and selecting (430) at least one image of the placed product for inclusion in the image database and / or the trained model if the measured weight corresponds to the target weight.

Description

[0001] System and method for verifying a product placed in a shopping cart and / or basket

[0002] 1. Technical area

[0003] A first aspect relates to a method, a computer program, and a system for verifying a product placed in a shopping cart and / or shopping basket. A second aspect relates to a method, a computer program, and a system for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart and / or shopping basket.

[0004] 2. Background

[0005] Shopping in a self-service store traditionally involves paying for goods at the checkout upon completion of the purchase. The goods selected by the customer are identified at the checkout, so that the customer pays accordingly. This identification is often carried out by staff, e.g., a cashier, using a barcode scanner and / or scales. Alternatively, or in addition, self-service checkouts are used, where there is no dedicated cashier; instead, the customer scans the goods and pays for them. Typically, an employee is located near such a self-service checkout to assist with any problems that arise and to monitor the process.

[0006] However, in both cases, staff must be assigned to actively identify products and / or to monitor such a process. Furthermore, traditional checkouts, which typically comprise a checkout belt, and self-service checkouts, which must be set up with separate payment and scanning terminals, require a lot of space. In particular, there is only a certain number of checkouts and / or terminals in a store, which is generally fewer than the number of customers present. This leads to queues forming at the checkout area, especially during peak times, as several customers are assigned to one checkout and there is not an individual checkout for each customer. This has led to the development of digital shopping carts, in which the products intended for purchase by the customer are identified as soon as they are placed in the shopping cart. In particular, this identification should take place without the assistance of staff.However, this requires the use of certain control mechanisms. These control mechanisms are intended to ensure that all products placed in the shopping cart are registered and match the products for which the customer pays at the end of the shopping process. The control mechanisms used so far can be divided into two classes.

[0007] The first class of digital shopping carts uses a checkweigher to ensure that the customer correctly records all items placed in the shopping cart. This checkweigher weighs the inserted product and compares the measured weight with a weight stored in a database for the respective scanned product. However, this is particularly problematic in the food sector, especially fast-moving consumer goods (products with a high product rotation, i.e. products for which the average storage time in a warehouse and / or on-shelves is short; typical examples: food and beverages, personal care products, cleaning products and, in particular, daily newspapers), and in the DIY and furniture stores. In these sectors, there are many products that have the same or at least a very similar weight, but differ significantly in price.The checkweigher cannot differentiate between such products, or can only do so inadequately, and consequently, it cannot be guaranteed that the product placed on the shopping scales corresponds to the item recorded. Another problem is that the weight of products is often subject to fluctuations, for example, due to production-related factors, so that a precise weight cannot be recorded.

[0008] The second class of digital shopping carts uses optical image recognition via a camera to ensure that the customer correctly records all items placed in the shopping cart. However, this proves difficult in practice, as in typical supermarkets, tens of thousands of products must be differentiated from one another, and the products can be viewed from all possible angles, from all sides, and partially obscured when placed in shopping carts.

[0009] Another problem with optical image recognition is that the product placed in the shopping cart must be visually accessible. If the product is not visible to the camera, it cannot be visually detected and identified accordingly. This optical inaccessibility of products in the shopping cart can be created either unintentionally or intentionally by the customer. On the one hand, the product can be positioned when placed in the cart in such a way that it is obscured by another product placed in the cart. On the other hand, the customer can intentionally hide a product in the packaging of another product. However, since the camera only checks the optical properties of the placed product, it cannot register the difference.

[0010] Another disadvantage of digital shopping carts, which are based either on a checkweigher or optical image recognition, is that despite the control mechanisms, staff are still required at some point in the shopping process. If a customer has a problem registering the product, or if the product is registered incorrectly, the staff must help and support the customer. However, qualified staff is difficult to find and expensive in times of a shortage of skilled workers.

[0011] KR 102323796 Bi concerns a shopping cart that recognizes products using artificial intelligence. The shopping cart has a camera for taking pictures of a product placed in the shopping cart. These images are transmitted to a server, which recognizes the product using artificial intelligence.

[0012] EP 3 262562 Bi relates to a system and method for identifying products in a shopping cart. This identification is based on optical image recognition software using a camera attached to the shopping cart and a comparison with a database containing product-specific information.

[0013] In light of this technical background, there is a need for improved methods by which a product placed in a shopping cart and / or shopping basket can be verified.

[0014] 3. Summary of the invention

[0015] This objective is achieved in a first aspect by a method for verifying a product placed in a shopping cart and / or shopping basket, comprising the following steps: receiving at least one image of the placed product; selecting a trained model based on a scanned product; and determining a probability that the at least one image of the placed product shows the scanned product based on the trained model.

[0016] The at least one received image of the inserted product is an image of the same product that is placed in the shopping cart and / or shopping basket and is preferably created in connection with an insertion process of the inserted product. This insertion process can include the product lying in the shopping cart, the act of placing it in a shopping cart, as well as the picking of a product, preferably from a shelf and / or from a device designed for product presentation. The at least one image can include several or a plurality of images and / or contain images of a video sequence. For example, the at least one image and / or the video sequence can be created by one or more cameras of the shopping cart and / or shopping basket. The scanned product for which a trained model is selected can differ from the inserted product, e.g.when the product placed in the shopping cart or basket is not the one previously scanned. The trained model is based on optical or optically accessible properties of the scanned product. Optically accessible properties include length, width, height, color design, and geometry. The probability that the scanned product does not differ from the product placed in the basket is quantified by the probability. The determination of the probability is based on the trained model and indicates how likely it is that at least one image of the placed product shows the scanned product. Verification of the placed product is based on the determined probability. In general, the scanned product can be a product billed to the customer at the end of the shopping process.Verification ensures that the inserted product actually corresponds to the scanned (and invoiced) product (and not, for example, a significantly more expensive product, for which the cheaper price of the scanned product would then be incorrectly invoiced).

[0017] In some of the following examples, reference is made specifically to a shopping cart. However, all aspects can also be applied to a shopping basket, even if this is not always explicitly stated. The term shopping cart refers to all carts that can typically be used to transport goods from a sales room to a checkout, e.g. platform carts, stake carts, pipe-hanger carts, magazine carts, roll containers, storage carts, etc. Alternatively, or additionally, all aspects can also be applied to permanently installed checkouts, e.g. self-service checkouts, scope checkouts, and / or self-checkout terminals. In general, the aspects described here can be applied regardless of the specific device, e.g. shopping cart and / or self-service checkout. In particular, the aspects described here can be applied to the control of any type of order picking.In particular, the type of order picking is not limited to retail and / or food retail.

[0018] The trained model for the scanned product can be based on an image database. In particular, selecting the trained model can involve retrieving the trained model from a memory. The selection can be made from a plurality of different trained models. For example, the trained model can be trained using a subset of the images from the image database. This allows the probability that the at least one image of the inserted product shows the scanned product to be determined by applying the trained model to the at least one image of the inserted product.

[0019] In particular, the image database can be an image database specific to the scanned product. A specific image database only contains images for a specific product. The specific image database can be a separate image database for each product, separate from the image databases of other products. Similarly, the trained model can be a model specifically trained for the scanned product. The specifically trained model can therefore, for example, be trained to recognize exactly the scanned product. For this purpose, in addition to the image database for the scanned product, images of other products can also be used as "negative examples" for training purposes. With a model specifically trained for the scanned product, for example, only a probability can be calculated which indicates that at least one image of the inserted product shows the scanned product.The hypothesis tested is whether the scanned product matches the inserted product. Hypotheses regarding products other than the scanned product may not be possible, for example, with a model trained specifically for the scanned product.

[0020] In some examples, a product can be scanned, e.g. with a barcode scanner on a shopping cart. A (possibly different) product can then be placed in a shopping cart and at least one image of the placed product can be created, e.g. by one or more cameras attached to the shopping cart. Based on the scanned product and / or information about the scanned product, a trained model can then be selected, e.g. a model trained separately for the scanned product. Using this model and the at least one image, a probability (e.g. in %) can then be determined as to whether the at least one image shows the scanned product (it is emphasized that statements such as "yes" / "no" and possibly "unclear" can also be interpreted as probabilities without a specific percentage having to be specified).

[0021] Some of the steps of the method can, for example, be carried out by a computer belonging to the shopping cart (e.g. a tablet mounted on the shopping cart). For example, all steps of the method can be carried out by a computer belonging to the shopping cart. For example, the trained model and any further trained models that can be selected can be stored in a memory of the shopping cart and / or its computer. Alternatively or additionally, some of the steps can be carried out by a server and / or in a cloud. For example, all steps of the method can be carried out by the server and / or the cloud. In other embodiments, a first part of the method can be carried out by the computer belonging to the shopping cart and a second part of the method can be carried out by the server and / or the cloud. The first and second parts can be disjoint.Alternatively, certain steps of the process may be performed by both the computer associated with the shopping cart and the server and / or the cloud.

[0022] For example, the computer belonging to the shopping cart can receive the at least one image of the inserted product, preferably from one or more cameras belonging to the shopping cart. Then, in a first alternative, the shopping cart and / or the shopping cart's computer can send the at least one image and information about the scanned product to the server and / or the cloud. For this purpose, a wireless communication means of the shopping cart and / or its computer can be used, for example (such as a transmitter / receiver with WLAN, Bluetooth, 3G, 4G, 5G, etc. transmission or reception function). The server and / or the cloud then selects a trained model based on the scanned product or the information about the scanned product. The server and / or the cloud determine a probability that the at least one image of the inserted product shows the scanned product based on the trained model.The determined probability can be returned to the shopping cart. The server and / or cloud can belong to a market associated with the shopping cart and / or to a corresponding international, national, or regional headquarters of a department store group.

[0023] In a second alternative, it is also possible for the shopping cart and / or the shopping cart's computer to send (only) the information about the scanned product to the server and / or the cloud. The server and / or the cloud then selects a trained model based on the scanned product or the information about the scanned product. The server and / or the cloud can then, for example, send the trained model to the shopping cart and / or its computer, which then determines the probability that at least one image of the inserted product shows the scanned product based on the trained model. In this case, too, the server and / or the cloud can belong to a market associated with the shopping cart and / or to a corresponding international, national, or regional headquarters of a department store group.

[0024] The computers, servers, and / or clouds described below may include one or more processors and one or more storage devices. Each processor may include one or more processor cores, and each processor may include one or more logic circuits for processing data and / or information. For example, each processor may include an arithmetic and logic unit (ALU), a control unit, and a plurality of registers. Each processor may include a cache memory. Each processor may include a system-on-chip (SoC) that includes a plurality of processor cores, random access memory (RAM / DRAM), graphics processing units (GPU), one or more controllers, and one or more communication modules / communication interfaces. Each processor may include a plurality of transistors. In general, a computer and / or a computer system may be configured to receive and / or forward data.This data can be read from machine-readable storage media such as hard disks, magnetic disks / floppy disks, solid-state drives (SSDs), magneto-optical disks (MODs), or optical disks. A computer program can be written in any programming language, in particular a compiled language and / or an interpreted language and / or a scripting language. Examples of possible languages ​​include C, C++, Fortran, Python, and Perl.

[0025] Using a specifically trained model maximizes the quality of the

[0026] Hypothesis testing minimizes the likelihood of incorrect verification and thus optimizes the security of the purchasing process. In particular, the specifically trained model can deliver a clear and accurate output with a very high probability. Generic optical image recognition models have the disadvantage that they often deliver unusable results, i.e., the conclusions drawn from the output of the image recognition model are not conclusive. This is primarily due to the fact that a global image recognition model is used for image recognition, which is trained on all products offered in a market. Since the product range of a market typically ranges from 40,000 to 60,000 products,000 products, the image recognition model for a given product provides a list of match probabilities for each product in the product range. However, it would require extremely complex training for a single model to reliably recognize all of the large number of products. Models trained specifically for individual products, in contrast, are much faster and more reliable to train. Furthermore, the runtime of the specifically trained model is minimized because the trained model is smaller and / or less complex. This leads to faster verification of the given product and thus a smoother and more pleasant shopping experience for the customer. In addition, calculating the probability can consume less energy, so the shopping cart needs to be loaded less frequently or at longer intervals, which minimizes cart downtime.In other embodiments, a specific model may be a model specifically trained for a product group of the scanned product. In this case, the calculated probability comprises a plurality of probabilities, each probability indicating how likely the at least one image of the inserted product matches a product in the product group of the scanned product. Alternatively, or additionally, the specific model may be a model specific to part of a product group. In general, the trained model may be specifically trained for a set of products associated with the scanned product.

[0027] The trained model can be a neural network. For example, it can be a feedforward neural network, preferably a convolutional neural network. The training of the specific trained model can be based on the specific image database. In particular, the specifically trained model can be trained using an image database, wherein the image database is preferably product-specific. For example, the specifically trained model can be trained based on a product-specific image database with a large number of images of the product (preferably from different directions, angles, with different lighting, shading, etc.). Alternatively, or additionally, the specifically trained model can also be trained based on a specific image database that is specific to a different product.For example, the specific model can receive images from the product-specific image database as "positive examples" and images from a product-specific database of a different product as "negative examples." Alternatively, or additionally, the negative examples can come from several different specific image databases.

[0028] The method may further comprise receiving a measured weight of the inserted product and comparing the measured weight with a target weight of the scanned product.

[0029] The target weight of the scanned product can be stored in a weight database and is a weight specific to the scanned product. Comparing the measured weight with the target weight allows the scanned product to be compared with the inserted product with regard to another product-specific property and thus represents a further indicator of match. If the measured weight deviates from the target weight, it is unlikely that the scanned product matches the inserted product. The weight of the inserted product is a property of the product inserted in the shopping cart, which is complementary to the optical properties that can be assigned to the inserted product by the at least one image. This makes it possible to differentiate between visually similar products that differ in weight.Similarly, a distinction can be made between products of similar weight which are visually different. This synergistic effect, which results from the combination of optical and weight properties, leads to more reliable verification of the inserted product and thus to a minimization of the possibility of deception by the customer. For example, a shopping cart and / or basket can be equipped with a scale that registers a change in weight so that the weight of a newly inserted product can be determined. The weight comparison can, for example, be carried out by the shopping cart and / or its computer, which also receives at least one image and / or information about the scanned product (e.g. a tablet mounted on the shopping cart). In other examples, however, the first aspect can also apply without (a means for) weighing an inserted product.In other words, no weight of the inserted product can be received. Even in such a case, however, the trained model may comprise a trained model that has been trained based on the image database described herein for automated training. In other words, a shopping cart and / or shopping basket and / or other elements described herein, such as self-checkouts, without weighing means may rely on a high-quality, trained model that has been trained based on a weight-verified image database (see method for automatically building an image database according to the second aspect).

[0030] In other embodiments, additional product-specific properties of the inserted product and / or desired properties of the scanned product can be received. These can include sensor-based properties such as data from an infrared and / or thermal sensor. Alternatively, or additionally, an RFID tag can be used. This RFID tag can be used in addition to the scan of the scanned product to ensure that the inserted product is the scanned product.

[0031] The use of the RFID tag data can take place before the evaluation of at least one image of the inserted product. The use of the RFID tag can be product- and / or product-group-specific. For example, only specific products and / or specific product groups can be equipped with an RFID tag. Specific products and / or specific product groups can include, in particular, alcohol-containing products and / or higher-priced products.

[0032] Receiving the at least one image of the inserted product and / or selecting the trained model and / or determining the probability may be performed based on the measured weight substantially matching the target weight.

[0033] A match between the measured weight and the target weight can be present if the measured weight does not deviate from the target weight by more than 20%, preferably not more than 10%. The tolerated deviation of the measured weight of the inserted product from the target weight of the scanned product can be a deviation specific to the scanned product. Alternatively, or additionally, the target weight can be a list of weights. For example, the list of weights can contain moments of a weight distribution. For example, the list can contain the first and second moments of a weight distribution, where the first moment is associated with the target weight and the second moment with the tolerated deviation. Alternatively, or additionally, a target weight can contain multiple weights and / or multiple tolerances.For example, the target weight may include a first target weight with a first tolerance and a second target weight with a second tolerance.

[0034] By conditioning the above-mentioned procedural steps on the substantial agreement of the weights, a synergistic effect is achieved in conjunction with the use of the image database specific to the scanned product and / or the specifically trained model. The first verification step is performed by comparing the weight of the inserted product with the target weight. If the two weights do not substantially agree, it is highly likely that the inserted product is not the scanned product. Therefore, calling the trained model for verification is not necessary, as the match has already been ruled out. This enables rapid and conclusive falsification of the match between the inserted product and the scanned product, as comparing two weights can be done faster than evaluating at least one image of the inserted product by the trained model.Furthermore, the significance of the probability calculated by the trained model is increased.

[0035] In general, the inserted product can be identified with the scanned product if the calculated probability is greater than a threshold, wherein preferably the threshold depends on the product group of the scanned product and / or the scanned product itself.

[0036] The identification of the inserted product with the scanned product involves equating the inserted product with the scanned product. In other words, the inserted product is considered to be the same product as the scanned product. The threshold, which includes a probability, guarantees identification with high certainty, i.e., the probability of incorrect identification of the inserted product with the scanned product is minimized, thus optimizing the verification of the inserted product. The threshold can be at a probability of at least 80%, preferably at least 90%, of matching the scanned product with the inserted product. The threshold can be a value specific to the product group of the scanned product.The product group of a product may include a set of products of similar nature and / or similar intended use and / or similar price level and / or similar target audience. For example, the product group of a scanned alcoholic beverage may include a subset of all alcoholic beverages in the product range. The threshold for part of the product group of alcoholic beverages may be higher than the threshold for part of the product group of non-alcoholic beverages.

[0037] Alternatively, or additionally, the threshold may be a value specific to the scanned product itself. The threshold specific to the scanned product is set individually for the scanned product. In other embodiments, a specific threshold may be set for any subset of products. In general, thresholds are variable and can be changed dynamically. Similarly, product groups are variable and can be changed dynamically. For example, a price change of a product may lead to a changed threshold and / or a changed product group membership. Furthermore, observed deceptive customer behavior may lead to a change in the threshold and / or product group membership.

[0038] The method may further comprise outputting the at least one image for manually checking whether the inserted product should be identified with the scanned product if the probability is below a first threshold but above a second threshold.

[0039] The manual inspection of the at least one image can be carried out by store staff. For this purpose, the at least one image of the inserted product is manually compared with the scanned product. During the manual inspection, the scanned product can be represented by a product name and / or an image associated with the scanned product. The inspection can take place during shopping and / or at the end of the shopping process. For example, at the end of a shopping process (with a full shopping cart), a store employee can use the at least one image to check whether the product actually shows the scanned product (without having to spend a long time searching for the product shown in the image in the shopping cart).

[0040] The manual check can take place if the probability is below a first threshold but above a second threshold. The first threshold is preferably below the threshold for identifying the inserted product with the scanned product. The first threshold can be less than a probability of 95%, preferably less than 90%. The first threshold can depend on the threshold for identification. The second threshold is below the first threshold and can be greater than a probability of 60%, preferably a probability greater than 70%. This specifies a lower bound on the probability at which identification is still considered possible. In general, the second threshold can depend on the first threshold and / or the threshold for identification.The first and second thresholds can, in particular, be product-specific and / or product group-specific and / or variable and / or dynamic. Furthermore, the combination of the first and second thresholds results in a probability interval that includes all probabilities that are greater than the second threshold and less than the first threshold.

[0041] If the calculated probability lies within this interval, a manual inspection can be scheduled, for example, with a test probability of at least 30%, preferably 50%. The test probability can be product- and / or product group-specific.

[0042] With a verification probability of at least 30%, preferably 50%, not every product with a probability below the identification threshold needs to be manually checked. This relieves staff and reduces the number of personnel required. At the same time, the first threshold guarantees a minimum level of security for the verification process. If the probability is below the first threshold, a manual check can always be performed. If the probability is above the first threshold, the inserted product can be identified with the scanned product. This allows, for example, the staffing situation and the risk of theft at each individual location to be addressed individually.In other embodiments, if the probability is below the second threshold, the inserted product can be considered different from the scanned product and both products cannot be identified with each other. In this case, the at least one image of the inserted product is always subjected to manual inspection. Alternatively, the at least one image can also be subjected to manual inspection in this case with an inspection probability of at least 30%, preferably at least 50%. If the probability is below the first threshold, an error message can be displayed and the customer's shopping process can be interrupted until the customer has corrected the inserted product. The error message can include a request to remove the inserted product and insert the scanned product. This leads to a reduction in the workload or rather, a reduction in the number of personnel required to operate the store.In other cases, the shopping process can continue and a manual check will only be carried out at the exit of the store.

[0043] The need for a manual inspection can be communicated to staff by the shopping cart and / or their computer and / or the server and / or the cloud. For example, staff handheld devices (e.g. tablets) can be provided for this purpose, on which the shopping carts to be inspected are displayed. The images can also be checked on staff handheld devices and / or on the shopping cart computers. In other cases, the need for a manual inspection can also be indicated directly on the shopping cart itself, e.g. by an appropriate light or similar. In some cases, this can be indicated (only) when the customer is about to complete their purchase, e.g. when it is detected that the cart is approaching a store's checkout area. This can be detected using Wi-Fi, Bluetooth, etc.

[0044] The method may further comprise scanning the scanned product, preferably with a scanner of the shopping cart and / or shopping basket and / or weighing the inserted product, preferably by placing it in a weighing area of ​​the shopping cart and / or shopping basket and / or creating the at least one image of the inserted product, preferably with one or more cameras of the shopping cart and / or shopping basket.

[0045] By scanning a product, the scanned product is identified with the product. The scanning is preferably carried out by a customer during the shopping process, for example using a handheld scanner. In general, however, the scanner can also be one or more scanners attached to the shopping cart, which are preferably attached to the shopping cart in such a way that they automatically perform a scan when a product is passed by. The scanned product is preferably a product that can be placed in the shopping cart and / or shopping basket. Scanning the scanned product can involve scanning a barcode and / or a QR code of the scanned product. In general, scanning a product can include capturing a feature that is unique to the scanned product.In other embodiments, scanning the scanned product may be based on creating the at least one image with one or more cameras.

[0046] The weighing of the inserted product can be based on a weight difference in the weighing area, wherein the weight difference comprises a weight before the inserted product is inserted and a weight after the inserted product is inserted. The weighing area is that part of the shopping cart and / or shopping basket for which a weight change can be registered. The weighing area can comprise a basket of the shopping cart. In other embodiments, the weighing area comprises a basket and at least part of a frame of the shopping cart. The weight difference is preferably determined when the shopping cart and / or shopping basket is stationary. The weighing in the weighing area can be dependent on the scan of the scanner. For example, the weighing area can only be active when the scanner is in use or a product is being scanned.By not continuously activating the weighing area, the cart's power consumption can be reduced, resulting in a longer usage period. This reduces energy costs and the cart's inactivity time. Alternatively, or additionally, the weighing area can be continuously weighed. Continuous weighing of the weighing area ensures that no product is placed in the shopping cart without being scanned. This guarantees that for every product placed in the shopping cart, a product is scanned, based on which the process for verifying the inserted product can be carried out. Weighing can be performed using load cells and / or force sensors.

[0047] The creation of the at least one image by at least one camera of the shopping cart and / or shopping basket can depend on the time of scanning and / or weighing. The at least one camera of the shopping cart permanently records at least part of the cart and saves the resulting image sequences on a memory. If the weighing area of ​​the cart registers a change in weight at a time, a section of the image sequence is selected which includes the time of the weight change. This section can include images of the image sequence before the weight change and / or after the weight change. For example, the section can begin at least 1 second, preferably at least 2 seconds before the registered weight change and / or extend at most 10 seconds, preferably at most 5 seconds beyond the time of the weight change.For example, the image material can be stored in a buffer memory that holds image material from a specific period in the past on a rolling basis.

[0048] By including the section of the image sequence that depicts the time of the weight change, it is ensured that the inserted product is depicted in one of the images in the image sequence. This allows the trained model to be provided with suitable image material for verification. This reduces the likelihood of falsification of the inserted product due to inappropriate image material, even though the inserted product and the scanned product are the same. This results in a more pleasant shopping experience for the customer, as the shopping process is interrupted less frequently.

[0049] The fact that the section of the image sequence includes the time of the weight change also results in the effect of being able to differentiate between the addition of a new product and the rearrangement of products already in the shopping cart and / or basket. For example, if the products in the shopping cart and / or basket are rearranged, this visually appears to the camera to be a completely different product arrangement. In this case, however, the weight can be used, since rearranging the products does not change the weight.

[0050] Image sequences that do not belong to a section assigned to a weight change are deleted from the memory. Thus, only images relevant for verification are saved, which prevents memory overflow. In particular, this allows the use of cost-effective memory, since the memory size does not have to be designed for the entire shopping process. Alternatively, or additionally, the section of the image sequence can be selected such that it includes the time of scanning. The method for verifying the product placed in a shopping cart and / or shopping basket can further comprise registering and / or perceiving a non-scanned product placed in the shopping cart. For example, the inserted non-scanned product can comprise a product that was placed in the shopping cart and / or shopping basket without a corresponding scan.The registration and / or perception can be based on a change in the measured weight. For example, based on the change in the measured weight, it can be registered and / or perceived whether a product has been placed in the shopping cart and / or shopping basket. Alternatively, or additionally, the registration and / or perception can be based on at least one image of the shopping cart and / or shopping basket. For example, based on the at least one image of the shopping cart and / or shopping basket, it can be registered and / or perceived whether a product has been placed in the shopping cart and / or shopping basket. In particular, the at least one image can comprise at least part of an insertion process of the product. Specifically, the at least one image can comprise a sequence of images, e.g., a sequence of images of the insertion process.In some embodiments, the camera may permanently capture at least a portion of the shopping cart and / or shopping basket.

[0051] In general, registering and / or detecting the insertion of a product may not require weighing. In addition, registering and / or detecting a non-scanned product placed in the shopping cart may include issuing an alarm and / or a notification. For example, the notification may include a request to scan the inserted product. The user may then, for example, complete the scanning and / or reject the request. In some examples, the shopping cart and / or shopping basket may be configured to request a manual review of the shopping cart and / or shopping basket after a certain number of rejected requests within a certain period of time and / or within a shopping transaction. This may, for example, include a check for fraudulent behavior by the user and / or the functionality of the shopping cart and / or shopping basket.

[0052] A further aspect relates to a computer program for verifying a product placed in a shopping cart and / or shopping basket, comprising instructions which, when executed by a computer, cause the computer to carry out the method described above.

[0053] The computer program can be executed on a computer associated with the shopping cart and / or shopping basket. Alternatively, or additionally, the computer program can be executed on a server and / or a cloud, wherein the server and / or the cloud are associated with a store and / or a headquarters. In other embodiments, a portion of the computer program can be executed on a computer associated with the shopping cart and another portion can be executed on a server and / or associated cloud associated with the store and / or headquarters.

[0054] Another aspect relates to a system for verifying a product placed in a shopping cart and / or shopping basket. This system comprises: means for receiving at least one image of the placed product; means for selecting a trained model based on a scanned product; means for determining a probability that the at least one image of the placed product shows the scanned product based on the trained model.

[0055] The verification system can be at least partially comprised by a shopping cart. For example, the means for receiving the at least one image of the inserted product and / or the means for selecting a trained model and / or the means for determining a probability can be part of the shopping cart, e.g., a computer (tablet) of the shopping cart. Preferably, the shopping cart comprises the means for receiving the image, the means for selecting a trained model, and the means for determining a probability. Alternatively, parts of the system can be detached from the shopping cart and located separately in a store and / or a central location (e.g., a server and / or a central location as described herein). In other embodiments, the system is located both on a shopping cart and separately in the store and / or a central location.For example, the shopping cart, as well as the market and / or the headquarters, can have means for determining a probability. The selection of the means for determining a probability to be used for verification can depend on the scanned product. For example, the means for determining the shopping cart can be used for a first part of the products in the product range, and the means for determining the market and / or the headquarters can be used for a second part of the products. The division of the product range into a first and a second part can depend on a sales frequency and / or a price level of the products. Alternatively or additionally, the means for determining can be located on a server and / or a cloud. In particular, the means for determining belonging to the market and / or the headquarters can be located on a server and / or a cloud.

[0056] The means for receiving the image may be configured to receive data from a wireless network, wherein the wireless network may be a WLAN and / or WMAN and / or WPAN and / or WWAN network. In other examples, Bluetooth and / or a cellular connection (3G, 4G, 5G, etc.) may also be used. Alternatively or additionally, the means for receiving the image may comprise physical connection elements, for example, copper and / or fiber optic cables.

[0057] The trained model can be a model specifically trained for the scanned product and can be trained based on an image database specific to the scanned product.

[0058] The verification system may further comprise means for receiving a measured weight of the inserted product and means for comparing the measured weight with a target weight of the scanned product.

[0059] The means for receiving the measured weight and / or the means for comparing can be part of the shopping cart. Alternatively, the means for receiving the measured weight and / or the means for comparing can be separate means located in the store and / or in a central location. The means for receiving the weight can at least partially coincide with the means for receiving the image. The means for receiving can be configured to receive data from a wireless network, wherein the wireless network can be a WLAN and / or WMAN and / or WPAN and / or WWAN network. Alternatively, or additionally, the means for receiving the measured weight can comprise physical connecting elements, for example copper and / or fiber optic cables. The means for comparing can be part of a computer and / or a processor.Furthermore, the means for comparing may at least partially coincide with the means for determining the probability. For example, the means for comparing and the means for determining the probability may be part of a computer, preferably a tablet, and / or a computer system.

[0060] Receiving the at least one image and / or selecting the trained model and / or determining a probability may be based on the measured weight substantially matching the target weight.

[0061] The verification system may further comprise at least one scanner for scanning the scanned product and / or a weighing means for weighing the inserted product and / or at least one camera for creating the at least one image.

[0062] The at least one scanner and / or the weighing device and / or the at least one camera can be part of the shopping cart. Preferably, the scanner, weighing device, and camera are part of the shopping cart. The weighing devices can comprise load cells and / or force sensors. Preferably, several cameras are mounted on the shopping cart such that at least some of the cameras optically capture the loading area of ​​the shopping cart.

[0063] Furthermore, the verification system may comprise means for outputting the at least one image for manually checking whether the inserted product should be identified with the scanned product if the probability is below a first threshold but above a second threshold.

[0064] Alternatively, or additionally, the at least one image can be output for manual inspection if the measured weight of the inserted product does not substantially match the target weight of the scanned product, but deviates within a certain tolerance range. Preferably, the output of the at least one image further includes the measured weight of the inserted product and / or the target weight.

[0065] The dispensing means may be part of the shopping cart and comprise a tablet computer. Alternatively, the dispensing means may be separate means not belonging to the shopping cart. The separate dispensing means may belong to a store and / or a central office. Preferably, the separate means is a tablet computer and / or computer. Another aspect relates to a shopping cart or shopping basket with a system described above. Furthermore, the shopping cart or shopping basket may have a memory, wherein the memory at least partially comprises the image database and / or the trained model and / or the weight database.

[0066] The method and system described above are based on selecting a trained model based on a scanned product, in particular a specific trained model trained using a specific database. A simple structure of such a specific database and / or a simple provision of appropriately trained models is an important element for the practical applicability of the aforementioned method.

[0067] According to a second aspect, a method and a system are provided with which an image database and / or a model for verifying a product placed in a shopping cart and / or shopping basket can be automatically constructed or automatically trained.

[0068] A method for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart and / or shopping basket may comprise the following steps: receiving a target weight of a scanned product; receiving a measured weight of the placed product; selecting at least one image of the placed product for inclusion in the image database and / or the trained model if the measured weight corresponds to the target weight.

[0069] Some of the steps of the method can be carried out by a computer belonging to the shopping cart. For example, all of the steps of the method can be carried out by the computer belonging to the shopping cart (e.g. a tablet mounted on the shopping cart). The at least one selected image can then be sent, for example, to a server and / or a cloud (e.g. using wireless communication means described herein) so that it can be included in the corresponding image database for the scanned product and / or the trained model can be (further) trained for the scanned product. Alternatively, or additionally, some of the steps of the method can be carried out by a server and / or a cloud. For example, all of the steps of the method can be carried out by the server and / or the cloud.In other embodiments, a first part of the steps can be performed by the shopping cart computer and a second part by the server and / or the cloud. The first and second parts can be disjoint. For example, the shopping cart computer can receive the measured weight of the shopping cart. It then transmits this measured weight to the server and / or the cloud, so that the server and / or the cloud receives the measured weight of the inserted product. The server and / or the cloud also receive the target weight of the scanned product, preferably by retrieving the target weight from a weight database. The server and / or the cloud then select the at least one image of the inserted product for inclusion in the image database. The at least one image can, for example, be selected from an image sequence for the inserted product transmitted by the shopping cart.Alternatively, the at least one image can be “selected” by the server and / or the cloud requesting the at least one image for the inserted product from the shopping cart.

[0070] In general, the method for automated building can be carried out on its own and / or in conjunction with the method for verification. An image database for verifying a product placed in a shopping cart is an image database that can be at least partially associated with a product. Partial association can comprise the association of multiple products with an image database. Automated building of an image database comprises adding further images to an existing image database and / or associating at least one image with one or a group of products for the first time. Automated building of an image database comprises automatically associating at least one image with a product or a group of products.

[0071] In connection with the process for verifying a product placed in a shopping cart, access to the aforementioned image database is already problematic, as it generally has to be created first. This is often not automated, but manual. One option for creating the image database is the centralized capture of product image data. However, this requires personnel, which makes the process very costly. Furthermore, many markets have regional products for which centralized capture does not make sense, so the capture must be carried out either by the supplier of the regional product themselves or by personnel in the regional market. Furthermore, the lighting and ambient conditions under which the product is optically captured in the regional market are different from the conditions under which the product is captured centrally.This results in an image database that doesn't reflect the realities of the market, resulting in an inaccurate and / or unusable image recognition program. Alternatively, the image data can also be generated by in-store personnel. However, this is also very costly and requires the database to be continually updated, as new products are frequently added to the product range.

[0072] According to the second aspect, the image database can be created and built by the customer during the shopping process. By checking the correspondence of the weights between the scanned and inserted product, assignments of incorrect images can be excluded with a very high degree of probability, allowing the creation of a high-quality image database that also takes into account the actual conditions in the market. As soon as customers scan a product and place it in the shopping cart, an image taken from the shopping cart can be directly stored in the image database for the scanned product, and the trained model for the product can be further trained. Errors caused by scanning a first product and then incorrectly inserting a second product are significantly reduced and / or avoided.

[0073] The resulting image database and / or models trained based on it can then be used not only in systems with scales, but also without (a means of) weighing an inserted product. In other words, a shopping cart and / or basket and / or other elements described herein, such as self-checkouts, without means of weighing can rely on a high-quality, trained model trained based on a weight-verified image database.

[0074] The target weight of the scanned product is a weight associated with the scanned product and may be a weight specific to that product. Generally, the target weight of the scanned product may be based on a weight database. A weight database may comprise a set of assignments, where an assignment assigns a target weight, preferably a product-specific target weight, to a product.

[0075] The measured weight of the inserted product is a weight measured during the insertion process of the inserted product. The target weight corresponds to the measured weight if its deviation does not exceed a predetermined threshold. This threshold can be the same for all products and can be a relative threshold. For example, the threshold can be at most 20%, preferably at most 10%, of the target weight of the scanned product. Alternatively, the threshold can be an absolute threshold, which can be at most 100 grams, preferably at most 50 grams. In particular, the threshold can be at most 10 grams or at most 5 grams. Alternatively, or additionally, the threshold can depend on the sensitivity or resolution of the weighing device. In other embodiments, the threshold can be a product- or product group-specific relative and / or absolute threshold.The product-specific threshold can be stored in the weight database together with the product-specific target weight. The threshold can be variable and / or dynamic.

[0076] Selecting the at least one image of the inserted product for inclusion in the image database comprises associating the at least one image of the inserted product with the scanned product or with a product group assigned to the scanned product. The at least one image of the inserted product was preferably created during a process of inserting the inserted product into the shopping cart and / or shopping basket. Similarly, selecting the at least one image of the inserted product for inclusion in the trained model comprises associating the at least one image of the inserted product with the scanned product or with a product group assigned to the scanned product. Inclusion in the trained model can include training the trained model with respect to the at least one recorded image.

[0077] At least one image of the inserted product is selected for inclusion in the image database if the measured weight corresponds to the target weight. This ensures that the weight of the inserted product matches the weight of the scanned product. In particular, this increases the probability that the scanned product is actually the inserted product. Thus, only those images are selected for inclusion in the image database that have a high probability of showing the scanned product. This control mechanism guarantees a high quality of the image database, resulting in an accurately trained model for the verification of the scanned product. In particular, it is this control mechanism that enables the automatic construction of the image database. If the measured weight does not match the target weight, the at least one image cannot be selected.

[0078] The model can be trained in batches using a batch of images. Each batch of images contains a defined number of images. Once this number of images has been reached in a batch, the model is trained using the images contained in the batch. This training can involve updating the previously trained model parameters based on the images contained in the batch. Alternatively, a model can be trained based on at least some of the images previously used for training and the batch of images. By training the model in batches, it is not necessary to train the model separately for each newly acquired image. This separate training of the model is particularly problematic when many images are to be included in the trained model in a short period of time, as each training session requires a certain amount of training time as well as computing resources.The training in bursts therefore reduces the load on the computer resources, which saves energy.

[0079] Training can take place, for example, on a server, a cloud, etc., that can be assigned to a specific market. Alternatively, training can be conducted regionally, nationally, and / or internationally, for example, for the markets of a department store group, depending on whether products are sold only regionally or nationally and / or internationally in a similar manner.

[0080] In other embodiments, the model can be trained based on the parameters trained up to the time the image was captured and the at least one image. This allows for quick and efficient incorporation of the image into the trained model, as the model does not need to be retrained on all existing images. Alternatively, incorporation into the trained model can include training the trained model on all images captured so far, along with the at least one image. The image database can be an image database specific to the scanned product, and / or the trained model can be a model specifically trained for the scanned product.

[0081] An image database specific to the scanned product contains only images that are exclusively associated with the scanned product. In other words, the product group assigned to the scanned product includes only the scanned product. Similarly, the specifically trained model is a model that is only associated with the scanned product. The training of the specifically trained model can be based on the specific database. For example, the specifically trained model can be trained such that only images associated with the scanned product are labeled with a first label, and all images not associated with the scanned product are labeled with a second label.

[0082] The use of product-specific image databases results in separate image databases that individually consume less storage space than a global and / or product group-specific image database. This leads to more effective and efficient use, as a large database does not need to be loaded or sent. Furthermore, the use of a product-specific database together with a product-specific trained model creates a synergistic effect, as a specific database can already be used for training.

[0083] Similarly, the use of specifically trained models results in less complex and more memory-friendly models. In particular, these models can have a short execution time, which allows for a faster result—i.e., a probability—for the verification of the inserted product. This leads to a smoother and more pleasant shopping process, as the customer doesn't have to wait long for the verification result. At the same time, the specifically trained model, due to its specification, delivers accurate and meaningful results from which direct conclusions can be derived.

[0084] Alternatively, or additionally, the specifically trained model can also be trained based on a specific image database specific to a different product. For example, the specific model can receive images from the product-specific image database as positive examples and images from a product-specific database of a different product as negative examples. Alternatively, or additionally, the negative examples can come from several different specific image databases.

[0085] The method for automated assembly may further comprise outputting the at least one image for manual checking as to whether a recording should be made if the measured weight does not correspond to the target weight, wherein preferably a deviation of the measured weight from the target weight does not exceed a threshold value.

[0086] If the measured weight does not match the target weight within a tolerance specified by the threshold, at least one image is subjected to a manual check. This manual check serves to determine whether the image of the inserted product matches the scanned product. If a match is found during the manual check, the image is added to the image database and / or the trained model. If no match is found, the image can be refused inclusion. Manually selected images can be prioritized during training because there is a higher probability that the at least one image of the inserted product matches the scanned product. Prioritization during training can include a higher weighting of the images. The output of at least one image can occur on the shopping cart itself and / or on a device in the store and / or in a central location.The output at the shopping cart can occur during shopping, preferably based on the time the product is placed in the shopping cart and / or at the end of the shopping trip. When output at the end of the shopping trip, the output can comprise a plurality of outputs. The device in the store is preferably a tablet computer. Furthermore, the output can occur in conjunction with a manual verification of the inserted product. The threshold can be an absolute or relative threshold and can be product- or product group-specific. Furthermore, the threshold can be variable and / or dynamic. Alternatively, the output for manual verification can occur in the event of a deviation between the two weights with a probability of at least 30%, preferably at least 50%.The output of the at least one image can correspond to the output of the at least one image in the process for verifying a product placed in a shopping cart or shopping basket. The manual check ensures that the at least one image of the placed product matches the scanned product. This guarantees the creation of a sufficiently large database, since the images are not immediately denied inclusion in the image database and / or the trained model in the event of a weight deviation. In particular, this also takes into account a manufacturer's weight change of a product, since the weight-based control mechanism is bypassed during manual checking. This can also contribute to an update of the weight database.

[0087] The method for automated assembly may further comprise scanning the scanned product, preferably with a scanner of a shopping cart and / or shopping basket. Alternatively, or additionally, the method may comprise weighing the inserted product, preferably by placing it in a weighing area of ​​the shopping cart and / or shopping basket, wherein the scanned product preferably corresponds to the inserted product. Alternatively, or additionally, the method may comprise creating the at least one image, preferably with at least one camera of the shopping cart and / or shopping basket.

[0088] By scanning a product, the scanned product is identified with the product and is preferably carried out by a customer. The scanned product is, in particular, a product that can be placed in the shopping cart and / or shopping basket. Scanning the scanned product can include scanning a barcode and / or a QR code and / or an RFID tag of the scanned product. In general, scanning a product can include capturing a feature unique to the scanned product. In other embodiments, scanning the scanned product can be based on creating the at least one image with at least one camera.

[0089] The weighing of the inserted product can be based on a weight difference in the weighing area, wherein the weight difference comprises a weight before the inserted product is inserted and a weight after the inserted product is inserted. The weighing area is that part of the shopping cart and / or shopping basket for which a weight change can be registered. The weighing area can comprise a basket of the shopping cart. In other embodiments, the weighing area comprises a basket and at least part of a frame of the shopping cart. The weight difference is preferably determined when the shopping cart and / or shopping basket is stationary. The weighing in the weighing area can be dependent on the scan of the scanner. For example, the weighing area can only be active when the scanner is in use or a product is being scanned.By not continuously activating the weighing area, the cart's power consumption can be reduced, resulting in a longer usage period. This reduces energy costs and the cart's inactivity time. Alternatively, or additionally, the weighing area can be continuously weighed. Weighing can be performed using load cells and / or force sensors.

[0090] The creation of the at least one image by at least one camera of the shopping cart and / or shopping basket can depend on the time of scanning and / or weighing. The at least one camera of the shopping cart permanently records a portion of the cart and saves the resulting image sequences on a memory. If the weighing area of ​​the cart registers a change in weight at a time, a section of the image sequence is selected which includes the time of the weight change. This section can include images of the image sequence before the weight change and after the weight change. For example, the section can begin at least 1 second, preferably at least 2 seconds before the registered weight change and / or extend at most 10 seconds, preferably at most 5 seconds beyond the time of the weight change.Because the section of the image sequence contains the time of the weight change, it is ensured that the inserted product is depicted in one of the images in the image sequence. This allows the image database and / or the trained model to be supplied with suitable image material for construction. Image sequences that do not belong to a section assigned to a weight change are deleted from the memory. This means that only images relevant for verification are saved, preventing memory overflow. In particular, this allows the use of cost-effective storage, as the memory size does not have to be designed for the entire shopping process. Alternatively, or additionally, the section of the image sequence can be selected so that it includes the time of scanning.

[0091] In general, there can be a synchronization process of the trained models and / or the image databases between the shopping cart and the server and / or the cloud. The synchronization process can take place between the shopping carts and the server and / or the cloud of a market. Alternatively, or additionally, the synchronization process can take place between shopping carts from a large number of markets, preferably between a network of regional markets and / or national markets and / or international markets. The synchronization process can be product-specific. This has the effect of accumulating large amounts of image data for a specific product in a short period of time, since many shopping carts, possibly from different markets, contribute to generating image data. Furthermore, the synchronization ensures that, upon completion of the synchronization process, all shopping carts have the same trained model.This ensures the quality of the trained model of each shopping cart for the verification of a product placed in the shopping cart.

[0092] A further aspect relates to a computer program for the automated construction of an image database, comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method described above.

[0093] Another aspect relates to a system for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart and / or shopping basket. This system comprises: means for receiving a target weight of a scanned product; means for receiving a measured weight of the placed product; means for selecting at least one image of the placed product for inclusion in the image database and / or the trained model if the measured weight corresponds to the target weight.

[0094] The image database can be a specific image database for the scanned product, and / or the trained model can be a model specifically trained for the scanned product. Furthermore, the target weight can be based on a weight database.

[0095] The automated assembly system may further comprise at least one scanner for scanning the scanned product and / or at least one weighing means for weighing the inserted product and / or at least one camera for creating the at least one image.

[0096] The system for automated assembly may further comprise means for outputting the at least one image for manual checking whether a recording should be made if the measured weight does not correspond to the target weight, and preferably if a deviation of the measured weight from the target weight does not exceed a threshold value.

[0097] Another aspect includes a shopping cart or shopping basket with a system as described above.

[0098] The shopping cart or shopping basket may further comprise a memory, wherein the memory at least partially comprises the image database, the trained model and / or the weight database.

[0099] Another aspect relates to a method for training a model to verify a product placed in a shopping cart and / or shopping basket. Training the model comprises training using images selected using one of the methods described herein. For example, a model specifically trained for a particular product may be trained using images of that product as "positive examples" and / or images of one or more other products as "negative examples," as described herein.

[0100] Finally, another aspect relates to a trained model and / or an image database for verifying a product placed in a shopping cart and / or shopping basket, which was trained and / or generated using images constructed using a method described herein.

[0101] Further aspects concern the use of the trained model and / or the image database by a shopping cart and / or a shopping basket, which can be configured as described herein, but also the use by other elements, such as permanently installed checkouts, e.g., self-service checkouts, scope checkouts, and / or self-checkout terminals. In general, the trained model and / or the image database can also be used, for example, in the control of any type of order picking that is not limited to retail and / or grocery stores.

[0102] A further aspect can be, for example, a shopping cart, a shopping basket and / or another element, such as a fixed cash register, e.g. self-checkout, scope checkout and / or self-checkout terminal, which is configured to use the trained model and / or the image database.

[0103] Another aspect relates to a method for verifying a product placed in a shopping cart and / or shopping basket, the method comprising: detecting a scanned product; tracking movement of the scanned product during at least a portion of an insertion process; determining, based on the tracking of the movement, whether the scanned product has been placed in the shopping cart.

[0104] The detection of the scanned product may include optical detection of the scanned product and may be performed by at least one camera and / or at least one camera system. Generally, the detection and / or the time of detection may be based on the time of scanning the scanned product.

[0105] The tracking of the movement of the scanned product can be carried out by a camera and / or a camera system. For example, the tracking of the movement can begin with the optical detection of the scanned product. For example, the camera and / or the camera system can be configured to detect a product arranged next to a scanner and then track this in the image. Preferably, the tracking of the movement of the product comprises the insertion process of the product and optionally the placing of the tracked product in the shopping cart and / or shopping basket. For example, the tracking of the movement can end with the completion of the insertion process. The completion of the insertion process can comprise removing a hand or arm and / or other limbs from the shopping basket or shopping cart. For example, the insertion process of a product inserted with a hand or arm can end when the product is in the shopping basket orshopping cart and / or the hand or arm has left the shopping basket or shopping cart. It is also possible that the tracking of the movement is alternatively or additionally coupled with the detection of the weight of the scanned product from the shopping basket or shopping cart. The scanned product can, for example, be tracked until a change in weight is detected. If this matches the expected change in weight for the scanned product, then it can be determined that the product has been placed in the shopping cart, after which tracking can be stopped (optionally after the removal of a hand or arm has been detected). The determination of whether the scanned product has been placed in the shopping cart is based on tracking the movement of the scanned product. The determination can be based on whether the optically detected product was placed in the shopping basket as a result of the movement and / or whether the detected product remains in the shopping basket.For example, the scanned product may not be considered to have been placed in the shopping cart if the scanned product moves in and out of the shopping basket or shopping cart, or if it does not move into the shopping cart at all. In general, determining whether the scanned product has been placed in the shopping cart does not require separate, e.g., optical, identification of the placed product. In other embodiments, the method may include identifying the placed product, preferably based on a trained model, e.g., as described herein.

[0106] The features described herein with respect to a method can also be implemented as features of a corresponding system, a computer program, a shopping cart, and / or a shopping basket, and vice versa. Furthermore, details of the first aspect can be combined with the second aspect, and vice versa.

[0107] 4. Short description of the characters

[0108] Exemplary embodiments of the invention are described below with reference to the figures. The figures show:

[0109] Fig. 1A: Flowchart of a possible embodiment of the method for verifying the inserted product;

[0110] Fig. 1B: Flowchart of another embodiment of the method for verifying the inserted product;

[0111] Fig. 2A: Possible embodiment of a shopping cart with the system for verifying the inserted product and / or for automatically building an image database and / or trained model;

[0112] Fig. 2B: Example of the optically accessible area of ​​a camera mounted on a shopping cart; Fig. 3A: Flowchart of a possible embodiment of the method for automatic setup or automatic training;

[0113] Fig. 3B: Flowchart of another embodiment of the method for automatic building or automatic training.

[0114] 5. Detailed description of preferred embodiments

[0115] Only a few possible embodiments of the invention are described in detail below. It should be understood that these exemplary embodiments can be modified and combined with each other in various ways, wherever compatible, and that certain features can be omitted where they are unnecessary.

[0116] Fig. 1A shows a possible embodiment of a method 100 for verifying a product placed in a shopping cart and / or shopping basket. The method includes receiving 110 at least one image of the placed product, selecting 120 a trained model based on a scanned product, and determining 130 a probability that the at least one image of the placed product shows the scanned product based on the trained model.

[0117] The scanned product is a product that was captured by a scanner 310 and that can be uniquely identified by the scan. The scanning of the scanned product preferably takes place before the inserted product is inserted. Scanning and insertion preferably always take place in pairs, i.e., scanning is always followed by insertion and insertion is always preceded by scanning. Unique identification can be achieved by a feature that is unique to the scanner, such as a barcode, an EAN code, a GTIN code, an RFID tag and / or a QR code. In particular, the scanned product is a product that will be invoiced to the customer at the end of the purchasing process. Since, from the customer's perspective, the amount of this invoice should be as low as possible, the scanned product can differ from the inserted product.For example, a cheap product may be scanned while a high-priced product is placed in the shopping cart and / or basket. It is therefore not immediately clear that the customer will actually pay for the items in the shopping cart at the end of the shopping process.

[0118] Receiving 110 the at least one image of the inserted product comprises receiving images that were created during the insertion process of the inserted product. The at least one image is preferably taken with a camera 330 of the shopping cart. The camera 330 continuously records, i.e., throughout the entire shopping trip, an area 335 of the shopping cart 300 that is visually accessible to the camera, wherein this area 335 comprises at least a portion of the shopping cart 301. The at least one image is a video sequence of the insertion process and is selected from the continuous recording. The selection of the video sequence can depend on the time of scanning and includes a period before the time of scanning, the time of scanning, and a period after the time of scanning. The period before the scanning, the time of scanning, and the period after the scanning are temporally related.As described herein, multiple cameras and / or camera systems may also be used on the shopping cart and / or shopping basket.

[0119] Selecting 120 the trained model based on the scanned product involves selecting a model specifically trained on the scanned product. This trained model is an image recognition model based on neural networks. For example, a convolutional neural network can be used, in particular a YOLO Nano convolutional neural network (see, for example, A. Wong, et al., 'YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detection,' in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurlPS Edition (EMC2- NIPS), Vancouver, BC, Canada, 2019 pp. 22-25. doi: 10.1109 / EMC2-NIPS53020.2019.00013). Selecting 120 the trained model can, in particular, comprise calling or loading an image recognition model from a memory. The training of such a model is based on appropriately prepared training data and a training phase.The correspondingly prepared training data is based on the image database obtained using the method for automatically building an image database.

[0120] The determination 130 of the probability that the at least one image of the inserted product shows the scanned product, based on the trained model for the scanned product, can be performed on a tablet computer 320 belonging to the shopping cart 300. The determination 130 is carried out by applying the trained model to the at least one image of the inserted product. Since the trained model is a model specifically trained for the scanned product, the result of the determination 130 only includes the probability that the at least one image of the inserted product shows the scanned product. Since the product billed to the customer corresponds to the scanned product, the information from the scanner, i.e., the identification of the scanned product, can be used as prior information for image recognition.In other words, the image recognition program does not calculate the probability for all products in the product range, but only for the scanned product.

[0121] The determination 130 can be carried out for a first group of products on the tablet computer 320 of the shopping cart 300 and for a second class of products on a server and / or a cloud. For example, only the trained models for products of the first group can be stored on the tablet computer 320. The products of the first group are products that are frequently sold on the market and for which the calculation 130 takes place frequently. In particular, the memory space of the tablet computer 320 is limited, so that not all trained models can fit there. The second group of products contains less frequently sold products and / or high-priced products for which very complex models are used. For example, the shopping cart can be connected to a wireless network and sends at least one image of the inserted product as well as information about the scanned product to a server and / or a cloud via this network.The information about the scanned product can be any type of information that enables identification of the scanned product from the server and / or the cloud. For example, the information about the scanned product can be the barcode of the scanned product and / or a pointer assigned to the scanned product. In this case, selecting the trained model can involve loading a model specifically trained for the scanned product. The probability determination 130 then takes place on the server and / or the cloud. Once the determination 130 is complete, the result is sent back to the shopping cart via the wireless network.

[0122] If a product from the second group is scanned—i.e., a product for which the probability is determined on a server and / or cloud—and a wireless connection cannot be established between the shopping cart and the server and / or cloud, the shopping cart attempts to establish a wireless connection at regular intervals. Once a connection is established, the data is transmitted. In particular, in this case, the shopping process is not interrupted, and the customer can continue shopping, since the probabilities for the inserted products only need to be available at the end of the shopping process.

[0123] The method may further comprise identifying 230 the inserted product with the scanned product. Identification 230 occurs if the probability is greater than a threshold value. The threshold value may be a probability of at least 80%, preferably at least 90%. If the probability calculation 130 is performed on the tablet computer 320 of the shopping cart, the identification of the products also takes place on the tablet computer 320 of the shopping cart. If the calculation 230 takes place on a server and / or a cloud, the identification may also take place on the server and / or the cloud, and the tablet computer is only informed via the wireless network whether or not identification takes place. Alternatively, the identification may take place on the tablet computer 320 based on the probability calculated on the server and / or the cloud.

[0124] The method 100 may further comprise outputting 240 of at least one image for manual verification of whether identification should occur. Outputting 240 occurs if the calculated probability is less than a first threshold but greater than a second threshold. The output may comprise a video sequence of the insertion process of the inserted product. Alternatively, the output may comprise individual images selected from the video sequence. The output may occur on the screen of the tablet computer 320. After outputting the video sequence, a manual input is requested from the tablet computer, which either identifies the inserted product with the scanned product or defines the inserted product as different from the scanned product. Alternatively, or additionally, outputting 240 may occur on a computer, preferably a tablet computer, managed by store staff and / or a central office.Dispensing can occur separately for each identification of the inserted product, i.e., if a product is recognized with a correspondingly high probability, dispensing takes place directly and immediately. Alternatively, dispensing can take place at the end of the shopping process, preferably during the activation of a payment function and / or a payment transaction, and comprise the joint dispensing of all inserted products that were not identified with a sufficiently high probability. Alternatively, dispensing 240 can depend on a spatial position of the shopping cart in the store. For example, dispensing 240 can occur when the cart is pushed into a payment area of ​​the store, which is preferably located in a vicinity of the store exit.

[0125] Alternatively, or additionally, dispensing 240 can occur with a dispensing probability. If the calculated probability for the inserted product lies between the first and second threshold values, dispensing 240 does not always occur, but only with a certain dispensing probability. This dispensing probability is at least 30%, preferably at least 50%. Alternatively, or additionally, dispensing 240 can always occur as soon as the calculated probability is less than the second threshold value.

[0126] Fig. 1B shows another embodiment of a method 200 for verifying a product placed in a shopping cart and / or shopping basket. The method 200 includes receiving 210 a measured weight of the placed product and comparing 220 the measured weight with a target weight of the scanned product. The receiving 210 of the measured weight takes place in conjunction with weighing the placed product. The weighing of the placed product can be performed by weighing means 340a, 340b of the shopping cart. The target weight of the scanned product is part of a weight database that includes target weights for all products in the product range. The weight database can be stored locally on the tablet computer 320 of the shopping cart 300, on a server, and / or in a cloud.Alternatively, or additionally, at least part of the weight database is located on the tablet computer 320 and another part is located on the server and / or in the cloud. The weight database can be dynamic, i.e., the target weights of the products represented in the database can be changed. The target weight of the scanned product can be the mean value of a weight distribution of the scanned product. Furthermore, the target weight can contain a variance of the weight distribution. In general, the method 200 can additionally include receiving 410 a target weight of a scanned product. The receiving 410 can occur before or after the receiving 210, but preferably before the comparison.

[0127] 220. The comparison 220 can take place on the tablet computer 320 or on a server and / or a cloud. If the comparison 220 takes place on the server and / or the cloud, the shopping cart sends the measured weight of the inserted product and an identifier of the scanned product to the server and / or the cloud via the wireless network. The identifier of the scanned product is configured such that the server and / or the cloud can retrieve the corresponding target weight from the weight database. In general, the comparison 220 can include calculating a difference between the measured weight and the target weight.

[0128] If the measured weight corresponds to the target weight, at least one image of the inserted product is received 110. For example, a video sequence is only selected if the two weights correspond. If the comparison 220 concludes that the two weights do not correspond, an error message can be output via the tablet computer 320. This error message prompts the customer to remove the inserted product from the shopping cart 301 and place a product corresponding to the scanned product into the shopping cart 301.

[0129] Fig. 2A shows a possible embodiment of a shopping cart 300 with the system for verifying the inserted product and the system for automatically building an image database and / or trained model. The shopping cart 300 includes a shopping basket 301, a lower storage level 302, a chassis 303, a handheld scanner 310, a tablet computer 320, a camera system 330, and a plurality of load cells 340a, 340b. The handheld scanner 310 is connected to the tablet computer 320 and configured to scan the scanned product. In other embodiments, the shopping cart may, for example, include exactly one load cell.

[0130] The camera system 330 comprises a plurality of cameras (not shown) which capture at least a portion of the shopping cart 301. The cameras can be configured to create at least one image of the inserted product. The optically accessible area 335 is the area of ​​the shopping cart captured by the cameras. In addition, the camera system 330 can be configured to also optically capture at least a portion of the lower storage level 302. Preferably, there is one camera for the shopping cart 301 and one camera for the lower storage area 302. In general, the cameras should be configured to have a frame rate of at least 24 fps, preferably at least 30 fps. The camera system is connected to the tablet computer 320.

[0131] The load cells 340a, 340b are connected to the tablet computer and configured to register a change in the weight of the shopping cart 301 and / or the lower storage level 302. In particular, the load cells can be configured to weigh a product placed in the shopping cart 301 and / or the lower storage level 302. In general, the load cells can register a change in the weight of the shopping cart.

[0132] 301 from the load cells to register a weight change of the lower storage level

[0133] 302 may be different. The load cells are configured to resolve a weight change with an accuracy of at least 5 grams, preferably at least 2 grams. In general, comparing 220 the measured weight with the target weight may depend on the accuracy of the load cells.

[0134] Furthermore, the shopping cart 300 and / or the tablet computer 320 have interfaces for sending and receiving data via a wireless connection, preferably a WLAN connection. The data can be sent to a server and / or a cloud. The tablet computer 320 includes a memory and processors, wherein at least a portion of the image database and / or a portion of the weight database is stored in the memory of the tablet computer 320. The memory of the tablet computer is, for example, at least 5 GB, at least 50 GB, at least 100 GB, or preferably at least 128 GB.

[0135] Fig. 2B shows a portion of the optically detectable area 335 of the shopping basket 301 of the shopping cart 300 from the perspective of a camera of the camera system 330, which is mounted near the tablet computer 320 and / or on the tablet computer 320 itself. In particular, Fig. 2B shows the process of inserting a product P into the shopping basket 301.

[0136] Fig. 3A shows a possible embodiment of a method 400 for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart 300 and / or shopping basket. The method comprises receiving 410 a target weight of a scanned product, receiving 420 a measured weight of the placed product, and selecting 430 at least one image of the placed product for inclusion in the image database and / or the trained model if the measured weight corresponds to the target weight. Preferably, the method 400 takes place concurrently with the method 200 and is carried out by the customer during the shopping process. The scanned product is preferably a product scanned with the scanner 310. The target weight of the scanned product is based on a weight database.Receiving 420 the measured weight of the inserted product takes place during an insertion process of the inserted product and is preferably performed by the load cells 340a, 340b. In general, method step 420 may correspond to method steps 210.

[0137] Fig. 3B shows another possible embodiment of a method 500 for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart 300 and / or shopping basket. The method comprises receiving 410 a target weight of the scanned product and receiving 420 a measured weight of the placed product. If the measured weight does not correspond to the target weight, at least one image is output 510 for manual checking as to whether a recording should be made. The output of the at least one image of the placed product can include a video sequence of the insertion process. The output can be on the display of the tablet computer 320 of the shopping cart 300 and / or on a computer associated with the store and / or a central office, which is preferably a tablet computer.Output 510 for manual inspection may at least partially coincide with output 240. If a recording is to be performed after output 510, the at least one image of the inserted product is selected 430 for inclusion in the database and / or the trained model. In particular, the identification of the inserted product with the scanned product during output 240 may lead to the selection 430 of the at least one image of the inserted product for inclusion in the image database and / or the trained model. If the measured weight corresponds to the target weight, selection 430 occurs directly, without output 510.

Claims

Claims 1. A method (400, 500) for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart and / or shopping basket, the method comprising: Receiving (410) a target weight of a scanned product; Receiving (420) a measured weight of the inserted product; Selecting (430) at least one image of the inserted product for inclusion in the image database and / or the trained model if the measured weight corresponds to the target weight.

2. The method (400, 500) for automated assembly according to claim 1, wherein the image database is an image database specific to the scanned product and / or wherein the trained model is a model specifically trained for the scanned product.

3. Method (400, 500) for automated assembly according to one of claims 1 or 2, wherein the target weight is based on a weight database.

4. A method (400, 500) for automated assembly according to any one of claims 1-3, further comprising: Scanning the scanned product, preferably with a scanner (310) of a shopping cart and / or shopping basket; and / or Weighing the inserted product, preferably by placing it in a weighing area of ​​the shopping cart and / or shopping basket, wherein the scanned product preferably corresponds to the inserted product; and / or creating the at least one image, preferably with a camera (330) of the shopping cart and / or shopping basket.

5. A method (400, 500) for automated assembly according to any one of claims 1-4, further comprising: if the measured weight does not correspond to the target weight, and preferably wherein a deviation of the measured weight from the target weight does not exceed a threshold value; Outputting (510) the at least one image for manual checking as to whether a recording should be made.

6. A computer program for the automated construction of an image database, comprising instructions which, when executed by a computer, cause the computer to carry out the method (400, 500) according to any one of claims 1-5.

7. A system for automatically building an image database and / or for automatically training a model for verifying a product placed in a shopping cart and / or shopping basket, comprising: Means for receiving (410) a target weight of a scanned product; Means for receiving (420) a measured weight of the inserted product; Means for selecting (430) at least one image of the inserted product for inclusion in the image database and / or the trained model if the measured weight corresponds to the target weight.

8. The automated assembly system of claim 7, wherein the image database is an image database specific to the scanned product and / or wherein the trained model is a model specifically trained for the scanned product.

9. The automated assembly system of claim 7 or 8, wherein the target weight is based on a weight database.

10. The automated assembly system of any of claims 7-9, further comprising: a scanner (310) for scanning the scanned product; and / or a weighing means (340a, 340b) for weighing the inserted product; and / or a camera (330) for creating the at least one image.

11. System for automated assembly according to any one of claims 7-10, further comprising: Means for outputting (510) the at least one image for manually checking whether a recording should be made if the measured weight does not correspond to the target weight, and preferably if a deviation of the measured weight from the target weight does not exceed a threshold value.

12. Shopping cart (300) or shopping basket with a system according to one of claims 7-11.

13. Shopping cart (300) or shopping basket according to claim 12, further comprising a memory, wherein the memory at least partially comprises the image database, the trained model and / or the weight database according to claim 9.

14. A method for training a model for verifying a product placed in a shopping cart and / or shopping basket, comprising: Training the model using images selected by a method according to any one of claims 1-5.

15. A trained model for verifying a product placed in a shopping cart and / or shopping basket, which model has been trained using images selected using a method according to any one of claims 1-5.

Images