Navigation in a supermarket

Abstract

Method for navigating a user (16) and thus a customer in a supermarket (10) using an application (“app”) running on his mobile device comprising the following steps: - Selecting a product (14D) using the app on the mobile device, - Establishing a connection between the app and a server (20) on which location information of the goods (14) of the supermarket (10) is stored in a database, - Determining the customer's location (22A) using their mobile device within the supermarket (10), - Querying the location information of the selected goods (14D) in the database, thereby determining a target position, - Navigation of the customer from their current location (22A) to the target location (22D) of the goods (14D) using the app on their mobile device, - wherein the customer, from his location (22A), takes a picture of a product (14A) and of an information field associated with the product using a camera of the mobile device, wherein the information field associated with the product is a QR code - wherein this image capture is transmitted from the mobile device to the server (20), - wherein the image capture on the server (20) is assigned to a product whose location information is known, wherein the assigned location information is extracted, - where the extracted location information from the image capture is defined as the user's location (22A), characterized by that an image analysis of the image capture is performed on the server (20), and that the server automatically selects the most resource-efficient method of image analysis for it, whereby the most resource-efficient method of image analysis is an analysis of the QR code, whereby further locations (22B, 22C) of the customer during his navigation to the destination (22D) are determined by means of a pedometer built into the mobile device, a gyroscope and in conjunction with map data of the supermarket.

Description

[0001] The invention relates to a method for navigating a user, i.e. a customer, in a supermarket and to a navigation system suitable for carrying out the method.

[0002] Using an application ("app") on their mobile device, the customer selects a product they wish to purchase in the supermarket. A connection is established between the mobile device and a server, which stores location information in a database in the form of position data of the supermarket's products. The corresponding data transfer from the server to the mobile device can be realized via the mobile internet, via hotspots distributed throughout the supermarket, and / or via NFC chips. Regarding the NFC chips, the supermarket can install corresponding NFC columns, preferably in the entrance area.

[0003] As part of the process, the customer's location within the supermarket is determined using their mobile device, allowing a starting position for navigation to be established. To determine a destination, the location information of the selected product within the supermarket is queried from the database. Once the customer's location and the destination location of the product are known, the customer is navigated from their location to the destination location of the product using the app.

[0004] Common outdoor navigation systems, such as those used for route guidance in cars, use satellite navigation to guide a user from their starting point to their destination. For satellite navigation, especially GPS navigation, to work, a connection between a plurality of satellites and the corresponding navigation device is necessary, which is as uninterrupted as possible. GPS navigation fails for navigation indoors, especially in a supermarket, because communication with the satellites is disrupted by the building's walls. However, especially in supermarkets, significantly more precise navigation is required than on the road.

[0005] Indoor navigation using Wi-Fi has been known for some time. This involves recording the transmission power of Wi-Fi access points. Based on the received Wi-Fi signals and their corresponding signal strength, the position of the navigation device can be triangulated quite accurately. Location determination using a so-called beacon is based on a similar process. This uses a Bluetooth connection, for example, and in particular according to the Bluetooth 4.0 LE (for "Low Energy") standard, and has the advantage that devices and positioning signals consume little power. Orientation works in a similar way to Wi-Fi, namely by measuring the reception strengths of the Bluetooth transmitters distributed throughout the building, i.e. the beacons. This process is used, for example, for orientation in museums. However, these well-known indoor navigation methods are complex, since there are numerous Wi-Fi access points or beacons throughout the building.Beacons must be installed to enable comprehensive navigation.

[0006] US 2016 / 0146614 A1 teaches that a system for computer-aided visual product recognition can assist a customer in navigating to a selected product in a retail store. A customer can specifically point the camera of a mobile computing device at one or more products displayed on the shelves of a store. Image recognition operations can be performed to compare the target product images with images from a pre-populated product image bank of known products. If a positive match is found, shelf location information of the identified target products can be used to determine the customer's current location. The target location in the store is defined by the shelf location for the selected product. A route can be calculated to navigate the customer from the current location to the target location.

[0007] The method is therefore based on the task of specifying a method and a navigation device with which indoor navigation can be realized using simple means.

[0008] This object is achieved by the features of the independent claims. Advantageous embodiments of the invention emerge from the dependent claims.

[0009] The key aspect of the invention is that the customer, from their location, uses a camera on their mobile device, in particular their smartphone, to take a picture of goods or information fields associated with the goods. The app transmits this image to the server using the mobile device, in particular via WLAN or mobile network. The server assigns the image to a product that is stored in its database. Since the location information for the corresponding product is stored in the database, the location information can be extracted by the server and defined as the customer's location. The customer's location and their destination, where the product desired by the customer is located, are now known, so that the customer can navigate using the app.

[0010] Preferably, the camera of the mobile device takes additional images "quasi in real time" of goods located along the navigation path while the customer is navigating, and their location information is also extracted from the server, so that the customer's current location can be determined and continuously updated in order to navigate the customer along the route, i.e., the navigation path. If information regarding the mobile device's spatial orientation is required for this navigation, a gyroscope built into the mobile device can preferably be used for this purpose. The position of the mobile device can be "calibrated" and verified using the respective images.

[0011] Preferably, pattern recognition or image analysis of the image recording is carried out on the server by a computing unit with a suitable algorithm implemented. Pattern recognition or image analysis is helpful in identifying certain features of the photographed product and being able to clearly assign these to a product in the database. For example, it may be useful to initially only analyze the outline of a bottle if it is not possible to assign a specific product in the database to the photograph, for example due to poor image quality. The pattern recognition then does not recognize that it is a bottle of brand XYZ, but can still navigate the customer to the drinks department because the shape of the bottle is recognized as a pattern. Pattern recognition or image analysis is also useful if information panels assigned to the products are being analyzed.This makes it possible to decode a QR code or barcode and extract the information contained within it, especially the location information. Price tags on goods can also be easily analyzed and assigned to products in the database. The algorithm on the server automatically selects the most resource-efficient method. For example, pattern recognition of a QR code can require less computing power than a complex image analysis of the product.

[0012] In a practical embodiment of the invention, the app prompts the customer to take another picture if the pattern recognition or image analysis of the previous picture cannot clearly identify a product. In a supermarket, similar products may be located at different locations, meaning that simply determining the "type" of a product is not enough to determine the user's location. A poor image may, for example, be caused by shaking during the picture being taken or by other customers being in the frame at the time the picture is taken and obscuring the product. Taking another picture can, if necessary, improve the quality of the image and clearly identify the product.If a clear assignment is not possible even after taking further pictures, the customer can also enter the name of the product interactively via his mobile device, for example by entering "I am in front of the Hochland cottage cheese", so that his location can be determined.

[0013] It can prove tedious for the customer to constantly take pictures of other products to update their location, so other navigation methods can be used as an alternative or support. These are particularly suitable when very precise spatial resolution is no longer necessary, because additional map material from the supermarket can also be used to support navigation. Thus, it may be sufficient if the customer, after their current location has been precisely determined, is initially navigated approximately in the direction of the destination or target position at the beginning of the navigation. At a later stage, i.e., shortly before the destination, the navigation is again realized using more precise location determination based on image analysis.One of these additional navigation methods can be implemented using a built-in pedometer and / or gyroscope in the mobile device, which can detect changes in the customer's location. This makes it possible to determine, at least approximately, how far the customer has moved and in which direction within the supermarket. If necessary, individual images can be taken to confirm the position and / or to adjust and calibrate the mobile device, or the algorithm stored on it, to the user's stride. Another of these navigation methods can be determined by measuring the reception strengths of radio transmitters distributed throughout the supermarket.The advantage here is that significantly fewer radio transmitters are required within the supermarket than with the described state-of-the-art technology, since the "precise" location determination is achieved via image capture, and only a rough orientation needs to be provided via the radio transmitters. All of these methods can also be combined.

[0014] Preferably, the selected product is visually highlighted in its surroundings at the target location on a device display and / or on virtual reality glasses. In augmented reality ("AR") style, the customer can use their smartphone display as an eye into the surrounding world, with the selected product being recognized and visually highlighted on the display. Since similar products are usually located next to each other on a supermarket shelf, this guides the customer to their selected product. Preferably, the customer can also be guided along the entire navigation path using augmented reality.

[0015] In a preferred embodiment of the invention, the customer is navigated by a graphic output on the display and / or by a voice output and / or by a vibration pattern on the mobile device or another mobile device that is connected to the mobile device. The graphic output on the display can guide the user to the appropriate changes of direction using arrows and metric information. With voice output alone, the customer advantageously does not necessarily have to hold the mobile device in their hand, but can also hear the voice commands through headphones. With a vibration pattern, the customer can, for example, leave their mobile device in their trouser pocket and be prompted to change direction along the route. A shortening interval between vibration pulses can thus indicate that a fork in the road has been reached and the direction must be changed.For example, upon reaching a fork in the road, a predetermined number of short pulses can indicate to the customer that they should take the left path, and a predetermined number of longer vibration pulses can indicate to the customer that they should take the right path to reach their destination. Preferably, the vibration pattern can be defined and set by the customer themselves via the app. It is also possible for the mobile device to be connected to a smartwatch, and for the navigation instructions to be implemented via the smartwatch. This also allows the user to avoid having to constantly hold the mobile device in their hand.

[0016] According to a further aspect of the invention, a system for navigation is provided which is designed to carry out the method according to the invention described above and comprises a mobile terminal with an app suitable for navigation, a database with market data on a server and a data connection between the server and the terminal.

[0017] The information fields assigned to the goods are preferably QR codes, barcodes, or other coded character or image sequences. This enables a high degree of flexibility in the provision of information and, where appropriate, easy pattern recognition. A smartphone, a tablet, and / or virtual reality glasses are preferably used as the mobile device. If a corresponding app cannot be installed on the virtual reality glasses, the virtual reality glasses can be used in conjunction with the smartphone, with the two communicating via Bluetooth, for example. All of these devices are characterized by the fact that they have a camera, which is a prerequisite for the method according to the invention.

[0018] Further advantages, features and developments of the invention emerge from the dependent claims and from the following description of a preferred embodiment.

[0019] The invention is explained in more detail below with reference to the accompanying drawings using a preferred embodiment. Fig. 1: shows an inventive method for navigation in a supermarket. Fig. 2: shows a visually highlighted product on a smartphone display.

[0020] Numerous features of the present invention are explained below using preferred embodiments. The present disclosure is not limited to the specifically mentioned feature combinations. Rather, the features mentioned here can be combined in any way to form embodiments of the invention, unless explicitly excluded below.

[0021] The invention is based on the idea of ​​a smart supermarket that, like other devices in the field of the Internet of Things (IoT), is intended to simplify the lives of customers. The basic idea is to provide a type of navigation system for shopping in a supermarket. The navigation system is implemented using the inventive method described in more detail below, which has the following properties: The main task is to navigate the customer to the products of their choice. In order to find the desired product as quickly as possible, the invention provides for the customer to create shopping lists before shopping. These shopping lists can take budgets and recipes into account, and product suggestions can be accepted and added to the shopping list.Time calculation is also preferably carried out, which calculates the time spent at checkouts in real time and can thus redirect the customer to a checkout where the waiting time is minimal. This is preferably implemented via an application, a so-called app, which runs on a smartphone. This also enables the seamless integration of health applications, such as Apple Health, which takes the current fitness level into account when navigating the supermarket. For example, the customer can be specifically redirected around the confectionery section if their weight development is heading in an unfavorable direction or if they indicate this in the app. Automatic allergy warnings can be issued, and the corresponding products can be avoided, but shopping can also be carried out according to a predetermined diet.

[0022] Augmented reality can make shopping even more interactive. The smartphone camera captures real-time images of the supermarket and displays the route in 3D on the smartphone display. Products can then be illuminated and, if necessary, provided with additional information (calories, contents, organic content, etc.). This can be particularly beneficial for young people and children when shopping. The app can also be useful for enabling cashless payments.

[0023] The invention enables the customer to monitor their spending and lifestyle, quickly find ingredients for recipes, cook at home, and live healthier. The method according to the invention can be used to deliver push notifications to a first-time consumer, display recommended products, and provide statistics and ratings on purchases, with data being conveniently collected and evaluated from the first-time consumer.

[0024] Implementing the process using the customer's own smartphone offers the advantage that there is no need to attach an additional navigation device to the shopping cart, which saves the supermarket hardware costs and eliminates the need to provide a separate power supply for these devices. This also prevents the supermarket's navigation devices from being stolen. The process can also provide for the customer to first be navigated to the supermarket of their choice. Upon arrival, their smartphone can connect to an NFC column in the supermarket using the installed app. This allows market data to be synchronized, as the NFC column has access to a database of market data stored on a server. Access to the database can also be via the mobile data network.The app also shows the customer a checklist, which can be checked off automatically or manually. Automatic processing of the checklist is possible because the mobile device "knows" that the customer has just reached the location of a specific item on their shopping list. Using the app, the customer can rate the supermarket. Preferably, all communication between the smartphone and the server assigned to the supermarket is encrypted.

[0025] Fig. 1 shows the inventive method for navigation in the supermarket 10, in which a plurality of shelves 12 are present in which the goods 14 or products 14 are stored. The customer 16 enters the supermarket through an entrance 18 with their smartphone on which the navigation app is installed. The customer 16 wishes to purchase the products 14B-D in the shortest possible time, i.e., via the shortest possible route, but they do not know where they are located. In order to enable the inventive method, the current location 22A of the customer 16 must be known as precisely as possible. For this purpose, the customer 16, after entering the supermarket 10, takes a picture of the product 14A with the camera of their smartphone.The app now sends this image recording either via a mobile data network or via the supermarket's WLAN to a server 20 which is assigned to the supermarket 10 and on which the location information of the goods 14 within the supermarket 10 is stored in a database.

[0026] A processing unit of the server 20 performs an image analysis or pattern recognition, searching for a match between the image of the product 14A and other products stored in the data set. For this purpose, images of the products or specific optically identifiable properties of the products can also be stored in the data set. If a clear match is found, the next step is to extract the location information of the clearly matching product 14A from the data set and mark it as the customer's current location 22A.

[0027] Using the previously created shopping list, the location information of the goods 14B-D is also extracted. An algorithm implemented on the server 20 then calculates an optimal navigation path 24 for the customer 16. The location information or the location position of the goods 14B is defined as a first intermediate destination 22B, the location information of the goods 14C is defined as a second intermediate destination 22C, and the location information of the goods 14D is defined as the target position 14D. The smartphone now guides the customer 16 along the optimized route past all of his selected goods 14B-D in the order 22B, 22C, and 22D.

[0028] By taking further pictures of goods 14 along the navigation path 24, the location of the smartphone can be determined again if necessary. However, the customer 16 can also see the entire navigation path 24 on the display of his smartphone as in Fig. 1 can be visually displayed, so that the customer only needs to memorize the navigation path 24 to find his or her goods 14. However, the larger a supermarket is, the more difficult it is for the customer 16 to perform this mental navigation task. In this case, the customer 16 is navigated along the entire navigation path 24 by their smartphone. For this purpose, the method can utilize a built-in pedometer and a built-in gyroscope of the smartphone in conjunction with map material of the supermarket. The radio strengths of WLAN hotspots within the supermarket 10 can also be used for navigation, with images of goods being used to determine the exact location. Navigation using the images alone is possible.

[0029] The smartphone's navigation instructions can be communicated to the customer 16 in graphical form, in the form of voice commands, and / or as a vibration pattern. The smartphone can also interact with other devices, such as a smartwatch or VR headset.

[0030] The customer 16 can also be navigated through the supermarket 10 using augmented reality. The camera of the smartphone or the VR headset records the surroundings of the supermarket surrounding the customer 16 in real time and displays them on the smartphone display or the VR headset along with the navigation instructions. At the same time, the images captured by the camera are transmitted to the server 20, preferably in compressed form, and the current location is determined by the algorithm by accessing the database.

[0031] Fig. 2 shows how augmented reality can be used to visually highlight the selected product in a shelf 12 on the display 26 of the smartphone. Fig. 2 shows that the selected product 14D, in contrast to the other products 14 in the product shelf 12, is optically highlighted and is additionally marked by optical position designations, in particular by arrows 28, so that the customer 16 can find the product 14D as easily and quickly as possible.

Claims

[1] Method for navigating a user (16), i.e. a customer, in a supermarket (10) with the aid of an application (“app”) running on his mobile device, comprising the following steps: - Selecting a product (14D) using the app on the mobile device, - Establishing a connection between the app and a server (20) on which location information of the goods (14) of the supermarket (10) is stored in a database, - Determining the customer's location (22A) using his mobile device within the supermarket (10), - querying the location information of the selected goods (14D) in the database, whereby a target position is determined, - Navigation of the customer from the location (22A) to the destination position (22D) of the goods (14D) using the app on his mobile device, - wherein the customer takes a picture of a product (14A) and of an information field assigned to the product from his location (22A) using a camera of the mobile device, wherein the information field assigned to the product is a QR code - wherein this image recording is transmitted from the mobile terminal to the server (20), - wherein the image recording on the server (20) is assigned to a product whose location information is known, wherein the assigned location information is extracted, - wherein the extracted location information of the image recording is defined as the location (22A) of the user, characterized by , that an image analysis of the image recording is carried out on the server (20), and that the server automatically selects the most resource-efficient method of image analysis for it, wherein the most resource-efficient method of image analysis is an analysis of the QR code, wherein further locations (22B, 22C) of the customer are determined during navigation to the destination (22D) by means of a pedometer built into the mobile terminal, a gyroscope and in conjunction with map material of the supermarket. [2] Method according to one of the preceding claims, characterized by that the app asks the user to take another picture if the pattern recognition of the previous picture cannot clearly identify a product. [3] Method according to one of the preceding claims, characterized by that further locations (22B, 22C) of the user during navigation to the destination (22D) are determined by measuring reception strengths of radio transmitters distributed in the supermarket (10). [4] Method according to one of the preceding claims, characterized by that the selected product (14D) is optically highlighted in its surroundings at the target position on a display of the mobile device and / or on virtual reality glasses. [5] Method according to one of the preceding claims, characterized by that the user is navigated by a graphic output on a display and / or a voice output and / or a vibration pattern on the mobile device or another mobile device that is connected to the mobile device. [6] System for navigating a customer in a supermarket using a smartphone for carrying out a method according to any one of claims 1-5, comprising: a smartphone with a camera and an app suitable for navigation, a server (20) assigned to the supermarket (10) on which a database with location information of the goods of the supermarket (10) is stored, an algorithm implemented on the server (20) for calculating a navigation path (24) and a data connection between the server (20) and the smartphone. [7] System according to claim 6, characterized by that the associated information fields are price labels of the goods, QR codes, bar codes and / or other codable character or image sequences.

Images