Route generation method

Abstract

A method for generating a route is provided. The present application also provides a system for in-store routing using existing store cameras and artificial intelligence as well as machine learning to enable user-generated shopping lists. The system uses pixel buffer comparisons of real-time imaged items compared to a database of machine learned images. The system also provides item identification and detection through machine learning to improve shopper experience. The system and method also include drone assistance schemes and radio signal articles and biological detection to improve accuracy. Other features to improve guidance and accuracy include landmark navigation and masking to improve the accuracy of item identification and detection. The system can be an independent self-service terminal.

Description

[0001] This application is a divisional application of the application filed on September 4, 2019, with application number 201980081100.2 and entitled "System, apparatus and method for item location, inventory creation, route planning, imaging and detection".

[0002] Cross-references to related applications

[0003] This application claims priority and benefit to U.S. Provisional Application Serial No. 62 / 691,927, filed June 29, 2018; U.S. Provisional Application Serial No. 62 / 828,153, filed April 2, 2019; U.S. Provisional Application Serial No. 62 / 720,560, filed August 21, 2018; U.S. Provisional Application Serial No. 62 / 776,671, filed December 7, 2018; U.S. Provisional Application Serial No. 62 / 792,044, filed January 14, 2019; U.S. Provisional Application Serial No. 62 / 794,058, filed January 18, 2019; U.S. Provisional Application Serial No. 62 / 839,234, filed April 26, 2019; and U.S. Utility Application Serial No. 16 / 458,795, filed July 1, 2019, all of which are incorporated herein by reference. Technical Field

[0004] This specification generally relates to systems and methods for assisting in item location (such as shopping or warehouse item location and tracking), detection, and route planning, and more specifically to systems and methods for using user-generated shopping lists and in-store SKU databases, machine learning, drone-assisted methods, image recognition, and detection to determine the fastest and most efficient routes through stores or warehouses to facilitate faster and more efficient checkout and / or identification processes. Background Technology

[0005] Shopping lists are well-known in the art. People typically create shopping lists on their mobile devices or using traditional paper. Mobile applications are increasingly used for online shopping and item retrieval information. Specifically, visiting a store's website to determine if a particular store has an item in stock and the price of each item is known in the art. However, these previously known shopping methods are outdated and often do not provide the most accurate and up-to-date information. A common problem is that users will check the availability of an item online to confirm that the product is available, only to find that it is unavailable once they arrive at the store. These existing systems are not updated in real time and do not use location information from a database of specific physical stores, making them inconvenient for users.

[0006] Therefore, there is a need for alternative and efficient shopping systems and methods for creating lists, verifying availability, and generating easily accessible route planning information for users. Furthermore, when using such systems and methods, the checkout process needs to be streamlined.

[0007] The use of various methods such as infrared and sonar to detect the physical presence of living organisms (humans, animals, etc.) and objects (any inanimate objects) is well known in the art. However, these methods cannot determine additional biophysical information and often fail to detect the presence of an object when it is located behind another physical object.

[0008] Therefore, there is a need for improved methods and systems to accurately detect the wavelengths, frequencies, and general properties of organisms and things. Summary of the Invention

[0009] A system for in-store route planning for a user-generated shopping list, the system comprising: a global positioning system (GPS) that determines the user's location to determine a specific store where the user is currently located; a processor connected to a SKU database of the specific store, the SKU database including product prices, product locations, and product availability information, wherein the processor is configured to determine the most efficient route based solely on location information from the SKU database of the specific store, with products categorized by aisle and / or location; and mapping the location information onto a predetermined store layout map using mapping points to determine the fastest route between the mapping points, wherein each mapping point specifies a specific product location. In some embodiments, the processor connects to the SKU database before the user enters the store location. The processor may be configured to display product availability information from the GPS and the SKU database as the availability of products in the shopping list to the user on a user display screen. One option is to provide the fastest route in an instruction list that includes product location information. In some embodiments, to optimize the provided routes for efficiency, the system is configured to create multiple lists, and / or the user can choose which list to start shopping from. In addition, in some embodiments, SKU database information is sent to the user in real time.

[0010] In another embodiment, a system, device, and system are provided that allows a user to check out at a store. The system includes: a mobile application that allows the user to generate a shopping list; a self-service terminal that wirelessly communicates with the mobile application, the self-service terminal having at least one camera communicating with a database of learned images, the database being continuously updated based on previous transactions within the store; and a processor that compares physically present items with the database of images, the processor comparing the physically present items with items in the user's shopping list on the mobile application, the processor comparing items in a shopping cart with the shopping list to confirm the items in the cart and / or the corresponding total payment. In some embodiments, the mobile application allows wireless payment. In some embodiments, the self-service terminal includes a scale configured to weigh items of variable weight, the scale being configured to send weight information to the mobile application. The self-service terminal may include multiple cameras. The self-service terminal may communicate with cameras physically spaced apart from itself. Furthermore, the self-service terminal may communicate with existing security cameras in the store. In some embodiments, the camera is a depth-sensing camera.

[0011] In another embodiment, a system is provided for authenticating items collected by a user in a store. The system includes: at least one camera in a shopping location within the store, the camera capable of viewing the user and their surroundings in a frame of the shopping location, the camera configured to detect training items picked up by the user in real-time, the camera detecting only the training items, wherein the training items are products having data stored in an image database corresponding to the store; and a processor communicating with the image database, the processor using machine learning to compare the real-time training items with the image database to determine specific products. In some embodiments, data collected from the real-time training items is stored in the image database for future use, thereby improving the accuracy of the system. Additionally, an option is to use pixel buffer comparison to authenticate the training items within the frame. In some embodiments, the processor determines an identifier and a confidence score. The system may also include a mobile application used by the user to generate a shopping list, the processor communicating with the mobile application to confirm that the correct products have been added to the shopping list by comparing the real-time collected data with the user's shopping list. The system may include a mobile application configured to generate a shopping list, and the processor communicates with the mobile application to add specific products to the shopping list.

[0012] In another embodiment, a system and method are provided for authenticating items during a checkout process, the system and method comprising: an image database containing a plurality of database images, the image database including a plurality of images of individual items; at least one camera configured to capture checkout photographs of the items during the checkout process for authentication of the items; and an authentication system configured to use pixel buffer comparison to compare the checkout images with the database images for authentication of the items, wherein the system is configured to store the checkout images in the image database such that the checkout images become database images, thereby increasing the image database to improve accuracy and precision through machine learning.

[0013] In another embodiment, a system and method using artificial intelligence to identify items in a store using a mobile device camera includes: an image database containing multiple database images, the image database including multiple images of individual items; the mobile device camera configured to view the images in real time; a processor configured to compare the images with the multiple database images in real time to determine an item that a user points the mobile device camera at; and a system configured to allow the user to interact with the identified items within an application, wherein the interaction functions include taps, voice, clicks, and / or gestures.

[0014] In another embodiment, a bag configured to weigh items includes: a holding section configured to hold a product selected by a user during shopping; a weighing device configured to weigh items placed in the holding section; an accelerometer configured to detect rotation or movement to prevent erroneous readings; and wherein the bag and corresponding system are configured to zero the weight reading after the user has weighed a first item, thereby allowing the user to weigh a second item in the same bag without removing the first item from the bag. The bag may also include a processor. In some embodiments, the processor communicates with a mobile application to send weight information related to the product. The processor may communicate wirelessly with the mobile application. The bag may also include storage capacity for storing weight information related to variable-weight items contained within the bag. In some embodiments, the bag may also include sensors for connecting and communicating with a checkout counter at a store.

[0015] In another embodiment, a drone-assisted system configured to assist users in shopping within a store includes: a drone communicating with a user's mobile device, the drone having light projection features, the drone communicating with a processor having specific locations of items in the store, and the drone being configured to receive the item location from the processor after the user requests the item location, the drone being configured to locate the item within the store and illuminate individual items within the store to indicate the exact location of the item to the user. In some embodiments, the drone includes an onboard camera and a corresponding processor. In some embodiments, the processor is configured to perform object detection and recognition via the camera. Furthermore, the processor may be configured to perform facial recognition via the camera. In some embodiments, the camera and the processor are configured to track the user in the store.

[0016] In another embodiment, a drone system includes: a shelving unit having a top shelf; and a wireless charging pad configured to charge a drone, the wireless charging pad being located on the top shelf. In some embodiments, the shelving unit is configured to hold items in a store. The shelving unit may be arranged to form an aisle in the store, and a power cable may extend through the shelving unit to power the wireless charging pad.

[0017] In another embodiment, a system for imaging objects includes: a radio transmitter configured to transmit a signal projected onto a desired object or area, the signal traveling through a solid surface and creating a return signal upon interaction with the surface or object, the return signal being received by a radio receiver; and a processor configured to interpret the signal received by the radio receiver to determine whether there are additional objects behind the object that cannot be seen by a conventional camera. In some embodiments, the system includes at least one camera, and data from the camera and signals received from the radio receiver are combined to provide a comprehensive image or calculation of all objects within the area.

[0018] In another embodiment, a detection system includes: an artificial intelligence program communicating with a processor and a radio transmitter configured to transmit signals to an object or living organism; a radio receiver configured to receive echoes from the radio transmitter; the processor configured to receive and interpret data received from the radio receiver, the processor communicating with the artificial intelligence program to interpret the data from the echoes; and the processor configured to determine a confidence threshold, wherein if the confidence threshold is met, the processor outputs the data in a predetermined desired format, and if the confidence threshold is not met, the radio transmitter transmits a new signal to reinforce the data. In some embodiments, the radio transmitter transmits signals through an object to detect objects invisible to a camera. In some embodiments, the signals are transmitted to a person or animal to detect biomagnetism, body frequencies, and / or body wavelengths. In some embodiments, the signals are transmitted to a person or animal to set bodily function parameters including neural commands. In some embodiments, the detection of biomagnetism, body frequencies, and / or body wavelengths is collected as data used to interpret the frequency of the wave to correlate that frequency with a specific bodily state. In some embodiments, the specific physical state is a mood, illness, and / or symptom. In some embodiments, the transmitted signal is a Wi-Fi signal. In some embodiments, the predetermined desired format is a visual representation of the detected object.

[0019] In another embodiment, a system for landmark navigation within a store includes: a mobile application on a mobile device having a user interface configured to allow a user to select landmark navigation options; a processor configured to process routes based on the user's shopping list or based on a specific object the user is purchasing; and a route matrix that evaluates distances between existing navigation points, wherein if the distances meet or exceed a landmark requirement threshold, the system inserts a landmark navigation point between qualified navigation steps to notify the user that a specific product is near a specific landmark. In some embodiments, the locations of the landmarks are used to provide the user with a visual map. In some embodiments, data is retrieved from at least one in-store camera, the data relating to specific locations of existing landmarks within the store.

[0020] In another embodiment, a system for improving image processing includes: at least one depth-sensing camera that finds a focus within a frame; and a processor configured to retrieve depth data from a corresponding region within the frame where the focus is located, the processor being configured to use the depth data to determine a background contained within the frame, wherein the processor then places a binary mask on the background to occlude unwanted images within the camera's field of view, thereby improving accuracy. In some embodiments, the mask visually occludes all images within the frame except for the focus. In some embodiments, data is retrieved only from the unoccluded focus within the frame to minimize noise within the frame.

[0021] A self-service terminal for store checkout is provided, comprising a main body equipped with both a first camera and a second camera, both communicating with a processor. The first camera is configured to detect and authenticate a user, and the second camera is configured to detect at least one product selected by the user. The processor uses machine learning to compare a pre-existing image database with data collected by the second camera to accurately detect the selected product, and the processor generates a list of items detected by the second camera. In some embodiments, the first camera is a biometric camera. In some embodiments, the user is automatically charged when the first or second camera detects that the user has left or when the user initiates checkout at the self-service terminal. In some embodiments, the self-service terminal also includes a display screen. The self-service terminal may further include a third camera configured to view the user's shopping cart when the third camera is pointed at to view the contents of the shopping cart. The third camera can communicate with the processor, which is configured to visually or audibly notify whether there are items left in the user's shopping cart.

[0022] A system for processing orders is provided, the system comprising: a processor; a first camera communicating with the processor, the first camera and the processor being configured to detect that a user is placing an order; an order input interface configured to accept orders, the order input interface communicating with the processor; and a second camera communicating with the first camera and the processor, the second camera being spaced apart from the first camera, wherein an order entered in the order input interface is marked as incomplete until the first camera or the second camera detects the user and the ordered item in the same pixel buffer. The first camera and / or the second camera may be a biometric camera. In some embodiments, payment is completed after the user places the order. In other embodiments, payment is completed when the user walks away from the first camera or the second camera.

[0023] A self-service terminal for store checkout includes a main body equipped with a camera and a display, the camera and the display communicating with a processor. The camera is configured to detect and authenticate a user, and the processor is configured to generate targeted marketing material using data associated with the user for display to the user on the display. In some embodiments, the material displayed to the user is an advertisement based on the user's shopping history. In some embodiments, the user can interact with the display using gestures, voice commands, applications, and / or physical interactions. Attached Figure Description

[0024] The embodiments illustrated in the accompanying drawings are illustrative and exemplary in nature and are not intended to limit the subject matter as defined in the claims. The following detailed description of the illustrative embodiments can be understood when read in conjunction with the following drawings, wherein the same structures are indicated by the same reference numerals, and in the drawings:

[0025] Figure 1 A flowchart is provided that details the advanced steps taken by a system according to one or more embodiments shown and described herein;

[0026] Figure 2 An exemplary diagram is depicted illustrating a passageway priority system according to one or more embodiments shown and described herein;

[0027] Figure 3 An exemplary store layout and corresponding routes are depicted according to one or more embodiments shown and described herein;

[0028] Figure 4 A flowchart is depicted detailing the store SKU connection and processing performed by a system according to one or more embodiments shown and described herein;

[0029] Figure 5 A screenshot of the center display screen according to one or more embodiments shown and described herein is depicted;

[0030] Figure 6 Screenshots of a list selection display screen are depicted according to one or more embodiments shown and described herein;

[0031] Figure 7 Screenshots of a manifest display screen depicting one or more embodiments shown and described herein;

[0032] Figure 8 A screenshot of a route display screen according to one or more embodiments shown and described herein is depicted;

[0033] Figure 9 A schematic diagram of a checkout device and system used when a user checks out and leaves a store, according to one or more embodiments shown and described herein;

[0034] Figure 10 Exemplary embodiments of a store item catalogue according to one or more embodiments shown and described herein are depicted;

[0035] Figure 11 The embodiments shown and described herein are based on, for example, Figure 10 An exemplary performance graph of the image catalog shown;

[0036] Figure 12 An exemplary improved store item catalog is depicted according to one or more embodiments shown and described herein;

[0037] Figure 13 The embodiments shown and described herein are based on, for example, Figure 13 An exemplary performance graph of the improved image catalog is shown;

[0038] Figure 14 A flowchart is depicted for illustrating an intelligent item recognition system (“IRIS”) according to one or more embodiments shown and described herein;

[0039] Figure 15 Exemplary screenshots depicting AI features of an object when a camera is pointed at it, according to one or more embodiments shown and described herein;

[0040] Figure 16 This is an exemplary model of IRIS in use according to one or more embodiments shown and described herein;

[0041] Figure 17This is a flowchart of the entire system according to one or more embodiments shown and described herein;

[0042] Figure 18 This is a schematic diagram of a system as widely disclosed herein, based on one or more embodiments shown and described herein;

[0043] Figure 19 These are exemplary photographic and graphical representations illustrating the location of a product as seen by a mobile application that highlights the product location to enable a user to easily locate the product, according to one or more embodiments shown and described herein.

[0044] Figure 20 It is a weight measuring device, accelerometer, pressure sensor, according to one or more embodiments shown and described herein. Exemplary diagram of a gravity bag, etc.;

[0045] Figure 21 An illustrative embodiment of a gravity bag in use according to one or more embodiments shown and described herein is depicted;

[0046] Figure 22 This is a generalized depiction of a drone shopping assistant based on one or more embodiments shown and described herein;

[0047] Figure 23 An exemplary model of a drone charging shelf according to one or more embodiments shown and described herein is depicted from a side view;

[0048] Figure 24 An overall exemplary diagram of a user using a drone-assisted system according to one or more embodiments shown and described herein is depicted, wherein a drone moves from a drone charging station along a flight path to a user location to illuminate a specific product for which the user has requested assisted positioning, wherein the illumination is a light projection from the drone to the specific product.

[0049] Figure 25 A flowchart is provided for describing a process for a radio transmitter detection system according to one or more embodiments shown and described herein;

[0050] Figure 26 A side-by-side comparison is depicted of a camera capturing a raw feed and capturing a feed having an active binary mask, according to one or more embodiments shown and described herein;

[0051] Figure 27 A schematic diagram of a landmark navigation system graphically illustrated according to one or more embodiments shown and described herein is provided.

[0052] Figure 28A flowchart is depicted for a landmark navigation system embodying one or more embodiments shown and described herein;

[0053] Figure 29 A flowchart of a self-service terminal and a corresponding self-service terminal according to one or more embodiments shown and described herein is illustrated;

[0054] Figure 30 An exemplary self-service terminal according to one or more embodiments shown and described herein is depicted;

[0055] Figure 31 An exemplary self-service terminal according to one or more embodiments shown and described herein is depicted; and

[0056] Figure 32 An exemplary checkout system and corresponding flowcharts are depicted according to one or more embodiments shown and described herein. Detailed Implementation

[0057] This application comprises two parts: a system, apparatus, and method; and a system and method for using a mobile application to create listings and optimize routes within a store. The second part relates to a checkout system and apparatus configured to streamline the checkout process for users while they are using the aforementioned mobile application.

[0058] This specification describes a system and method for providing a one-stop shopping experience for shoppers, particularly at grocery stores. In high-level overview, the system and method include a list creation system, a connection to a Global Positioning System (GPS) to determine the location information of a specific store, a connection to the store's SKU system to provide real-time availability, location, and price information, and route planning functionality to provide the most efficient and fastest route throughout the store using information retrieved only from an SKU database containing product information specific to the physical store. The route planning system retrieves items from the user-generated list. Upon activation, the system determines the fastest route the user can take throughout the store based solely on SKU information specific to the user's store.

[0059] Figure 1 A flowchart is disclosed that typically maps a system to a store SKU system or connects items on a user-generated grocery list to the store SKU system using a mobile application. System 100 enables users to create one or more lists of items, such as grocery items, within a mobile application. Users may have multiple lists, thus enabling them to manage multiple lists, share lists with group members, or create specific lists for specific events.

[0060] In the first step 102, the system communicates with GPS to determine the user's exact geographic location. The system uses this geographic location to determine the specific store where the user is currently located. In step 104, once the geographic location is established, the system references data from the determined store's SKU system. The store's SKU system stores information such as product pricing, availability, and location within the store.

[0061] The system described in this manual has a particular advantage: it allows the system to connect directly to a specific store's SKU system. The SKU system provides real-time and completely accurate data regarding the price, availability, and location of items in that specific store. Similar systems cannot provide real-time and accurate data regarding product availability, pricing, and location because they are not directly connected to a specific store's SKU system.

[0062] At step 106, the user is prompted to select a shopping list from multiple lists within the mobile app. At step 108, after the SKU system interacts with the user's selected list, the system notifies the user if an item on the list is unavailable. At step 110, this availability determination occurs in real time and can even occur before the user enters the store. This data will show whether the item on the user's list is in stock and its current aisle location within the store where the user is located. Additionally, a payment system 114 (such as...) can be provided to the user within the mobile app. or (Payment) and make it available to users.

[0063] Then, the system determines the most efficient route based on the user's shopping list. In step 112, the route is calculated based on the location of items on the user's shopping list. For example... Figures 2 to 4 As discussed in further detail, this aisle prioritization system uses data from the store's SKU system. Specific SKU information, including location information within a specific store, is communicated to the application.

[0064] Then, the system and application generate, for example, based on the exact location of each specific product. Figure 2 The shown is the aisle priority matrix. (Reference) Figure 2 and 3 Items on a user's shopping list are directly associated with their SKU numbers. Each SKU number in the store is associated with a location name or aisle name. For example... Figure 2 The matrix shown categorizes items on a user's shopping list based on SKU and therefore aisle name. Items are then categorized by aisle name. In this embodiment, all items located in, for example, aisle 1 are sequentially located as follows: Figure 2Within the matrix shown, this grouping of items by aisle name is used to calculate data for specific routes customized for the user.

[0065] exist Figure 2 In the example shown, the user has a list containing four items: milk, bread, cereal, and soda. After the system determines which store the user is actually in based on GPS information, each item on the user's list is assigned a specific SKU number, such as shown in matrix 205. The SKU number of each item on the user's list is associated with a specific location (also known as an aisle name). Each item on the user's list is also associated with a quantity (also known as inventory availability).

[0066] Then, the data from matrix 205 is transmitted to the aisle priority system as shown in matrix 207. The position information of the matrix shown in 205 is compared with other items on the list in aisle priority matrix 207. Then, the products on the user list are organized by aisle name and grouped together by aisle name.

[0067] As illustrated in matrix 207, bread is categorized in the bakery, cereals in aisle 3, milk in aisle 8, and soda in aisle 5. Then, the grouping by product location, as shown in matrix 207, is combined with... Figure 3 The store layouts shown in 200 are compared. Each store location will have a separate store layout map, which will specify the user's route based on aisle priorities such as those shown in matrix 207.

[0068] refer to Figure 3 Layout 200 specifies that users should start from entrance 202 and follow route 206 as specified by the aisle priority system (if most efficient). Figure 2 In the further example shown, the user in this particular example will first stop at bakery 208 and then continue through the remaining aisles 210. In this particular embodiment, the user starts from bakery 208, proceeds to aisle 3, then to aisle 8, and finally through aisle 5. The user is then guided to checkout 214 and through exit 204. In this particular example, aisle priority will be assigned, and the user will first obtain the item at bakery 208 and then continue through aisles 1 to 4. The user will then proceed to delicatessen 212 and then through aisles 8, 7, 6, and 5, where items in aisle 9 can be picked up between any of aisles 7, 6, or 5. It should be noted that, as Figure 2 and 3 The examples shown are merely illustrative and are not intended to limit the scope of this specification.

[0069] Figure 4The SKU and list communication between the store database and the processor and the user's mobile device is further discussed. Flowchart 300 illustrates the steps of first obtaining the list items with SKU numbers at step 302, where at step 304 the numbers are used to retrieve store product information, including the store's location, from the SKU database of that specific store. Then, at step 306, the system inputs the information of each specific listed item into the aisle priority route planning system. The system then uses an aisle priority system, such as those described above, to construct the most efficient route to pick up and leave each item in the shortest possible time and distance, as shown in step 308.

[0070] Figures 5 to 8 This shows an example application visual that will be displayed on a user's mobile device. Figures 5 to 8 The progress chart shows a high-level flowchart of the screens that will be displayed to the user. Upon opening the application, the first screen prompts the user. A login screen is then displayed, allowing the user to enter login information and log in to the system to use any of the aforementioned applications throughout the system. The central screen 406 allows the user to connect to specific store groups, lists, or route information. Screen 408 displays one or more lists created by the user, screen 410 displays the items included in the lists, and screen 412 displays route planning features and an overview to efficiently collect items from the user's lists.

[0071] Such as Figure 5 Screen 406, as shown, displays the central screen. Central screen 406 is where users can access and view all the functions included within the system and applications. On central screen 406, users can view notifications related to sales, adjustments to the grocery list, out-of-stock items, etc. Users can then search for items, change their settings, access their lists, access their groups, and enable grouping at button 424. Button 420 allows users to change grouping settings, such as who can access and obtain the list. Button 422 allows users to view lists such as those shown in list screen 408. Notifications are displayed in a notification window as indicated by reference numeral 426. Settings information is available at reference numeral 429.

[0072] Figure 6 List screen 408 is displayed. List screen 408 allows users to create new lists at reference mark 440, view weekly lists 442, or view additional specific lists such as those indicated at reference marks 444, 446, and 448. A search function 428 is also displayed on list screen 408.

[0073] Figure 7A list of items on the user's grocery list 450 is displayed. The item names and quantities are listed on the user's list 450. A search function 428 and settings information 429 are also displayed on the route display screen 410. This display screen 410 appears once the user selects an item on the list screen 408. The user can scroll through the list and make changes as needed. The list will display quantities or weights (if applicable) and detailed brand-specific information. Below the list, a health rating can be included, rating and assigning scores to product information based on nutritional information. The user can then begin route planning by pressing the Start Route Planning button 452. Selecting the Start Route Planning button 452 opens a new display screen 412, which displays the most efficient route based on the user's shopping list within the store.

[0074] Figure 8 The route planning function is displayed on screen 412. Screen 412 displays local store information 460 and list names 462. Location information for each item or group of items is shown at step 470. The route planning function described herein distinguishes itself from the prior art by using its aisle priority system. The route planning system of this specification is used for aisle-by-aisle guidance in a store to ensure shoppers save time and avoid wandering around the store while searching for specific items. The route planning function is more significant and novel than simply walking through aisles: it uses GPS and / or geolocation to directly connect to the store to communicate the user's specific location to the system. This information determines the specific store the user is in. Once the GPS system has determined the specific store the user is in, the GPS system is no longer needed because the location information for each specific product is sent to the system based solely on the SKU number within that specific store.

[0075] A payment button 472 may also be provided in any of the above-described displays. In some embodiments, a user may pay for an item when placing it in a shopping cart, or may pay for it using specific technology at checkout when a mobile payment system is enabled. Systems that use RFID systems and / or cameras to verify purchases made in the store may also be provided at checkout, as will be discussed below.

[0076] The aforementioned mobile app and system allow users to create lists, and then the mobile app generates the most efficient route for shoppers based on store data and using an aisle prioritization system (all discussed above and in the accompanying diagrams). Self-service checkout kiosks work in conjunction with the mobile app to make the checkout process completely seamless. Within the app, users are granted access to Apple's [services / mechanisms]. (or a similar procedure). In this embodiment, the mobile application accepts payment when the user is at the self-service checkout terminal (also known as the bagging station).

[0077] Each self-service terminal includes multiple stereo cameras programmed to be linked to a computer. When the computers in each bagging station are linked, they detect shopping items based on the item's hue and saturation values, as well as detailed RGB and size information. If the camera's robotic vision does not detect an item that is on the shopper's list, an alert or notification is created for both the user and the store. Similarly, if an additional item not on the user's shopping list is found in the user's cart or bag, an alert / notification is sent to both the user and the store. This prevents theft and accidental overcharging. The cameras communicate with the application, and the application communicates with the cameras. It is a purely communication and item catalog system.

[0078] As discussed above, stores capable of working with mobile applications include at least one of the self-service kiosks described herein. (See appendix) Figure 9 An exemplary self-service terminal is shown. The self-service terminal 500 includes a main body or housing 502. The housing 502 includes a plurality of stereo cameras (or depth-sensing cameras) 504, all of which are connected to and linked to a computer or other processor. The self-service terminal 500 also includes a light ring 508 or other similar flashing device that allows the system to more easily define and view the contents A, B within the basket portion 512 of the trolley 510. The self-service terminal 500 also includes a wireless scale 506 for weighing products or other bulk food items.

[0079] Each of the 500 self-service terminals will include a camera and access via Wi-Fi, cellular, and / or wireless communication. The host computer communicates with the application. The camera will be programmed to know detailed information about every item in the store. This information includes: SKU, item name, HSV, RGB values, and its 3D dimensions. Camera 504 will use a series of advanced algorithms to accurately detect and identify items in the shopper's cart and bags, as they are transferred from cart to bag. Camera 504 and the camera system will communicate with the application to determine the expiration dates of the shopper's items.

[0080] As an example, a user's shopping list contains two items: bananas and apples. If the camera system detects a cereal box in cart 510, the system will automatically notify the shopper and allow them to remove the product from the cart or add the item to their shopping list, thus adjusting the total purchase price. If the user ignores the error, self-service terminal 500 will illuminate its light to warn store staff of the item detection error.

[0081] The camera employs a suite of advanced algorithms that communicate with the user's grocery inventory within the mobile application. The first operation used by the camera is robot vision via Python and OpenCV. This allows the camera to track objects based on their HSV, RGB, and dimensions. The camera system will utilize multiple object tracking algorithms. One algorithm is real-time and is used in conjunction with high-quality product detection and product isolation, known as KCF (Kernel Correlation Filter). Another algorithm is TLD (Tracking, Learning, and Detecting) for searching for specific items that are only relevant to the inventory. These detection algorithms and methods are used in conjunction with the camera and can be used individually or together as needed and permitted by the system.

[0082] The KCF algorithm is programmed to find all items in the store and catalog the items it detects when a new user approaches. The cataloged items should exactly match the active user's shopping list. If an item not on the list is found, the KCF system will send a warning / notification.

[0083] The TLD algorithm is programmed to find items that are only on the user's grocery list. If an item is missing from the cart, the TLD system sends an alert / notification. The system provides real-time grocery product detection and deep learning to prevent store theft and products hidden in bags or carts.

[0084] In some embodiments, the system described herein utilizes machine learning to eliminate item theft and unexpected overcharging of customers. Figures 10 to 13 The system described here is shown and described for improving system performance by leveraging machine learning. The system discussed and shown here creates a seamless checkout process.

[0085] In some embodiments, the verification and detection process for identifying an item involves using machine learning models (including, but not limited to, image classifiers and object detectors, and pixel buffer comparisons). A machine learning model, such as an image classifier or object detector, takes an input image and runs an algorithm to determine the probability that the image or an object within the image matches a training object in the model. The model then outputs an identifier along with a confidence score or multiplier for each identifier. For the output to be considered reliable, the confidence score needs to reach a desired threshold. It is worth noting that while the model is running, it will continuously output identifiers and confidence scores at a rate of several times per second, even if the training object is not present in the image frame. However, a well-trained model will never assign a high confidence score to an image that does not contain a training object. Therefore, setting a high confidence threshold ensures high accuracy.

[0086] The second aspect of the aforementioned verification method involves pixel buffer comparison. Individual images, frames, or machine learning model outputs from given images can be held for future use; these images are defined as buffers. As the model runs, previous model outputs that have reached a confidence threshold are placed into buffers and / or moved through a series of buffers. Holding the outputs within these buffers allows for comparison of the current model output with previous model outputs. This output comparison is beneficial because it includes providing parameters for certain actions and further enhancing the accuracy of the output.

[0087] In an illustrative manner, the system begins with no training item within the camera frame, where the machine learning model attempts to determine the presence of any training item in the image. When no training item is present in the frame, the model outputs the identifier of the most likely item or the closest match to the associated confidence score. In this case, the model is well-trained and only assigns low confidence scores to these outputs that do not reach the confidence threshold considered reliable. Then, a training item enters the camera's field of view. The model, based on its training and algorithm, begins to identify the training item and outputs a higher confidence score to the appropriate identifier. The confidence score reaches or exceeds the confidence threshold required for further action by the procedure. This output is then placed in a buffer. The model then outputs a high confidence score for the same item again. Remember, the model is creating multiple outputs, meaning a single item will likely remain in the model's frame for multiple identifications. After creating subsequent model outputs that meet the required confidence threshold, the new output is then compared to the results of the previous output, and certain parameters are consulted for action. In this case, if the identifier of the previous output is the same as the identifier of the new output, the system can consider both outputs to be results for the same item still within the camera frame. In other cases, the identifier of the new output is different from the identifier of the previous output, thus notifying the system that a new item has entered the camera frame.

[0088] The checkout system and item authentication process illustrated and described herein rely on machine learning. Machine learning discovers algorithms that collect item data, giving the system insights and the ability to predict or identify items or objects based on the collected data. The more the system runs, the more data it collects, and the more questions it generates, resulting in higher prediction / accuracy rates. In other words, the system continues to collect images and other data to continuously improve the system's accuracy and product detection.

[0089] This system uses machine learning to create a catalog to index items in the store. It is a fully functional grocery store item inspection system that performs with 100% accuracy and 100% recall. Using high-quality 360° photographs of items, the camera system fully understands the items users retrieve from the shelves and the items in shoppers' carts.

[0090] This item authentication system connects directly to applications, systems, and software such as those described herein. The self-service kiosks (also known as escalators) described and shown here have built-in cameras. The system can also utilize cameras already installed in the store facility to further enhance accuracy throughout the system and provide additional angles when taking photos.

[0091] The self-service terminal will use machine learning, eliminating the need to manually input item color codes into the system data. Traditional methods typically rely solely on color for item authentication. This system utilizes actual photographs, thereby improving the accuracy of item authentication. The machine learning system studies and learns every detail of each item, thus achieving a high level of accuracy in item authentication.

[0092] Figure 10 An exemplary embodiment of a store item catalog according to one or more embodiments shown and described herein is depicted. In this embodiment, such as Figure 10 As shown, a smaller number of images (usually 5 to 6) of each item are cataloged. This is illustrated through examples and in review. Figure 11 At that time, there were 5 cataloged Fruits. The image. Figure 11 The embodiments shown and described herein are based on, for example, Figure 10 The image catalog shown is an exemplary performance graph. Based on this performance graph and in this Fruit Loops example, 88.9% precision is achieved with 100% recall. However, as shown in Cinnamon Toast... The example demonstrates 100% accuracy and recall when cataloging 10 images. In full functionality, and after a certain time, hundreds of photos will be cataloged, eliminating errors. The more systems used, the better the performance.

[0093] Figure 12 Exemplary improved store item catalogs according to one or more embodiments shown and described herein are depicted, in which a greater number of images are used. As shown, in these embodiments, 15 images are cataloged. 100% accuracy is achieved through high-quality images and data; without high-quality data, accuracy may be imperfect. Quantity is not a guarantee of higher accuracy. Photographs that reveal detail under different lighting and camera angles are key to success. Figure 13 The embodiments shown and described herein are based on, for example, Figure 12 The figure shows an exemplary performance graph of the improved image catalog. As shown in the figure, in some embodiments, 100% accuracy is achieved when using 15 (or more) images.

[0094] Figure 14 and 15 It provides users with artificial intelligence features included within the application's improved functionality. For example... Figure 14 The AI ​​features within the application communicate with the camera on the user's mobile device. When the system is granted access to the mobile device's camera, it processes the live camera image and precisely identifies the object the user is pointing at. The system then allows the user to interact with the identified object within the application. Permitted interaction types include taps, voice recognition, voice activation, clicks, and / or gestures.

[0095] As described above and as Figures 14 to 16 The system and corresponding functions shown herein refer to specific items that have been trained and programmed in a store-specific item catalog and / or store-specific model. As described here and as... Figures 14 to 16 The system 1000 shown promotes higher efficiency and faster response times. Due to the store-specific catalog, faster response times and efficiency can be provided.

[0096] By way of example, if a user wants to add items to their grocery list, they begin the process by accessing the app 1002 on their mobile device 1004. The user selects the app and opens the system in the live view screen 1006. When the user points to the actual and tangible product 1008 (e.g., When a product is selected, system 1010 identifies its detailed information. This information is extracted from the product's store-specific catalog 1012 (based on information contained in a database) and data 1014. The system then processes this information and makes it accessible to the user within the application. Therefore, no barcode is required. The system processes and collects information and displays product information 1016 to the user without any barcode, relying entirely on image data and product database photo information.

[0097] Similarly, such as Figure 15 As shown, etc., depict Banana. The user selects the program and opens the system in the live view 1002. When the user points to the live product (e.g., When encountering a banana, the system will identify the item's detailed information (1016) and automatically determine the specific item. Bananas. This information is extracted from product 1008's store-specific catalog (based on information contained in a database). The system then processes this information and makes it accessible to users active within the application. Again, no barcodes or enabled user interaction are required.

[0098] This intelligent object recognition system (also known as IRIS or iris) 1020 communicates with both system 1010 and cloud 1022. Cloud 1022 is configured to store and collect additional data using cameras or other visual cues 1024 from both store 1026 and user mobile devices and applications 1002. This collection of information, data, and images is handled by IRIS 1020 and system 1010 and incorporated into a catalog 1012 that includes product and data 1014. This data collection through machine learning exponentially improves the accuracy of the entire IRIS system by collecting vast amounts of information, data, and images to compare with real-world products. Figure 15 As shown, etc.

[0099] Figure 16 The document describes multiple sources of data collection utilized by the iris system 1020. The iris system 1020 uses data 1014 already present in the store 1026 database to collect data from the store 1026. This information includes SKU information, images, pricing information, color images, product data, and any other applicable information required for the operation of the iris 1020.

[0100] System 1020 further communicates with self-service terminal 500, which includes a camera 1024 for visual perception. When a user checks out, the camera 1024 of self-service terminal 500 collects information. This information is transmitted back to iris 1020 and stored. All information is stored on hard drive 1052, such as... Figure 17 As shown, etc.

[0101] System 1020 further communicates with the user's device and mobile application 1002. The mobile device includes vision such as camera 1024. As the user collects information and data, the information and data are transmitted back to system 1020 and subsequently stored.

[0102] System 1020 also communicates with store 1026, which uses camera system 1024 to collect data. When the system is in use, such as when determining whether a user has removed an item from a shelf or other display within the store, the star-shaped camera system using camera 1024 collects images. These images, collected during the period when the user terminates the removal of the item, are collected by camera 1024, communicated to system 1020, and subsequently stored.

[0103] Figure 17The overall and general operation of the iris system 1020 is disclosed and described. System 1020 communicates with a data storage unit 1052. Data in the data storage unit 1052 is collected in the manner discussed above. Data 1014 may be collected by user 1050. Furthermore, in order to use the system, user 1050 undergoes a facial scan 1051 to verify their identity. This scan is required to operate the mobile application and the automatic checkout and billing creation described above and foregoing. In some embodiments, facial scan 1051 is performed using the mobile application 1002.

[0104] Figure 18 This illustration depicts user 1050 shopping in a store using this system and iris 1020. In the illustration, user 1050 logs into his account using facial scanning, and the system connects to the user's shopping list. In this embodiment, the user's list includes 10 items, including but not limited to cake mix and birthday candles. In this embodiment, the user is guided through the store using an aisle priority system to easily find the cake mix and birthday candles. Furthermore, in this embodiment, the user is utilizing iris or this system 1020 to easily identify items when the user removes them from shelves or displays.

[0105] In some embodiments and as Figure 26 As shown, additional masking processing can be employed to improve visual detection. In self-service kiosks or other checkout areas, a system using depth-sensing hardware, cameras, and software is provided. This system uses a biometric focusing mask, which creates a sharp focus for machine learning frameworks to extract information from (e.g., ...) Figure 26 Image retrieval data (shown in the side-by-side comparison). This allows high-traffic areas to remain highly accurate because a mask is generated around the desired active shopper, thus focusing only on the desired active shopper. The system searches for the assigned focus within a given frame / image. In some embodiments, the focus is the user's face. The system then retrieves depth data from the corresponding region in the image where the user's face is located. Using the depth data, the system is able to determine which regions constitute the background of the image or parts not intended to be considered for classification. Once the background has been calculated, the system places a binary mask over the background region of the image (in...). Figure 26 (Depicted on the right side). After applying the mask, a machine learning model is applied to process the image. The benefit of applying this binary mask and then processing the image is that this method effectively removes any background "noise" that might interfere with obtaining accurate output from the machine learning model.

[0106] In another aspect of this specification, a landmark navigation system is provided to help users easily locate products based on known landmarks within the store (such as sushi stands or cookie advertisements). Figure 27 and28 The diagram illustrates a landmark navigation system. Landmark navigation systems are designed to assist users in navigating through a store by using points of interest, landmarks, marketing stalls, departments, or other distinguishable or salient features within the retail space. This system is intended to complement, rather than replace, a primary SKU-based navigation system. It is particularly useful for users in large stores where items are expected to be located far apart. These landmarks are implemented as navigation points along the user's route to help them move along the path without becoming disoriented or lost.

[0107] Through examples and references Figure 27 The diagram illustrates a store layout with various areas. Store 2000 includes a deli 2002, restrooms 2004, a sushi stand 2006, a soda stand 2008, a pharmacy 2011, and entrances 2022 and exits 2024. Multiple self-service checkout kiosks 2020 are also provided. Cookie stands 2012 and clothing stores 2010 are located near aisles 2014 and 2016. Navigation can include "Proceed past the pharmacy, soda stand, and then to aisle 12" when the landmark system is activated and the user is guided along a route instead of simply receiving the instruction "Proceed to aisle 12". In this case, the pharmacy and soda stand are included as landmarks or navigation points to assist the user in reaching their destination, aisle 12. While users may already be able to navigate to their destination without landmarks, some would certainly consider this the desired assistance.

[0108] As mentioned above, the landmark navigation system is as follows: Figure 28 The operation is as follows. As a first step, the user activates a landmark navigation request (landmark system) on a device with a user interface. The system then processes the active route or initiates route planning based on the user's list or the items the user is searching for. The route matrix evaluates the distances between existing navigation points. If the distances meet or exceed the landmark requirement threshold, the system inserts a landmark navigation point between qualified navigation steps. The route then proceeds as normal.

[0109] Figures 20 to 21 The gravity bag 1100 shown in this specification is an example. The gravity bag is a solution for variable-cost items (e.g., products whose cost is determined by weight). The gravity bag 1100, as shown here, connects to this system and allows the user to weigh a product (or other variable-weight item) and leave it in the bag while continuing to weigh new items. After the user has finished weighing the current item in the bag, the bag is processed and zeroed, allowing the user to continue using the bag and the previously weighed items within it.

[0110] like Figure 20The gravity bag 1100 shown includes a bag portion 1112, which is a standard bag made of polyester, cotton, nylon, or any other suitable material configured to hold items of variable weight. The bag also includes a load sensor 1114 that tracks the weight of the bag as the user adds items. The gravity bag 1100 also includes a chip 1116 that measures the weight and... A signal is sent to the device. In this embodiment, the signal (or other similar wireless signal) is sent to the user's mobile device via a mobile application. Information relating to the weight of the item recently added to the gravity bag 1100 is sent to the mobile application. The gravity bag 1100 also includes an external pressure sensor 1118. The pressure sensor 1118 may also be a magnet for checkout counter detection. An accelerometer 1120 is used if the bag is rotated, twisted, and / or rotated to prevent misreading. Furthermore, a battery 1122 (or multiple batteries) is used to power the various components discussed herein to operate the gravity bag 1100.

[0111] In this example, a user has three onions on their list. The user grabs the onions, and a bag weighs them. The onions cost $1.03 per pound. The weighted bag reads 15 ounces of onions and then (after the user instructs the app to weigh them) saves the data in the mobile app. The bag then zeros out at 15 ounces. This process is repeated, allowing more weight-based items to be added, weighed, and correctly priced. The bag's structure... Figures 22 to 24 As shown in the image.

[0112] Figures 22 to 24 This specification relates to a drone-assisted system. The drone system 1200 of this specification is configured to work directly with the mobile applications described herein. In this embodiment, the drone 1202 is a mini-drone and is configured to wirelessly charge on a charging pad 1208 atop a shelf unit 1204 having multiple shelves 1206. A wire 1210 connects the wireless charging pad 1208 to a power source. In some embodiments, the charging pad 1208 is located on top of an aisle shelf. This placement allows the drone 1202 to charge in the exact area where the shopper is located, such that if the shopper remains in the aisle for an extended period, the user's dedicated drone can land and receive rapid charging before the paired user continues with the drone.

[0113] Upon activation based on a user request guided by the drone, the user device sends navigation pulses, thereby signaling the user's location to the scheduled drone. The activated drone then pairs with the user via a device connected to the user, facial recognition of the user, or a combination of both. Drone 1202 utilizes live video feeds from an onboard camera to detect, identify, and track the user.

[0114] In some embodiments, the drone is equipped with a light projection 1218 capable of projecting images, colors, or outlines onto a surface spaced apart from the drone (such as indicated by reference numeral 1216). Figure 24 In the illustrated embodiment, the drone projects light onto a single product 1214 on a shelf 1206 within an aisle. When a user requests it via a mobile app, the drone locates products that the user cannot otherwise pinpoint and highlights the specific item the user is looking for using light projection.

[0115] Pairing the drone's onboard camera with computer vision software allows the drone to perform object detection and recognition, as well as facial recognition. These capabilities, combined with AI programming, enable the drone to guide the user to a requested item and indicate its precise location (such as item highlighting as described above). AI programming also facilitates the drone charging process by recognizing low power levels and subsequently guiding the drone back to the charging pad and assigning a replacement drone to continue user service.

[0116] It should also be noted that this system and method are intended for use with a user's mobile device (such as a cellular phone). In this specification, the system is intended for use in both the user's home and store, thereby preventing the store from possessing the personal device due to its integration into a self-service terminal system or personal device system.

[0117] Another element of this system is the brand AURA. TM This object detection system utilizes radio frequency (RF) to determine objects invisible to conventional cameras. The system includes the use of RF or Wi-Fi to determine objects invisible to conventional cameras or other detection components. A series of signals are projected onto a desired object or area (such as in a store or warehouse) using a radio transmitter. These signals travel across a solid surface, and as they interact with the surface or object, they produce return signals received by a radio receiver (essentially a miniature radar). For example, these signals are transmitted through shelving units to determine if there are additional objects behind these objects visible to the camera. These signals are categorized and converted into usable data that details the object. By using a camera in an appropriate context within the system, the system will detect whether there are additional objects behind objects visible to the camera. These signals will utilize the signals reflected from the object to give the object's measurement and potential quantity. These signals will provide the system with the ability to give the accurate location and triangulation values ​​of the detected object.

[0118] The object detection system uses an artificial intelligence (“AI”) program. The AI ​​program communicates with a radio transmitter. The radio transmitter sends a signal in a space such as a warehouse. The signal is configured to pass through solid objects, such as vertical surfaces on shelves. The radio waves return to a radio receiver. The radio receiver includes a processor configured to determine whether the object is behind a solid surface (such as a vertical surface on a shelf) or behind existing products on the shelf. The signal data is received by the detection system, and the AI ​​program interprets the data to determine the presence of an object. The detection system makes the decision and performs the detection. The system then determines a confidence threshold, and when this threshold is met, the detection system outputs the data in a specified format (such as graphically, graphically, or via some other signal such as an audible signal). The detection system then repeats the same process. If the desired threshold is not met, the detection system sends a new RF to augment the dataset and determine the presence of an object. The detection system then establishes parameters for the next RF transmission, and the radio transmitter sends a signal. The process then continues as with previous signal transmissions and receptions.

[0119] The system also communicates with the aforementioned existing consumer technologies and is capable of detecting customer location and triangulation values ​​in real time.

[0120] By way of example, the signal transmitter begins sending a signal throughout a room or designated area (such as a shopping area or warehouse) upon setup. As the signal travels and comes into contact with an object, it continues to travel through the object, generating a rebound signal, known as a return signal, which travels to the receiver. The speed and quantity of the return signals inform the system of several metrics, including the size of the object, its relative position, and the number of objects. When this data is paired with a camera image, the system is able to detect objects that are not visible to the camera (i.e., hidden behind products visible to a conventional camera).

[0121] It should be noted that all of the above systems can be applied to any area within a store, warehouse, production facility, etc., where it is desired to locate, track, visualize, consider, image, detect, and / or inspect any item. Any of the aforementioned artificial intelligence, camera, radio frequency, and other systems can be used to locate, track, visualize, consider, image, detect, and / or inspect any item within a store, warehouse, production facility, etc.

[0122] In another aspect of this specification, Wi-Fi is used to detect biomagnetism, body frequencies, and body wavelengths, etc. This system relies on miniature radar. Using a radio transmitter, a series of signals are transmitted and projected onto a desired object or area (such as those described above). These signals travel across a solid surface, and as they interact with the surface or object, they produce a return signal received by a radio receiver. This current aspect of the application focuses on biomagnetic radio frequency and electroencephalography (EEG) to set parameters for bodily functions and neural commands without wearable technology. The system is tethered to the user's body and uses AI commands and Wi-Fi signals to transmit the user's specified signals (AURA). TM Isolate only the user's body area.

[0123] This system operates similarly to the human eye and how the ganglion cells within the eye interpret light in response to its frequency and amplitude. Wi-Fi signals interpret wave frequencies to associate them with specific biological states (mood, illness, ailment, or other physical conditions). The frequency of a light wave determines its hue, while the amplitude of that frequency determines its brightness. The eye's response corresponds to that of light waves because the pupil acts as a window to ganglion cells and many other elements.

[0124] This system is designed to use radio frequency to monitor and communicate with ganglion cells. The system serves as a window into these cells. Ganglia act as relay stations for neural signals in a connection-point manner: nerve plexuses begin and end at ganglion cells. Ganglia serve as response points in the nervous system, which is why they are the first neurons in the retina to respond with action potentials in conjunction with this method and system.

[0125] This system further utilizes Wi-Fi as an electromagnetic frequency to communicate with ganglion cells and nerve plexuses within the human body. Using Wi-Fi, the system detects autonomous brain wavelengths and related motor skills (such as whole-body movement and fine motor skills). This system acts as a bridge within the nervous system, relaying brain commands to ganglia that have been severed due to spinal cord injury, nervous system disorders, or illness.

[0126] Using AI, this system employs a body-understood language and programs that language within a Wi-Fi signal to allow seamless communication flow between repeaters in the brain and nervous system. This enables the use of programmable Wi-Fi (AURA) without surgical implants or wearable devices. TM It serves as a means of connecting the body and computers to fix spinal cord injuries and other somatic and autonomic nervous system problems.

[0127] Figure 29A self-service terminal and corresponding flowcharts according to one or more embodiments shown and described herein are depicted. Self-service terminal 1300 includes a main body 1302 and a plurality of cameras 1306, 1308, 1310 all mounted on the main body 1302. The self-service terminal at 1300 also includes a display 1304 mounted for user viewing. A processor 1312 is included within self-service terminal 1300, or is accessible via a cloud or other server, or is a spaced-apart processor or system. Figure 29 In the embodiment shown, camera 1306 is a biometric camera configured to scan the user’s face or collect other biometric data related to the user, including but not limited to retinal scan information, facial scan information, infrared information, etc.

[0128] The processor 1312 communicates with multiple cameras 1306, 1308, 1310, the display 1304, and any other necessary hardware to complete transactions. For example... Figure 29 As shown in the flowchart, the system activates when a user approaches the self-service terminal. One of several cameras (most typically a biometric camera 1306) detects the user. If biometrics are used, facial scanning, retinal scanning, or other methods can be used to detect the user. Facial scans or other biometric data are used to connect the user to a user-created account within the system. This user-created account may contain payment information, shopping history, favorites, disliked items, biographical information, or any other information typically created with a typical store account. The user-created account may also be associated with a user's photo or other linked image. Once the user is detected, user data can be used to generate user-specific marketing advertisements and / or experiences to be displayed to the user. The processor determines what content (such as advertisements) should be displayed and shows that material to the user on the self-service terminal. The user can then interact with the advertisements on the display using gestures, voice commands, apps, and / or physical self-service terminal displays. The processor can display information such as customized advertisements or products that are compared and taken into account the user's created account. Customized advertisements or products can also be displayed based on the content currently present in the user's shopping cart during the current checkout exchange.

[0129] In some embodiments, cameras 1308 and 1310 continue to detect items in the user's shopping cart. The system uses machine learning and / or compares a database available to processor 1312 to determine the items located in the cart. The processor is configured to collect data to improve the system's accuracy. Upon detection of each item, the processor generates a list. This list may be displayed on the self-service terminal's display 1304. The user then proceeds to checkout via a user-initiated checkout (such as a button) or when any of cameras 1306, 1308, or 1310 detects the user has left. Any of these actions will result in automatic deduction from the user's created account. The transaction is then complete.

[0130] Now for reference Figures 30-31 A self-service checkout terminal 1400 is provided, in which trolley cameras 1408 and 1410 are used to ensure that all items contained in the trolley 1450 are recorded. The self-service terminal 1400 generally includes a main body 1402, which has multiple cameras 1404 and 1406 and trolley cameras 1408 and 1410. Along with a display 1420, bagging / rack / strip positions 1412 and 1414 may also be provided. Cameras 1408 and 1410 are configured to point and angle downwards toward the trolley 1450 to detect whether the trolley 1450 still contains any items. If the trolley 1450 still contains one or more items, the processor will notify the user and / or store audibly and / or visually that the trolley 1450 still contains items to prevent theft. For example, if camera 1408 detects items in trolley 1450, the processor can cause a portion of self-service terminal 1400 to be aluminumned and / or emit a loud noise to alert of potential theft.

[0131] Now for reference Figure 32A checkout system 1500 is provided, featuring a first camera 1502 and a second camera 1506. An order input interface 1504 is also provided. The camera located on the first and / or second camera may be a biometric camera 1508. A processor 1510 communicates with the first camera 1502, the second camera 1506, and the order input interface 1504. The system within the processor 1510 is activated when the camera detects a user using biometric data and / or determines whether the user is placing an order. The user is identified using a mobile application, biometrics, and / or account login. Payment may be completed after the user places an order with an employee via direct input or verbal communication. Alternatively, payment may be completed after the user picks up their item. The order is marked as incomplete until the user and item are connected to the same pixel buffer (this can be done using either or both of camera 1 and / or camera 2). Category verification, product verification, and / or item authentication are then performed via either camera. Category verification is performed when the processor determines that the user has at some point picked up goods of a general category (such as the size of a beverage). Product verification and / or item authentication are performed using machine learning and data comparison systems as described above. Once a pickup is confirmed, the order is completed or payment is made. If any camera detects the user leaving, the order and / or payment can also be completed.

[0132] It should be noted that the terms “substantially” and “about” can be used here to indicate the degree of uncertainty attributable to any quantitative comparison, value, measurement or other representation.

[0133] These terms are also used here to express, quantitatively, the degree to which something may change relative to the reference without causing a change in the fundamental function of the subject under discussion.

[0134] While specific embodiments have been shown and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter.

[0135] Furthermore, although various aspects of the claimed subject matter are described herein, these aspects need not be used in combination. Therefore, the appended claims are intended to cover all such variations and modifications within the scope of the claimed subject matter.

Claims

1. A computer-implemented method for generating the most efficient route for a user to navigate through a store to purchase products on a user-generated shopping list generated on the user's mobile device, the method comprising: Access the Global Positioning System to determine the geographic location of the user's mobile device, and determine the identity of the store where the user is located based on the determined geographic location; Access a database of SKUs of products that can be sold at the identified stores, the SKU database including records of at least some of the products that can be sold at the identified stores, the records including at least price, product location and product availability information; The shopping list is generated from the accessed SKU database, and at least one of the most efficient and fastest routes for the user to traverse the store to purchase products on the shopping list is generated from the store layout map of the identified stores, wherein the locations of each product in the SKU database correspond to the mapped locations on the store layout map. as well as While the user is shopping in the identified store, instructions are repeatedly sent to the user's mobile device to continuously guide the user to the mapped locations of the various products on the shopping list based on the product availability information and at least one of the determined most efficient route and fastest route.

2. The computer-implemented method according to claim 1 further includes: Connect to the SKU database before the user enters the store.

3. The computer-implemented method according to claim 1 further includes: When the user approaches the next item on the shopping list in the identified store, light is projected onto that next item.

4. The computer-implemented method according to claim 3 further includes: The drone, which communicates with the user's mobile device, the SKU database, the store layout map, and the shopping list, includes a light projection mechanism, and wherein the computer-implemented method further includes: when the user approaches the next product on the shopping list in the identified store, the drone projects light onto the next product.

Images