Shopping cart monitoring using computer vision

Abstract

A system for monitoring shopping carts uses a camera to generate images of shopping carts moving in a store. In some embodiments, the camera can be additionally or alternatively mounted to a shopping cart and configured to image the contents of the shopping cart. The system can use the captured image data, and / or other types of sensor data (e.g., store location where a certain item was added to the basket), to classify items detected in the shopping cart. For example, a trained machine learning model can classify items in the cart as “non-item,” “high theft risk item,” “electronic item,” etc. When a shopping cart approaches a store exit without any associated payment transaction indication, the system can optionally use the associated item classification data, in combination with other data such as cart path data, to determine whether to perform an anti-theft action, such as locking the cart wheels or activating an alarm.

Description

[0001] Priority Statement

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 988,174, filed March 11, 2020, which is incorporated herein by reference. Background Technology Technical Field

[0004] This disclosure generally relates to systems and methods for tracking the movement and status of movable shopping baskets, including but not limited to motorized and non-motorized (e.g., human-powered) shopping carts and handheld shopping baskets, using computer vision and machine learning techniques.

[0005] Description of related fields

[0006] Fencing systems exist to prevent shopping cart theft. Typically, these systems consist of wires buried in the sidewalks of store parking lots to define the outer boundary of areas where shopping carts are permitted. When a shopping cart is pushed across this wire, a sensor located in one or near one of the wheels detects the electromagnetic signal generated by the wire, activating a braking mechanism in the wheel to lock or prevent rotation. To unlock the wheel, service personnel typically use a handheld remote control to send an unlock signal. These types of fencing systems present challenges. Summary of the Invention

[0007] Systems for monitoring shopping carts use cameras to generate images of shopping carts moving through a store. In some implementations, the camera may be attached to the shopping cart additionally or alternatively and configured to image the contents of the cart. The system can use the acquired image data and / or other types of sensor data (e.g., the store location of an item added to a basket) to categorize items detected in the shopping cart. For example, a trained machine learning model can categorize items in the shopping cart as “non-merchandise,” “high-risk theft items,” “electronic goods,” etc. When the shopping cart approaches a store exit without any associated payment transaction indication, the system may optionally combine it with other data, such as cart path data, using associated item categorization data to determine whether to perform anti-theft actions, such as locking the cart wheels or activating a store alarm. The system can also compare the categorization of the shopping cart contents with payment transaction records (or summaries thereof), for example, to detect underpayment events.

[0008] One aspect of the invention is a system for reducing push-out theft, comprising: a camera for capturing images of the contents of a shopping cart associated with a store; and a computing system comprising one or more processors. The computing system is programmed to at least implement: an item classifier that assigns a category to items detected in the images using a trained machine learning model; and an exit manager that determines whether to authorize the shopping cart to leave the store based at least on (1) the category assigned to at least one item in the shopping cart by the item classifier, and (2) a flag indicating whether the at least one item has been paid for. The camera may be mounted at a fixed location within the store or on the shopping cart. The item classifier may be configured to classify items as goods and non-goods, and / or assign goods category classifications to items. The item classifier may be configured to classify items added to the shopping cart at least in part based on the store location of the shopping cart at the time the items were added.

[0009] Another aspect of this disclosure is a method for reducing push-out theft. The method includes: monitoring the location of a shopping cart within a store; detecting an event of an item being added to the shopping cart; assigning a category to the item based on at least one of: (1) the location of the shopping cart in the store at the time of the event, determined by the monitoring, and (2) an image of the item in the shopping cart captured by a camera; subsequently, detecting that the shopping cart is leaving the store without an associated payment transaction indicator; and, in response to determining that the shopping cart is leaving, determining whether to perform anti-theft actions, at least in part based on the category assigned to the item. The method can be performed under the control of program instructions executed by one or more processors. The task of detecting the event of an item being added may include detecting vibrations caused by the addition of the item to the shopping cart using a vibration sensor mounted to the shopping cart, or may include detecting changes in weight measurements generated by a scale on the shopping cart. In some embodiments, the shopping cart includes a camera module having a camera and a processor, and the task of detecting the event includes having the camera module compare a first image captured by the camera with a subsequent second image captured by the camera to determine whether the contents of the shopping cart have been changed. A trained machine learning module may be used to assign the category.

[0010] Another aspect of this disclosure is a camera module configured to be mounted on a shopping cart to capture images of the cart's contents. The camera module includes: a camera; a processor; and a wireless transceiver. The processor is configured to assess, at least in part, whether an item has been added to the shopping cart by comparing a first image captured by the camera with a subsequent second image captured by the camera. The processor can also be configured to, in response to determining that an item may have been added to the basket, use the wireless transceiver to upload the second image or an identifier of the second image over a wireless network for analysis. The processor can receive input from a basket vibration sensor and can use that input to trigger the camera to capture an image. The processor can be configured to mark the second image prior to uploading it over the wireless network to indicate areas showing changes in the basket's contents. The camera module may also include a cart position sensing module, in which case the processor can use the output of the cart position sensing module to disable the camera when the shopping cart is not in the product area.

[0011] Another aspect of the invention is a method for monitoring shopping carts, comprising: receiving wheel rotation event data from each of a plurality of shopping carts in a store via a wireless network; generating shopping cart motion data for each of the plurality of shopping carts based on the wheel rotation event data of the respective shopping cart, wherein the shopping cart reports wheel rotation event data associated with a cart identifier; capturing an image sequence showing the movement of the shopping cart in the store by means of at least one camera installed in the store; generating shopping cart motion data for imaging the shopping cart by analyzing the image sequence; and comparing the shopping cart motion data generated from the image sequence with the shopping cart motion data of each of the plurality of shopping carts generated from the wheel rotation event data. For a first shopping cart, the wheel rotation event data may include rotation event data of the front wheels and rotation event data of the rear wheels, and the method may include using the event data of the front wheels and the rear wheels in combination to detect the steering of the first shopping cart. This detected steering may be associated in a timely manner with the steering detected in the image sequence. The method may further include, in response to detecting a match between (1) shopping cart motion data generated from an image sequence and (2) shopping cart motion data generated from wheel rotation event data of a first shopping cart among a plurality of shopping carts, associating a cart identifier of the first shopping cart with an imaged optical track of the shopping cart.

[0012] Another aspect of this disclosure is a system for monitoring shopping carts, comprising: a shopping cart wheel assembly including wheels, wheel rotation sensors, a processor, and a wireless transceiver, the shopping cart wheel assembly being configured to report wheel rotation event data on a wireless network; a camera mounted at a store location and configured to capture image sequences showing the movement of the shopping cart within the store; and a computing system programmed to use the image sequences and the reported wheel rotation event data to determine whether the shopping cart wheel assembly is part of a shopping cart imaged in the image sequences. The computing system may be configured to compare cart motion data detected from the image sequences with cart motion inferred from the wheel rotation event data. Furthermore, the computing system may be configured to use timestamps contained in the wheel rotation event data to at least infer changes in cart speed over time. In some implementations, the shopping cart wheel assembly is one of two shopping cart wheel assemblies for a first shopping cart, one mounted at the front wheel position and the other at the rear wheel position. Both wheel assemblies are configured to report their respective wheel rotation event data on a wireless network. In this case, the computing system can be configured to compare the wheel rotation event data of the front and rear wheel assemblies to detect a turn made by the first shopping cart. The computing system can then promptly correlate the detected turn with turns detected in an image sequence. Attached Figure Description

[0013] Figure 1A and Figure 1B An example operation of the cart enclosure system is illustrated schematically. Figure 1A In the image, a shopping cart full of goods is attempting to leave the store, and anti-theft measures have been implemented to prevent theft (e.g., the cart's wheels are locked or an alarm is activated). Figure 1B The cart was empty and no anti-theft measures were taken.

[0014] Figure 1C Various types of anti-theft system components are shown, which can be deployed inside and around stores to track movable shopping baskets, such as motorized and non-motorized (e.g., manually pushed) shopping carts, handheld shopping baskets, and motorized mobile carts. Movable shopping baskets can be imaged using a computer vision unit (CVU) or a camera transceiver unit (CTU) to determine, for example, whether they are empty or at least partially loaded with goods.

[0015] Figure 2A An example of a shopping cart with a navigation system and one or more smart wheels is shown.

[0016] Figure 2BAn example of a shopping cart with a smart positioning system mounted on the handle is shown. In this figure, the child seat of the cart is in the open position (sometimes referred to as the child seat down).

[0017] Figure 2C This is a side view of a shopping cart equipped with a camera module.

[0018] Figure 2D yes Figure 2C A top-down view of the shopping cart.

[0019] Figure 2E It shows Figure 2C and Figure 2D A component of one embodiment of a camera module.

[0020] Figure 3 Components of one embodiment of a protective system for a shopping basket are shown.

[0021] Figure 4A This illustration schematically depicts an anti-theft system that uses computer vision technology to identify whether a shopping basket is at least partially loaded with goods and is leaving the store. The shopping basket can be attached to a manually pushed shopping cart, a motorized trolley, or carried by the shopper.

[0022] Figure 4B This schematically illustrates another implementation of the anti-theft system.

[0023] Figure 5 The diagram schematically illustrates the side view (left) and top view (right) of the camera of a computer vision unit positioned to determine the location of a shopping basket.

[0024] Figure 6A , Figure 6B and Figure 6C This illustration schematically depicts an example of the placement and orientation of computer vision units (CVUs) and secondary cameras near the entrance / exit of a retail store. The number and arrangement of the CVUs and secondary cameras, as well as the shape and size of their respective fields of view (FOV, indicated by dashed or dotted lines), are intended to be illustrative and not limiting. In other implementations, the layout may differ to achieve security purposes for the retail facility.

[0025] Figure 7 This illustration schematically depicts an example of the path taken by a shopping basket near the entrance / exit of a retail store. Empty baskets are shown without a shaded line, while baskets that are at least partially loaded are shown with a shaded line. Symbols along the path taken by the shopping basket (in this example, a shopping cart) indicate the possibility of theft.

[0026] Figure 8An example of a processing pipeline used to train machine learning (ML) models is illustrated.

[0027] Figure 9 An example of a processing pipeline for analyzing images obtained from an anti-theft system is illustrated schematically.

[0028] Figure 10 An example of a processing pipeline for real-time event detection or live streams from an anti-theft system is illustrated schematically.

[0029] Figure 11 An example of a pipeline for business intelligence (BI) analysis of image data from an anti-theft system is illustrated schematically.

[0030] Figure 12 The processing pipeline in a CVU is illustrated schematically.

[0031] Figure 13 A method is shown that can be implemented by a camera or camera module mounted on a trolley to capture and process images of the contents of the trolley.

[0032] Figure 14 This demonstrates how to generate item classifications for captured images.

[0033] Figure 15 A method is shown to associate an imaged cart path with a unique cart ID using data collected from the cart's wheel assembly.

[0034] Figure 16 It shows in Figure 15 The example dataset can be maintained and used in the methods.

[0035] Figure 17 A method is shown for determining whether the contents of an image of a shopping cart leaving a store match a payment transaction record.

[0036] Figure 18 It shows what can be used to implement Figure 17 A set of components of the method.

[0037] Figure 19 It shows that it can be used Figure 18 The system and Figure 17 The types of data records generated and used in the method.

[0038] Throughout the accompanying drawings, reference numerals may be reused to indicate correspondences between reference elements. The drawings are provided to illustrate examples of the implementations described herein and are not intended to limit the scope of this disclosure. Detailed Implementation

[0039] Overview

[0040] While existing cart containment systems can prevent shopping cart theft, some of these systems may fail to detect other types of shopping-related misuse. For example, a thief might push a shopping cart, at least partially loaded with goods or merchandise, out of the store without paying for the goods or merchandise (this type of theft is sometimes called "push-out" theft). A cart containment system may not be able (or only be able to) determine whether a cart pushed out of the store is empty (in which case there is no threat of merchandise theft or only a limited threat) or loaded with merchandise (in which case there may be a significant threat of merchandise theft). If the cart containment system is triggered every time a cart leaves the store (loaded or unloaded), it could result in numerous false alarms, as the system would be triggered even when an empty cart leaves the store.

[0041] False alarms can be reduced by determining whether a shopping cart passes through an operating checkout lane before the shopper attempts to leave the store. If so, the shopper is likely to have purchased items, and the cart containment system can be configured not to trigger in this case. If the cart does not pass through an operating checkout lane (or is present in the lane for insufficient time to actually pay), the cart containment system can be configured to trigger upon leaving. However, even in this case, false alarms can still occur because the shopper might push the empty cart back out of the store for some non-theft reason (e.g., choose another cart (e.g., one with less wheel vibration), return to a parked car to retrieve shopping bags or shopping lists, etc.). A containment system that detects cart passage through an operating checkout lane might require specific hardware in each checkout lane to detect the cart's passage, path, speed, distance traveled, dwell time, etc. Such hardware would increase the cost of these devices. Furthermore, this method may have significant limitations for retail stores that have implemented mobile payment systems, where shoppers do not need to pay through fixed checkout lanes but can instead use mobile applications (e.g., on the shopper's smartphone) to pay for goods.

[0042] While it is often possible to detect theft using Electronic Article Surveillance (EAS) systems (e.g., including EAS towers at store exits), the cost and burden of attaching EAS tags to merchandise is often impractical (especially in the case of grocery stores).

[0043] Retail stores may want to identify whether a shopping cart approaching the exit is at least partially loaded with items from the store (e.g., groceries, health products, alcohol, etc.), and if so, whether the cart has previously passed through the checkout lane or the shopper has paid via mobile payment. A cart containment system is capable of imaging the cart basket using cameras installed in the store (and / or cameras mounted on the cart) and of analyzing the images using computer vision and machine learning techniques to determine, for example, whether the cart basket is empty (e.g., lower threat of theft) or at least partially loaded with goods (e.g., higher threat of theft). The image can be a still image or a frame or multiple frames from a video. In some embodiments, the system can also use the image to classify or identify items in the basket; for example, detected items can be classified as merchandise rather than non-merchandise, or as high-risk items. The system can also take cart location data into account when classifying items; for example, if the cart has not entered the store's electronics section, the system may not (or with reduced probability) classify detected items as electronic items.

[0044] If the system detects that a trolley, at least partially loaded, is attempting to leave the store without any sufficient tags indicating that the items have been paid for, it can trigger a trolley containment system to perform anti-theft actions (e.g., braking or locking the trolley's wheels to prevent its movement, displaying an alarm or message to the shopper to return to the store, activating the store's video surveillance system or alarm, notifying store security personnel, etc.). In embodiments where the system categorizes the items detected in the shopping trolley, this categorization can also be considered when deciding whether to perform an anti-theft action and / or what type of anti-theft action to perform. For example, the system can score a given trolley based on the likelihood of it containing high-risk items, the value of the goods contained in the trolley, etc., and consider this score when deciding whether to perform a specific anti-theft action.

[0045] Figure 1A and Figure 1B An example of the trolley enclosure system in operation is illustrated schematically. Figure 1A and Figure 1B The functional components shown will be further described below (see, for example) Figure 1C , Figure 4A and Figure 4B ).exist Figure 1AIn this system, the computer vision unit (CVU) or cart transceiver unit (CTU) includes a camera 410 capable of imaging the store area near the store exit. The store area is within the field of view (FOV) of the camera 410. The CVU or CTU can perform (or communicate with another system to perform) computer vision analysis of the images from the camera 410. The computer vision analysis can determine the loading status of the carts, such as whether the shopping cart is empty, at least partially loaded with goods, or full of goods. The CVU or CTU can communicate with a door manager 450 that controls the anti-theft functions of the cart enclosure system. An example of a door manager 450 is available from Gatekeeper Systems, Inc. (Ford Hillland, CA). Anti-exit systems, and those described, for example, in U.S. Patent Nos. 8,463,540, 9,731,744, and 10,232,869; the entire contents of which are incorporated herein by reference. Other examples of door managers and anti-theft systems described in U.S. Patent Nos. 5,881,846 or 7,420,461; the entire contents of which are incorporated herein by reference.

[0046] like Figure 1A As shown, if a trolley at least partially loaded with goods is approaching the store exit without a sign indicating that the goods have been paid for, the CVU or CTU can signal the door manager 450 to activate anti-theft functions (e.g., locking or braking one of the trolley's wheels, sounding an alarm, activating the store surveillance system, etc.). Conversely, as Figure 1B As shown, if the shopping cart is essentially empty, the risk of theft is low, and the CVU or CTU may take no action or signal to the access control manager 450 that no anti-theft action will be taken. As mentioned above, the decision to take anti-theft action can also depend on the type or identities of the items detected in the cart.

[0047] therefore, Figure 1A and Figure 1B Example trolley containment systems can advantageously reduce or prevent theft of goods from the store while reducing or preventing false alarms caused by empty trolleys being pushed out of the store for some reason (e.g., switching the trolley to another, returning the shopper's cart to retrieve shopping bags or shopping lists, etc.). Furthermore, in some implementations, the system can limit anti-theft actions to scenarios where expensive or high-risk items are detected in the trolley.

[0048] In some implementations, a separate door manager 450 is not used, and the CVU transmits the anti-theft signal to the shopping basket, shopping cart, or store surveillance system.

[0049] While many shoppers use shopping carts in retail stores, the computer vision technologies described in this article are not limited to shopping carts and can be applied to any movable shopping basket, including human-powered shopping carts, motorized carts with baskets, or handheld shopping baskets carried by shoppers. Furthermore, these computer vision technologies are not limited to retail applications and can be used to determine whether other types of carts are loaded with items, objects, or merchandise, such as warehouse, industrial, or utility carts, luggage carts, medical, hospital, or pharmacy carts, wheelchairs, strollers, or hospital beds.

[0050] Various examples and implementations are described below. These examples and implementations are intended to illustrate the scope of this disclosure and are not intended to be limiting.

[0051] Retail store scenario example

[0052] Figure 1C An example of an anti-theft system 400 is shown. The anti-theft system shown is deployed in a store to track or control the movement of a shopping cart 30 and prevent the theft of goods from the cart. However, the components and methods of the invented anti-theft system can be used in other applications, such as tracking baggage carts in airports or carts in warehouses.

[0053] The system includes a set of cart transceivers (CTs) that communicate bidirectionally with a set of wireless access points (APs) to establish a bidirectional radio frequency (RF) communication link with the shopping cart 30. In one example, each cart transceiver (CT) is fully contained within one of the standard-sized (e.g., 5-inch diameter) wheels 32 (typically the front wheel) of each shopping cart 30, and locks the wheel together with a braking unit capable of being braked by the cart transceiver. Examples of braking units that can be used for this purpose are described in U.S. Patent Nos. 6,362,728, 8,820,447, 8,602,176, or 8,973,716; the entire contents of each of these are incorporated herein by reference. (For the purpose of detailed description, the term "cart transceiver" refers collectively to the RF transceiver of the cart and the associated sensor circuitry.) Alternatively, a progressive or partial braking unit may be used, which is additionally capable of preventing wheel rotation without locking the wheel.

[0054] Some circuitry of the trolley transceiver (CT) can alternatively be located elsewhere on the shopping trolley 30. For example, as described below, some or all of the transceiver circuitry can alternatively be included in the display unit attached to the handle of the shopping trolley or the front of the trolley, or in a camera module mounted to the trolley (such as...). Figure 2C-2E(As shown in the diagram). As another example, some or all of the circuitry, including the sensor circuitry, can be encapsulated in the wheel assembly (e.g., the swivel wheel or fork) rather than included in the wheel itself or the handlebars or frame of the cart. The CT can be included in the frame or body of a motorized mobile cart. The CT is not limited to use on a cart; it can also be attached to a handheld shopping basket (e.g., on the side or bottom of the basket or in the handle).

[0055] The access point (AP) is typically responsible for communicating with the cart transceiver (CT) to retrieve and generate cart status information, including information indicating or reflecting the cart's position. Types of cart status information that can be retrieved and monitored include, for example, whether wheels 32 are locked or unlocked, whether the cart is moving; the average rotational speed of the wheels (which can be sensed using rotation sensors in wheels 32); whether the cart detects specific types of position-related signals, such as VLF, EAS, or magnetic signals (described below); whether wheels 32 are slipping; the CT's battery level and general wheel "health"; and the number of lock / unlock cycles the cart has experienced since a certain reference time. In some examples, the cart may include sensors capable of determining whether its shopping basket is at least partially loaded (e.g., by analyzing the cart's vibration data), and the CT may convey the loading status (e.g., empty, partially loaded, fully loaded) to the AP. (In contrast to other wheels of the shopping cart, the term "wheel 32" is used herein specifically to refer to the wheel that includes the electronic equipment described herein.) The access point (AP) is also capable of generating and / or relaying commands to the cart transceiver (CT), including lock and unlock commands (or other types of anti-theft commands) sent to a specific shopping cart.

[0056] exist Figure 1C In the example shown, all access points (APs) communicate wirelessly with the central control unit (CCU) directly or via intermediate access points. The central control unit can be implemented as a desktop computer or hardware server comprising a wireless transceiver card or wired connection to an external transceiver unit. The CCU is typically responsible for collecting, storing, and analyzing cart status information, including location information, collected by the access points (APs). In addition to data retrieved from the cart transceiver (CT), the CCU may also collect data generated by the access points, such as measurements of the signal strength of detected cart transmissions. Some or all of the collected data is preferably stored by the CCU along with associated event timestamps.

[0057] Figure 1C The system shown can include one or more computer vision units (CVUs), for example, referencing Figure 4A and Figure 4BThe CVU 1000 is described. A CVU may include a camera (still camera or video camera), an image processor, and a transceiver configured to communicate with an AP, CCU, or CT. As further described below, the CVU (used alone or in conjunction with a CCU or AP) is capable of analyzing images of a shopping basket (captured by the camera) to determine the basket's loading status, such as empty, partially loaded, or full. The CVU can be placed near store entrances / exits (e.g., to image shopping baskets entering or leaving), near checkout counters 34 (e.g., to image shopping baskets in the checkout aisle), or near other areas of the retail store (e.g., in areas where high-value items are stored). In some examples, the CVU 1000 includes a camera transceiver unit (CTU), which may include a subset of fewer components than those of the CVU. For example, a CTU may include a camera and an RF transceiver (or a wired Ethernet connection) without an image processor. An apparatus may contain any number of CVUs or CTUs. In some implementations, the use of a CTU is more cost-effective (because each unit does not include an image processor), and image processing functions are offloaded to a CCU (or main CVU). The choice between using a CVU or CTU, and the corresponding arrangement of the CVU or CTU, will depend on the details of the installation in any particular retail store (e.g., the location or number of exits or entrances, the location or number of checkout lanes, the physical dimensions or layout of the store interior, the number of customers, the presence or location of high-value items, etc.). For example, one installation might primarily or exclusively use a CTU and offload image processing to a single CVU or CCU. However, another installation might primarily use a CVU. Yet another installation might utilize a CVU in an area storing high-value items to enable local image processing and utilize a CTU in other areas of the store. Many installation options can be chosen to suit the specific needs of a particular retail store.

[0058] In embodiments where camera modules are mounted on shopping carts, the CVU or CCU can additionally or alternatively analyze images generated by the cameras mounted on the cart. In some implementations, these cart-mounted camera modules may be the sole source of image data in the system (e.g., the CVU and CTU may be omitted or there may be no cameras).

[0059] The CCU or CVU can analyze the collected data in real time to make decisions, such as whether to send a locking command to a specific cart 30, whether to activate the store's video surveillance system, or whether to send an alarm message to personnel. Figure 1A and Figure 1BAn example of a CVU or CTU that communicates with a door manager and takes appropriate anti-theft actions when necessary is shown. The door manager may include an access point (AP) for communicating with a cart transceiver in the cart wheels, as described herein. For example, as a cart approaches or passes a store exit, the CCU or CVU may analyze the cart's recent history (e.g., path and speed) to assess whether a customer is attempting to leave the store without paying. The CCU (or CVU) may analyze camera images to assess whether a shopping basket leaving the store is at least partially loaded or whether the basket has passed checkout station 34. (The access point may additionally or alternatively be responsible for making such an assessment.) Based on the result of this assessment, the CCU may send a locking command to the cart (typically via the access point) or may not issue a command authorizing the cart to leave. As another example, if the CCU detects a rapid increase in the number of carts in operation, the CCU may alert personnel (e.g., via the store LAN) that additional checkout stations may need to be opened. As another example, if the CVU detects that a shopping basket is leaving the store and is at least partially loaded (or contains items at high risk of theft), the CVU can send an alert to store staff, activate an alarm, display a warning to the shopper via a display screen (or smart navigation module) on the basket, or transmit a locking command to the smart wheels of the cart to actuate the brakes (e.g., to prevent the cart from moving).

[0060] The task of analyzing images preferably involves using one or more trained machine learning models. Different training models can be created and used for different types of classification tasks (e.g., whether the cart is empty, whether the cart contains high-risk items, etc.) and different image types (e.g., photos taken from cameras mounted on the cart and not mounted on the cart). To train the machine learning model, a group of human labelers can review and label or “mark” images of carts taken in a particular store or a group or chain of stores using similar shopping carts and selling similar goods. Examples of labels include “empty cart,” “containing only non-merchandise items,” “containing children,” “containing merchandise,” “partially loaded,” “fully loaded,” “containing high-risk items,” and “containing electronic items.” These labeled images can be used to train and validate the machine learning model. In some cases (especially when using cameras mounted on the cart), labeled images of carts containing specific items known to be frequently stolen can be generated, and these images can be used to train one or more models to detect these items in the cart.

[0061] CCU can also run data mining and reporting software that analyzes data collected over time to detect important movement patterns and trends. For example, CCU can generate reports showing how customers typically move through the store, how much time they spend in each aisle or other shopping area, the loading levels of shopping baskets leaving the store, and data on theft incidents (e.g., full or partially loaded shopping baskets leaving the store without paying). This information can be used, for example, to adjust the store layout or the size or number of shopping baskets offered to shoppers.

[0062] The CCU (or CVU) can additionally or alternatively transmit data it collects via a cellular or wireless network (e.g., the Internet) to remote nodes that process analysis and reporting tasks. For example, the CCU (and possibly one or more access points or CVUs) may have an autonomous WAN link that uses cellular data services, such as General Packet Radio Service (GPRS), to transmit collected data to remote nodes for analysis and reporting. This functional component can be used to monitor the health of the system from a remote facility. The system may also be able to be tested and configured via a WAN link from the remote facility.

[0063] like Figure 1C As shown, the CCU (or CVU) can connect to various other types of systems existing within the store. For example, the CCU or CVU can connect to a pre-existing alarm system and / or video surveillance system, in which case the CCU or CVU can be configured to activate an audible alarm or video camera upon detecting an unauthorized exit event (in various implementations, the video camera in the surveillance system may be different from or the same camera in the CVU). As another example, the CCU or CVU can connect to a pre-existing central storage computer that maintains information about the status of the store's checkout cash registers or mobile payment platforms; this information can be retrieved and used by the CCU or CVU, as described below, to assess whether a customer has paid for goods through an operating checkout lane or using a mobile payment application or mobile payment point.

[0064] In some implementations of the system, the CCU can be omitted. In these implementations, the access point (AP) can perform all the other real-time analytics functions that might be handled by the CCU. For example, an access point or CVU installed near a store exit might be able to detect a customer attempting to leave the store without paying (or determine that the cart's basket is at least partially loaded) and decide whether to send a locking command to the cart. To accommodate centralized and distributed installations, each access point or CVU can operate with or without a CCU. Implementations that omit the access point are also possible, allowing the CCU or CVU to communicate directly with the cart transceiver. Many variations in the components and circuitry of distributed network connections can be envisioned.

[0065] The cart transceiver (CT), access point (AP), computer vision unit (CVU), checkout gate (CB), camera module mounted on the cart, and central control unit (CCU) can all operate as the only addressable node on the wireless tracking network. For example... Figure 1C As shown, another type of node that may be included in the network is a handheld mobile control unit (MCU). The mobile control unit is designed to enable store personnel to unlock individual carts by pressing a button on the MCU. The mobile control unit may also include functions for retrieving and displaying various types of cart status information, configuring the wheel / cart transceiver and updating its firmware, and controlling the motorized cart retrieval unit 40 (see the discussion of cart retrieval unit 40 below).

[0066] In some implementations, various types of nodes (e.g., cart transceivers, access points, central control units, computer vision units, cart-mounted camera modules, and motion control units) can communicate with each other using a non-standard wireless communication protocol that allows the cart transceiver to operate with a very low duty cycle without needing to maintain synchronization with the access point when not in operation. Therefore, the cart transceiver can operate for extended periods (e.g., several years) using a relatively small battery mounted in wheel 32. Details of a particular wireless communication protocol described in U.S. Patent No. 8,463,540, “Two-Way Communication System for Tracking Locations and Statuses of Wheeled Vehicles,” the entire disclosure of which is incorporated herein by reference.

[0067] Each cart transceiver (CT) is preferably capable of measuring received signal strength based on the RSSI (Received Signal Strength Indication) value of a transmission it receives on the wireless tracking network. The system can use these RSSI measurements in various ways. For example, the cart transceiver can compare the RSSI value transmitted by the access point to a threshold to determine whether to respond to the transmission. The cart transceiver can also report this RSSI value (along with the cart transceiver's unique ID) to the access point so that the system can estimate the location of the shopping cart or the distance to the shopping cart. As another example, the cart transceiver can be programmed to generate and report RSSI values ​​transmitted from other nearby cart transceivers; this information can then be used to estimate the number of carts queuing at the checkout lane, in the cart storage structure, near the store entrance / exit, in the cart pile retrieved by the mechanized cart retrieval unit 40, or elsewhere.

[0068] Figure 1C The diagram shows three checkout stations 34, each including a checkout register (REG), which typically includes a merchandise scanner. In this particular example, each checkout station 34 includes an access point (AP) that can be mounted to a pre-existing pole (if present) indicating the checkout lane number. Each such access point can include a connection or sensor that enables it to determine whether each checkout station is currently operational. This information is useful for assessing whether a customer passing through the checkout lane has paid. Several different methods that can be used to sense the operational / inactive status of a checkout station are described below. Each access point placed at checkout station 34 can communicate with nearby shopping carts / trolley transceivers, such as those queuing in the corresponding checkout lane, using a directional antenna (see Figure 2 discussed below).

[0069] In some implementations, stores can utilize the end of the checkout lane, at the store exit (e.g., Figure 1C As shown), checkout barriers (CBs) are used in areas with high-value items. CBs typically include gates, barriers, or revolving doors that will be locked unless the customer is allowed to leave the checkout lane or store or high-value area (e.g., the customer has already paid for the items). The CB can then be unlocked to allow the customer to leave (e.g., by pushing a rotating gate to allow passage). After leaving, the gate will rotate to close and lock to prevent other customers from leaving without paying. CBs can communicate with the store's CCU, CVU, CTU, checkout cash register, mobile payment point 35 (described below), etc., to receive commands to unlock (or lock) the barrier.

[0070] Figure 1C A mobile payment point 35 is also schematically shown. The mobile payment point does not need to be fixed in a physical location within the store, but can represent a wireless network connection that allows shoppers to pay for items in their shopping baskets. For example, shoppers can access a mobile payment application (e.g., on their smartphones or on a communication display installed in their shopping baskets or trolleys), which can electronically record the items or goods in the basket and provide mobile payment options (e.g., payment via credit or debit card). The mobile payment point 35 is capable of wireless communication with APs, CCUs, CVUs, etc., thereby enabling it to record payments and transmit them to [the relevant authorities / entities]. Figure 1CAppropriate components of the system are shown. For example, the CVU can detect (via computer vision image analysis as described herein) that a loaded shopping basket is about to leave the store. The CVU can access payment information to determine whether the shopper associated with the leaving basket has paid for the items in the basket. If the shopper has paid (e.g., via mobile payment point 35 or through cash register 34), the system can allow the shopping basket to leave the store without triggering anti-theft actions. However, if the shopper has not paid, the system can trigger anti-theft actions (e.g., activating an alarm or store surveillance system, sending a locking command to the cart wheels, notifying store staff, etc.).

[0071] Access points can be additionally or alternatively installed on various other fixed and / or mobile structures near the store. For example, such as Figure 1C As shown, access points can be installed in shopping cart storage structures 36 (two are shown) in a store parking lot. These access points installed on the parking structures can be used to detect and report the number of carts stored in their respective areas, and can also be used to enable in-store access points, CVUs, or CCUs to communicate with carts that are out of range.

[0072] Figure 1C The system shown can include other optional components. For example, a power-assisted (mechanized) cart retrieval unit or trolley 40, which can be a cart pusher or cart puller, can be used to retrieve shopping carts and return them to the cart storage location 36. The store can include a pair of conventional EAS (Electronic Goods Surveillance) towers at the store exit, or additionally or alternatively at the end of each checkout lane. While EAS towers are not required to achieve the various functions described herein, the system can take advantage of their ubiquitous presence in retail stores. For example, each cart transceiver (CT) can include an EAS receiver for detecting its passage between the pair of EAS towers and can be configured to report EAS detection events on a wireless tracking network; this information can also be taken into account when assessing whether a departing customer has paid.

[0073] Figure 1C The example store configuration is also shown as having an extremely low frequency (VLF, typically below 9 kHz) signal line 44 buried in the sidewalk along the perimeter of the parking lot or near the store exit. Such a signal line can be used to define the boundary of an area where shopping carts are permitted to be present. The wheels 32 of the shopping cart can include a VLF receiver that detects the VLF signal and engages a brake when the cart is pushed over the signal line 44. Although in Figure 1C It is not shown in the diagram, but a VLF line can also be set at the store exit so that all carts passing through the exit must cross this line, and / or at other locations of concern.

[0074] Although this system does not require the use of VLF signal line 44, it is preferably capable of using one or more VLF lines as a mechanism for monitoring the cart's position. Specifically, the cart transceiver (CT) preferably includes a VLF receiver. The VLF receiver may be able to detect codes transmitted on the VLF lines, thus enabling the use of different lines to uniquely identify different areas or boundaries. When a VLF signal is detected, the cart transceiver can take various actions depending on the situation. For example, the cart transceiver may attempt to report a VLF detection event on the wireless tracking network and then wait for a command instructing whether to engage the brake. If no command is received within the pre-programmed time period (e.g., 2 seconds) in this example, the cart transceiver may automatically engage the brake. The VLF detection event can be reported to the CVU on the wireless tracking network, and the CVU can image the cart or basket to determine its loading status. If it is determined that the cart or basket is not loaded, theft is unlikely, and a braking command may not be sent (or the brake may be instructed not to engage). Conversely, if it is determined that the cart or basket is at least partially loaded and is leaving the store, the CVU can send a braking or locking command or some other type of anti-theft command to the cart. For example, for a hand-held basket (which does not have locking wheels), anti-theft commands could include warning commands (e.g., activating a light or alarm on the basket to alert a shopper), commands to activate the store's video surveillance system (to obtain video of a potential theft), signals to alert store security personnel, etc. Such anti-theft commands can be used additionally or alternatively with wheeled carts.

[0075] Further reference Figure 1C Optionally, one or more magnetic markers or magnetic strips (MAGs) may be provided above or below the store floor to provide additional or alternative location tracking mechanisms. As shown, these magnetic markers may be placed in key locations, such as at each checkout lane and store exit. Although not shown in Figure 1, one or more magnetic markers may also be placed in parking lots and / or shopping aisles. Each magnetic strip is capable of having a unique magnetic image, which can be sensed by optional magnetic sensors included in or attached to the shopping basket or shopping cart 30 in the wheel 32. Thus, the magnetic marker serves as a magnetic barcode for identifying a specific location. In one implementation, as the cart 30 passes a magnetic marker, a cart transceiver (CT) transmits the detected magnetic code or information from which that code can be derived over a wireless tracking network. Further details on how magnetic markers are sensed and used are described in U.S. Patent No. 8,046,160, “Navigation Systems and Methods for Wheeled Objects,” the entire disclosure of which is incorporated herein by reference.

[0076] Figure 1C The system shown can include other or alternative functions or components. For example, the system can implement the techniques and functions for low-energy positioning of movable objects described in U.S. Patent No. 9,606,238, the entire disclosure of which is incorporated herein by reference. These techniques can be used to track the position of a shopping basket as it moves through a store environment. The movement of the shopping basket can be tracked using techniques (e.g., dead reckoning) described in the aforementioned combined U.S. Patent Nos. 8,046,160, 9,731,744, or 10,232,869; the entire contents of each of these are incorporated herein by reference.

[0077] It is clear from the preceding discussion that Figure 1C Many of the components shown are optional and may or may not be included in a given system setup. For example, in some setups, magnetic markers, EAS towers, checkout gates, and / or VLF signal lines may be omitted. Additionally, access points or CCUs can be omitted. A CTU can be replaced by a CVU, and vice versa. Furthermore, the components shown may differ from the illustrated arrangement. For example, VLF signal lines (e.g., replacing the shown magnetic markers and EAS towers) may be placed in the checkout lanes and / or store exits / entrances to enable the cart to detect checkout events and exit / entry events separately. Furthermore, other types of signal transmitters and detectors / receivers can be used to monitor cart position. For example, ultrasonic transmitters / receivers can be used to track cart position, or the store may include radio frequency (RF) detectors (e.g., located on the ceiling) that detect RF signals from the cart and use direction of arrival (DOA) technology to determine the cart's position.

[0078] Evaluation of Customer Payment Technology Implementation Examples

[0079] The system supports a variety of different methods to assess whether a customer has left the store without paying. The specific method used may vary considerably depending on the type and location of the system components included in a given device. For example, if the store does not include any Electronic Goods Surveillance (EAS) towers, magnetic markers (MAGs), or VLF lines, the determination can be made solely or primarily based on cart location / path information determined according to CT-AP communication, optionally considering historical wheel speed records as an additional factor. If EAS towers, magnetic markers, and / or VLF signal lines are provided, they can be used as additional or alternative sources of information for decision-making. The system may include a Computer Vision Unit (CVU) near checkout lane 34, and the CVU is capable of analyzing images of the checkout lane to determine whether a shopper has passed through the lane, interacted with the store checkout staff or the store payment system, remained in the lane for a sufficient amount of time to indicate the checkout and payment process (e.g., more than 1 minute, 2 minutes, 3 minutes, 5 minutes, or longer), provided payment information, etc. The CVU can analyze the images to determine whether a shopper is approaching the exit from the direction of checkout lane 34 or from another direction where payment is unlikely. See below for reference. Figure 4A Further described, one or more additional secondary cameras 410a can be located within the facility to monitor the movement of shopping carts (e.g., through checkout lanes or payment points or from locations storing high-value items). As the cart moves from the field of view of one secondary camera to another (or CVU or CTU), the system is able to hand over tracking of the cart to the next camera to provide a substantially continuous path for the cart. The CVU (or CCU) can access payment information from the mobile payment point 35 to determine whether a departing shopper has paid for the items in the shopper's basket. U.S. Patent No. 8,463,540 describes other (or alternative) techniques for assessing whether a departing customer has paid, the entire disclosure of which is incorporated herein by reference. Many combinations or variations of the foregoing can be used to determine whether a shopper associated with a shopping basket (e.g., a shopper pushing a manually propelled shopping cart) may have already paid for the items in the shopper's basket.

[0080] Example of a shopping basket on a shopping cart

[0081] Figure 2A The functional components of an example shopping cart 30 with a shopping basket 205 are shown. The shopping cart 30 is manually propelled and includes a smart positioning system 210 and one or more anti-theft wheels or anti-theft components 215 (capable of braking, locking, or preventing the rotation of the wheels or the movement of the cart). The smart positioning system 210 can be mounted on the handle of the cart 30 (e.g., as shown in the image). Figure 2A and Figure 2BThe anti-theft wheel 215 can be mounted in or on the trolley (e.g., at the front of basket 205), or can be installed elsewhere on or in the trolley (e.g., at the front of basket 205). The anti-theft wheel 215 can be a smart locking wheel, such as a wheel that, in addition to a locking or braking mechanism, has sensors (e.g., for sensing VLF lines), a wireless communication system (e.g., a trolley transceiver CT), and / or a processor. A smart positioning system 210 can be used to track the position of the shopping trolley 30, which can estimate the trolley's position using dead reckoning or vibration detection techniques. For example, the smart positioning system 210 can include components or functions described in U.S. Patent Nos. 8,046,160, 9,731,744, or 10,232,869, the entire contents of each of which are incorporated herein by reference.

[0082] The functions of the navigation system and the anti-theft system can be distributed between the smart positioning system 210 and the smart locking wheel 215. For example, one or both of the smart positioning system 210 and the smart locking wheel 215 can have the capability to detect leaving / entering events; the anti-theft function of wheel locking can be located in the smart locking wheel 215, while the anti-theft function of user warning can be located in the smart positioning system 210. Furthermore, although... Figure 2A A shopping basket 205 for a manually propelled shopping cart is shown, but similar techniques described herein are applicable to shopping baskets on motorized or mobile shopping carts or handheld shopping baskets carried by shoppers. For example, a smart positioning system 210 can be attached to (or integrated into) a motorized or mobile shopping cart, or attached to a handheld shopping basket.

[0083] like Figure 2B As shown, some shopping carts include a user-adjustable child seat that can move between a closed position and an open position. In the open position (e.g....) Figure 2B As shown), shoppers can place children (or other items) on the seat section. In many strollers, shoppers can push the metal frame of the child seat 1620 away from the handle 1610 of the stroller 30, which causes the seat section to move into a horizontal position. Therefore, the open position is sometimes referred to as the child seat lowered. Figure 2B A shopping cart 30 is shown with the child seat 1620 in either the open or lowered position. As will be further described below, a computer vision unit (CVU) can image the shopping cart 30 to determine whether the cart's loading is (at least in part) attributable to a child placed in the child seat 1620 or to an object (e.g., a handbag) placed on the child seat, rather than store merchandise.

[0084] In some embodiments, the shopping cart 30 may include a camera or camera module capable of imaging the contents of the basket 205. Figure 2C (side view) and Figure 2D (Top view) shows an example of such a cart 30, illustrating a camera module 217 mounted on the inner front edge or front wall of a basket 215, preferably on the front lip. In the example shown, both the horizontal field of view (HFOV) and vertical field of view (VFOV) of the camera are 90 degrees. The camera's HFOV and VFOV are preferably in the range of 45° to 135°, more preferably in the range of 60° to 120° or 75° to 105°, with optimal values ​​depending on the size of the basket 205 and the camera's mounting position. As described above, the system can use images (which may include video) captured by the camera module 217 to identify or classify products or other items placed in the basket 205. The camera module 217 may include a processor for receiving and analyzing images, and / or may include a transceiver for transmitting images to a CVU, CCU, or other node for analysis.

[0085] As discussed further below, one or more types of sensors on the shopping cart (e.g., vibration sensors, scales, or motion sensors that sense movement within the basket) can be used to detect events where a customer may add items to the cart 30. This event may trigger the camera module 217 to capture an image of the shopping cart contents immediately after the event occurs. Using a trigger event to initiate image acquisition keeps the camera or camera module 217 in a "off" or low-power state (thus saving battery power) when no items are added to the cart, and also reduces the number of images stored and analyzed. Power consumption can be further reduced by activating the camera module 217 or its camera only when the shopping cart 30 is in a specific area, such as a merchandise area or a high-risk theft area; for example, the camera module 217 may avoid generating images when the cart is in a store parking lot.

[0086] The item addition event can also serve as a system trigger to capture one or more additional types of event metadata, such as the cart's location, event timestamp, wheel rotation speed, recent wheel speed history, and / or basket weight measurement (if the shopping cart includes a scale, as described below). Furthermore, this event may also trigger the uploading and / or analysis of the captured images or image sets (as described below). Another event that can be used to trigger image capture is when the cart transitions from a stationary state to a moving state while it is in a merchandise area; for example, if wheel rotation is initiated when the cart is in a merchandise area (or a specific type of merchandise area), the wheel assembly can generate a signal that causes camera module 217 to generate an image of the cart.

[0087] Figure 2E It shows Figure 2C and Figure 2DOne embodiment of the camera module 217 is shown. In the illustrated embodiment, the camera module 217 includes a lens 217A (e.g., a fisheye lens or Fresnel lens), an imager 217B, an image preprocessor 217C, a control processor or MCU 217D, a wireless transceiver 217E, a battery 217F, and memory (not shown). Although a single camera (imager) is shown, in some embodiments, the camera module 217 may include multiple cameras (e.g., two, three, or four) spaced horizontally and / or vertically, allowing imaging of the cart contents from multiple perspectives. As shown, the camera module 217 may also include or be connected to a shock / vibration sensor 217G, such as an accelerometer, capable of sensing cart vibrations caused by adding items to the basket 205. Figure 2E In this embodiment, the control processor 217D is configured to analyze vibration graphs of detected shock or vibration events to determine whether the event is likely a result of cart movement compared to adding an item to basket 205. U.S. Patent No. 10,232,869 describes examples of components and processes for detecting and analyzing cart vibrations, the disclosure of which is incorporated herein by reference. In some embodiments, the shock / vibration sensor 217G may be physically separate from the camera module 217 (e.g., it may be mounted in the bottom of the basket or in the wheel assembly), in which case its output or analysis of its output may be wirelessly transmitted to the camera module 217.

[0088] When an event that may indicate the addition of an item is detected, the control processor 217D can initiate the capture and initial processing of an image (or a set of two or more images if the camera module includes multiple spaced-apart cameras). In some cases, the task of analyzing the newly captured image is divided between the camera module 217 and an external node such as a CVU. For example, the camera module's control processor 217D or image preprocessor 217C can compare the image with recently captured images to determine if the contents of the cart may have changed. If a change in the cart's contents is detected, the control processor can mark the changed areas in the image (e.g., by drawing boundaries around the altered portion or the newly detected item) and then transmit the marked image (along with the cart ID and any captured event metadata, including the image capture timestamp) to the CVU or other remote node for further analysis.

[0089] Subsequently, the CVU or other remote nodes can use one or more trained machine learning models, or other types of software components (e.g., rule-based classifiers), to classify newly added items, for example, by assigning items one or more categories such as "goods," "non-goods," "high-risk items," "electronics," etc. In some cases, the task of analyzing images may also involve using OCR (Optical Character Recognition) to identify text on product packaging; in this case, any identified text can be used or considered for classifying or identifying items. Images captured by the shopping cart's camera module 217 during the shopping process, along with the results of related image analysis, can be maintained by the CCU (or another processing node) in conjunction with the shopping cart's ID. This data, along with any other data collected for the cart during the shopping process, can ultimately be used to determine whether to authorize parking and leaving the store.

[0090] In some embodiments, the shopping cart 30 may include a built-in scale (not shown) configured to weigh the contents of items placed in the basket 205. For example, the scale may be part of a structure that mechanically connects the basket 205 to a cart frame or part of a cart frame that supports the basket. An example of a cart with a built-in scale is disclosed in U.S. Patent Publication 2015 / 0206121 A1, the disclosure of which is incorporated herein by reference. The scale may alternatively be incorporated into one or more wheel assemblies of the cart. The scale may be electrically or wirelessly coupled to a processor on the cart, such as a control processor 217D for a camera module or a processor for a smart wheel assembly, such that the output of the scale is transmitted to the processor. As discussed below, the system may use the item weight measurement, combined with other collected data, such as cart images and / or cart location, to identify or classify items added to the cart. An example method for classifying items in the shopping cart basket and making a departure authorization decision based on this classification is described below.

[0091] In some system implementations, the shopping cart 30 in the store may only be equipped with the camera module 217, and therefore may lack smart wheel components or other navigation components. In this implementation, the camera module 217 may include location-tracking electronics capable of communicating with an access point (AP), a fixed transmitter, or a transponder in the store to monitor the cart's location. For example, the camera module 217 may be able to receive signals from smart shelves or smart tag systems included in some stores, thereby enabling it to identify nearby merchandise. Another example is that the camera module 217 may include electronics typically included in a cart transceiver (CT), in which case the camera module can use RSSI measurements to determine proximity to the store access point, as described in the patent mentioned above. As discussed below, Bluetooth and other wireless standards can alternatively be used to monitor the cart's location.

[0092] Example of an intelligent positioning system / Implementation method of an intelligent braking wheel

[0093] Figure 3 A component set 300 for an example tracking system for a shopping basket (e.g., a basket on a shopping cart or motorized trolley, or a handheld shopping basket) is shown. This example component set 300 includes the following components: (1) a smart positioning system 210; (2) smart locking wheels 215; (3) fixed function components 385 associated with store exits and / or entrances, checkout lanes, high-value areas, and the location of the wheels 215, which can be reset or updated; (4) a system configuration and control device 390; (5) an RF beacon or other RF function component 395; and (6) a computer vision unit (CVU) 1000 or a camera transceiver unit (CTU) 1001. Reference Figure 4A and Figure 4B The CVU and CTU are described further.

[0094] The intelligent positioning system 210 includes (1) sensor elements 315 for determining the trolley's heading and speed (e.g., magnetometer and / or accelerometer), and optionally determining the system's temperature (e.g., temperature sensor); (2) optional sensors 320 that provide data that can infer the wheel's rotational rate (e.g., the sensor does not need to be close to the wheel); for example, a vibration sensor; (3) a processor and memory 325; (4) a communication system 330 that (e.g., via an RF link) communicates with the intelligent locking wheel 315, system configuration and control device 390, RF beacon or other RF functional component 395, and / or CVU 1000; and (5) optional detectors 310 configured to determine that the trolley is passing through a store's exit / entrance (exit / entry event), and in some embodiments, to determine whether the movement is leaving or entering the store. In some systems, the circuitry in the wheel performs the actual detection function; the intelligent positioning system communicates with the detection circuitry in the wheel to obtain exit / entry information. Some systems may use detector 360 as the primary detector and detector 310 as the secondary detector; (6) an indicator 335 (e.g., visual and / or audible) that provides a notification to the user that the stroller is in a warning area and / or is about to lock. The indicator may include a display configured to output text or images (e.g., to the user that a perimeter boundary is nearby and the wheels will lock if a wheeled object moves outside the perimeter boundary). The indicator may include a light (e.g., a light-emitting diode (LED)) that illuminates or flashes as a notification to the user. The indicator may include an audible alarm or notification. In some embodiments, the indicator includes a speech synthesizer capable of outputting a human-understandable message such as “The stroller is approaching the boundary and is about to lock.” The indicator may include a speaker for outputting audible notifications. The smart positioning system 210 may also include a light detector 333 for detecting ambient light features for navigation purposes, or a vertical position detector 337 (e.g., a pressure sensor) for determining which layer of a multi-layered structure the smart positioning system is located on. The function of these components is further described in U.S. Patent Nos. 9,731,744 or 10,232,869, which are incorporated above.

[0095] Figure 3An example is shown in which the smart positioning system 210 is used with a wheeled cart that includes smart locking wheels 215 (although this is not required, the system 210 can be used on a handheld basket). Wheel 215 includes (1) a locking mechanism (e.g., a brake) 380 configured to prevent wheel rotation when the locking mechanism is activated (or the cart itself is translated); (2) a wheel rotation detector 375, such as a tuning fork and a striking pin (e.g., a part that strikes the tuning fork as the wheel rotates); (3) a processor and memory 370; (4) a communication system 365 configured to communicate with the smart positioning system 210, system configuration and control device 390, RF beacon or other RF functional component 395, and / or CVU 1000 or CTU 1001; (5) an optional detector 360 configured to detect leave / entry events, and in some embodiments, to detect whether movement is leaving or entering a store; and (6) an optional direction / omnidirectional wheel angle detector 383 configured to detect the direction of the (omnidirectional) wheel.

[0096] Fixed functional component 385 can be associated with store entrances and exits, checkout lanes, areas containing high-value items, and locations where the trolley's position can be reset or updated. The proximity of these functional components can be detected by detectors in a smart positioning system or smart locking wheels. The fixed functional component can be used to provide an accurate reference position to the smart positioning system (e.g., to reset any accumulated dead reckoning errors). Fixed functional component 385 may include VLF lines, access points, RF fields generated for warnings or locking, checkout gates, EAS towers, magnetic markers, or electromagnetic markers. CVU 1000 or CTU 1001 can communicate with fixed functional component 385 to provide appropriate signals when a shopping trolley approaches the fixed functional component (e.g., to provide a lock or unlock signal to a checkout gate or trolley transceiver, or to provide a position signal to reset or update the trolley's position).

[0097] The system configuration and control device 390 is capable of performing housekeeping tasks such as configuration and control. The device 390 is capable of communicating with the communication system 330 in the smart positioning system and / or the communication system 365 in the smart locking wheels. The system configuration and control device 390 may include a CCU (e.g., see reference 365). Figure 1C (as described), or in some cases, including CVU 1000.

[0098] RF beacons or other RF features 395 are capable of transmitting RF signals for entry / exit detection and / or precise location determination.

[0099] CVU 1000 or CTU 1001 can be used with, for example, reference... Figure 1CThe described smart locking wheel 215, smart positioning system 210, RF beacon or other RF functional component 395 and / or system configuration and control device 390 or central control unit (CCU) communicate wirelessly. Additionally or alternatively, the CVU or CTU can communicate with the CCU or controller 390 using a wired LAN connection, such as Ethernet.

[0100] The system described herein can be implemented with more or fewer features / components than those described above. Furthermore, the system can be implemented with different configurations than those described above; for example, the rotation detector can be implemented in one of the smart positioning system and the smart locking wheel, and the RF beacon can communicate with one of the communication systems 330 and 365, instead of both. Additionally, Figure 3 The components in the diagram can be combined, rearranged, separated, or configured in ways different from those shown.

[0101] The intelligent positioning system 210 can be positioned at one or more locations within a wheeled object. For example, some or all of the intelligent positioning system can be positioned in the handle, frame, casters, wheels, etc., of a trolley. For motorized shopping carts or mobile trolleys, the intelligent positioning system 210 can be attached to the frame or body of the cart, or combined with other electronic circuitry for operating the cart. The intelligent positioning system described herein can be used in applications other than cart enclosures. For example, the system can be used to estimate the position, path, or speed of a wheeled object. Moreover, in cart enclosure applications, the cart can include one or more wheels configured to prevent movement of the cart when activated, for example by including wheel brakes. For example, when the brakes are activated, the wheels can lock or prevent rotation. U.S. Patents US 8,046,160, US 8,558,698, and US 8,820,447 describe examples of cart wheels capable of preventing movement of the cart, the entire disclosure of which is incorporated herein by reference in its entirety.

[0102] Further description of the functionality of system 300 can be found in U.S. Patent Nos. 9,731,744 and 10,232,869, the entire contents of each of which are incorporated herein by reference.

[0103] Example of a theft prevention system using computer vision

[0104] Figure 4A An anti-theft system 400 is schematically illustrated, which uses computer vision technology to identify whether a shopping basket, at least partially loaded, is leaving a store. System 400 includes a computer vision unit (CVU) 1000, which may be generally similar to the reference design. Figure 1CThe CVU described. The CVU 1000 can be located near the surveillance area 440, such as near the entrance / exit, checkout lane 34, or areas in the store with high-value goods (such as alcohol, health products, medicines), etc.

[0105] exist Figure 4A In the illustrated embodiment, CVU 100 communicates with door manager 450, which in some such embodiments may perform reference operations. Figure 3 The system configuration and control device 390 are described above. For example, the door manager 450 can communicate with the communication system 330 in the smart positioning system 210 and / or the communication system 365 in the smart locking wheel 215, and issue anti-theft commands (e.g., to lock or brake the wheels, activate an alarm or warning, etc.). The door manager 450 may include a central control unit (CCU) (e.g., see reference). Figure 1C As described, or in some cases, a component of the CVU 1000, or in some cases, capable of communicating with the CCU. The door manager 450 can control fixed functional components 385 used at the store exit, such as defining one or more VLF lines or RF fields (e.g., such VLF or RF signals can be detected by a cart transceiver near the exit). Fixed functional components 385 can include checkout gates (CBs) located, for example, at the exit or checkout lane. The door manager (or CVU) may have an autonomous WAN link that uses cellular data services, such as General Packet Radio Service (GPRS), Long Term Evolution (LTE), or 5G New Radio (5GNR), to transmit collected data about store exit events to the CCU or a remote node (e.g., refer to...). Figure 4B The cloud platform 470 described is used for analysis and reporting. For example, remote nodes can be accessed by authorized store personnel (e.g., via a web browser) who can view statistics on exit incidents (e.g., theft) or images or videos of exit incidents (e.g., videos of shoppers attempting to exit to steal).

[0106] exist Figure 4A In the illustrated system 400, an RF field and a VLF line are used to provide warning and locking zones. When a shopping basket crosses the warning zone, an unauthorized shopping basket leaving the store can first receive a warning (e.g., audible or visual, and displayed, for example, by the smart positioning system 210), and then receive a locking signal (e.g., a command to activate the wheel brakes) if the basket crosses the locking zone. In other implementations, neither an RF field nor a VLF line is used, but only one of the RF field or VLF line. Similarly, in other implementations, only one of the warning and locking zones is utilized. Additionally or alternatively, a checkout gate (CB) can be used.

[0107] The CVU 1000 can communicate with payment points, such as checkout cash registers 34 or mobile payment points 35, to access payment information related to shopping baskets in the monitored area. As described herein, theft may occur when a shopping basket containing items attempts to leave the store without any indication that the customer has paid for the items. Therefore, the CVU 1000 can use information from the payment point to at least partially determine whether payment has been made for the items in the loaded shopping basket.

[0108] The CVU 1000 can include a camera 410 oriented to image a monitored area 440. The camera 410 can include a video camera capable of generating an image set 430, which is used by an image processor 420 to analyze basket activity within the monitored area 440. The image set 430 can include video, one or more frames of video, or a selection of images acquired by the camera. The camera can include a grayscale camera, a color camera (e.g., RGB), or a camera capable of imaging in the non-visible portion of the electromagnetic spectrum. For example, the non-visible portion can include the infrared (IR) region (which may be advantageous for imaging dark entrances or at night, where (optionally) an IR light source can be used to illuminate the entrance) or the ultraviolet (UV) region (which may be advantageous for imaging through glass entrances / exits or windows). Using a camera 410 that provides imaging in both the visible and invisible portions of the electromagnetic spectrum allows the CVU or CCU to perform multispectral or hyperspectral image analysis, which can better track or classify carts or goods based on their unique spectral reflectance characteristics. For example, multispectral imaging can be used to detect smuggled items based on the specific color of their packaging. This detection can be performed using a relatively small number of spectral bands (e.g., 7 to 9) under a wide variety of store lighting conditions and can be achieved by a CMOS imager and a Bayer mask or a set of filters for each spectral band. Camera 410 may include a depth camera that acquires images and depth data (e.g., distance from the camera) of objects in the images, and can be advantageously used for depth sensing and motion tracking of the basket. The depth camera may include a stereo camera comprising two or more spaced-apart image sensors that determine depth information via stereo technology.

[0109] In some implementations, the CVU (or component) can be powered by Power over Ethernet (PoE). In some implementations, camera 410 includes a video camera operating at 20 or more frames per second, providing an image resolution of 4 megapixels or greater (e.g., 1920×1080 or greater), and streaming video using the Real-Time Streaming Protocol (RTSP). Video can be compressed using the H.264 protocol for efficient bandwidth communication. In some implementations, such a camera is available from Hikvision Digital Technology Co., Ltd. (Industrial City, CA).

[0110] Camera 410 may include multiple cameras. For example, CVU 1000 or CTU 1001 may include imaging camera 410, and system 400 may include one or more secondary cameras 410a separated from camera 10 in the CVU or CTU. Secondary cameras 410a may be housed in the same housing as the CVU or CTU, or may be physically separate from the CVU or CTU. Secondary cameras 410a may be configured to have a field of view that at least partially overlaps with camera 410 (e.g., they may be used for image processing and basket loading sorting). The use of one or more secondary cameras 410a may allow system 400 to track basket 205 in areas outside the field of view of camera 410. For example, one or more secondary cameras 410a may be placed near checkout points or storage areas containing high-value items, allowing system 400 to track the movement of basket 205 in these areas before or after it enters the field of view of camera 410 of the CVU or CTU (e.g., for loading sorting). Secondary camera 410a can be placed near the store exit to enable tracking of baskets near or beyond the exit. The distance between camera 410 and secondary camera 410a can depend on the field of view, lens size, and height above the floor of the retail facility, etc., of these cameras. In various embodiments, the distance between secondary camera 410a and camera 410 can range from about 10 cm to about 1 m or more.

[0111] Some or all of CVU 1000, CTU 1001, or secondary camera 410a are installed near the store exit because this is where thieves would attempt to leave with unpurchased goods. In many retail facilities, store exits have a significant amount of glass, such as glass doors and windows. The use of glass allows sunlight (during the day) into the retail facility and can provide shoppers with a bright and pleasant shopping experience. However, sunlight can cause glare, flash, or reflections from the floor, metal surfaces, and metal shopping carts. Such glare, flash, or reflections can create artifacts in images used for motion tracking or computer vision analysis. Therefore, in some implementations, some or all of cameras 410, 410a can include a polarizing lens or filter 411 to reduce glare, flash, or reflections in the acquired images. For example, the polarization direction in the lens or filter 411 can be orthogonal to the path of sunlight reflected from the facility's floor.

[0112] Image processor 420 may include a hardware processor (e.g., a CPU or a graphics processing unit (GPU)) to perform the image analysis and object recognition tasks described herein. In some embodiments, image processor 420 may include an Edge Tensor Processing Unit (Edge TPU) available from Google, Inc. (Mountain View, CA) that supports TensorFlow Lite machine learning and computer vision models.

[0113] The CVU 1000 can include an RF communication node 425 (e.g., a transceiver) to communicate with other components of the system 400 (e.g., a door manager 450, a payment point, or a shopping basket (e.g., a smart positioning system 210 or locking wheel 215)). The RF communication node 425 can communicate with a reference... Figure 1C The system described communicates with any component. In addition to or alternatively to RF node 425, the CVU can include a wired LAN connection, such as Ethernet. For example, the CVU 1000 can be linked to the CCU via Ethernet.

[0114] In some implementations, the CVU 1000 (or CTU 1001) can be provided as a System-on-Module (SoM) board configured to perform machine learning inference or image classification models and provide wireless connectivity. An example of a SoM board is the Coral Dev Board, available from Google, Inc. (Mountain View, CA). The Coral Dev Board includes a CPU, GPU, Edge TPU coprocessor for machine learning models, onboard memory, and wireless connectivity (e.g., Bluetooth 4.2, IEEE 802.11b / g / n / ac 2.4 / 5GHz). In some such implementations, a camera 410 can be attached to the SoM board for a compact setup.

[0115] Figure 4A An example of a camera transceiver unit (CTU) 1001 is also shown. The CTU 1001 can be considered a type of CVU 1000 without the image processor 420, but other components are substantially the same as described herein. Using the CTU 1001 can provide a more cost-effective device because image processing functionality can be offloaded to a CCU or CVU. Thus, a device can include a CCU or one or a few CVUs for image processing, with additional CTUs placed throughout the facility to capture images of entrances, exits, payment points, high-value areas, etc. Typically, the CTU 1001 and CVU 1000 can be used interchangeably in a device. Therefore, it should be understood that references to CVU include references to CTU, and references to CTU include references to CVU. Thus, the functionality of system 400 can be distributed among CCUs, CVUs, CTUs, or door managers to provide a suitable and cost-effective security solution for any particular facility. Additionally or alternatively, some or all of the image processing (or other program functions) can be performed by a remote hardware processor (e.g., in the cloud). These components can communicate via wired or wireless LAN or WAN. Many variations are expected, and the specific examples and figures described herein are intended to be illustrative rather than limiting.

[0116] In some examples, CVU 1000 or CTU 1001 may include an inertial measurement unit (IMU, e.g., an accelerometer) capable of determining whether the CVU or CTU is mounted horizontally. Viewing the images streaming from the CVU or CTU can determine that the position, orientation, and focus of camera 410 are correct. Changes in IMU readings can indicate that the CVU or CTU has been tilted or rotated after installation, allowing for corrective actions. For example, the CVU or CTU can be physically leveled. Additionally or alternatively, changes in device orientation can be corrected by compensating for angular (or rotational) changes in the image using computer vision techniques.

[0117] The anti-theft system 400 can include additional sensors 460 to provide additional or different functions. For example, additional sensors 460 can include ultrasonic sensors, time-of-flight (ToF) sensors, radar sensors (e.g., millimeter-wave radar), or lidar sensors (scanning or non-scanning). In some cases, such sensors are provided as part of a depth camera that performs both imaging and proximity or ranging functions. The camera 410 of the CVU or CTU can include a depth camera or a pair of stereo cameras for depth functions. Sensor 460 can be used to provide distance data from the sensor to the cart (or the goods in the cart). A further description of using such additional sensors 460 to provide three-dimensional (3D) imaging of the shopping basket 205 or the goods is provided below.

[0118] Figure 4B Another implementation of the anti-theft system 400 is illustrated schematically. References have been made. Figure 4A Many components of this implementation of system 400 are described. This implementation uses a wireless cellular gateway for bidirectional communication between CVU 1000 and cloud platform 470. Cloud platform 470 can be located remotely from the facility where CVU 1000 is located. Cloud platform 470 can process images obtained from CVUs at multiple retail facilities.

[0119] As described above, the CVU acquires an image of the shopping basket 205 within the field of view of its camera 410. The CVU's processor 420 is capable of executing a machine learning or computer vision object detection model to determine the loading status (e.g., empty, at least partially loaded, or full) of a shopping basket attempting to leave the store, and is capable of changing the door lock status based on the detection that a cart at least partially loaded is attempting to leave the retail store without paying. For example, the door lock status can be changed to activate the wheel locking mechanism 380 of the smart locking wheels 215 of a shopping cart detected as having unpaid items.

[0120] The CVU is capable of collecting and storing images of the shopping basket locally and transmitting these images via cloud platform 470 for storage and analysis. The CVU and cloud platform 470 can communicate via an autonomous WAN gateway 465 using, for example, cellular data services such as General Packet Radio Service (GPRS), Long Term Evolution (LTE), or 5G New Radio (5GNR). Gateway 465 can provide wired or wireless network access to cloud platform 470 and can be a Virtual Private Network (VPN) on a municipal wireless (e.g., WiFi) network.

[0121] The cloud platform 470 can include processors and memory for storing and analyzing images collected by the CVU. For example, at box 472, the image set can be labeled to provide training data for updating the machine learning or computer vision object detection model used by the CVU. At box 474, the labeled image data can be used to update or generate a new object detection model. The updated or new model can be transmitted back to the CVU via WAN link 465.

[0122] In some implementations, cloud platform 470 can provide real-time event detection or real-time streaming 476, where event logs (e.g., a database of images of successfully or unsuccessfully identified theft incidents) can be viewed and analyzed for troubleshooting or to improve the performance of system 400. Cloud platform 470 can provide dashboards (e.g., accessible via the Internet), where authorized retail facility managers or system administrators can view event logs, access data tags or training modules 472, 474, perform system maintenance or upgrades, etc.

[0123] refer to Figures 8 to 11 Additional workflows and processing pipelines that can be executed (at least in part) by the cloud platform 470 are described.

[0124] Image processing technology examples for anti-theft systems

[0125] For reference Figure 4A and Figure 4B The CVU 1000 of the anti-theft system 400 is capable of imaging the monitored area 440 and acquiring an image set 430 of exit events. Exit events can include shopping baskets 205 leaving the store through an exit. In many retail stores, the exit is also the entrance through which shoppers can enter the store with shopping baskets; in this implementation, exit events can include shopping baskets entering or leaving the store (because the camera 410 can typically image the entire exit / entrance area and capture shoppers entering or leaving).

[0126] Image set 430 is passed to image processor 420, which is capable of applying computer vision, machine learning, or object recognition techniques (described herein) to image set 430 to perform some or all of the following various image recognition tasks in various implementations.

[0127] Image processor 420 is capable of classifying objects in an image set into one of the following categories (any of which can be referred to as the loading state of a basket): (a) a shopping basket containing goods; (b) a shopping basket not containing goods (e.g., the shopping basket is not necessarily empty; for example, a shopping cart 1620 with an open child seat may still contain a child, a handbag, etc.); and (c) other objects besides the shopping basket (e.g., a shopper). The loading state can represent a range of values ​​associated with the amount of goods loaded in the shopping basket. For example, the range can be a number (e.g., 1 to 5, where 1 is empty and 5 is full), a rating (e.g., A to E, where A represents full and E represents empty), or some other type of score, discriminant, or semantic classifier, or a probability scale of multiple loading levels (e.g., full, 3 / 4 full, 1 / 2 full, 1 / 4 full, or empty). The loading state can include a weighted score or value indicating the amount of goods loaded, as well as an estimate of the value of the load (e.g., whether the load includes high-value items). For example, a basket partially loaded with high-value items (e.g., wine bottles) may be considered more loaded than a basket fully loaded with bulky, inexpensive items (e.g., tissues), because the partially loaded basket results in a greater financial loss for the store.

[0128] Loading status can be determined using computer vision or machine learning techniques as described herein. In some implementations, loading status can be weighted to reflect the presence of high-value items in the basket (which tends to increase loading status) or the absence of high-value items in the basket (which tends to decrease loading status). For example, computer vision or machine learning techniques can be trained to identify the presence of high-value items (e.g., wine bottles) in the basket, and if present, the loading status increases because the presence of high-value items generally indicates a higher overall value in the basket. As another example, loading status can represent the presence of high-value items compared to the presence of other types of goods (e.g., low-value goods), as identifying the highest monetary value of the goods in theft scenarios may be advantageous.

[0129] Image processor 420 can distinguish between different types or sizes of shopping carts 30, whether the cart is manually propelled or motorized (e.g., a motorized cart with a shopping basket 205), and whether the object is a shopper carrying a handheld shopping basket 205. In some implementations, image processor 420 may not perform facial recognition (or attempt to identify personally identifiable features or information) on individuals in the image to protect their privacy. Image analysis performed by processor 420 (or cloud platform 470) can be configured to fully comply with data privacy laws and regulations (e.g., the California Consumer Privacy Act or the EU's General Data Protection Regulation (GDPR)).

[0130] The image processor 420 is able to distinguish shopping carts pushed (or carried) by store employees rather than shoppers (e.g., by recognizing that the person is wearing a store uniform). This can be effective in anti-theft logic because the likelihood of theft is greatly reduced if a store employee is pushing (or carrying) a (full) loaded basket out of the exit.

[0131] Image processor 420 is capable of determining the path of an object (e.g., changes in location over time) within the time period covered by the image set. For example, when referencing... Figure 5 Further described, the CVU 1000 is capable of determining the coordinates of an object over time (e.g., Cartesian x, y coordinates) and calculating the object's path (e.g., see Figure 6). When the shopping basket 205 is outside the FOV of the CVU's camera 410, the image processor 420 can analyze the images collected by the secondary camera 410a.

[0132] If the determined path of the shopping basket containing the goods indicates that the basket is heading towards or passing through an exit, the anti-theft system 400 can transmit an anti-theft signal to the shopping basket. As mentioned above, the anti-theft signal can include instructions to lock the smart wheel, activate an alarm (audible or visible), notify store personnel, activate the store's video surveillance system, etc.

[0133] In some implementations, after objects in image set 430 have been classified as objects of interest (e.g., a shopping basket containing goods), the actions of the anti-theft system 400 (e.g., how to convey anti-theft commands) can depend on the CVU1000 (or door manager 450) and the shopping basket (e.g., Figure 3The communication type between the smart positioning system 210 or the smart locking wheel 215 shown. For example, this action can depend on whether the system 400 attempts to use unicast or multicast addressing to the shopping basket. Examples of unicast and multicast command transmission techniques to the smart wheel 215 or the smart positioning system 210 are described in U.S. Patent No. 9,963,162, Cart Monitoring System Supporting Unicast and Multicast Command Transmissions to Wheel Assemblies, the entire disclosure of which is incorporated herein by reference.

[0134] Unicast addressing

[0135] In unicast addressing, commands from the anti-theft system 400 can be addressed to a specific shopping basket that has a specific unicast address on the store's tracking network. The CVU 1000 can use various techniques to associate shopping basket 205 with a specific unicast address.

[0136] For example, each shopping basket can be encoded with its unicast address via an optically readable tag, such as a barcode, Aruco tag, etc., placed on the shopping basket (or cart). The optically readable tag can encode the unicast address so that the camera 410 can detect it in the visible or infrared (IR) spectrum (e.g., IR tags are less distracting to shoppers and are less likely to be damaged by potential thieves because they may be invisible to the human eye). The image processor 420 is capable of detecting and decoding the unicast address of the shopping basket from an image including the optically readable tag.

[0137] Additional or alternative technologies can be used to associate an identified shopping basket with its specific unicast address. For example, a retail facility may include radio positioning infrastructure capable of identifying RF transmissions as originating from a specific unicast address (e.g., because the RF transmission itself includes the unicast address). The radio positioning infrastructure is able to detect RF transmissions from the basket (identified as objects of interest by the CVU), thus enabling system 400 to establish an association between the basket and its unicast address.

[0138] The radio positioning infrastructure can include a triangulation system that gives the basket's location (e.g., x, y coordinates) at a given time. The triangulation system can include a system with an RF receiver that measures the angle of arrival (AHA) of the RF signal emitted from an RF tag on the basket to estimate the location (e.g., a smart positioning system available from Quuppa LLC (Arlington, VA)). The CVU can detect the basket's location or path in the AHA system, and using the knowledge of the RF receiver's location (which does not need to be co-located with the CVU), system 400 can correlate the AHA-estimated location or path with the CVU-estimated location or path to infer the basket's unicast address. Therefore, system 400 can be integrated with the facility's existing location-based services or real-time tracking systems.

[0139] Radio positioning infrastructure can include systems that measure the radial distance to a basket, for example, via time-of-flight or phase unwrapping, followed by phase slope or impulse response methods. System 400 can correlate the optical path or position measured by the CVU with the change in radial distance over time (from the radial distance measurement node) to infer the unicast address of the basket.

[0140] In some devices, the shopping basket has an internal mechanism for measuring its own movement (e.g., a dead reckoning navigation system described in U.S. Patent No. 9,731,744, incorporated herein by reference, such as Intelligent Navigation System 210), and the self-detected movement of the basket can be correlated with the path of the basket detected by the CVU 1000. This correlation can be used to associate the unicast address of the shopping basket with the basket identified via image processing.

[0141] For example, in some implementations, a common time base exists between the anti-theft system 400 and the shopping basket, which can be updated using RF synchronization between the clock on the shopping basket and the clock of the anti-theft system 400. For a manually propelled cart, the wheel rotation count provides an approximate speed of the cart over time. System 400 can correlate the speed-over-time data with the optical path determined by the CVU to provide an association for inferring the unicast address of the cart. As another example, a basket including the smart positioning system 210 can determine the quasi-orientation of the basket relative to time (e.g., indoor geomagnetic field distortion), and this quasi-orientation can be additionally or alternatively correlated with the optical path determined by the CVU 1000 to infer the unicast address.

[0142] As another technique for associating an optically tracked shopping basket with its unicast address, the shopping basket can be configured to measure environmental features that vary along the path of the basket. This signature, varying with spatial location, may be known and can be used to correlate the basket's path (based on environmental features) with the basket's path (optically determined by the CVU) to infer the basket's unicast address. For example, environmental features may include a magnetic field in the store. A magnetic field near the wheel level can be mapped. The magnetic field map can be used to infer the unicast address of the cart wheel because the wheel can include a magnetometer to measure its local magnetic field, which can be compared to the map. The magnetic field map can be determined, and the system 400 can be trained using a cart with a known unicast address. Subsequent machine learning techniques can be applied to update the magnetic field map over time due to changes within the store (e.g., movement of a magnetic shelf).

[0143] The method of associating a unicast address with a shopping basket can begin when the basket enters the monitoring area 440, which can be done before the shopping basket has been categorized for its loading status (e.g., empty or loaded) or identified as an object of interest by the anti-theft system 400.

[0144] Regardless of the technology used, once the unicast address of the suspicious basket is known, the anti-theft system 400 (e.g., CVU 1000 or door manager 450) is able to send a potential theft message to a communication system (e.g., system 330 or 365). In the following illustrative example, the suspicious shopping basket is associated with a shopping cart (e.g., “cart 2345”) having smart locking wheels 215. The anti-theft system 400 is able to send a message to communication system 330 or 365, such as “Cart 2345, you appear to contain merchandise: if you detect a warning zone or exit signal but do not have permission to leave, issue a warning and then lock.” In this example, the message is directed to a specific suspicious shopping basket (associated with cart 2345), and if the cart attempts to leave the store (e.g., by entering a warning zone or locked zone), the smart locking wheels 215 (or smart positioning system 210) is commanded to issue a warning (e.g., in a warning zone) and then lock (if cart 2345 enters a locked zone) unless cart 2345 has permission to leave. If the shopping cart 2345 passes through the store's operating checkout lane 34, or if the goods are paid for at the mobile payment point 35, the shopping cart 2345 may have previously obtained permission to leave from the anti-theft system 400. In this case, the shopping cart 2345 is allowed to leave the store (without warning or locking) because payment has (most likely) been made for the goods in the cart's basket.

[0145] In some implementations, the suspicious shopping basket is not associated with a wheeled trolley and can be carried by hand, for example. Similar considerations apply, but potential theft alerts might include store security alarms, store surveillance system activation, etc. (because the basket does not have locking wheels).

[0146] Multicast addressing

[0147] In some implementations, the retail store may not have implemented unicast addressing, or the unicast address of a specific object of interest may be unknown (e.g., the unicast association technology described above cannot provide the unicast address of the object of interest). In such implementations or situations, the anti-theft system 400 can use non-unicast technology to convey anti-theft signals. For example, multicast addressing of shopping baskets near the monitored area 440 can be used. Multicast addressing can be state-based, where multicast messages address all shopping baskets with a specific state. For example, the state of a shopping cart wheel could be whether it is locked or unlocked; for example, the state of a shopping basket could be whether it is moving, etc. Therefore, multicast commands can be sent to all transceivers whose state is moving or unlocked, etc.

[0148] As an example, if a shopping basket containing goods is approaching an exit, and the basket is not authorized to exit (e.g., has a leave permit), the anti-theft system 400 can infer a potential exit theft. In some implementations, the exit configuration of system 400 is by default in "monitoring" mode, where an image of the exit is obtained by CVU 1000, and the smart wheels 215 are not locked if they cross the exit. In this example, because a potential exit theft has been identified, the anti-theft system 400 (e.g., CVU 1000 or door manager 450) can switch the exit configuration from monitor mode to "lock without permission" mode just before the suspicious shopping basket is about to leave through the exit. The "lock without permission" command can be multicast to all transceivers near the monitored area, and if the suspicious shopping cart does not have a leave permit, its smart wheels 215 will be locked to prevent theft. After the wheels are locked, system 400 can switch the exit configuration back to monitor mode.

[0149] Shopping basket tracking path

[0150] Figure 5 The side view (left) and top view (right) of the camera 410 of the computer vision unit 1000, positioned to determine the location of the shopping basket 205, are schematically shown. In this example, the shopping basket 205 is part of the shopping cart 30, but this is for illustration only and does not limit the tracking capability. Tracking can be performed in a coordinate system, such as... Figure 5The position or path of basket 205 (e.g., position change over time) is determined in the Cartesian x, y, z coordinate system 510 shown. In some implementations, only two horizontal coordinates (e.g., x and y) are tracked (e.g., horizontal movement), because cart movement typically occurs on a horizontal plane (e.g., at a constant height z).

[0151] The position of basket 205 can be represented as the center of the basket as measured in image coordinates. In short, some implementations project the known position and optical field of view (FOV) of camera 410 onto a plane at a height h determined by the type of basket being tracked (e.g., the height h of a full shopping cart differs from that of an empty shopping cart; if a given device includes multiple types, the height of different sizes / models of shopping carts may differ; the height of motorized carts differs; the height of hand-held baskets differs).

[0152] Figure 5 The geometry of the imaging environment is shown. Camera 410 is positioned at a height h0 above the floor (in some cases, the camera is mounted on the ceiling of the facility). The camera has a vertical FOV (vfov) centered at an angle to the vertical line, and a horizontal FOV (hfov) centered at an angle to the y-direction. The center pixel of the image of basket 205 can be measured by image processor 420 at angles h0 and h0. The center pixel of basket 205 does not necessarily need to be in the center of the camera image (even if camera 410 is maneuverable). By measuring angles h0 and h0, the image is processed using... Figure 5 Given the geometry shown, system 400 (e.g., CVU 1000) is able to convert angle measurements into position coordinates (e.g., x, y, and (optionally) z).

[0153] Figure 5 An example scenario with one camera 410 is shown. In other embodiments, multiple cameras 410, 410a (e.g., 2, 3, 4, 5, 6 or more) can be used to image the monitored area 440. Figure 6A , Figure 6B and Figure 6C Some example arrangements of CVUs are shown. As mentioned above, CTUs can generally be used interchangeably with CVUs to capture images of the monitored area, and in other implementations, one, some, or all of the CVUs shown in the figure can be replaced by CTUs.

[0154] exist Figure 6AIn this example, three CVUs, 1000a, 1000b, and 1000c, are positioned to image a monitored area 440 at the store entrance / exit. In this example, an (optional) VLF line is placed at the store entrance / exit. In other configurations, additionally or alternatively, RF warnings or locking field settings can be generated via RF antennas located near the store entrance / exit or checkout gates, or EAS towers or other anti-shop-lifting devices can be positioned near the exit. CVU 1000a is positioned away from and towards the entrance / exit to capture images of shopping baskets entering or leaving. CVUs 1000b and 1000c are positioned on either side of the entrance / exit and oriented inwards to capture images of shopping baskets moving, particularly towards the entrance / exit. The three CVUs 1000a, 1000b, and 1000c provide overlapping coverage of the monitored area (e.g., overlapping FOVs of their cameras, where FOV is schematically shown with dashed and dotted lines). In this configuration, using multiple CVUs reduces the likelihood that a single CVU will fail to capture an image of a suspicious shopping cart attempting to leave through the entrance / exit. Furthermore, a shopper pushing a shopping cart might obstruct a particular CVU's view of the shopping cart. For example, a departing shopper might block CVU 1000a's view of the cart (because the shopper is between the cart and the CVU), but CVUs 1000b and 1000c will have a good view of the cart (because the cart is located between the shopper and these CVUs).

[0155] As mentioned above, some implementations can use a secondary camera to image other areas of the store. Figure 6A An example of a secondary camera 410a-1 located near the payment point is shown, whose FOV points to the FOV of CVU 1000a and 1000c (and partially overlaps with the FOV of CVU 1000a and 1000c). Figure 6AAn example of a secondary camera 410a-2 located near the area containing high-value items is also shown, its field of view (FOV) pointing towards (and partially overlapping with) the FOVs of CVUs 1000a and 1000b. Images from camera 410a-1 can be used by the anti-theft system 400 to track the path of the shopping basket as it leaves the checkout point and moves toward the exit, and images from camera 410a-2 can be used to track the path of the shopping basket as it leaves the high-value items area and moves toward the exit. Since the basket originates from the checkout area, system 400 can use information from camera 410a-1 as an indication that the customer has paid for the items in the shopping basket. Since the basket originates from the high-value items area, system 400 can use information from camera 410a-2 as an indication that the customer's shopping basket contains high-value items. In this example, because the fields of view (FOV) of cameras 410a-1 and 410a-2 at least partially overlap with the FOVs of at least some of the other CVUs, system 400 is able to maintain continuity of basket tracking as the basket leaves the FOV of one of the secondary cameras 410a-1 and 410a-2 and enters one or more FOVs of the CVUs. This handover advantageously provides continuity of basket tracking and reduces or eliminates misidentification as the basket moves from one camera's FOV to the next. (See reference...) Figure 6C Furthermore, this handover can improve the efficiency of system 400 because path tracing is typically less demanding than load status determination, and CVU can be primarily used for load status determination rather than tracing.

[0156] Figure 6B Alternative arrangements of CVU 1000a, 1000b, and 1000c are shown. In this example, the orientation of CVU 1000a is similar to... Figure 6A As shown. However, CVUs 1000b and 1000c are oriented away from the entrance / exit and towards other areas of the store. For example, CVU 1000b faces the area containing high-value items (e.g., wine, medicine, health products, etc.), while CVU 1000c faces the payment point (e.g., checkout lane 34). In this example, the FOVs of the individual cameras of CVUs 1000a, 1000b, and 1000c do not overlap. In this configuration, CVU 1000a can be used to identify whether a shopping basket approaching the entrance / exit is already loaded. CVU 1000b can be used to identify shopping baskets that have visited the high-value area, while CVU 1000c can be used to identify shopping baskets approaching the exit from the payment point (this can indicate that payment has been made for the items in the basket).

[0157] CVUs 1000a, 1000b, and 1000c and secondary camera 410a are able to communicate with each other and share information that can help determine whether a shopping basket, at least partially loaded, is approaching an exit in an attempt to evade theft. This exchanged information helps the anti-theft system continue tracking the basket as it moves from one FOV to another, or from one secondary camera to another.

[0158] In this example, CVU 1000b can identify whether the basket contains high-value items when it leaves the high-value items area, and CVU 1000c can determine whether the basket originated from the store's checkout area. In this case, the likelihood of not paying for the items in the cart is lower, and system 400 can issue a leave permission to the basket. If the basket attempts to leave the store, and CVU 1000c has not yet identified the basket as originating from (or passing through) the checkout point, the shopper is more likely to attempt to steal high-value items, and system 400 may not issue a leave permission to the basket and will take anti-theft actions (e.g., locking the wheels).

[0159] Figure 6C Another example arrangement of a CVU 1000a and a secondary camera 410a near a store exit is shown. In this example, images from the CVU 1000a are analyzed to determine the loading status of a shopping basket near the store exit. Images from the secondary camera 410a are used to determine the path the shopping basket takes as it leaves the FOV (field of view) of the CVU 1000a (shown as a dashed line) and moves through the FOV of the camera 410a (shown as a double-dotted line). Because the FOVs at least partially overlap, the anti-theft system 400 is able to have a high degree of confidence that the shopping basket identified by the CVU 1000a is the same basket tracked by the camera 410a. Figure 6C The arrangement shown is advantageous because the image processing performed by the CVU 1000a to determine the loading status of the cart is computationally more complex and places a heavier burden on the processor than processing images from the secondary camera 410a to determine the path of the basket toward the exit. Therefore, the CVU 1000a is primarily used for determining the loading status (e.g., fully loaded, partially loaded, or empty), while the secondary camera images are used for path determination, which is not a computationally difficult task.

[0160] Note that, although in Figure 6A and Figure 6B Three CVUs are shown in the image. Figure 6COne CVU is shown, but this is only for illustration; other numbers of CVUs (e.g., 2, 4, 5, 6, or more) can be used. Furthermore, in some embodiments, only a single CVU exists (e.g., with an image processor and an RF communication node), and one or more of the other CVUs shown can be replaced by secondary cameras. CVUs can be placed at one, some, or all of the store exits, or additionally or alternatively, in other store locations (e.g., in high-value item areas, near checkout points, etc.). CTUs can replace some or all of the CVUs. Many variations in the arrangement and orientation of CVUs, CTUs, or secondary cameras are conceivable. Again, these configurations of CVUs (and cameras) are shown as examples to illustrate various anti-theft situations and store-specific requirements that can be advantageously addressed by various embodiments of the anti-theft system 400, and are not intended to be limiting.

[0161] Various factors can influence the choice of the number and arrangement of CVUs, CTUs, or secondary cameras in a retail store setup. These factors can include the FOV of the cameras, CVUs, and CTUs; the store's ceiling height (typical location for mounting CVUs, CTUs, and cameras); the typical speed at which shopping baskets move within the area imaged by these components; the distance between the checkout point or high-value item and the store exit; the need for basic continuous tracking of the shopping baskets; and the time scale (e.g., approximately 100 ms) it takes for the system 400 to identify a shopping basket as a theft risk and take anti-theft actions (e.g., locking the shopping cart wheels).

[0162] 3D imaging

[0163] A 3D image of a shopping basket can provide more information for machine learning or computer vision classifiers to analyze, and can enable anti-theft systems to classify the loading status of the basket more accurately or consistently. A 3D image can include a two-dimensional (2D) image plus distance or depth information in a plane perpendicular to the 2D image.

[0164] An empty shopping basket has a flat bottom, while a basket that is at least partially loaded will have items extending above the flat bottom. Therefore, a non-empty shopping basket has a 3D topological map that is significantly different from that of an empty basket. This topological map can be used at least partially to determine that the basket is not empty, but it can also provide information about the type of items in the basket (e.g., the topological map of baby diapers differs due to their generally cubic packaging compared to the topological map of bottled wine). Therefore, in some embodiments, the anti-theft system 400 includes sensors that provide depth information. (See reference...) Figure 4A The sensor 460 may include a depth camera, a stereo pair of cameras, an ultrasonic sensor, a time-of-flight sensor, a lidar (scanning or non-scanning), a millimeter-wave radar, etc.

[0165] For example, two cameras can be used to form a stereoscopic image of a shopping basket, and stereoscopic imaging technology can be used to derive depth information. Since cameras, CVUs, etc., are often mounted on the ceiling of retail facilities, and the ceiling height above the bottom of the shopping basket is in the range of approximately 3m to 5m, it is estimated that the pair of stereoscopic cameras can be placed approximately 20cm to 50cm apart to provide sufficient stereoscopic separation. The pair of stereoscopic cameras can be arranged inside the housing of the CVU or CTU, or the camera 410 of the CVU or CTU can be paired with a nearby secondary camera 410a to provide stereoscopic depth sensing functionality.

[0166] Time-of-flight (ToF) sensors (sometimes called time-of-flight arrays or non-scanning lidar) can be used with imaging cameras to provide 3D images of shopping baskets. A ToF sensor may include a laser for emitting light pulses and timing circuitry for measuring the time between the emission and the light signal reflected back from an object. The measured time (and speed of light) provides the distance to the object. Integrated ToF camera modules combining ToF sensors and imaging cameras are available from Analog Devices, Inc. (Norwood, MA). Analog Devices also offers the ADDI9036 CCD ToF signal processor and the AD-96TOF1-EBZ 3D ToF development platform.

[0167] In some implementations, in addition to or alternatively to an optical ToF sensor, an acoustic ultrasonic sensor can be used to provide distance / depth information. For example, an ultrasonic ranging sensor operating at a high frequency (e.g., greater than 100 kHz) can be aimed with the line of sight of an imaging camera. The ultrasonic sensor can have a relatively narrow field of view (providing sufficient resolution for objects in the basket) and a sufficiently high pulse repetition rate (providing sufficient temporal resolution due to basket movement). The high-frequency structure of the reflected ultrasonic pulses is different when scattered from the bottom of the basket (typically a metal or plastic mesh) compared to when scattered from the surface of items in a non-empty basket. The ultrasonic signal can be used to simply detect the presence of items in the basket, or (with a sufficiently narrow FOV) to identify the depth profile of items in the basket.

[0168] In some implementations, radar sensors can be used to provide depth sensing capabilities. An example of a radar sensor is the RETINA 4D image radar, available from Smart Radar System (Gyeonggi Province, South Korea), which provides 3D position and velocity information. The RETINA radar can generate 4D point clouds, which can be analyzed, for example, using neural networks, to identify objects within the point cloud.

[0169] 3D images can be used to train machine learning or computer vision models, and the added depth information (compared to 2D images) can help provide a more accurate or stable estimate of the loading status of a shopping basket. Furthermore, since different types of goods will have different depth features, machine learning or computer vision models can learn to distinguish between different types of goods and can be configured to incorporate this type of information into the loading status (e.g., a higher loading status for a basket containing wine bottles than for a basket containing produce).

[0170] Identify children in shopping carts

[0171] For reference Figure 2B The shopping cart 30 may include a child seat 1620, in which a shopper can place a child. The child can also be placed in the shopping basket 205 itself. Thieves might use a child to conceal items, distract, or disguise their intentions (e.g., parents with children are less likely to be perceived as thieves). Therefore, some implementations of the anti-theft system 400 can be configured to determine the presence of a child (including an infant) in the shopping cart.

[0172] This determination can be made by analyzing images (2D or 3D) acquired by system 400 (e.g., from a CVU, CTU, or secondary camera). Items placed in the stroller (in the shopping basket 205 or child seat 1620) tend not to move relative to the stroller, while children do tend to move relative to the stroller (e.g., moving their head, arms, or legs, rocking their bodies, etc.). Therefore, the discriminant of whether an object in the stroller is a child is whether the object is moving relative to the stroller. The temporal sequence (2D or 3D) of the images can be analyzed to determine which objects, if any, are moving relative to the stroller itself through translational or rotational motion. For example, the position of an object relative to a fixed location in the stroller (e.g., relative to the handle 1610, relative to the side wall of the basket, etc.) can be compared over the duration of the time sequence to distinguish whether an object is moving relative to the stroller and thus whether the object is likely a child. In the case of 3D images, depth information can provide a discriminant for a child because a child's depth features (head, arms, torso, legs) differ from those of typical retail store merchandise.

[0173] Shopping basket path examples and potential theft

[0174] Figure 7An example of a path taken by a shopping basket near the entrance / exit of a retail store is illustrated schematically. Empty shopping baskets shown are not shaded, while at least partially loaded baskets are shaded. In this illustrative example, the shopping basket is part of shopping carts 30a-30e, but this is for illustration only and not a limitation. Symbols on the path taken by the shopping basket (in this example, a shopping cart) indicate the possibility of theft by exit, as will be further described below. The path taken by shopping carts 30a-30e can be determined by the CVU as described above. For example, Figures 6A-6C The CVU configuration shown can be adapted to image store entrances / exits, as well as areas where high-value items are located and payment points are located.

[0175] exist Figure 7 The paths of shopping carts 30a-30e are annotated with symbols that graphically represent attributes and theft potential. Circles are used for entering shopping carts, hollow circles indicate empty shopping carts, and solid circles indicate shopping carts that are at least partially loaded. Squares are used to indicate leaving shopping carts, hollow squares indicate empty shopping carts, and solid squares indicate shopping carts that are at least partially loaded. A solid quadrilateral star indicates a leaving cart with a high probability of theft.

[0176] Stroller 30a is entering the store and is determined to be unloaded. The path of stroller 30a is marked with a hollow circle. Stroller 30b also enters the store, and the CVU determines that the stroller is at least partially loaded. This could be due to a child or handbag being in an open child seat, or due to the presence of items stored outside the store that the shopper has placed in their shopping basket. Strollers 30d and 30e are leaving the store. Stroller 30e is unloaded and there is no possibility of theft.

[0177] The shopping cart 30d is loaded, but because it is approaching the exit from the direction of the checkout point, there is no possibility that the cart 30d will be stolen. To further confirm the payment status of the shopping cart 30d, the anti-theft system 400 can query the checkout point (e.g., cash register 34 or mobile payment point 35) to determine whether the shopping cart 30d has actually passed through an operating checkout lane (e.g., spent sufficient time in the lane to indicate payment) or paid for the items in the cart basket using a mobile payment application. If so, the system 400 can issue a permission for the cart 30d to leave. In an implementation using a checkout gate (CB), the command could be to unlock the CB to allow departure. If there is no further payment indication, the cart 30d may attempt to simulate payment by passing through from the direction of the checkout point, or quickly pass through the checkout point without spending sufficient time to indicate that payment has occurred, and the system 400 can issue an anti-theft command to the transceiver of the cart 30d. Assuming the cart 30d is coming from the direction of the checkout point, this command could be a warning command (rather than a locking command). In implementations using a checkout barrier (CB), the command could be to keep the CB locked to prevent the cart 30d from leaving. The anti-theft system 40 could issue a command to store staff to reach the associated CB or exit to determine if the shopper has actually paid for the goods.

[0178] The system determines that cart 30c is already loaded and is approaching the exit from the direction of the high-value items rather than the checkout point. Cart 30c represents an increased likelihood of theft and is marked with a solid star. The anti-theft system 400 can query the checkout point to determine if cart 30c previously passed through it. If so, cart 30c may have subsequently entered the high-value items area and placed high-value items in the shopping cart basket without payment. The system 400 may refuse to issue permission for cart 30c to leave (or refuse to open the checkout gate), instead issuing a locking command to shopping cart 30d or alerting store staff to attempt to prevent potential theft of goods from the store.

[0179] The foregoing example is intended to illustrate, rather than limit, the logic that can be performed by the implementation of the anti-theft system 400 using the image processing capabilities of the CVU.

[0180] although Figure 7 The symbols shown are generally intended for description, but in some implementations, the images (e.g., often videos) acquired by the CVU can be represented using symbols similar to... Figure 7 The indicated markings (or annotations) are overlaid. For example, it is possible to process (e.g., via image processor 420 or remote processing node) video acquired by the CVU to display the cart path. It is possible to use different colors or symbols to distinguish the paths of carts entering or leaving, or carts with different theft probabilities (e.g., similar to...). Figure 7(The symbols shown). Authorized store personnel have access to the annotated images to view the exit event in which the rollout occurs.

[0181] Examples of machine learning techniques used in anti-theft systems

[0182] In some implementations, the CVU 1000 (e.g., image processor 420) implements one or more object recognizers that can capture received data (e.g., image acquisition) and identify or draw points, label images, and attach semantic information to objects (e.g., shopping cart, motorized trolley, shopping basket, empty, loaded, etc.).

[0183] Object recognizers can identify shopping baskets, shopping carts, motorized shopping carts, or mobile trolleys; items and merchandise within the shopping basket (which can be included in the lower basket at the bottom of the shopping cart); the presence of objects in the child seat of the cart (e.g., a child or a handbag); markers associated with the user pushing or operating the cart (e.g., a specific style of shirt (e.g., plaid or striped), specific trousers, skirt, jacket, or hat); user characteristics (e.g., facial features, body features); and so on. Object recognizers can identify store personnel, for example, by recognizing that the person is wearing a store uniform, identifying tags, etc. In some implementations, for privacy purposes, object recognizers do not recognize facial or body features.

[0184] Object recognizers can identify entrances / exits, checkout lanes, or other objects in a store. One or more object recognizers can specifically identify objects with certain characteristics. For example, one object recognizer can be used to identify a shopping basket, while another can be used to identify items or merchandise within the basket, and yet another can be used to identify characteristics of the user associated with the shopping basket, etc.

[0185] By analyzing multiple sequential images (e.g., frames from a video), an object recognizer can determine the path of a shopping cart as it enters or leaves the store. In some cases, the object recognizer may classify shopping cart behavior, such as entering or exiting, rather than determining the path (or something other than determining the path).

[0186] Object recognition can be performed using a variety of computer vision techniques. One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include: Scale Invariant Feature Transform (SIFT), Speed-Up Robustness (SURF), Oriented Fast and Rotated BRIEF (ORB), Binary Robustness Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenface method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean Shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) technique, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), Bundle adjustment, Adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), Feature Point Histogram, various machine learning algorithms (e.g., Support Vector Machine, Associative Vector Machine, k-Nearest Neighbor Clustering, Naive Bayes, Neural Networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and so on.

[0187] Object recognition can be performed additionally or alternatively using various machine learning algorithms. After training, the machine learning algorithms can be stored by a CVU 1000 (e.g., a graphics processor 420). Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression, logistic regression (e.g., Logit model)), instance-based algorithms (e.g., learned vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., Naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., prior algorithms), artificial neural network algorithms (e.g., perceptron), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), ensemble algorithms (e.g., stacked generalization), and / or other machine learning algorithms.

[0188] Machine learning models can include neural networks, such as convolutional neural networks, recurrent or recurrent neural networks, stacked autoencoders, etc. Neural networks can include deep neural networks with many layers (e.g., more than 3, 5, 10, or more layers). Neural networks can include convolutional layers, pooling layers, fully connected layers, classifier layers (e.g., softmax), activation functions (e.g., rectified linear units), lossy layers, etc. Weights in neural networks can be learned using supervised or unsupervised learning techniques.

[0189] Individual machine learning models can be customized for specific applications or installations. For example, the CVU 1000 can store a default model for analyzing images of shopping baskets. This default model can be used as a starting point for generating additional models specific to the conditions at the CVU installation location. For instance, when installed in a specific store with a particular exit, the object recognizer can learn specific features of the exit the CVU is monitoring, and specific features of the shopping baskets, shopping carts, etc., used in that particular retail store. The CVU can update its computer vision, machine learning, or object recognition algorithms using supervised or unsupervised training techniques applied to images acquired after installation. Therefore, the default computer vision, machine learning, or object recognizer can be specific to the specific environment of the images it analyzes. The computer vision, machine learning, or object recognizer can continue to learn over time and become more efficient and accurate in its object recognition tasks.

[0190] In some implementations, machine learning techniques can use TensorFlow. TM Alternatively, use TensorFlow Lite (available at www.tensorflow.org) or Torch. TM It can be obtained from torch.ch, or implemented in Python, PyTorch. TM Each of these (available at pytorch.org) provides an open-source software library for programming machine learning applications, such as image recognition using neural networks. Neural networks can include convolutional neural networks (CNNs) and can be deep networks (e.g., artificial neural networks comprising 3 or more layers, where each layer is trained to extract one or more features from an image). In some implementations, convolutional layers can be followed by one or more fully connected (layers), support vector machines (SVMs), softmax layers, or other types of classification layers. For example, the output of a neural network could be a classification, such as the loading status of a shopping basket. As described in this paper, the loading status could indicate whether the basket is loaded or unloaded, or empty, partially loaded or full, a score, a numerical range, etc. As another example, the output of a neural network could be a classification: the shopping basket is entering the store (e.g., this does not indicate a potential theft) or leaving the store (e.g., this indicates a potential theft).

[0191] Machine learning techniques can be trained using supervised or unsupervised learning methods. For example, training images of shopping baskets entering or leaving a store exit can be obtained (e.g., from a CVU, CTU, or store surveillance system) and classified according to their loading status, such as empty, partially loaded, or fully loaded. In some cases, the training images are segmented to include the front of the shopping basket but exclude the area of ​​the child seat 1620 (see...). Figure 2BThis advantageously reduces training or classification problems because machine learning techniques do not need to process images of children (or handbags or other non-commodity items) that may be included in the child seat 1620.

[0192] Semantic information, such as the type of shopping basket (e.g., manual cart, motorized cart, hand basket), whether the basket enters or leaves the store, and whether there are high-value items in the basket, can also be categorized in the training images. In some implementations, the training images are manually annotated / classified. This training data can then be used by machine learning techniques (e.g., convolutional neural networks with fully connected layer classifiers) to learn how to analyze and classify new images. Training can be performed using a reference... Figure 4B The cloud platform 470 described is used for execution.

[0193] During the use of the anti-theft system 400, images of the shopping cart can be continuously acquired and categorized / annotated, and these images can be used to further train or update machine learning techniques. Images of the shopping cart's path can be examined or analyzed (e.g., see...). Figure 7 This improves the ambiguous classification or identification of loaded or unloaded carts. Cart location or path data obtained using, for example, RF technology can be compared with location or path data determined by image processing techniques to improve the system's location or path determination (e.g., using machine learning training methods).

[0194] Therefore, machine learning techniques can be specialized over time for the actual retail store environment where the anti-theft system 400 is installed, which can advantageously achieve improved accuracy, efficiency, or improved performance in identifying potential theft situations.

[0195] Example of a processing pipeline and workflow for an anti-theft system

[0196] Figure 8 An example of a processing pipeline for training machine learning (ML) models is shown. This pipeline can be configured, for example, by referring to Figure 1. Figure 4A and Figure 4BThe described anti-theft system 400 is executed by a cloud platform 470. The pipeline receives image data collected from multiple CVUs, CTUs, secondary cameras, store surveillance systems, etc. The image data can originate from a single retail facility or multiple retail facilities. The pipeline can be accessed or controlled via an ML training application (app) 800, which can be a web-based interface to the cloud platform 470. For example, a system administrator can use the ML training app to begin training an ML model, look up information about the status of CVUs (e.g., from a database 810 of CVU information), and communicate updated ML models to specific CVUs or CVUs. Because the received image data can come from one or more specific stores, the trained ML model can be customized for that specific store(s). For example, some stores may use shopping baskets with plastic mesh (instead of wire mesh), and the ML model can be trained on images from those stores to not only better recognize baskets with plastic mesh structures but also better identify store merchandise placed in such baskets.

[0197] exist Figure 8 In the pipeline shown, at point 1, the image dataset is uploaded from one or more CVUs to the cloud platform 470 via WAN gateway 465. At point 2, the platform is notified of the receipt of the new image dataset. At point 3, the image dataset is prepared and submitted for data labeling (which can be performed by a human classifier). For example, it can identify shopping baskets in the image and classify them based on their loading status. At point 4, the labeled imaging dataset can be used to train an ML model. The training can be used to generate new ML models or update existing ML models.

[0198] At point 5 of the pipeline, MLapp 800 can be used to select a CVU with labeled data for training or to provide other control commands to the cloud platform 470. At point 6, the ML training instance can be initiated and executed by the computing engine in the cloud platform 470. The computing engine can train new or updated ML models, and at point 7, the trained model can be saved. At point 8, the CVU can be notified that a new or updated ML model is available, and at point 9 of the pipeline, the trained model can be downloaded via the CVU.

[0199] Figure 9 An example of a processing pipeline for analyzing images obtained from an anti-theft system is illustrated schematically. The pipeline can be configured from a reference... Figure 4BThe described cloud platform 470 performs the operation. The pipeline receives image data collected from multiple CVUs, CTUs, secondary cameras, store surveillance systems, etc. The image data can originate from a single retail facility or multiple retail facilities. As described above, the image data generally includes video footage of potential theft incidents in the retail store. At points 1a and 1b of the pipeline, image data from CVUs, CTUs, secondary cameras, or other store surveillance systems is periodically uploaded to the cloud platform 470, for example, via a WAN gateway 465, and stored in cloud storage. At point 1c, loading status detection events determined by the CVUs can be additionally or alternatively uploaded to the cloud platform 470. Loading status detection events can include a timestamp of the event and a determined loading status of the shopping basket involved in the event (e.g., empty, partially loaded, full, etc.). The image data can include annotations, such as the area around the shopping basket or cart involved in the event, the area around the merchandise in the cart, the bounding box around the customer using the shopping basket, etc. At point 2, a new upload can be notified to the cloud platform 470.

[0200] At points 3a and 3b in the pipeline, cloud platform 470 can analyze image data to determine, for example, image metadata and store the metadata in cloud database 810 (e.g., a Structured Query Language (SQL) database). The metadata can include inferred metadata determined from the CVU detection event. Inferred metadata can include, for example, the position of the shopping basket in the image frame, the time of the event, how long it takes to perform the loading status determination, the confidence value associated with the ML model's confidence that the loading status can be correctly inferred from the image data, the ML model weights and parameters used in the image analysis, etc. The metadata can also include image metadata that correlates image data from the CTU, secondary cameras, or store surveillance video (which may not be relevant to the loading status determination) that leads to the theft event with image data from the CVU (which will be associated with the loading status determination). Image metadata can include information about the correlation between image data from the CTU, secondary cameras, or store surveillance video and image data from the CVU.

[0201] Figure 10 An example of a processing pipeline for real-time event detection or live streams from an anti-theft system is illustrated schematically. The processing pipeline can be used to provide a reference. Figure 4B The described real-time event detection or live stream 476. The processing pipeline can use the reference... Figure 8 The MLapp 800 is described for management. Because real-time streaming of image data can utilize a significant amount of bandwidth from the WAN gateway 465, the live streaming function is only activated when needed, for example, by a system administrator or authorized field service personnel or retail store staff for the diagnosis, testing, installation, or maintenance of the security system 400.

[0202] At point 1 of the pipeline, live streaming detection is enabled, and at point 2, the live streaming server is started. At point 3, image data from CVU, CTU, secondary cameras, or store surveillance systems is streamed to cloud platform 470 via WAN gateway 465. MLapp 800 can include a streaming player 840 capable of displaying the streaming image data. In various implementations, WAN gateway 465 can implement one or more streaming protocols, such as Real-Time Streaming Protocol (RTSP), HTTP Live Streaming (HLS), or Real-Time Messaging Protocol (RTMP).

[0203] Figure 11 An example of a pipeline for business intelligence (BI) analysis of image data from an anti-theft system 400 is illustrated schematically. At point 1 of the pipeline, an ETL (Extract, Transform, Load) process can be used to access database 810 and run the necessary BI queries on the image data and metadata stored by cloud platform 470. At point 2 of the pipeline, the image data can be downloaded from the cloud platform to a streaming viewer 840, which can be a component of an MLapp 800 for interacting with cloud platform 470. As described above, cloud platform 470 can store image data (e.g., video) of possible push-out theft incidents in a retail store. The image data can include video 852 of push-out theft incidents obtained from CTU, secondary cameras, or store surveillance video (which may not be related to loading status determination) and video 854 of push-out theft incidents obtained from CVU (which will be associated with loading status determination, such as whether the shopping basket is empty, partially loaded, or full). Authorized store personnel or system administrators can view the push-out theft videos, run analyses on the image data and metadata, etc. For example, as Figure 11 As shown in the example, surveillance video 852 shows that a push-out theft is occurring, and the associated CVU video of the event marks the shopping basket as full. This example represents a push-out theft detection successfully performed by the anti-theft system (by...). Figure 11 (The checkmark in the text indicates this). BI queries can include information about system-generated false positives (e.g., innocent incidents incorrectly identified as theft) or false negatives (e.g., incorrectly identified pushback thefts), statistics on frequency and timing, or estimated loss amounts from pushback thefts, etc.

[0204] Reference Figures 8 to 11 The example pipelines described are intended to be illustrative rather than exclusive. The cloud platform 470 can be configured to perform one, some, or all of the functions of these pipelines in various implementations.

[0205] Figure 12An example of a processing pipeline in a CVU 1000 is illustrated schematically. The processing pipeline is at least partially executed by the CVU's processor 420. Many of the functions of the CVU 1000 have already been described above and will not be repeated here. In this example pipeline, the CVU is configured to provide a loading state for a shopping cart 30, which includes a shopping basket 205 for storing goods. In this example, for illustrative purposes, the loading state is either empty or full, but as described herein, a more general loading state can be provided. The processor 420 is programmed to execute a classification engine 1202 and a detection engine 1204.

[0206] At point 1a of the pipeline, image data (e.g., video of a monitored area of ​​a store) is sent to classification engine 1202, which executes an ML cart classification model. The classification model can be trained to recognize the presence (or absence) of carts in images and their locations within the images. The cart classification model can segment images to identify the shopping basket portion of a cart. Since images typically include things other than carts, the classification model can also classify images based on whether they contain people (e.g., shoppers pushing carts, store staff, or children in carts) or other living objects (e.g., service animals). At point 2 of the pipeline, the classified images (along with the classification metadata determined by the classification engine) can be stored in the cart image dataset. Classified images can be annotated with bounding boxes around the objects classified in the images (e.g., carts, baskets, shoppers, etc.). In some cases, images that do not contain shopping carts are not stored, which helps reduce memory usage.

[0207] At point 1b of the pipeline, image data is transmitted to detection engine 1204, which executes an ML detection model to determine the loading status of the cart (or basket). The ML classification model can be different from the ML detection model, which advantageously allows each of these ML models to be trained for its respective task. In other implementations, the ML classification model and the ML detection model are part of the same ML model, which advantageously allows for integrated training of the ML models due to the overlap between the classification and detection tasks. In some implementations, classification engine 1202 executes before detection engine 1204. If classification engine 1202 determines that there is no cart in the image, detection engine 1204 can be omitted, which advantageously saves power and processing cycles and improves efficiency. In other implementations, classification engine 1202 acts as a preprocessor and only executes detection engine 1204 if a cart is detected in the image. This also advantageously saves power and processing cycles and improves efficiency.

[0208] At point 3 in the pipeline, the loading status (e.g., "FULL" or "EMPTY" in this example) is communicated to the door manager 450 (e.g., see reference). Figure 1A , Figure 1B and Figure 4A (Described as follows) to take appropriate anti-theft actions on the cart. For example, if the detection engine determines that the cart is full, the locked state can be communicated to the door manager 450, which in turn transmits a locking command to the smart locking wheels 215 of the cart. If the detection engine determines that the cart is empty, the locked state can be disabled and communicated to the door manager 450, which may either take no action to actuate the wheel locks or transmit an unlocking command to the smart locking wheels 215 of the cart.

[0209] At point 4 of the pipeline, detection metadata (e.g., loading status) can be transmitted to the cloud platform 470.

[0210] Figure 12 The processing pipeline schematically shown can be used in various implementations of the CVU 1000, such as in Figures 8 to 11 The pipeline illustrated in the diagram is capable of interoperating with the cloud platform 470. As explained below, Figure 12 The pipeline shown can be expanded to additionally consider the categorization of items detected in shopping carts.

[0211] Image capture using an onboard camera module

[0212] Figure 13 A method for capturing, preprocessing, and uploading images of the contents of a stroller, which can be implemented by a camera module 217, is shown. For example, under firmware control, this method can be implemented by the camera module's control processor 217D. Figure 2E Control. In some embodiments, the camera module can wirelessly communicate with the wheel assembly of the cart and use information obtained from the wheel assembly to implement the method. This information may include, for example, wheel rotation data, cart position data, etc. The method can alternatively be implemented without communicating with the wheel assembly.

[0213] As shown in box 1310, camera module 217 can activate its camera / imager 217B in response to a shopping cart entering the store, and can then capture an initial image. Any items in the cart at this time are typically non-merchandise items, such as wallets, reusable shopping bags, backpacks, or children's items. In box 1320, the camera module awaits a triggering event representing a possible addition of merchandise to the cart basket. Examples of this triggering event, as previously described, include: (1) detecting an impact / vibration event with a vibration curve corresponding to the characteristics of an item addition event; (2) initiating cart movement in the merchandise area; (3) if the cart includes a scale, and the change in basket weight exceeds a threshold; and (4) if the cart includes a motion detector, detecting movement inside the basket.

[0214] In response to a triggering event, camera module 217 captures an image of the basket contents (box 1330) and compares this image with a recently captured image (box 1340). Based on this comparison or difference calculation, the method determines whether it is possible to add a new item to the basket (box 1350). If a new item is detected, the image—optionally tagged to indicate the area of ​​the newly added item—is preferably uploaded to the CVU or other node for further analysis, along with event metadata of the capture event. (If no new item is detected, the camera module may discard the image.) To reduce wireless bandwidth consumption, a simplified representation of the image, such as a compressed version of the image, may be uploaded alternatively. For example, event metadata may include one or more of the following: (1) the location of the shopping cart (e.g., store department), (2) the classification of the shock / vibration event or other triggering event that triggered the image acquisition, (3) if the shopping cart contains a scale, a weight measurement, or a weight difference relative to a previous weight measurement, and (4) wheel rotation data. Since the receiving entity knows the unique RF ID of the transmitting camera module (which may also serve as the cart ID), the receiving entity can associate the image and metadata with a specific cart.

[0215] The process can then wait for the next triggering event. Figure 13 The process shown may continue until, for example, the shopping cart passes through an operating checkout area or leaves the store. In embodiments where camera module 217 includes multiple cameras, each camera can be used to capture a corresponding image when a trigger event occurs, and one set of images can be compared with another set of images in block 1340.

[0216] Classification of stroller contents

[0217] Figure 14This illustrates a method that a CVU or other processing node can use to analyze images captured by the trolley's camera module 217 (or, if the camera module has multiple cameras, a set of two or more images). As described above, this image can be a preprocessed image labeled (e.g., with bounding boxes) to identify pixels or pixel regions representing newly detected items. Figure 14 As shown, the feature extraction module first processes the image to extract features useful for item classification or recognition. Examples of such features include one or more of the following: item color, item shape, item size, text extracted via OCR, and item texture. For at least some features, feature extraction can only be applied to the labeled image region corresponding to the newly added item.

[0218] The extracted features are then passed to one or more trained machine learning models or classifiers 1420, preferably along with associated event metadata. The trained machine learning model 1420 uses this data to determine one or more item categories for the newly added item. These categories may correspond to the labels or tags used for model training described above (e.g., "electronic items," "commodity items," "non-commodity items," "high-risk theft items," etc.). In some cases, the category may include a unique product identifier for the item.

[0219] Event metadata can be used in various ways to inform item categorization methods. For example, if the metadata specifies the store department where the item was added to the cart, ML model 1420 can suppress categories that do not correspond to that department; for example, when the item is added, the category "electronics" might be suppressed if the cart is in the clothing department.

[0220] Although the item sorting task is Figure 14 The classification is shown as being performed by one or more trained machine learning models, but classification can be performed additionally or alternatively without using a machine learning model. For example, rule-based systems or decision trees can be used to determine the item classification. In some implementations, a default classification can be specified for newly added items using store location. For example, a rule could specify that if an item is added to a shopping cart in a high-theft-risk area (e.g., the liquor department), the item should be classified as a high-theft-risk item by default, or should be classified as a high-theft-risk item unless the item can be identified as a non-high-theft-risk item from its image.

[0221] This method can also be used when the cart lacks a camera module, or when the camera module's field of view is obstructed. For example, in an implementation where the cart lacks a camera, the addition of an item to the cart can be detected via a vibration sensor in the wheel assembly (or, if the cart includes a scale, by detecting changes in the basket's weight). In response to this event, a processor in the wheel assembly or elsewhere can determine whether the cart was (or was) in a high-risk theft area when the item was added. If an item was added in a high-risk theft area, the cart's log can be updated to reflect the likelihood that the cart contained a high-risk theft item.

[0222] The task of categorizing newly added items can also take into account the categorization of other items already detected in the cart. For example, information about items or item types that are frequently purchased together (e.g., printers and ink cartridges) can be used to increase the probability of certain categories.

[0223] In some embodiments, images captured by the CVU and / or CTU can also be analyzed to classify items in the cart. This method may be similar to... Figure 14 The method shown can be used to categorize multiple items shown in a single image of the cart. Furthermore, since these images are taken from a greater distance, more detailed categorization can be used (e.g., goods versus non-goods). Further, in some cases, the categorization generated by cameras mounted on or off the cart may be a summary (multi-item) of the cart contents rather than a single item; for example, a summary categorization such as "goods," "clothing," or "beverages" could be assigned to a summary of the cart contents.

[0224] If the shopping cart eventually approaches the store exit without any payment event indication, the categorization of items collected during the shopping process can be used in various ways to determine whether to implement anti-theft measures and / or select one or more specific anti-theft measures to implement (e.g., wheel locks, activating store alarms, activating exit barriers, etc.). As a simple example, if all items added to the cart are categorized as non-merchandise items, no anti-theft action will be implemented. As another example, an anti-theft action (or a specific type of anti-theft action, such as a wheel lock) can only be implemented if at least one "high-risk for theft" item is detected in the cart.

[0225] More sophisticated methods might involve scoring the shopping cart in terms of overall theft risk. For example, a score could be generated by summing the prices of any detected high-risk items, or by counting the detected high-risk items. Furthermore, anti-theft actions could be automatically taken if a large number of units of the same item (e.g., ten or more) are detected in the cart, especially if the item is not typically purchased in such quantities. The scoring algorithm could also consider the time the cart spends in specific store areas; for example, if the cart spends a relatively long time in a high-risk merchandise area, the algorithm might increase the score to reflect the increased theft risk, even if the system does not detect any items being added to the cart in that high-risk area.

[0226] In some embodiments, multiple scores can be maintained for highlighting items during the shopping process. For example, one score could represent the probability that the cart contains items that have not yet been paid for (“unpaid” items as referred to herein), while another score could represent the probability that the cart contains items at high risk of theft. In one process, the scores can be updated in real time as events occur. For example, if the cart visits a working checkout counter, the probability that it contains unpaid items might decrease to a lower value; however, if the cart returns to the merchandise area without leaving the store, the probability might increase.

[0227] Alternatively, a separate score can be maintained to represent the likelihood that the cart contains one or more non-merchant items, such as wallets, backpacks, or reusable shopping bags. A cart determined to likely contain only non-merchant items may be considered to have a slightly higher theft risk than a truly empty cart (e.g., because the non-merchant classification of the detected items may be incorrect).

[0228] Comparative imaging of the cart contents and transaction records

[0229] In an embodiment where the system constructs a data record describing the imaged items detected in the trolley basket (e.g., the number of items detected, the product category of these items, the product ID that the items can identify, etc.), the system can compare this record with associated payment / checkout transactions to assess whether the customer has paid for all items. If no corresponding payment transaction is found, or if a significant discrepancy is detected (e.g., one or more high-priced items are detected in the trolley but are not included in the partially matching transaction record), appropriate anti-theft actions can be initiated.

[0230] Figure 17An embodiment of this method is illustrated. This method can be implemented via a CCU, CVU, CTU, and / or other processing nodes of the system as the shopping cart approaches the store exit. In block 1710, the method receives a notification that a cart containing system-categorized items is approaching the store exit. In blocks 1720 and 1730, the system retrieves the cart's path history, compares its checkout point locations to identify the payment points the cart has passed through, and looks up the most recently processed payment transactions at those payment points. In some cases, payment transactions may be anonymized and / or generalized to protect customer privacy. In block 1740, the method determines whether any of these payment transactions "matches" a record of the cart's contents. To determine if a match is found, subtle differences can be ignored. If no match is found (indicating that the customer likely did not pay for the items in the cart), the cart may be prevented from leaving the store (block 1750). If a match is found, the cart can be allowed to leave the store. Various other types of anti-theft actions can be performed additionally or alternatively. For example, if a transaction record is found to be almost a match for the contents of a shopping cart, but one or more high-value (or high-risk) items in the cart are missing from the transaction record, the system can prompt the clerk to check the high-value items in the customer's cart and receipt.

[0231] Figure 18 It shows what can be used to implement Figure 17 A set of system components for the method. The door manager 450 shown can be referenced. Figure 1A As described above. Two illustrative payment processing systems, 1810A and 1810B, communicate their respective transaction records to a payment transaction anonymization system or node, 1820. For example, one payment processing system may handle self-checkout transactions, while the other may handle recorder / cashier-based transactions. Another payment processing system may handle payments made via a customer's smartphone. Database 1830 stores the in-store coordinates of payment points available to customers. In addition to anonymizing transaction records by removing any personally identifiable information (including retail store loyalty numbers, etc.), the anonymization system preferably (1) associates each transaction record with the in-store coordinates of the payment point used, and (2) generates payment transaction metadata to facilitate the matching process. This enables the CCU (or CTU or CVU) to identify the most recent transaction record matching the path taken by the cart in the store checkout area and to perform the comparison described above. Figure 18 In the example shown, the CTU or CVU instructs the door manager to perform any anti-theft actions.

[0232] Figure 19 It shows that it can be used Figure 17 The types of data records generated and used in the method. Figure 19The top box shows an example of an anonymous and enhanced transaction record generated by the payment transaction anonymization system 1820 for a given transaction. This anonymous transaction record includes an anonymous transaction ID, a transaction timestamp, a payment point ID and associated in-store coordinates, a payment point dwell time indication, a payer risk prediction (described below), other payment event metadata, an estimated physical quantity of the paid goods, a list of high-theft-risk items paid, and a list of highly optically distinguishable items paid. The payer risk prediction may be, for example, a risk score based on the payment instrument used by the customer and / or other information about the customer (e.g., whether they used a loyalty number). The above are just examples of the types of information that may be included in an anonymous transaction record. Furthermore, in some embodiments, actual transaction records may be used for comparison instead of anonymous records.

[0233] Figure 19 The lower half of the diagram illustrates the data collected by a cart monitoring system (e.g., via a CVU or CTU) for a given shopping cart process. In this example, the data includes the cart path ID, a summary of the cart contents (as shown in the example in the lower left box), the cart's RFID ID, and the cart's path history (as shown in the example in the lower right box). In this example, the cart spent 148 seconds at self-pay station #4. Therefore, the summary of the cart's contents can be compared with recent anonymous transaction records corresponding to this self-pay station. The confidence score for a specific product category can be used to determine the weight assigned to each such product category when determining if a match exists.

[0234] Emergency Exit Usage

[0235] In some implementations, a CVU or CTU can be installed to capture images of store emergency exits, including paths leading to them. These images can be used to detect push-out theft attempts via emergency exits. The CVU or CTU can be mounted on a wall above the emergency exit or suspended from the ceiling. The images captured by this CTU or CVU can be used to classify cart contents as described above, or to associate the imaged cart or cart track with the cart ID using the association methods described above.

[0236] The images captured by this CVU or CTU can also be analyzed to detect events where a cart containing goods (or goods of a specific category) passes or begins to pass an emergency exit. When such an event is detected, the associated door manager can lock the cart's wheels and / or perform another anti-theft action. Typically, no anti-theft action is taken if the cart containing goods is only parked or moved near an emergency exit.

[0237] Further imaging features

[0238] The performance of the imaging subsystem (see discussion above) may include one or more of the following characteristics. For example, this can improve the speed, performance, and / or accuracy of the system in identifying, mapping, and / or classifying the contents of the stroller basket.

[0239] Image correction for reference geometry

[0240] Assuming the focal plane of any given camera is centered at a known xyz position (the coordinate system is arbitrary, as long as it is consistent: in this paper, positive x is assumed to be a consistent direction meaningful for shopping cart tracking, such as alignment with the checkout lane, y is horizontal and orthogonal to x, and z is locally vertical), assuming the angles between the camera's line of sight and the xyz coordinate system are known angles theta and phi, and assuming the floor of the shopping cart basket is nearly horizontal (<< 5 degrees), we can obtain the original pixel regions in the pixel xy coordinate space corresponding to all pixels included within the shopping cart basket outline, and convert those pixels into projections of the pixels onto a constant-radius sphere. In some implementations, a good sphere radius value is the distance from the center of the cart basket outline to the camera's focal plane when the camera's line of sight intersects the center of the cart basket outline.

[0241] If, for some reason, the focal plane rotates around the viewing axis, causing the pixel's x and y coordinates to be non-horizontal, then additional rotation of the pixel region may be necessary. This correction takes into account not only the geometry from the camera to the cart basket but also any refractive non-ideals in the camera lens.

[0242] The specific transform and resampling to be used depends, among other things, on the availability of computational power, the effective size of the pixels in the region of interest, and the accuracy of the known specific camera orientation. Some implementations use a bilinear (trapezoidal) transform computed at 25% and 75% of each edge. Certain variants use other algorithms (e.g., bicubic, Lanczos method).

[0243] One benefit of performing the correction is that the shopping cart basket itself and, more importantly, the contents of the basket are approximately the same regardless of the geometry of the camera to the shopping cart. This, in turn, leads to higher fidelity models in the feature space of ML (machine learning) inference applications (e.g., RCNN).

[0244] Frame-to-frame differential

[0245] Difference between image frames, where each frame is segmented to extract pixels corresponding to the area of ​​the shopping cart basket, enables the detection of relative motion of objects in the basket (e.g., a child), which can be used to influence higher-level behavior of the system. For example, if a child is temporarily detected in the shopping cart, intuitive inferences used to infer a possible theft might be adjusted. In some embodiments, color changes of corresponding pixels in consecutive frames, without altering edge positions, may represent lighting changes that can be corrected for in color space transformations (see below).

[0246] Multi-frame image fusion

[0247] Since shopping carts are generally in motion relative to cameras mounted on non-carts, multi-frame image fusion can be applied to still image sequences followed by super-resolution processing to improve image quality for the next stage of the image processing pipeline. Additional information can be collected and used to improve the accuracy of this method; for example, data on the incremental motion of the cart across a set of frames can be used to improve super-resolution processing.

[0248] Color space transformation

[0249] In some implementations, the standard RGB or YUV color space generated by most cameras may not be optimal for distinguishing different kinds of shopping cart basket contents, especially considering potentially highly variable lighting conditions (such as when one part of the shopping cart basket is under direct sunlight while another part is under artificial light or reflected sunlight). Therefore, color space transformation can be performed.

[0250] An example color space transformation is to infer a local illumination volume that varies over time at the basket surface (e.g., based on the color of background pixels near the shopping cart, such as floor pixels in the previous frame) and extract a binary or ternary color space vector for each basket pixel, where the color vector is a reradiative surface in, for example, the SV color space, which will give the observed pixel the condition given the inferred light source.

[0251] As a subset of the more general approach described in the previous paragraph, adjustments based on some average illuminance volume can capture most of the differences between day / night, sunny / cloudy days, etc.

[0252] Another example of color space transformation could be contrast expansion, which can be performed, for example, in the RGB color space or in a 1:1 linear transformation color space at the brightness of a single component. Two illustrative such color spaces are YUV or Y'UV and HSL.

[0253] Extended color space

[0254] The color space of the camera used in the system (typically employing a CMOS CCD as the underlying focal plane array) can be extended to near-infrared or near-ultraviolet light. This can be achieved in some embodiments by replacing the lens in the off-the-shelf camera with a lens lacking the color filter typically included in such cameras, allowing non-visible wavelengths to pass through. Because the imagers and other integrated circuits included in off-the-shelf cameras are typically capable of handling near-infrared and near-ultraviolet light, color space extension can be achieved without additional modifications to the camera. Color space extension can also be achieved by using non-standard Bayer masks or beam splitters and band-specific filters. For example, these non-standard Bayer masks are available from ON Semiconductor (Phoenix, AZ) for use in their other commodity imaging chips. Extending the camera's color space can improve the accuracy of the image analysis tasks described herein.

[0255] 3D images of the contents of the shopping cart basket

[0256] Some embodiments are configured to generate three-dimensional images of the shopping cart basket and / or its contents. Three-dimensional images of the shopping cart basket and / or its contents can improve the system's ability to infer the presence or absence of unpaid items. In various embodiments, the three-dimensional image will not be a volume, but a surface, where the color (possibly in the extended color space described above) is derived from a two-dimensional image of the camera, and the relative height of the surface along a vector from the camera to the shopping cart basket is determined by one of the methods described below or by another method.

[0257] There are several practical methods for generating such images. In some embodiments, sensor 460 can be used to generate images. In some embodiments, one or more of the following are used to generate images:

[0258] Stereo camera: Assuming the known geometry of the camera to the shopping cart basket, as described above, plus the known separation between the two focal planes of the stereo array, a stereo image can be calculated.

[0259] Time-of-flight array: A laser diode time-of-flight array configured in parallel with a camera, such as Analog Devices' (Norwood, MA) ADDI9036, is capable of providing three-dimensional surface maps.

[0260] Millimeter-wave (MMW) radar: Millimeter-wave radars, such as the Smart Radar System (Gyeonggi Province, South Korea) RM68-01, are capable of providing one-dimensional (range versus angle) surface maps. This surface map can be combined from multiple frames of images from a moving shopping cart to generate the desired three-dimensional surface map (this can be considered a special case of inverse synthetic aperture radar or ISAR). In some variants, two-dimensional millimeter-wave radar arrays, such as the Smart Radar System's "Retina" radar, are capable of providing a complete three-dimensional surface map in a single scan.

[0261] Ultrasound: A single-source linear element receiver ultrasonic array is capable of providing the same line cutting of shopping cart basket contents as linear millimeter-wave radars such as the aforementioned RM68-01.

[0262] Combination of imaging effects: Any subset of the various image processing techniques listed above can be combined.

[0263] Enhanced training set / model space

[0264] Special elements of training settings

[0265] In various implementations, the training set (of images) can be supplemented not only with images of ordinary carts loaded with goods and other carts without goods (e.g., empty carts), but also with images of carts containing objects known to be easily confused with goods in naive classification algorithms. Examples of such items include: wallets, backpacks, and empty shopping bags (e.g., reusable shopping bags).

[0266] Shopping cart geometry deduction

[0267] Shopping carts are three-dimensional objects. Under typical geometric constraints where cameras can be mounted, the distance from the camera to the cart is usually less than 10 meters. If the three-dimensional geometry of the shopping cart is known, then image segmentation that extracts only the cart basket can become more efficient.

[0268] In some cases, a given facility (e.g., a store) may include only a limited number (e.g., 1, 2, or 3 types) of different models of shopping carts at a given time. A shopping cart geometry model library can be populated by importing mechanical CAD models from shopping cart manufacturers and then learning over time the population of a given store. In some implementations, the library can be expanded over time by capturing image sequences of carts whose basket geometry is not recognizable. In some embodiments, the 3D cart geometry can be determined and / or inferred from an image stream via offline image processing techniques.

[0269] Improved transfer learning for different lighting conditions

[0270] As an alternative to or in combination with the color space transformation and / or the extended color space described above, the combined operation can be performed on images captured for the purpose of training the set, so as to expand the training set to include those images under different lighting conditions.

[0271] Correlation between cart image and radio frequency address

[0272] In some cases, shopping carts are instructed to stop via a wireless command that addresses an individual cart, such as through a unique MAC address (sometimes a multicast address as described above). A shopping cart is deemed to need to stop based at least in part on analysis of the footage (e.g., because there is a high probability that the cart contains unpaid items). Therefore, a cart whose position within the camera's field of view (e.g., the pixel coordinates of the cart's center) is known can be associated with a known radio frequency address.

[0273] The above describes some methods for performing this correlation, such as those discussed in the "Unicast Addressing" discourse. In some embodiments, the radio positioning system described above may include RSSI information. While RSSI is generally significantly less accurate than some of the other techniques mentioned, in certain situations, a series of short-range RSSI nodes (e.g., distributed near the camera's field of view) can provide sufficient accuracy to eliminate which cart has which RF address in the field of view. In some implementations, as an alternative or supplement to the phase slope method, the impulse response method can be used to process a series of phase difference measurements as a function of frequency. In some variations, as the shopping cart passes under or near various elements of a store's lighting structure or other structure, optical characteristics are used as an alternative or supplement to magnetic field characteristics. For RF positioning antenna arrays that are physically co-located or close to a camera suspended from the ceiling looking down, angle of arrival (AoA) processing on the radio frequency often provides a better correlation with image-based cart tracking data than phase-based ranging.

[0274] One method that can be used to identify shopping carts displayed in images captured by a CTU and / or CVU is to correlate cart motion detected in a series of images with cart speed detected by the cart's wheel assembly. (This correlation is not required for images captured by any camera module 217 mounted on the cart, as these camera modules transmit the images they capture along with a corresponding unique cart identifier.) For example, a CVU or a group of adjacent CVUs can capture a series of images showing multiple carts moving within an area of ​​a store, and the image sequence can be analyzed at multiple times when the cart is within the camera's field of view to estimate the position and orientation of each cart. Note that for shopping carts with all wheels (typically four) omnidirectional, their orientation is generally not the same as the direction of the cart's velocity vector. For such carts, the cart velocity vector associated with the image sequence can be estimated by applying a filter, such as a low-pass finite impulse response filter or a Kalman filter, to a specific reference point on the cart in each image. While capturing these images, the wheel assembly of each such cart can measure the rotational rate of its respective wheel and report it on the network (along with a unique cart / wheel ID). (Since the wheel diameter is known—typically 5 inches—the wheel rotational rate can be converted into cart speed.) By performing a best fit between the motion history from the images and the wheeled shopping cart speed versus time, the system (e.g., CVU or CCU) is able to map a specific cart displayed in a specific image to a cart ID.

[0275] In one embodiment, the trolley's wheel assembly detects the passage of magnets as the wheel rotates by using a magnetically activated switch (such as the Coto Technology (North Kingston, RI) RR122-1B52-511 Tunnel Magnetoresistive (TMR) switch) on the non-rotating portion of the wheel assembly, and by recording rotation detection timestamps. To improve the resolution of wheel rotation detection, multiple magnets (instead of a single magnet) can be mounted at equal intervals around the inner circumference of the wheel, thereby detecting rotation events more frequently. If N equally spaced magnets are used within the wheel, incremental rotation can be detected every πD / N inches. For example, for a 5-inch diameter wheel and N=3 magnets, speed can be measured every six inches of travel. As an alternative to using magnets, eddy currents induced in one or more conductive targets rotating with the wheel can be detected, for example, using an LDC0851 differential inductive switch from Texas Instruments (Dallas, TX) and a suitable inductor in the non-rotating portion of the wheel assembly.

[0276] If a given cart includes rotation sensors in both the front and rear wheels, the sensed rotation rates of these two wheels can be compared to determine if the cart is likely to turn. The steering detected in this way can be matched with steering events detected in the image sequence during the association process.

[0277] In some cases, the wheel assembly can also track and report the trolley's direction. In this case, the association process can also compare the trolley's reported direction with the trolley's direction determined from the image sequence.

[0278] like Figure 15 The method described above is illustrated. For example, this method can be implemented by a CVU, CCU, CTU, other processing nodes, or a combination of nodes. In block 1510, the method analyzes image sequences captured by one or more cameras mounted on non-carts (e.g., cameras on one or more CVUs or CTUs) and assigns optical trajectory IDs to carts detected in these images. At this point, the unique IDs of these carts (e.g., wheel assembly IDs or RF IDs) are not yet known. The term "trajectory" in this description generally refers to the detected cart path and associated data, such as image timestamps. In block 1520, the method generates a PVT (position, velocity-time) and heading time series for each optical trajectory. In block 1530, the method obtains wheel rotation detection timestamps from carts near the camera. Cart wheel assemblies can automatically / periodically or upon receiving a query for such data can report these timestamps over the network. For example, an access point (AP) near the camera can send a command instructing all wheel assemblies within a certain RSSI-based range of the access point to report their wheel rotation data for the last N seconds.

[0279] In box 1540, the method uses interpolation to convert each rotation detection timestamp sequence into a sequence corresponding to the image acquisition time interval. In box 1550, the method finds a best fit between (A) the probability of position, velocity, and orientation change for each inserted trolley and (B) the optical position, velocity, and heading of each trolley. In box 1560, if one or more matches with sufficient confidence are found, the trolley-to-track assignment table is updated. This method can be repeated continuously as image sequences and wheel rotation data are acquired. As mentioned above, the method can also be implemented without considering the trolley orientation.

[0280] Figure 16 It shows that it can be made by Figure 15The method maintains example data tables. Three tables, labeled "Trolley RFID #1," "Trolley RFID #2," and "Trolley RFID #3," contain wheel rotation timestamp data for each of the three trolleys. In this example, each such trolley reports this data for both the front and rear wheels, although this method could be implemented using data from only one wheel. Each table contains rotation timestamps for both the front and rear wheels. Because trolley RFID #2 has reported that the front wheels rotated more than the rear wheels within the same time period, the trolley has likely turned (and is therefore more likely to match a track containing a change in heading that occurred approximately at the same time as the excessive front wheel rotation). Three tables labeled "Shopping Trolley Image Tracks" contain corresponding datasets for three trolleys / paths (A, B, and C) detected in an image sequence. For example, the "Trolley Image Track A" table contains five entries, each including location data, heading data, and a timestamp for a specific trolley track. Figure 16 The table at the bottom shows Figure 15 The results of the "Best Match" search are displayed as percentage values ​​representing the probability of a match. In this example, cart RFID #1 is most likely to match image track B, while cart RFID #2 is most likely to match image track A. A lower match confidence or probability for cart RFID #2 could be used as a basis for allowing the cart to leave the store, even if it may contain unpaid items. The available data is insufficient to match cart RFID #3 to any of the three image tracks, for example, because cart RFID #3 is not within the field of view of any camera.

[0281] In embodiments where the shopping cart 30 includes a camera module 217, another method for identifying the imaged cart is to compare an image captured by the onboard camera with an image of the cart captured by a CTU or CVU. For example, when the cart is imaged by a CTU or CVU near a store exit, that image can be compared with the most recent image captured by the camera module 217 of a nearby operating shopping cart. If a matching image is found from the onboard camera module (indicating similar contents of the shopping cart), the shopping cart ID or address associated with that matching image can be associated with the image from the CTU or CVU. The task of comparing two images from different cameras is preferably to warp the images to a common reference plane based on known geometry involved (e.g., camera position and vector to the cart basket, lens transfer function to pixel address, etc.). A neural network-based comparison engine can then compare the two warped images based on features extracted from the images, for example.

[0282] In some embodiments, the cart can be marked or labeled with a barcode or other readable ID that can be detected in images captured by the CTU or CVU. In such embodiments, the above association process can be avoided, or used only when the readable ID cannot be read.

[0283] In some embodiments, an RF address is identified as a ephemeral peer device, associated with, and / or linked to another ephemeral peer device. For example, a person pushing a shopping cart may have a smartphone or other electronic device, and the system or other store infrastructure can be configured to track smartphones within the store with a degree of accuracy. By associating a given shopping cart RF address with a given smartphone ID, the tracking movement of the smartphone can be correlated with the movement of the cart. The specific tracking ID of the smartphone can depend on the protocol used by the tracking infrastructure. For example, the ID might be an IMEI when tracking via a cellular network, an IEEE 802.11 MAC ID when tracking via an 802.11 access point, a BLE advertising UUID, etc.

[0284] In some embodiments, the shopping cart can be equipped with hardware and software that enables the retail store's tracking infrastructure to track the cart. This additional hardware might be, for example, an IEEE 802.11n or 802.11ac transceiver to utilize the tracking capabilities within the store's IEEE 802.11 infrastructure. (The technical characteristics of 802.11ac, compared to 802.11n, facilitate transmitter localization.) Location tracking can also be achieved using Bluetooth. For example, Bluetooth standard version 5.1 includes angle-of-arrival radio positioning, while the proposed version 5.3 adds standard ranging capabilities. Another example is the ability to track cart location using the UWB (Ultra-Wideband) standard. As described above, hardware for tracking cart location using these and other technologies can be incorporated into the cart's wheel assembly, camera module, and / or mounted on other parts of the cart.

[0285] Other information

[0286] The various illustrative logic blocks, modules, and methods described herein can be implemented or executed by a machine, such as a computer or computer system, a processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. A processor can be a microprocessor, a controller, a microcontroller, a state machine, a graphics processor, a tensor processor, a combination thereof, etc. A processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors or processor cores, one or more graphics or stream processors, one or more microprocessors combined with a DSP, or any other such configuration. The various processors and other computing devices of the systems described herein operate collectively as a computing system, preferably under the control of executable program instructions stored in non-volatile memory devices and / or other types of non-temporary storage devices.

[0287] Furthermore, some implementations of the object positioning system disclosed herein are mathematically, computationally, or technically complex enough that they may require dedicated hardware (e.g., FPGA or ASIC) or one or more physical computing devices (using appropriate executable instructions) to perform the functions, for example, due to the amount or complexity of the computations involved (e.g., analyzing image data using computer vision or machine learning) or in order to provide results substantially in real time (e.g., determining whether a shopping basket near a store exit is at least partially loaded) (e.g., so that anti-theft actions can be performed before or simultaneously with the shopping basket near the store exit).

[0288] The blocks or states of the methods described herein can be directly contained in hardware, embodied in software modules stored in non-transitory memory and executed by a hardware processor, or embodied in a combination of both. For example, each of the above processes can also be contained in a software module (stored in non-transitory memory) executed by one or more machines, such as a computer or computer processor, and fully automated by that software module. Modules can reside in non-transitory computer-readable media such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, optical disks, memory capable of storing firmware, or any other form of computer-readable medium (e.g., storage). The computer-readable medium can be coupled to a processor such that the processor can read information from and write information to the computer-readable medium. Alternatively, the computer-readable medium can be integrated into the processor. The processor and the computer-readable medium can be in an ASIC. The computer-readable medium can include non-transitory data memory (e.g., hard disks, non-volatile memory, etc.).

[0289] Processes, methods, and systems can be implemented in a networked (or distributed) computing environment. For example, a central control unit, base station, or other components of a cart-based enclosure system can be implemented in a distributed, networked computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourcing computing networks, the Internet, and the World Wide Web. Networks can be wired or wireless, terrestrial or satellite, or any other type of communication network.

[0290] Based on the examples, certain actions, events, or functions of any process or method described herein can be performed in a different order, and can be added, combined, or omitted entirely. Therefore, in some examples or implementations, not all described actions or events are necessary for the practice of the process. Furthermore, in some examples or implementations, actions or events can be performed simultaneously rather than sequentially, for example, through multithreading, interrupt handling, or via multiple processors or processor cores. In any apparatus, system, or method, no element or action is necessary or indispensable for all examples or implementations, and the disclosed apparatus, system, and method can be arranged differently than those shown or described.

[0291] Unless otherwise expressly stated or otherwise understood in the context, the conditional language used herein, such as “can,” “may,” “may,” “for example,” etc., is generally intended to convey that some examples or implementations include certain features, elements, and / or states, while other examples or implementations do not include certain features, elements, and / or states. Therefore, such conditional language is generally not intended to imply that one or more examples or implementations require features, elements, and / or states in any way, or that one or more examples or implementations necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or states are included in or will be executed in any particular example or implementation in any particular embodiment. The terms “including,” “having,” etc., are synonyms and are used inclusively in an open-ended manner, without excluding other elements, features, actions, operations, etc. Furthermore, the term “or” is used in its inclusive meaning (rather than its exclusive meaning), and therefore, when used, for example, to connect a series of elements, the term “or” means one, some, or all of the elements in the list.

[0292] Unless otherwise specifically stated, connecting phrases such as "at least one of X, Y, and Z" are generally understood according to the context in which they are used, and the terms used to express clauses, terms, etc., may be X, Y, or Z. Therefore, such connecting phrases are generally not intended to imply that some example or implementation requires the presence of each of at least one of X, at least one of Y, and at least one of Z. In this specification and the appended claims, the articles "a" or "the" refer to one or more elements when referring to an element, unless the context clearly indicates otherwise.

[0293] While the above detailed description has shown, described, and pointed out novel features applicable to various examples or implementations, it will be understood that various omissions, substitutions, and changes can be made to the form and details of the illustrated logic blocks, modules, and processes without departing from the spirit of this disclosure. As will be appreciated, certain examples of the invention described herein can be implemented without providing all the features and benefits set forth herein, as some features can be used or implemented separately from other features.

Claims

1. A system for reducing push-out theft, comprising: A camera, mounted at a fixed location in the store, is positioned to capture images of the contents of multiple shopping carts associated with the store, wherein the multiple shopping carts have their own distinct identifiers, and each of the shopping carts includes at least one wheel assembly that senses and reports wheel rotation events over a wireless network; and A computing system comprising one or more processors, the computing system being programmed to at least implement: An item classifier that assigns a category to detected items in the shopping cart based on analysis of the image using a trained machine learning model; and An exit manager determines whether to authorize the shopping cart to leave the store based at least on (1) the classification assigned to at least one item in the shopping cart by the item classifier, and (2) a flag indicating whether the at least one item has been paid for. The computing system is further programmed to: Identify a shopping cart identifier from the image sequence captured by the camera; Using the wheel rotation event, the identifier of the photographed shopping cart is matched with the cart trajectory detected in the image sequence captured by the camera; The identifier that matches the cart's trajectory is used to send an instruction to the photographed shopping cart, the instruction being based on the exit manager's departure authorization decision.

2. The system of claim 1, wherein the item classifier is configured to classify items detected in the shopping cart into goods and non-goods.

3. The system of claim 1, wherein the item classifier is configured to classify items detected in the shopping cart by product category.

4. The system of claim 1, wherein the item classifier is configured to classify the items added to the shopping cart at least in part based on the store location of the shopping cart when the items are added.

5. The system of claim 1, wherein the computing system is further programmed to: Based on the monitored location of the shopping cart, identify potential payment points for the goods in the shopping cart; and The transaction records executed at the payment point are compared with the item classification assigned by the item classifier to the items detected in the shopping cart.

6. The system of claim 5, wherein the computing system is further programmed to detect, based on the comparison, an underpayment event in which a customer pays less than all the items in the shopping cart.

7. The system of claim 5, wherein the computing system is further programmed to determine whether the shopping cart returns to the merchandise area after payment, rather than leaving the store first.

8. The system of claim 1, wherein the camera has a wavelength response including near-infrared and / or near-ultraviolet light.

9. A method for reducing push-out theft, comprising: Use cameras installed in fixed locations to film the store's exit; The location of a shopping cart within the store is monitored. The shopping cart is one of a plurality of shopping carts associated with the store. The shopping cart includes wheel assemblies that sense and report wheel rotation events on a wireless network. The plurality of shopping carts have their own distinct identifiers. The event of detecting an item being added to the shopping cart; The items are classified based on at least one of the following: (1) the location of the shopping cart in the store when the event occurred, as determined by the monitoring, and (2) an image of the items in the shopping cart captured by the camera; Determine the identifier of the shopping cart; Using the reported wheel rotation events, the identifier of the shopping cart is matched to the cart trajectory detected in the sequence of images captured by the camera; Subsequently, without any associated payment transaction flags, it was detected that the shopping cart was leaving the store; In response to determining that the shopping cart is leaving, at least in part based on the category assigned to the items, it is determined whether to perform anti-theft actions; and The identified identifier is used to send information to the shopping cart, the information being based on a determination of whether to perform an anti-theft action; The method is executed under the control of program instructions executed by one or more processors.

10. The method of claim 9, wherein detecting the event comprises detecting vibrations caused by an item being added to the shopping cart using a vibration sensor mounted to the shopping cart.

11. The method of claim 9, wherein the shopping cart includes a camera module having a camera and a processor, and wherein detecting the event includes the camera module comparing a first image captured by the camera with a subsequent second image captured by the camera to determine whether the contents of the shopping cart have been changed.

12. The method of claim 11, wherein the camera module further includes a wireless transceiver, and the method further includes, in response to detecting a change in the contents of the basket, the camera module uploading the second image or an identifier of the second image on a wireless network for analysis.

13. The method of claim 9, wherein assigning a category to the articles comprises using a trained machine learning module to classify the articles into commodities and non-commodities.

14. The method of claim 9, wherein specifying the category of the article includes determining a commodity identifier for the article.

15. The method of claim 9, wherein assigning a category to the article includes assigning a commodity type to the article.

16. The method of claim 9, wherein the classification is specified at least in part based on images of items in the shopping cart captured by a camera.

17. The method of claim 9, wherein the camera is mounted on the basket of the shopping cart.

18. The method of claim 9, wherein the classification is specified at least in part based on the location of the shopping cart in the store at the time the event occurs.

19. The method of claim 9, further comprising: Based on the monitored location of the shopping cart, potential payment points for the goods in the shopping cart are identified; and The transaction record executed at the payment point is compared with the item category assigned to one or more items detected in the shopping cart.

20. A method for monitoring a shopping cart, comprising: Receive wheel rotation event data from each of the multiple shopping carts in the store via wireless network; For each of the plurality of shopping carts, shopping cart motion data is generated based on the wheel rotation event data of each shopping cart, wherein the shopping cart reports wheel rotation event data associated with a cart identifier; Capture a sequence of images showing shopping carts moving within the store using at least one camera installed in the store; By analyzing the image sequence, shopping cart motion data for imaging is generated. The shopping cart motion data used for imaging is compared with the shopping cart motion data of each of the plurality of shopping carts generated from the wheel rotation event data. and The comparison is used to associate the wheel rotation event data and the associated shopping cart identifier with one of the multiple shopping carts captured in the image sequence.

21. The method of claim 20, wherein the wheel rotation event data includes a rotation event timestamp, and the method includes using the timestamp to generate data reflecting the cart speed.

22. The method of claim 20, wherein the wheel rotation event data includes rotation event data of the front wheels and the rear wheels of the first shopping cart, and the method includes using the event data of the front wheels and the rear wheels in combination to detect the steering of the first shopping cart.

23. The method of claim 20, further comprising: In response to the detection of a match between (1) shopping cart motion data generated from the image sequence and (2) shopping cart motion data generated from the wheel rotation event data of the first shopping cart among the plurality of shopping carts, the cart identifier of the first shopping cart is associated with the optical trajectory of the imaged shopping cart.

24. The method of claim 20, wherein motion data generated from the wheel rotation event data of the shopping cart specifies at least a shopping cart speed that varies over time.

25. The method of claim 20, further comprising using a trained machine learning model to determine from the image whether the shopping cart contains goods.

26. The method of claim 20, further comprising classifying items in the shopping cart at least in part based on the image using a trained machine learning model.

27. The method of claim 26, further comprising using at least the classification to determine the theft risk associated with the shopping cart.

28. The method of claim 26, further comprising comparing the classification with a payment transaction record to assess whether all items in the shopping cart are included in the payment transaction.

29. A system for monitoring shopping carts, comprising: A shopping cart wheel assembly including a wheel, a wheel rotation sensor, a processor, and a wireless transceiver, the shopping cart wheel assembly being configured to report wheel rotation event data over a wireless network; A camera is installed in the store location and configured to capture a sequence of images showing the movement of shopping carts in the store; and The computing system is programmed as follows: Using the image sequence and the reported wheel rotation event data, a match is detected between shopping cart motion data generated from the image sequence and shopping cart motion data generated from the wheel rotation event data to determine whether the shopping cart wheel assembly is part of the shopping cart imaged in the image sequence. and In response to determining that the shopping cart wheel assembly is part of the shopping cart imaged in the image sequence, an association is established between the cart identifier and the photographed shopping cart.

30. The system of claim 29, wherein the computing system is configured to compare cart motion data detected from the image sequence with cart motion inferred from wheel rotation event data.

31. The system of claim 29, wherein the computing system is configured to use timestamps contained in the wheel rotation event data to at least infer the change in cart speed over time.

32. The system of claim 29, wherein the shopping cart wheel assembly is one of two shopping cart wheel assemblies of a first shopping cart, one of which is mounted at a front wheel position and the other at a rear wheel position, and both wheel assemblies are configured to report their respective wheel rotation event data on a wireless network, wherein the computing system is configured to compare the wheel rotation event data of the front wheel assembly and the rear wheel assembly to detect a turn made by the first shopping cart, and to promptly associate the detected turn with a turn detected in the image sequence.

33. The system of claim 29, wherein the wheel assembly is configured to detect and report multiple partial rotation events each time the wheel completes a full rotation.

34. The system of claim 29, wherein the computing system is further programmed to use the image to determine whether the shopping cart contains goods.

35. The system of claim 29, wherein the computing system is further programmed to use the image to generate a classification of items in the shopping cart.

36. The system of claim 35, wherein the computing system is further programmed to compare the classification with payment transaction records associated with the shopping cart.

37. The system of claim 35, wherein the computing system is further programmed to use the classification to determine the theft risk associated with the shopping cart.

Images