Shopping basket monitoring using computer vision and machine learning

Abstract

A system for monitoring a shopping basket (e.g., a human-propelled cart, a basket on a motorized cart, or a hand-held basket) can include a computer vision unit that can image a surveillance area (e.g., an exit of a store), determine whether the basket is empty or loaded with merchandise, and assess the likelihood that the merchandise has been stolen. The computer vision unit can include a camera and an image processor programmed to execute a computer vision algorithm to identify the shopping basket and determine the loading state of the basket. The computer vision algorithm can include a neural network. The system can identify an at least partially loaded shopping basket that is leaving a store without an indication that the merchandise has been paid for, and perform an anti-theft action, such as activating an alarm, notifying store personnel, activating a store surveillance system, activating an anti-theft device associated with the basket (e.g., locking the shopping cart wheels), etc.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of priority of U.S. Patent Application No. 62 / 728,583, filed on September 7, 2018, entitled "SHOPPING BASKET MONITORING USING COMPUTER VISION AND MACHINE LEARNING", the entire content of which is incorporated herein by reference. Background Art Technical Field

[0003] The present disclosure generally relates to systems and methods for tracking movable shopping baskets, including but not limited to motorized and non - motorized (e.g., human - powered) shopping carts and handheld shopping baskets, using computer vision and machine learning techniques to track the movement and status thereof. Background Art

[0005] There are enclosure systems for deterring theft of shopping carts. Typically, these systems include wires embedded in the sidewalks of a store parking lot to define the outer boundary of an area where shopping carts are permitted to be used. When a shopping cart is pushed over this wire, a sensor in or near one of the wheels detects an electromagnetic signal generated via the wire, thereby activating a braking mechanism in the wheel to lock or prohibit the wheel from rotating. To unlock the wheel, an attendant typically uses a handheld remote control to send an unlock signal to the wheel. Some such enclosure systems present challenges. Summary of the Invention

[0006] A system for monitoring a shopping basket (e.g., a human - propelled cart, a motorized shopping cart or a basket on a motorized cart, or a hand - held basket) can include a computer vision unit that can image a monitoring area (e.g., an exit of a store), determine whether the basket is empty or loaded with merchandise, and evaluate the likelihood of merchandise theft. The computer vision unit can include a camera and (optionally) an image processor programmed to execute computer vision algorithms to identify the shopping basket in an image and determine the loading state of the basket. The loading state can include, for example, semantic class labels (e.g., full, partially full, empty), an estimated numerical value indicating the quantity of merchandise in the basket (e.g., in a range from 1 to 5, where 1 is empty and 5 is full), a score (which can weight the quantity of merchandise and the value of the merchandise), etc. In some implementations, the image processor can be arranged separately from the computer vision unit.

[0007] Computer vision algorithms can include neural networks. The system is capable of identifying shopping baskets that are leaving the store, determining the loading status (e.g., at least partially loaded), determining that there is no indication that the customer has paid for the merchandise, and performing anti-theft actions, such as activating an audible or visual alarm, notifying store personnel, activating the store surveillance system, activating an anti-theft device associated with the basket (e.g., locking the shopping cart wheels), etc.

[0008] The systems and methods disclosed herein can be applied to a variety of applications, including but not limited to retail stores (e.g., supermarkets or large retail outlets). Such systems and methods can be applied to tracking baskets or carts in indoor and outdoor environments and in, for example, retail environments, transportation (e.g., airports, trains, subways, bus stops) environments, medical (e.g., hospitals or clinics) environments, or warehouse environments. Such systems and methods can be used in applications where it may be desirable to identify whether a shopping cart, trolley, basket, etc. is at least partially loaded with items or objects.

[0009] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the specification, the drawings, and the claims. The Summary of the Invention and the following detailed description are not intended to limit or restrict the scope of the subject matter of the invention.

[0010] Brief Description of the Drawings

[0011] Figure 1A and Figure 1B schematically illustrates an example operation of a cart enclosure system. In Figure 1A , a shopping cart filled with merchandise is attempting to leave the store, and an anti-theft action is performed to prevent the merchandise from being stolen (e.g., the wheels of the shopping cart are locked or an alarm is activated). In Figure 1B , the cart is empty and no anti-theft action is taken.

[0012] Figure 1C illustrates various types of anti-theft system components that can be deployed inside and around a store to track movable shopping baskets, such as motorized and non-motorized (e.g., human-pushed) shopping carts, hand-held shopping baskets, and motorized mobility carts. A computer vision unit (CVU) or a camera transceiver unit (CTU) can be used to image the movable shopping baskets, e.g., to determine whether they are empty or at least partially loaded with merchandise.

[0013] Figure 2A illustrates an example of a shopping cart having a navigation system and one or more smart wheels.

[0014] Figure 2BAn example of a shopping cart with a smart positioning system mounted on the handle of the shopping cart is shown. In this figure, the child seat of the cart is in the open position (sometimes referred to as child seat down).

[0015] Figure 3 Components of an example containment system for a shopping basket are shown.

[0016] Figure 4A Schematically illustrated is a theft deterrent system that uses computer vision techniques to identify whether a shopping basket is at least partially loaded with merchandise and is leaving a store. The shopping basket can be attached to a human-propelled shopping cart, a motorized mobility cart, or can be hand-carried by the shopper.

[0017] Figure 4B Another implementation of the anti-theft system is schematically shown.

[0018] Figure 5 Schematically showing a side view (left) and a plan view (right) of a camera of a computer vision unit positioned to determine the location of a shopping basket.

[0019] Figure 6A 、 Figure 6B and Figure 6C Examples of placement and orientation of computer vision units (CVUs) and secondary cameras near the entrance / exit of a retail store are schematically shown. The number and arrangement of CVUs and secondary cameras, as well as the shape and size of their respective fields of view (FOVs, represented by dashed or dotted lines) are intended to be illustrative and not limiting. In other implementations, the layout may differ to meet the security objectives of the retail facility.

[0020] Figure 7 Schematically illustrated are examples of paths taken by shopping baskets near an entrance / exit of a retail store. Empty baskets are shown without shading, at least partially loaded baskets are shown with shading. Symbols on the path taken by a shopping basket (in this example a shopping trolley) indicate the potential for push theft.

[0021] Figure 8 Schematically illustrates an example of a processing pipeline for training a machine learning (ML) model.

[0022] Figure 9 Schematically shows an example of a processing pipeline for analyzing images obtained from an anti-theft system.

[0023] Figure 10 Schematically shows an example of a processing pipeline for real-time event detection or a live stream from an anti-theft system.

[0024] Figure 11An example of a pipeline for business intelligence (BI) analysis of image data from an anti-theft system is schematically shown.

[0025] Figure 12 An example of a processing pipeline in the CVU is schematically shown.

[0026] Throughout the drawings, reference numerals may be reused to indicate corresponding relationships between reference elements. The drawings are provided to illustrate examples of the implementations described herein and are not intended to limit the scope of the disclosure. Detailed Description

[0027] Overview

[0028] Although existing cart enclosure systems can be used to prevent shopping cart theft, some such systems may not be able to detect other types of shopping-related misuse. For example, a thief may push a shopping cart at least partially loaded with groceries or merchandise out of the store without paying for the groceries or merchandise (this type of theft is sometimes referred to as "push-out" theft). The cart enclosure system may not have the ability (or only have a limited ability) to determine whether a shopping cart pushed out of the store is empty (in which case there is no threat or only a limited threat of merchandise theft) or loaded with merchandise (in which case there may be a significant threat of merchandise theft). If the cart enclosure system is triggered every time a cart leaves the store (loaded or unloaded), it may result in many false alarms because the system will be triggered even when an empty cart leaves the store.

[0029] False alarms can be reduced by determining whether a shopping cart has passed through an active store checkout lane before attempting to leave the store. If so, it is likely that the shopper has purchased the merchandise, and the cart enclosure system can be configured not to trigger in such a case. If the cart does not pass through an active store checkout lane (or does not spend enough time in the lane to actually make a payment), the cart enclosure system can be configured to trigger when leaving. However, even in this case, false alarms may still occur because a shopper may push an empty shopping cart outside the store for some non-theft reason (e.g., to select another cart (e.g., one with less wheel wobble), to return to a parked car to retrieve grocery bags or a shopping list, etc.). A cart enclosure system that detects carts passing through active cash registers may require specific hardware devices in each checkout lane to detect the passage, path, speed, travel distance, dwell time, etc. of the cart in the checkout lane. Such hardware increases the cost of these devices. Additionally, this method may have significant limitations for retail stores that have implemented a mobile payment system, in which case shoppers do not need to pass through a fixed checkout lane to make a payment but can instead use a mobile application (e.g., on the shopper's smartphone) to pay for the merchandise.

[0030] While it is often possible to detect shoplifting using an Electronic Article Surveillance (EAS) system (e.g., including an EAS tower at a store exit), the cost and burden of attaching EAS tags to merchandise are often impractical (especially in the case of grocery stores).

[0031] A retail store may wish to identify whether a shopping cart approaching an exit is at least partially loaded with items from the store (e.g., groceries, health products, liquor, etc.), and if so, whether the cart has previously passed through a checkout aisle or the shopper has made a payment via a mobile payment method. A cart enclosure system can image the shopping cart basket using cameras installed in the store and can use computer vision and machine learning techniques to analyze the image to determine, for example, whether the cart basket is empty (e.g., a lower threat of theft situation) or at least partially loaded with merchandise (e.g., a higher threat of theft situation). The image can be a still image or one or more frames in a video.

[0032] If the system detects that a cart that is at least partially loaded is attempting to leave the store without any indication that the items have been paid for, the cart enclosure system can be triggered to perform an anti-theft action (e.g., braking or locking the wheels of the cart to prevent its movement, displaying an alert or message to the shopper to return to the store, activating the store video surveillance system or an alarm, notifying store security personnel, etc.).

[0033] Figure 1A and Figure 1B Schematically shows an example operation of the cart enclosure system. Figure 1A and Figure 1B The features shown in Figure 1C , Figure 4A and Figure 4B will be further described below (see, for example, Figure 1A ). In Figure 1A , a Computer Vision Unit (CVU) or a Cart Transceiver Unit (CTU) includes a camera 410 capable of imaging a store area near a store exit. The area of the store is within the field of view (FOV) of the camera 410. The CVU or CTU can perform (or communicate with another system to perform) computer vision analysis of the image from the camera 410. The computer vision analysis can determine the loading status of the cart, e.g., whether the shopping cart is empty, whether it is at least partially loaded with merchandise, whether it is full of merchandise, etc. The CVU or CTU can communicate with a door manager 450 that controls the anti-theft functions of the cart enclosure system. An example of a door manager 450 is available from Gatekeeper Systems, Inc. (Foothill Ranch, CA) A prevention ejection system, as described, for example, in U.S. Patent Nos. 8,463,540, 9,731,744, and 10,232,869; the entire contents of which are incorporated herein by reference. Other examples of door managers and anti-theft systems are described in U.S. Patent Nos. 5,881,846 or 7,420,461; the entire contents of which are incorporated herein by reference.

[0034] As Figure 1A shown, if a cart that is at least partially loaded with merchandise is approaching a store exit without a marker indicating that the merchandise has been paid for, the CVU or CTU can communicate a signal to activate the anti-theft function to the door manager 450 (e.g., locking or braking one of the cart wheels, sounding an alarm, activating the store surveillance system, etc.). Conversely, as Figure 1B shown, if the shopping cart is substantially empty, the risk of theft is low, and the CVU or CTU can either take no action or communicate a signal to the access control manager 450 not to take anti-theft action.

[0035] Thus, Figure 1A and Figure 1B the example cart enclosure system can advantageously reduce or prevent theft of merchandise from the store while reducing or preventing false alarms of empty carts being ejected from the store for some reason (e.g., switching carts for another cart, returning to the shopper's vehicle to retrieve a shopping bag or shopping list, etc.).

[0036] In some implementations, a separate door manager 450 is not used, and the CVU transmits the anti-theft signal to the shopping basket, shopping cart, or store surveillance system.

[0037] Although many shoppers use shopping carts in retail stores, the computer vision techniques described herein are not limited to shopping carts and can be applied to any movable shopping basket, including human-powered shopping carts, motorized mobility carts with baskets, or handheld shopping baskets carried by shoppers. Moreover, these computer vision techniques are not limited to retail applications and can be applied to determine whether other types of carts are loaded with items, objects, or merchandise, such as warehouse, industrial, or utility carts, luggage carts, medical, hospital, or pharmacy carts, wheelchairs, baby strollers, or hospital beds, etc.

[0038] Various examples and implementations are described below. These examples and implementations are intended to illustrate the scope of the present disclosure and are not intended to be limiting.

[0039] Example Retail Store Scenario

[0040] Figure 1CAn example of an anti-theft system 400 is shown. The illustrated anti-theft system is deployed in a store to track or control the movement of shopping carts 30 and prevent theft of merchandise in the carts. However, the inventive components and methods of the anti-theft system can be used in other applications, such as tracking luggage carts in an airport or carts in a warehouse.

[0041] The system includes a set of cart transceivers (CTs) that communicate bidirectionally with a set of wireless access points (APs) to establish a two-way radio frequency (RF) communication link with the shopping carts 30. In one example, each cart transceiver (CT) is fully contained within one of the standard-sized (e.g., 5-inch diameter) wheels 32 (typically the front wheels) of the respective shopping carts 30, together with a braking unit that can be actuated by the cart transceiver to lock the wheel. Examples of braking units that can be used for this purpose are described in U.S. Patent No. 6,362,728, U.S. Patent No. 8,820,447, or U.S. Patent No. 8,602,176, or U.S. Patent No. 8,973,716; the entire content of each of which is incorporated herein by reference. (For purposes of detailed description, the term "cart transceiver" collectively refers to the RF transceiver of the cart and the associated sensor circuitry). Alternatively, a progressive or partial braking unit can be used that is additionally capable of prohibiting rotation of the wheel without placing the wheel in a locked state.

[0042] Some of the circuitry of the cart transceiver (CT) can alternatively be provided elsewhere on the shopping cart 30. For example, as described below, some transceiver circuitry can alternatively be included in a display unit attached to the handle of the shopping cart or the front of the cart. As another example, some or all of the circuitry, including the sensor circuitry, can be encapsulated within the wheel assembly (e.g., the caster or fork of the wheel) rather than being included in the wheel itself or the handle or frame of the cart. The CT can be included in the frame or body of a motorized mobility cart. The CT is not limited to use on carts and can also be connected to a handheld shopping basket (e.g., on the side or bottom of the basket or in the handle).

[0043] An access point (AP) is typically responsible for communicating with a cart transceiver (CT) to retrieve and generate cart status information, including information indicating or reflecting the cart's location. The types of cart status information that can be retrieved and monitored include, for example, whether wheels 32 are in a locked or unlocked state, whether the cart is moving; the average rotational speed of the wheels (which can be sensed using rotational sensors in wheels 32); whether the cart has detected a particular type of location-related signal, such as a VLF, EAS, or magnetic signal (as described below); whether wheels 32 are slipping; the battery charge of the CT and the general "health" of the wheels; and the number of lock / unlock cycles the cart has experienced since a certain reference time. In some examples, the cart can include sensors capable of determining whether its shopping basket is at least partially loaded (e.g., by analyzing the cart's vibration data), and the CT can communicate the load status (e.g., empty, partially loaded, full) to the AP. (Contrary to the other wheels of the shopping cart, the term "wheels 32" is specifically used herein to refer to wheels that include the electronic devices described herein.) The access point (AP) is also capable of generating and / or relaying commands to the cart transceiver (CT), including lock and unlock commands (or other types of anti-theft commands) sent to a particular shopping cart.

[0044] In Figure 1C the example shown, all access points (APs) communicate wirelessly with a central control unit (CCU) either directly or via intermediate access points. The central control unit can be implemented as a desktop computer or a hardware server that includes a wireless transceiver card or is wired to an external transceiver unit. The CCU is generally responsible for collecting, storing, and analyzing the cart status information, including location information, collected by the access points (APs). In addition to the data retrieved from the cart transceiver (CT), the CCU can also collect data generated by the access points, such as signal strength measurements of the detected cart transmissions. Some or all of the collected data is preferably stored by the CCU along with associated event timestamps.

[0045] Figure 1C The system shown in Figure 4A and Figure 4BThe described CVU 1000. The CVU can include a camera (a still camera or a video camera), an image processor, and a transceiver configured to communicate with an AP, a CCU, or a CT. As further described below, the CVU (used alone or in combination with a CCU or an AP) is capable of analyzing an image of a shopping basket (captured by the camera) to determine the loading state of the shopping basket, e.g., empty, partially loaded, or fully loaded. The CVU can be placed near a store entrance / exit (e.g., imaging shopping baskets entering or exiting), near a checkout counter 34 (e.g., imaging shopping baskets in a checkout aisle), or near other areas of a retail store (e.g., in an area where high-value items are stored). In some examples, the CVU 1000 includes a camera transceiver unit (CTU), which can include a subset of all the components of the CVU. For example, the CTU can include a camera and an RF transceiver (or a wired Ethernet connection) without including the image processor. A device can include any number of CVUs or CTUs. In some implementations, the use of CTUs is more cost-effective (since each unit does not include an image processor), and the image processing function is offloaded to the CCU (or the main CVU). The choice of whether to use a CVU or a CTU and the choice of the corresponding arrangement of CVUs or CTUs will depend on the details of the devices in any particular retail store (e.g., the location or number of exits or entrances, the location or number of checkout aisles, the physical size or layout of the store interior, the number of customers, the presence or location of high-value items, etc.). For example, one installation can primarily or solely use CTUs and offload the image processing to a single CVU or CCU. However, another installation can primarily use CVUs. Yet another installation can utilize CVUs in an area where high-value items are stored in order to be able to perform image processing locally and utilize CTUs in other areas of the store. Many device options can meet the needs of a particular retail store.

[0046] The CCU or CVU can analyze the collected data in real time for decision-making purposes, such as whether to send a lock command to a particular cart 30, whether to activate a store video surveillance system, or whether to send alert information to a person. Figure 1A and Figure 1BShows an example of a CVU or CTU that communicates with a door manager and takes appropriate anti-theft actions when needed. The door manager may include an access point (AP) for communicating with a cart transceiver in the cart wheels as described herein. For example, when a cart is approaching or passing through a store exit, the CCU or CVU may analyze the cart's recent history (e.g., path and speed) to evaluate whether a customer is attempting to leave the store without paying. The CCU (or CVU) may analyze camera images to evaluate whether a shopping basket leaving the store is at least partially loaded or whether the basket has passed through the checkout station 34. (The access point may additionally or alternatively be responsible for making such a determination.) Based on the result of this determination, the CCU may send a lock command to the cart (typically via the access point), or may not issue a command authorizing the cart to leave. As another example, if the CCU detects a rapid increase in the number of active carts, the CCU may warn personnel (e.g., via the store LAN) that additional checkout stations may need to be opened. As another example, if the CVU detects that a shopping basket is leaving the store and is at least partially loaded, the CVU may send an alert to store personnel, activate an alarm, display a warning to the shopper via a display on the basket (or a smart navigation module), or communicate a lock command to the smart wheels of the cart to activate the brakes (e.g., prohibit the movement of the cart).

[0047] The CCU may also run data mining and reporting software that analyzes the data collected over time to detect meaningful traffic patterns and trends. For example, the CCU may generate a report that shows how customers typically move through the store, how much time they spend in each aisle or other shopping area, the loading level of shopping baskets leaving the store, data on theft incidents (e.g., fully or partially loaded shopping baskets that leave the store without payment), etc. This information can be used, for example, to adjust the store layout or to adjust the size or quantity of shopping baskets provided to shoppers.

[0048] The CCU (or CVU) may additionally or alternatively communicate the data it collects via a cellular network or a wireless network (e.g., the Internet) to a remote node that processes analysis and reporting tasks. For example, the CCU (possibly along with one or more access points or CVUs) may have an autonomous WAN link that uses a cellular data service, such as General Packet Radio Service (GPRS), to communicate the collected data to a remote node for analysis and reporting. This feature can be used to monitor the health of the system from a remote facility. The system may also be able to be tested and configured via the WAN link from the remote facility.

[0049] As Figure 1CAs shown, the CCU (or CVU) can be connected to various other types of systems present in the store. For example, the CCU or CVU can be connected to a pre-existing alarm system and / or video surveillance system. In this case, the CCU or CVU can be configured to activate a sound alarm or video camera when an unauthorized exit event is detected (in various implementations, the video camera in the surveillance system may be different from or the same as the camera in the CVU). As another example, the CCU or CVU can be connected to a pre-existing central storage computer that maintains information about the status of the store's checkout cash registers or mobile payment platforms; as described below, this information can be retrieved and used by the CCU or CVU to evaluate whether a customer has paid for merchandise through an active checkout lane or using a mobile payment application or mobile payment point.

[0050] In some implementations of the system, the CCU can be omitted. In these implementations, the access point (AP) can implement all of the real-time analysis functions that might otherwise be handled by the CCU. For example, an access point or CVU installed near the store exit may be able to detect that a customer is attempting to leave the store without paying (or determine that the cart's basket is at least partially loaded) and decide whether to send a lock command to the cart. To accommodate both centralized and distributed devices, each access point or CVU can operate with or without the CCU. Implementations that omit the access point are also possible, such that the CCU or CVU communicates directly with the cart transceiver. Many variations of distributed network-connected components and circuitry can be envisioned.

[0051] The cart transceiver (CT), access point (AP), computer vision unit (CVU), checkout barrier (CB), and central control unit (CCU) are all capable of operating as uniquely addressable nodes on a wireless tracking network. As Figure 1C shown, another type of node that may be included in the network is the handheld mobile control unit (MCU). The mobile control unit is designed to enable store personnel to unlock individual carts by pressing a button on the MCU. The mobile control unit can also include functions for retrieving and displaying various types of cart status information, for configuring the wheels / cart transceiver and updating its firmware, and for controlling the motorized cart retrieval unit 40 (see the discussion of the cart retriever 40 below).

[0052] In some implementations, various types of nodes (e.g., cart transceivers, access points, central control units, computer vision units, and mobile control units) can communicate with each other using a non-standard wireless communication protocol that enables the cart transceivers to operate at a very low duty cycle without maintaining synchronization with the access point when inactive. As a result, the cart transceivers are able to operate for a long time (e.g., several years) using a relatively small battery installed in the wheels 32. Details of a specific wireless communication protocol described in U.S. Patent No. 8,463,540, "Two-Way Communication System for Tracking Locations and Statuses of Wheeled Vehicles," the entire disclosure of which is incorporated herein by reference in its entirety.

[0053] Each cart transceiver (CT) is preferably capable of measuring the received signal strength based on the RSSI (Received Signal Strength Indication) value of transmissions received on the wireless tracking network. The system can use these RSSI measurements in various ways. For example, the cart transceiver can compare the RSSI value of an access point transmission with a threshold to determine whether to respond to the transmission. The cart transceiver can also report this RSSI value (along with the unique ID of the cart transceiver) to the access point so that the system can estimate the location of the shopping cart or the distance to the shopping cart. As another example, the cart transceiver can be programmed to generate and report the RSSI values of transmissions from other nearby cart transceivers; this information can in turn be used to estimate the number of carts queuing at checkout lanes, in cart storage structures, near store entrances / exits, in stacks of carts retrieved by the mechanized cart retrieval unit 40, or elsewhere.

[0054] Figure 1C Three checkout stations 34 are shown, each checkout station 34 including a checkout register (REG), which typically includes a merchandise scanner. In this particular example, each checkout station 34 includes an access point (AP), which can be mounted to a pre-existing pole indicating the number of the checkout lane, if present. Each such access point can include a connection or sensor that enables it to determine whether the respective checkout station is currently active. This information is useful for assessing whether a customer passing through the checkout lane has paid. Several different methods that can be used to sense the active / inactive state of the checkout station are described below. Each access point placed at the checkout station 34 can communicate with nearby shopping carts / cart transceivers using a directional antenna, such as those queuing in the respective checkout lane (see Figure 2 discussed below).

[0055] In some implementations, a store can utilize a checkout barrier (CB) at the end of a checkout lane, at the store exit (as shown in Figure 1C ), in an area with high-value items, etc. The CB generally includes a gate, obstacle, or turnstile that will be locked unless the customer is allowed to leave the checkout lane or store or high-value area (e.g., the customer has paid for the items). The CB can then be unlocked to allow the customer to leave (e.g., by pushing a gate that rotates open to allow passage). After leaving, the gate rotates closed and locks to prevent other customers from leaving without paying. The CB can communicate with the store's CCU, CVU, CTU, checkout cash register, mobile payment point 35 (described below), etc. to receive commands to unlock (or lock) the barrier.

[0056] Figure 1C The mobile payment point 35 is also schematically shown. The mobile payment point does not have to be fixed to a physical location in the store and can represent a wireless network connection that allows a shopper to pay for the items in the shopper's basket. For example, a shopper can access a mobile payment application (e.g., on the shopper's smart phone or on a communication display installed in the shopping basket or cart) that can electronically record the items or merchandise in the basket and provide mobile payment options (e.g., pay by credit card or debit card). The mobile payment point 35 can communicate wirelessly with an AP, CCU, CVU, etc., thereby being able to record the payment and convey it to Figure 1C the appropriate components of the system shown. For example, the CVU can detect (via the computer vision image analysis described herein) that a loaded shopping basket is about to leave the store. The CVU can access payment information to determine whether the shopper associated with the departing basket has paid for the items in the basket. If the shopper has paid (e.g., via the mobile payment point 35 or at the cash register 34), the system can allow the shopping basket to leave the store without triggering an anti-theft action. However, if the shopper has not paid, the system can trigger an anti-theft action (e.g., activate an alarm or store surveillance system, send a locking command to the cart wheels, notify store personnel, etc.).

[0057] Access points can additionally or alternatively be installed on various other fixed and / or mobile structures near the store. For example, as shown in Figure 1C , access points can be installed on shopping cart storage structures 36 (two are shown) in the store parking lot. These access points installed on the parking structure can be used to detect and report the number of carts stored in their respective areas and can also be used to enable the in-store access points, CVU, or CCU to communicate with the carts that would otherwise be out of range.

[0058] Figure 1CThe system shown can include other optional components. For example, a powered assist (mechanized) cart retrieval unit or cart 40, which can be a cart pusher or a cart puller, can be used to retrieve a shopping cart and return it to the cart storage location 36. The store can include a pair of conventional EAS (electronic article surveillance) towers at the store exit, or additionally or alternatively at the end of each checkout aisle. Although EAS towers are not required to implement the various functions described herein, the system can take advantage of their prevalence in retail stores. For example, each cart transceiver (CT) can include an EAS receiver for detecting its passage between a pair of EAS towers and can be configured to report an EAS detection event on the wireless tracking network; this information can in turn be considered when evaluating whether a departing customer has paid.

[0059] Figure 1C The exemplary store configuration in also shown as having very low frequency (VLF, typically below 9 kHz) signal lines 44 buried in the sidewalk along the perimeter of the parking lot or near the store exit. Such signal lines can be used to define the boundaries of the area in which shopping carts are permitted. The wheels 32 of the shopping cart can include a VLF receiver that detects a VLF signal when the cart is pushed over the signal line 44 and engages a brake. Although not shown in Figure 1C it can also be provided at the store exit such that all carts passing through the exit must cross this line, and / or at other locations of concern.

[0060] Although the present system does not require the use of the VLF signal line 44, the system preferably is capable of using one or more VLF lines as a mechanism for monitoring the position of the cart. Specifically, the cart transceiver (CT) preferably includes a VLF receiver. The VLF receiver may be capable of detecting the codes transmitted on the VLF line, so as to be able to use different lines to uniquely identify different areas or boundaries. When a VLF signal is detected, the cart transceiver may take various measures according to the situation. For example, the cart transceiver may attempt to report the VLF detection event on the wireless tracking network and then wait for a command indicating whether to engage the brake. If no command is received within the pre-programmed time period (e.g., 2 seconds) in this example, the cart transceiver may automatically engage the brake. The VLF detection event may be reported to the CVU on the wireless tracking network, and the CVU may image the cart or shopping basket to determine its loading status. If it is determined that the cart or basket is not loaded, the likelihood of theft is low, and a brake command may not be sent (or it may be indicated that the brake is not engaged). On the contrary, if it is determined that the cart or basket is at least partially loaded and the cart or basket is leaving the store, the CVU may send a brake or lock command or some other type of anti-theft command to the cart. For example, for a hand basket (which does not have locking wheels), the anti-theft command may include a warning command (e.g., the light or alarm on the basket may be activated to warn the shopper), a command to activate the store video surveillance system (to obtain video of a potential theft event), a signal to alert the store security personnel, etc. Such anti-theft commands may be used additionally or alternatively with the wheeled cart.

[0061] Further reference Figure 1C Optionally, one or more magnetic markers or magnetic strips (MAG) may be provided above or below the store floor to provide an additional or alternative position tracking mechanism. As shown in the figure, these magnetic markers may be provided at key locations, such as at each checkout aisle and store exit. Although not shown in FIG. 1, one or more magnetic markers may also be provided in the parking lot and / or shopping aisles. Each magnetic strip can have a unique magnetic pattern that can be sensed by an optional magnetic sensor included in the wheels 32 or attached to the shopping basket or shopping cart 30. Thus, the magnetic markers serve as magnetic barcodes for identifying specific locations. In one implementation, when the cart 30 crosses a magnetic marker, the cart transceiver (CT) transmits the detected magnetic code or information from which this code can be derived on the wireless tracking network. U.S. Patent No. 8,046,160, "Navigation Systems and Methods for Wheeled Objects", describes additional details on how to sense and use magnetic markers, the entire disclosure of which is incorporated herein by reference in its entirety.

[0062] Figure 1C The system shown can include other or alternative features or components. For example, the system can implement the techniques and functions for low-power location of movable objects described in U.S. Patent No. 9,606,238, the entire disclosure of which is incorporated herein by reference in its entirety. These techniques can be used to track the location of a shopping cart as it moves through a store environment. Techniques (e.g., dead reckoning) described in the above-incorporated U.S. Patent No. 8,046,160 or U.S. Patent No. 9,731,744 or U.S. Patent No. 10,232,869 can be used to track the movement of the shopping cart; the entire content of each of which is incorporated herein by reference.

[0063] It will be apparent from the foregoing discussion that Figure 1C many of the components shown are optional components that may or may not be included in a given system installation. For example, in some installations, the magnetic markers, EAS towers, checkout barriers, and / or VLF signal lines can be omitted. Additionally, the access points or CCUs can be omitted. The CTU can be replaced with a CVU, and vice versa. Further, the components shown can be different from the arrangements shown. For example, VLF signal lines can be provided in the checkout lanes and / or store exits / entrances (e.g., in place of the magnetic markers and EAS towers shown) to enable the cart to detect checkout events and leaving / entering events, respectively. Additionally, other types of signal transmitters and detectors / receivers can be used to monitor the cart location. For example, ultrasonic transmitters / receivers can be used to track the cart location, or the store can include radio frequency (RF) detectors (e.g., located on the ceiling) that detect RF signals from the cart and use direction-of-arrival techniques to determine the cart's location.

[0064] Example Techniques for Evaluating Customer Payment

[0065] The system is capable of supporting multiple different methods for evaluating whether a customer has left the store without paying. One or more specific methods used may vary significantly based on the type and location of the system components included in a given device. For example, if the store does not include any Electronic Article Surveillance (EAS) towers, Magnetic Tags (MAG), or VLF lines, the determination can be made solely or primarily based on cart position / path information determined according to CT-AP communication, optionally considering the wheel rotation speed history as an additional factor. If EAS towers, magnetic tags, and / or VLF signal lines are provided, they can be used as additional or alternative information sources for making decisions. The system can include a Computer Vision Unit (CVU) near the checkout lane 34, and the CVU is capable of analyzing images of the checkout lane to determine whether a shopper has passed through the lane, interact with the store checkout attendant or the store payment system, has stayed in the lane for a sufficient amount of time indicating the checkout and payment process (e.g., more than 1 minute, 2 minutes, 3 minutes, 5 minutes or longer), provided payment information, etc. The CVU can analyze the images to determine whether the shopper is approaching the exit from the direction of the checkout lane 34 or another direction where payment is less likely. As further described below with reference to Figure 4A Additional one or more secondary cameras 410a can be located in the facility to monitor the movement of the cart (e.g., through the checkout lane or payment point or from a location storing high-value items). When the cart moves from the field of view of one secondary camera to another secondary camera (or CVU or CTU), the system is capable of transferring the tracking of the cart to the next camera to provide a substantially continuous path of the cart. The CVU (or CCU) can access payment information from the mobile payment point 35 to determine whether a shopper leaving has paid for the items in the shopper's basket. U.S. Patent No. 8,463,540 describes other (or alternative) techniques for evaluating whether a departing customer has paid, the entire disclosure of which is incorporated herein by reference in its entirety. Many combinations or variations of the foregoing can be used to determine whether a shopper associated with a shopping basket (e.g., a shopper pushing a manually propelled shopping cart) is likely to have paid for the items in the shopper's basket.

[0066] Example Shopping Basket on Shopping Cart

[0067] Figure 2A FIG. shows the features of an example shopping cart 30 with a shopping basket 205. The shopping cart 30 is manually propelled and includes an intelligent positioning system 210 and one or more anti-theft wheels 215 (which can brake, lock, or prohibit the rotation of the wheels or the movement of the cart). The intelligent positioning system 210 can be mounted on the handle of the cart 30 (e.g., as Figure 2A and Figure 2Bas shown), or can be installed in the cart or elsewhere on the cart (e.g., at the front of the basket 205). The anti-theft wheel 215 can be a smart locking wheel, such as a wheel having a sensor (e.g., for sensing VLF lines), a wireless communication system (e.g., the cart transceiver CT), and / or a processor in addition to a locking mechanism or a braking mechanism. A smart positioning system 210 can be used to track the position of the shopping cart 30, which can utilize dead reckoning or vibration detection techniques to estimate the position of the cart. For example, the smart positioning system 210 can include components or functions described in U.S. Patent No. 8,046,160, U.S. Patent No. 9,731,744, or U.S. Patent No. 10,232,869, the entire content of each of which is incorporated herein by reference.

[0068] The functions of the navigation system and the anti-theft system can be distributed between the smart positioning system 210 and the smart locking wheel 215. For example, one or both of the smart positioning system 210 and the smart locking wheel 215 can have leave / enter event detection capabilities; the anti-theft function of wheel locking can be located in the smart locking wheel 215, while the anti-theft function of user warning can be located in the smart positioning system 210. In addition, although Figure 2A a shopping basket 205 for a manually propelled shopping cart is shown, similar techniques described herein apply to motorized shopping carts or shopping baskets on motorized carts or handheld shopping baskets carried by a shopper. For example, the smart positioning system 210 can be attached (or integrated into) a motorized shopping cart or a motorized shopping cart, or attached to a handheld shopping basket.

[0069] As Figure 2B shown, some shopping carts include a user-adjustable child seat that can move between a closed position of the child seat and an open position of the child seat. In the open position (as Figure 2B shown), the shopper can place a child (or other item) on the seat portion. In many carts, the shopper can push the metal frame of the child seat 1620 away from the handle 1610 of the cart 30, which causes the seat portion to move to a horizontal position. Thus, the open position is sometimes referred to as the child seat down. Figure 2B A shopping cart 30 is shown in which the child seat 1620 is in the open or child seat down position. As will be further described below, a computer vision unit (CVU) can image the shopping cart 30 to determine whether the loading of the cart is (at least partially) attributable to a child placed in the child seat 1620 or an object (e.g., a handbag) placed on the child seat rather than store merchandise.

[0070] Example Smart Location System / Smart Brake Wheel Implementations

[0071] Figure 3 Component set 300 for an example tracking system for a shopping basket (e.g., a basket on a shopping cart or mobility cart or a handheld shopping basket) is shown. The example component set 300 includes the following components: (1) an intelligent positioning system 210; (2) intelligent locking wheels 215; (3) fixed features 385 associated with store exits and / or entrances, checkout aisles, high-value areas, locations where the wheels 215 can be reset or updated, etc.; (4) a system configuration and control device 390; (5) an RF beacon or other RF feature 395; and (6) a computer vision unit (CVU) 1000 or a camera transceiver unit (CTU) 1001. Reference Figure 4A and Figure 4B The CVU and CTU are further described.

[0072] The intelligent positioning system 210 includes (1) sensor elements 315 for determining the heading and speed of the cart (e.g., magnetometer and / or accelerometer), and optionally for determining the temperature of the system (e.g., temperature sensor); (2) optional sensors 320 that provide data from which the wheel rotation rate can be inferred (e.g., the sensor does not need to be close to the wheel); for example, a vibration sensor; (3) a processor and memory 325; (4) a communication system 330 that communicates (e.g., via an RF link) with the intelligent locking wheel 315, the system configuration and control device 390, an RF beacon or other RF feature 395, and / or the CVU 1000; (5) an optional detector 310 configured to determine that the cart is passing through an exit / entrance of a store (departure / entry event), and in some embodiments, to determine whether the movement is leaving or entering the store. In some systems, the circuitry in the wheel performs the actual detection function; the intelligent positioning system communicates with the detection circuitry in the wheel to obtain departure / entry information. Certain systems may use detector 360 as the primary detector and detector 310 as the secondary detector; (6) an indicator 335 (e.g., visual and / or audible) that provides a notification to the user to indicate that the cart is in a warning zone and / or about to be locked. The indicator may include a display configured to output text or an image (e.g., outputting a warning to the user that the containment boundary is near and the wheel will lock if the wheeled object moves outside the containment boundary). The indicator may include a light (e.g., light-emitting diode (LED)) that illuminates or flashes as a notification to the user. The indicator may include an audible alarm or notification. In some embodiments, the indicator includes a speech synthesizer capable of outputting a humanly understandable message such as "The cart is approaching the limit and will soon be locked". The indicator can include a speaker for outputting an audible notification. The intelligent positioning system 210 may also include a light detector 333 for detecting ambient light features for navigation use, or a vertical position detector 337 (e.g., a pressure sensor) for determining which floor of a multi-story structure the intelligent positioning system is located on. The functions of these components are further described in U.S. Patent No. 9,731,744 or No. 10,232,869 incorporated hereinabove.

[0073] Figure 3An example is shown where the intelligent positioning system 210 is used with a wheeled cart including intelligent locking wheels 215 (although this is not necessary, and the system 210 can be used on a handheld basket). The wheel 215 includes (1) a locking mechanism (e.g., a brake) 380 configured to prohibit rotation of the wheel when the locking mechanism is activated (or the cart itself is translated); (2) a wheel rotation detector 375, e.g., a tuning fork and a striker (e.g., a part that strikes the tuning fork as the wheel rotates); (3) a processor and memory 370; (4) a communication system 365 configured to communicate with the intelligent positioning system 210, the system configuration and control device 390, an RF beacon or other RF feature 395, and / or the CVU 1000 or CTU 1001; (5) an optional detector 360 configured to detect departure / entry events and, in some embodiments, to detect whether the movement is leaving or entering the store; and (6) an optional heading / caster angle detector 383 configured to detect the heading of the (caster) wheel.

[0074] The fixed feature 385 can be associated with the exits and entrances of the store, checkout aisles, areas where high-value items are located, locations where the position of the cart can be reset or updated, etc. The proximity of these features can be detected by detectors in the intelligent positioning system or the intelligent locking wheel. The fixed feature can be used to provide an accurate reference position to the intelligent positioning system (e.g., for resetting any accumulated dead reckoning position errors). The fixed feature 385 can include a VLF line, an access point, an RF field generated for warning or locking, a checkout barrier, an EAS tower, a magnetic marker, or an electromagnetic marker, etc. The CVU 1000 or CTU 1001 can communicate with the fixed feature 385 to provide an appropriate signal when the shopping cart approaches near the fixed feature (e.g., providing a locking or unlocking signal to the checkout barrier or the cart transceiver, or providing a position signal to reset or update the position of the cart).

[0075] The system configuration and control device 390 can perform housekeeping tasks such as configuration and control. The device 390 can communicate with the communication system 330 in the intelligent positioning system and / or the communication system 365 in the intelligent locking wheel. The system configuration and control device 390 can be a device including a CCU (e.g., as described with reference to Figure 1C ), or in some cases, including the CVU 1000.

[0076] The RF beacon or other RF feature 395 can transmit RF signals for entry / exit detection and / or precise position determination.

[0077] The CVU 1000 or CTU 1001 can communicate with, for example, as referenced Figure 1CThe described intelligent locking wheel 215, intelligent positioning system 210, RF beacon or other RF feature 395 and / or system configuration and control device 390 or central control unit (CCU) communicate wirelessly. Additionally or alternatively, the CVU or CTU can use a wired LAN connection, such as Ethernet, to communicate with the CCU or controller 390.

[0078] The systems described herein can be implemented with more or fewer features / components than those described above. Also, the systems can be implemented with a configuration different from that described above. For example, the rotation detector can be implemented in one of the intelligent positioning system and the intelligent locking wheel, and the RF beacon can communicate with one of the communication systems 330 and 365 instead of both. Additionally, Figure 3 the functions of the components in can be combined, rearranged, separated, or configured differently than shown.

[0079] The intelligent positioning system 210 can be arranged at one or more positions in a wheeled object. For example, some or all of the intelligent positioning systems can be arranged in the handle, frame, caster, wheels, etc. of a cart. For a motorized shopping cart or a mobility cart, the intelligent positioning system 210 can be attached to the frame or body of the cart or integrated with other electronic circuits for operating the cart. The intelligent positioning systems described herein can be used in applications other than cart enclosures. For example, the system can be used to estimate the position, path, or speed of a wheeled object. Also, in a cart enclosure application, the cart can include one or more wheels configured to prohibit the cart from moving when activated, such as by including wheel brakes. For example, when the brakes are activated, the wheels can be locked or prevented from rotating. U.S. Patents US 8,046,160, US 8,558,698, and US 8,820,447 describe examples of cart wheels that can prohibit the cart from moving, and the entire disclosures of all of these patents are incorporated herein by reference in their entirety.

[0080] Further descriptions of the functions of system 300 can be found in U.S. Patent No. 9,731,744 or U.S. Patent No. 10,232,869, the entire contents of each of which are incorporated herein by reference.

[0081] Example Anti-Theft System Using Computer Vision

[0082] Figure 4A Schematically shown is an anti-theft system 400 that uses computer vision technology to identify whether an at least partially loaded shopping basket is leaving a store. The system 400 includes a computer vision unit (CVU) 1000, which can be generally similar to that referred to Figure 1CThe described CVU. The CVU 1000 can be located near the surveillance area 440, such as near an entrance / exit, a checkout aisle 34, an area in the store with high-value goods (such as wine, health products, pharmaceuticals), etc.

[0083] In Figure 4A the implementation shown, the CVU 100 communicates with the door manager 450, which in some such implementations can perform the functions of the reference Figure 3 system configuration and control device 390 described. For example, the door manager 450 can communicate with the communication system 330 in the intelligent positioning system 210 and / or the communication system 365 in the intelligent locking wheel 215, and issue anti-theft commands (such as to lock or brake the wheels, activate an alarm or warning, etc.). The door manager 450 can include a central control unit (CCU) (such as, with reference to Figure 1C the one described), or in some cases, can be a component of the CVU 1000, or in some cases, can communicate with the CCU. The door manager 450 can control the fixed feature 385 used at the store exit, such as one or more VLF lines or RF fields that can define a warning area or a locking area (for example, such VLF or RF signals can be detected by a trolley transceiver near the exit). The fixed feature 385 can include a checkout barrier (CB) located, for example, at the exit or the checkout aisle. The door manager (or CVU) can have an autonomous WAN link that uses cellular data services, such as General Packet Radio Service (GPRS), Long Term Evolution (LTE), or 5G New Radio (5G NR) to convey the collected data about store exit events to the CCU or a remote node (such as, with reference to Figure 4B the described cloud platform 470) for analysis and reporting. For example, the remote node can be accessed by authorized store personnel (such as via a web browser), and they can view statistics about exit events (such as theft situations) or images or videos of exit events (such as a video of a shopper trying to push out stolen goods).

[0084] In Figure 4A the system 400 shown, RF fields and VLF lines are used to provide a warning area and a locking area. When a shopping basket crosses the warning area, an unauthorized shopping basket leaving the store can first receive a warning (such as auditory or visual, and for example displayed by the intelligent positioning system 210), and then receive a locking signal (such as an instruction to activate the wheel brake) if the basket crosses the locking area. In other implementations, neither RF fields nor VLF lines are used, or only one of RF fields or VLF lines is used. Similarly, in other implementations, only one of the warning area and the locking area is utilized. Additionally or alternatively, a checkout barrier (CB) can be used.

[0085] The CVU 1000 is capable of communicating with a payment point, such as a checkout cash register 34 or a mobile payment point 35, in order to access payment information related to shopping baskets in the monitored area. As described herein, theft may occur when a shopping basket with items attempts to leave the store without any indication that the customer has paid for the items. Thus, the CVU 1000 is capable of using information from the payment point to at least partially determine whether the items in a loaded shopping basket have been paid for.

[0086] The CVU 1000 can include a camera 410 that is oriented to image the monitored area 440. The camera 410 can include a video camera capable of generating an image set 430 that is used by an image processor 420 to analyze shopping basket activity in the monitored area 440. The image set 430 can include video, one or more frames of the video, or a selection of images acquired by the camera. The camera can include a grayscale camera, a color camera (e.g., RGB), or a camera capable of imaging in a non-visible portion of the electromagnetic spectrum. For example, the non-visible portion can include the infrared (IR) region (which may be advantageous for dark entrances or night imaging, where (optionally) an IR light source can be used to illuminate the entrance) or the ultraviolet (UV) region (which may be advantageous for imaging through glass in / out doors or windows). Using a camera 410 that provides imaging in both the visible and non-visible portions of the electromagnetic spectrum can allow the CVU or CCU to perform multispectral or hyperspectral image analysis, which can better track or classify a cart or merchandise based on the unique spectral reflection characteristics of the cart or merchandise. For example, multispectral imaging can be used to detect theft-prone items based on the specific color of their packaging. Such detection can be performed using a relatively small number of spectral bands (e.g., 7 to 9) under a variety of store lighting conditions and can be implemented by a CMOS imager and a Bayer mask or a set of filters for each spectral band. The camera 410 can include a depth camera that acquires an image and depth data of an object in the image (e.g., the distance from the camera) and can be advantageously used for depth sensing and motion tracking of the basket. The depth camera can include a stereo camera that includes two or more spaced-apart image sensors that determine depth information via stereoscopic techniques.

[0087] In some implementations, the CVU (or component) can be powered by Power over Ethernet (POE). In some implementations, the camera 410 includes a video camera operating at a speed of 20 or more frames per second, providing an image resolution of 4 megapixels or greater (e.g., 1920×1080 or greater), and streaming video using the Real-Time Streaming Protocol (RTSP). The H.264 protocol can be used to compress the video for efficient bandwidth communication. In some implementations, such a camera can be obtained from Hikvision Digital Technology Co., Ltd. (City of Industry, CA).

[0088] The camera 410 can include multiple cameras. For example, the CVU 1000 or the CTU 1001 can include the imaging camera 410, and the system 400 can include one or more secondary cameras 410a spaced apart from the camera 10 in the CVU or CTU. The secondary cameras 410a can be included in the same housing as the CVU or CTU, or can be physically separated from the CVU or CTU. The secondary cameras 410a can be configured to have at least a partially overlapping field of view with the camera 410 (e.g., it can be used for image processing and shopping basket loading classification). The use of one or more secondary cameras 410a can allow the system 400 to track the shopping basket 205 in areas outside the field of view of the camera 410. For example, the (one or more) secondary cameras 410a can be placed near the payment point or the storage area containing high-value items, so that the system 400 can track the movement of the shopping basket 205 in these areas before or after the shopping basket 205 enters the field of view of the camera 410 in the CVU or CTU (e.g., for loading classification). The secondary cameras 410a can be placed near the store exit to enable tracking of the basket near or beyond the exit. The distance between the camera 410 and the secondary cameras 410a can depend on the field of view of these cameras, the lens size, the height above the floor of the retail facility, etc. In various implementations, the secondary cameras 410a can be spaced apart from the camera 410 by a distance in the range of about 10 cm to about 1 m or greater.

[0089] Some or all of the CVU 1000, CTU 1001, or secondary camera 410a are installed near the store exit, as this is the location where thieves will attempt to leave with unpurchased merchandise. In many retail facilities, there is a large amount of glass at the store exit, such as glass doors, glass windows, etc. The use of glass allows sunlight (during the day) to enter the retail facility and can provide a bright and pleasant shopping experience for shoppers. However, sunlight can cause glare, flashes, or reflections from the floor, metal surfaces, and metal shopping carts. Such glare, flashes, or reflections can create artifacts in the images used for motion tracking or computer vision analysis. Therefore, in some implementations, some or all of the cameras 410, 410a can include a polarizing lens or filter 411 to reduce glare, flashes, or reflections in the acquired images. For example, the polarization direction in the lens or filter 411 can be orthogonal to the path of sunlight reflected from the floor of the facility.

[0090] The image processor 420 can include a hardware processor (e.g., a CPU or a graphics processing unit (GPU)) to perform the image analysis and object recognition tasks described herein. In some implementations, the image processor 420 can include an Edge TPU available from Google, Inc. (Mountain View, CA), which supports TensorFlow Lite machine learning and computer vision models.

[0091] The CVU 1000 can include an RF communication node 425 (e.g., a transceiver) to communicate with other components of the system 400 (e.g., the door manager 450, a payment point, or a shopping basket (e.g., the smart positioning system 210 or the locking wheel 215))). The RF communication node 425 can communicate with any component of the system described in the reference Figure 1C and can communicate with any component of the system. Additionally or alternatively to the RF node 425, the CVU can include a wired LAN connection, such as Ethernet. For example, the CVU 1000 can be linked to the CCU via Ethernet.

[0092] In some implementations, the functionality of the CVU 1000 (or CTU 1001) can be provided as a system-on-module (SoM) board that is configured to execute machine learning inference or image classification models and provide wireless connectivity. An example of an SoM board is the Coral Dev Board available from Google, Inc. (Mountain View, CA). The Coral Dev Board includes a CPU, a GPU, an Edge TPU coprocessor for machine learning models, on-board memory, and wireless connectivity (e.g., Bluetooth 4.2, IEEE 802.11b / g / n / ac 2.4 / 5 GHz). In some such implementations, the camera 410 can be connected to the SoM board for a compact setup.

[0093] Figure 4A An example of the camera transceiver unit (CTU) 1001 is also shown. The CTU 1001 can be considered a type of CVU 1000 without the image processor 420, but the other components are generally the same as those described herein. Using the CTU 1001 can provide a more cost-effective device because the image processing function can be offloaded to the CCU or CVU. Thus, a device can include a CCU for image processing or one or a small number of CVUs, while additional CTUs are placed throughout the facility to capture images of entrances, exits, payment points, high-value areas, etc. Generally, the CTU 1001 and the CVU 1000 can be used interchangeably in a device. Thus, it should be understood that references to the CVU include references to the CTU, and references to the CTU include references to the CVU. Thus, the functionality of the system 400 can be distributed among the CCU, CVU, CTU, or gate manager to provide a suitable and cost-effective anti-theft device for any particular facility. Additionally or alternatively, some or all of the image processing (or other program functions) can be performed by a remote hardware processor (e.g., in the cloud). These components can communicate via a wired or wireless LAN or WAN. Many variations are contemplated, and the specific examples and figures described herein are intended to be illustrative rather than restrictive.

[0094] In certain examples, the CVU 1000 or CTU 1001 can include an inertial measurement unit (IMU, e.g., an accelerometer) that can be used to determine whether the CVU or CTU is mounted in a horizontal orientation. Viewing the images streamed from the CVU or CTU can determine that the position, orientation, and focus of the camera 410 are correct. Changes in the IMU readings can indicate that the CVU or CTU has tilted or rotated after installation, and corrective actions can be taken. For example, the CVU or CTU can be physically leveled. Additionally or alternatively, the orientation change of the device can be corrected by compensating for the angular (or rotational) change of the image using computer vision techniques.

[0095] The anti-theft system 400 can include additional sensors 460 to provide additional or different functionality. For example, the additional sensors 460 can include ultrasonic sensors, time-of-flight (ToF) sensors, radar sensors (e.g., millimeter-wave radar), or lidar sensors (scanning or non-scanning). In some cases, such sensors are provided as part of a depth camera that performs both imaging and proximity or ranging functions. The camera 410 of the CVU or CTU can include a depth camera or a pair of stereo cameras for depth functions. The sensor 460 can be used to provide distance data from the sensor to the cart (or merchandise) in the cart. Further description of using such additional sensors 460 to provide three-dimensional (3D) imaging of the shopping basket 205 or merchandise is provided below.

[0096] Figure 4B Another implementation of the anti-theft system 400 is schematically shown. Many of the components of this implementation of the system 400 have been described with reference to Figure 4A This implementation uses a wireless cellular gateway for two-way communication between the CVU 1000 and the cloud platform 470. The cloud platform 470 can be located at a location remote from the facility where the CVU 1000 is located. The cloud platform 470 can process images obtained from CVUs at multiple retail facilities.

[0097] As described above, the CVU acquires an image of the shopping basket 205 within the field of view of its camera 410. The processor 420 of the CVU can execute a machine learning or computer vision object detection model to determine the load status (e.g., empty, at least partially loaded, or full) of the shopping basket attempting to leave the store, and can change the door lock status based on the detection that a cart that is at least partially loaded is attempting to leave the retail store without paying. For example, the door lock status can be changed to activate the wheel locking mechanism 380 of the smart locking wheels 215 of the shopping cart detected to have unpaid merchandise.

[0098] The CVU can locally collect and store images of the shopping basket and communicate the images through the cloud platform 470 for storage and analysis. The CVU and the cloud platform 470 can communicate through an autonomous WAN gateway 465 using, for example, cellular data services such as General Packet Radio Service (GPRS), Long-Term Evolution (LTE), or 5G New Radio (5G NR). The gateway 465 can provide wired or wireless network access to the cloud platform 470 and can be a virtual private network (VPN) over a municipal wireless (e.g., WiFi) network.

[0099] The cloud platform 470 can include a processor and memory for storing and analyzing the images collected by the CVU. For example, a set of images can be labeled at block 472 to provide training data for updating a machine learning or computer vision object detection model used by the CVU. At block 474, the labeled image data can be used to update or generate a new object detection model. The updated model or new model can be communicated back to the CVU via the WAN link 465.

[0100] In some implementations, the cloud platform 470 can provide real-time event detection or real-time streaming 476, where event logs (e.g., a database of images of theft events that were successfully or unsuccessfully identified) can be viewed and analyzed for troubleshooting or to improve the performance of the system 400. The cloud platform 470 can provide a dashboard (e.g., which can be accessed via the Internet), where an authorized retail facility manager or system administrator can view the event logs, access the data labeling or training modules 472, 474, perform system maintenance or upgrades, etc.

[0101] Reference Figures 8 to 11 Additional workflows and processing pipelines that can be (at least partially) performed by the cloud platform 470 are described.

[0102] Example Image Processing Techniques for Anti-Theft Systems

[0103] As reference Figure 4A and Figure 4B As described, the CVU 1000 of the anti-theft system 400 can image the surveillance area 440 and acquire a set of images 430 of an exit event. An exit event can include a shopping basket 205 leaving the store through the exit. In many retail stores, the exit is also the entrance through which shoppers can enter the store with a shopping basket, and in such an implementation, an exit event can include a shopping basket entering or leaving the store (since the camera 410 can typically image the entire exit / entrance area and capture shoppers entering or leaving).

[0104] The set of images 430 is communicated to the image processor 420, which can apply computer vision, machine learning, or object recognition techniques (described herein) to the set of images 430 to perform some or all of the following various image recognition tasks in various implementations.

[0105] The image processor 420 can classify the objects in the image set into one of the following (any of which can be referred to as the loading state of the basket): (a) a shopping basket containing merchandise; (b) a shopping basket not containing merchandise (e.g., the shopping basket is not necessarily empty, e.g., a shopping cart 1620 with an open child seat may still contain a child, a handbag, etc.); (c) other objects other than the shopping basket (e.g., a shopper). The loading state can represent a range of values associated with the loading amount of the shopping basket. For example, the range can be a number (e.g., 1 to 5, where 1 is empty and 5 is full), a grade (e.g., A to E, where A represents full and E represents empty), or some other type of score, discriminator, or semantic classifier, or a probability scale of multiple loading levels (e.g., full, 3 / 4 full, 1 / 2 full, 1 / 4 full, or empty). The loading state can include a weighted score or value indicating the loading amount, as well as an estimated value of the load (e.g., whether the load includes high-value items). For example, the loading state of a basket partially loaded with high-value items (e.g., a wine bottle) may be higher than that of a basket full of bulky and inexpensive items (e.g., paper towels) because the partially loaded basket causes a greater monetary loss to the store.

[0106] The loading state can be determined by computer vision or machine learning techniques as described herein. In some implementations, the loading state can be weighted to reflect the presence of high-value items in the shopping basket (which tends to increase the loading state) or the absence of high-value items in the shopping basket (which tends to decrease the loading state). For example, computer vision techniques or machine learning techniques can be trained to identify the presence of high-value items (e.g., wine bottles) in the shopping basket, and if present, the loading state will increase because if high-value items are present, the value of the merchandise in the shopping basket will tend to be higher in price. As another example, the loading state can represent the presence of high-value merchandise as compared to the presence of other types of merchandise (e.g., low-value merchandise) because it may be advantageous to identify theft situations where the monetary value of the merchandise is greatest.

[0107] The image processor 420 is capable of distinguishing between different types or sizes of shopping carts 30, whether the cart is human-propelled or motorized (e.g., a motorized cart with a shopping basket 205), and whether the subject is a shopper carrying a handheld shopping basket 205. In some implementations, the image processor 420 may not perform facial recognition (or attempt to identify personally identifiable features or information) on individuals in the image to protect the privacy of these individuals. The image analysis performed by the processor 420 (or cloud platform 470) can be set to fully comply with data privacy laws and regulations (e.g., the "California Consumer Privacy Act" or the European Union's "General Data Protection Regulation (GDPR)").

[0108] The image processor 420 is able to distinguish a shopping cart being pushed (or carried) by a store employee rather than a shopper (e.g., by recognizing that the person is wearing a store uniform). This can play a role in theft prevention logic because if a store employee is pushing (or carrying) a loaded basket out of an exit, the likelihood of a push-out theft is greatly reduced.

[0109] The image processor 420 can determine the path of the object (e.g., the change in position over time) during the time period covered by the image set. For example, as referenced Figure 5 As further described, the CVU 1000 can determine the coordinates of an object (e.g., Cartesian x, y coordinates) as a function of time, and calculate the path of the object (e.g., see FIG. 6 ). When the shopping basket 205 is outside the FOV of the CVU's camera 410, the image processor 420 can analyze the images collected by the secondary camera 410a.

[0110] If the determined path of the shopping basket containing merchandise indicates that the shopping basket is heading toward or passing through an exit, the anti-theft system 400 can communicate an anti-theft signal to the shopping basket. As described above, the anti-theft signal can include instructions to lock the smart wheel, activate an alarm (audible or visible), notify store personnel, activate a store video surveillance system, etc.

[0111] In some implementations, after an object in the image set 430 has been classified as an object of interest (e.g., a shopping basket containing merchandise), the action of the anti-theft system 400 (e.g., how to communicate the anti-theft command) can depend on the CVU 1000 (or door manager 450) and the shopping basket (e.g., Figure 3The type of communication between the smart positioning system 210 or the smart locking wheel 215 shown in [figure]. For example, this action can depend on whether the system 400 attempts to use unicast or multicast addressing for the shopping basket. U.S. Patent No. 9,963,162, Cart Monitoring System Supporting Unicast and Multicast Command Transmissions to Wheel Assemblies, describes examples of unicast and multicast command transmission techniques to the smart wheel 215 or the smart positioning system 210, the entire disclosure of which is incorporated herein by reference.

[0112] Unicast Addressing

[0113] In unicast addressing, commands from the anti-theft system 400 can be addressed to a specific shopping basket that has a specific unicast address on the store's tracking network. The CVU 1000 can use a variety of techniques to associate the shopping basket 205 with a specific unicast address.

[0114] For example, each shopping basket can be encoded with its unicast address via an optically readable marker disposed on the shopping basket (or cart), such as a barcode, an Aruco marker, etc. The optically readable marker can encode the unicast address so that the camera 410 can detect it in the visible spectrum or the infrared (IR) spectrum (e.g., IR markers are less intrusive to shoppers and are also less likely to be damaged by potential thieves since they may be invisible to the human eye). The image processor 420 can detect and decode the unicast address of the shopping basket from an image that includes the optically readable marker.

[0115] Additional or alternative techniques can be used to associate the identified shopping basket with its specific unicast address. For example, a retail facility can include a radio positioning infrastructure that can identify an RF transmission as originating from a specific unicast address (e.g., because the RF transmission itself includes the unicast address). The radio positioning infrastructure can detect an RF transmission from the basket (identified by the CVU as an object of interest), so the system 400 can establish an association between the basket and its unicast address.

[0116] The radio positioning infrastructure can include a triangulation system that gives the position of the basket (e.g., x, y coordinates) at a given time. The triangulation system can include a system with an RF receiver that measures the angle of arrival of RF signals transmitted from an RF tag on the basket to estimate the position (e.g., the Smart Location System available from Quuppa LLC, Arlington, VA). The CVU can detect the position or path of the basket in the angle-of-arrival system and, using the knowledge of the position of the RF receiver (which does not need to be co-located with the CVU), the system 400 can associate the position or path estimated by the angle of arrival with the position or path estimated by the CVU to infer the unicast address of the basket. Thus, the system 400 can be integrated with existing location-based services or real-time tracking systems of the facility.

[0117] The radio positioning infrastructure can include a system that measures the radial distance to the basket, for example, via time-of-flight or phase unwrapping, followed by phase slope method. The system 400 can correlate the optical path or position measured by the CVU with the variation of the radial distance over time (from the radial distance measurement node) to infer the unicast address of the basket.

[0118] In some installations, the shopping basket has an internal mechanism that measures its own movement (e.g., the dead reckoning navigation system described in U.S. Patent No. 9,731,744 incorporated herein, such as the Smart Navigation System 210), and the self-detected movement of the basket can be correlated with the path of the basket detected by the CVU1000. This correlation can be used to associate the unicast address of the shopping basket with the basket identified via image processing.

[0119] For example, in some implementations, there is a common time base between the anti-theft system 400 and the shopping basket, and this common time base can be updated using RF synchronization between the clock on the shopping basket and the clock of the anti-theft system 400. For a manually propelled cart, the rotation count of the wheels provides an approximation of the speed of the cart over time. The system 400 can correlate the data of the speed variation over time with the optical path determined by the CVU to provide an association for inferring the unicast address of the cart. As another example, a basket including the Smart Location System 210 can determine the quasi-heading of the basket relative to time (e.g., indoor geomagnetic field distortion), and this quasi-heading can additionally or alternatively be correlated with the optical path determined by the CVU 1000 to infer the unicast address.

[0120] As another technique for associating an optically tracked shopping basket with its unicast address, the shopping basket can be configured to measure environmental characteristics that vary along the path of the basket. The variation of the characteristic with spatial position may be known and can be used to correlate the path of the basket (based on the environmental characteristic) with the path of the basket (optically determined by the CVU) to infer the unicast address of the basket. For example, the environmental characteristic can include the magnetic field in the store. The magnetic field near the level of the wheels can be mapped. The magnetic field map can be used to infer the unicast address of the cart wheels because the wheels can include magnetometers to measure their local magnetic field, which can be compared with the map. The magnetic field map can be determined and the system 400 can be trained using carts with known unicast addresses. Subsequent machine learning techniques can be applied to update the magnetic field map that changes over time due to changes within the store (e.g., movement of ferromagnetic racks).

[0121] The process of associating the unicast address with the shopping basket can begin when the basket enters the monitoring area 440, which can be before the shopping basket has been classified as to its load state (e.g., empty or loaded) or identified as an object of interest by the anti-theft system 400.

[0122] Regardless of the technique used, once the unicast address of a suspect basket is known, the anti-theft system 400 (e.g., the CVU 1000 or the door manager 450) can send a potential theft message to the communication system (e.g., system 330 or 365). In the following illustrative example, the suspect shopping basket is associated with a shopping cart having smart locking wheels 215 (e.g., "cart 2345"). The anti-theft system 400 can send a message to the communication system 330 or 365 such as "Cart 2345, you appear to contain merchandise: If you detect a warning zone or an exit signal but do not have permission to leave, issue a warning and then lock." In this example, the message is directed at a specific suspect shopping basket (associated with cart 2345), and if the cart attempts to leave the store (e.g., by entering a warning zone or a locking zone), the command smart locking wheel indicator 215 (or the smart positioning system 210) is ordered to provide a warning (e.g., in the warning zone) and then lock (if cart 2345 enters the locking zone), unless cart 2345 has permission to leave. If cart 2345 passes through an active checkout lane 34 of the store, or if the merchandise is paid for at a mobile payment point 35, then cart 2345 may have previously obtained permission to leave from the anti-theft system 400. In this case, cart 2345 is allowed to leave the store (without warning or locking) because the merchandise in the cart's basket has (presumably) been paid for.

[0123] In some implementations, the suspect shopping basket is not associated with a wheeled cart and can be carried by hand, for example, by the shopper. Similar considerations apply, but the potential theft message could be a store security alert, store surveillance system activation, etc. (since the hand basket has no locking wheels).

[0124] Multicast Addressing

[0125] In some implementations, the retail store may not have implemented unicast addressing, or the unicast address of a particular object of interest is unknown (e.g., the above unicast association techniques do not provide the unicast address of the object of interest). In such an implementation or situation, the anti-theft system 400 is able to use non-unicast techniques to convey the anti-theft signal. For example, it is possible to use multicast addressing for the shopping baskets near the surveillance area 440. Multicast addressing can be based on a state, where the multicast message is addressed to all shopping baskets in a particular state. For example, the state of a shopping cart wheel can be whether it is locked or unlocked; for example, the state of a shopping basket may be whether it is moving, etc. Thus, a multicast command can be sent to all transceivers whose state is moving or unlocked, etc.

[0126] As an example, if a shopping basket containing merchandise is approaching an exit, where if the basket is not authorized to exit (e.g., has a leave permit) by the anti-theft system, the anti-theft system 400 can infer that there may be a case of push-out theft. In some implementations, the exit configuration of the system 400 is default in the "monitoring" mode, where an image of the exit is obtained by the CVU 1000 and the smart wheel 215 is not locked if it crosses the exit. In this example, since a possible case of push-out theft has been identified, the anti-theft system 400 (e.g., the CVU 1000 or the door manager 450) can switch the exit configuration from the monitoring mode to the "lock if no permit" mode just before the suspect shopping basket is about to leave through the exit. A "lock if no permit" command can be multicast to all transceivers near the surveillance area, and if the suspect shopping cart does not have a leave permit, its smart wheel 215 will be locked to prevent theft. After the wheel is locked, the system 400 can switch the exit configuration back to the monitoring mode.

[0127] Tracking Path of Shopping Basket

[0128] Figure 5 A side view (left) and a plan view (right) of the camera 410 of the computer vision unit 1000 positioned to determine the location of the shopping basket 205 are schematically shown. In this example, the shopping basket 205 is part of the shopping cart 30, but this is for illustration only and does not limit the tracking ability. It is possible in a coordinate system, for example Figure 5Determine the position or path (e.g., the change of position over time) of the basket 205 in the Cartesian x, y, z coordinate system 510 shown. In some implementations, only two horizontal coordinates (e.g., x and y) are tracked (e.g., horizontal movement) because the cart movement typically occurs on a horizontal plane (e.g., at a constant height z).

[0129] The position of the basket 205 can be represented as the center of the basket measured in image coordinates. Briefly, some implementations project from the known position and optical field of view (FOV) of the camera 410 to a plane at a height h determined by the category of the tracked basket (e.g., a full shopping cart has a different height h than an empty shopping cart, and if a given device includes multiple types, the heights of different sizes / models of shopping carts may vary; motorized carts have different heights; hand baskets have different heights).

[0130] Figure 5 The geometry of the imaging environment is shown. The camera 410 is at a height h above the floor (in some cases, the camera is mounted on the ceiling of the facility). The camera has a vertical field of view (vfov) with its center at an angle 0 to the vertical line, and a horizontal field of view (hfov) with its center at an angle θ to the y - direction. The center pixel of the image of the basket 205 can be measured by the image processor 420 at angles 0 and θ. The center pixel of the basket 205 does not have to be at the center of the camera image (even if the camera 410 is steerable). By measuring the angles and θ and using the geometry shown in Figure 5 , the system 400 (e.g., CVU 1000) can convert the angle measurements into position coordinates (e.g., x, y, and (optionally) z).

[0131] Figure 5 An example scenario with one camera 410 is shown. In other implementations, multiple cameras 410, 410a (e.g., 2, 3, 4, 5, 6 or more) can be used to image the surveillance area 440. Figure 6A , Figure 6B and Figure 6C Some example arrangements of the CVU are shown. As described above, the CTU can generally be used interchangeably with the CVU to capture images of the surveillance area, and in other implementations, one, some, or all of the CVUs shown in the figure can be replaced with CTUs.

[0132] In Figure 6AIn [the figure], three CVUs 1000a, 1000b, and 1000c are positioned to image a surveillance area 440 at a store entrance / exit. In this example, an (optional) VLF line is placed at the store entrance / exit. In other devices, additionally or alternatively, an RF warning or lock field setting can be generated by an RF antenna located near the store entrance / exit or checkout barrier, or an EAS tower or other anti-store lifting device can be set up near the exit. CVU 1000a is located away from the entrance / exit and is oriented towards the entrance / exit to obtain images of shopping baskets entering or leaving. CVUs 1000b and 1000c are located on either side of the entrance / exit and are oriented inwards to obtain images of shopping baskets moving especially towards the entrance / exit. The three CVUs 1000a, 1000b, and 1000c provide overlapping coverage of the surveillance area (e.g., the FOVs of their cameras overlap, where the FOVs are schematically shown with dashed and dotted lines). In this configuration, using multiple CVUs can reduce the likelihood that a single CVU will not obtain an image of a suspicious shopping basket attempting to leave through the entrance / exit. Also, a shopper pushing a shopping cart may block a particular CVU's view of a shopping basket. For example, a departing shopper may block CVU 1000a's view of the basket (because the shopper is between the basket and the CVU), but CVUs 1000b and 1000c can see the basket well (because the basket is between the shopper and these CVUs).

[0133] As described above, some implementations can use secondary cameras to image other areas of the store. Figure 6A An example of a secondary camera 410a-1 located near a payment point is shown, whose FOV is directed towards (and partially overlaps) the FOVs of CVUs 1000a and 1000c. Figure 6AAlso shown is an example of a secondary camera 410a-2 located near an area with high-value items, whose FOV points towards the FOVs of CVUs 1000a and 1000b (and partially overlaps with the FOVs of CVUs 1000a and 1000b). Images from camera 410a-1 can be used by anti-theft system 400 to track the path of a shopping basket leaving the payment point and moving towards the exit, and images from camera 410a-2 can be used to track the path of a shopping basket leaving the high-value item area and moving towards the exit. Since the basket comes from the payment point area, system 400 can use the information from camera 410a-1 as an indicator that the customer has paid for the items in the shopping basket. Since the basket comes from the high-value item area, system 400 can use the information from camera 410a-2 as an indicator that there are high-value items in the customer's shopping basket. In this example, because the FOVs of cameras 410a-1 and 410a-2 at least partially overlap with the FOVs of at least some of the other CVUs, system 400 can maintain the continuity of shopping basket tracking as the shopping basket leaves the FOV of one of secondary cameras 410a-1 and 410a-2 and enters the FOV of one or more of the CVUs. This handover can advantageously provide continuity of tracking of the shopping basket and reduce or eliminate misidentification as the shopping basket moves from the FOV of one camera to the next. As will be further described with reference to Figure 6C As further described, this handover can improve the efficiency of system 400 because path tracking is generally less intensive than load state determination, and the CVUs can be mainly used for load state determination rather than tracking.

[0134] Figure 6B An alternative arrangement of CVUs 1000a, 1000b, 1000c is shown. In this example, the orientation of CVU 1000a is similar to Figure 6A that shown. However, the orientations of CVUs 1000b and 1000c are away from the entrance / exit and towards other areas of the store. For example, CVU 1000b is oriented towards the area where high-value items (such as wine, pharmaceuticals, health supplements, etc.) are located, while CVU 1000c is oriented towards the payment point (such as checkout lane 34). In this example, the FOVs of the respective cameras of CVUs 1000a, 1000b, and 1000c do not overlap. In this configuration, CVU 1000a can be used to identify whether a shopping basket approaching the entrance / exit has been loaded. CVU 1000b can be used to identify shopping baskets that have been to the high-value area, while CVU 1000c can be used to identify shopping baskets approaching the exit from the payment point (which can indicate that the items in the basket have been paid for).

[0135] CVUs 1000a, 1000b, and 1000c and secondary camera 410a are able to communicate with each other and share information that can help determine if a partially loaded shopping basket is approaching an exit in a theft situation. This exchanged information can help the anti-theft system continue to track the basket as it moves from the FOV of one of the CVUs to another or from a secondary camera to a secondary camera or CVU when the basket moves from one FOV to another.

[0136] In this example, CVU 1000b can identify if a basket contains high-value items when the basket leaves the high-value item area, and CVU 1000c can determine if the basket is from the payment point area of the store. In this case, the likelihood of not paying for the items in the cart is low, and system 400 can issue a leave permission to the basket. If the basket attempts to leave the store and CVU 1000c has not identified the basket as coming from (or passing through) the payment point, the shopper is more likely to be attempting to steal high-value items, and system 400 may not issue a leave permission to the basket and take anti-theft actions (e.g., lock the wheels).

[0137] Figure 6C Another example arrangement of CVU 1000a and secondary camera 410a near a store exit is shown. In this example, images from CVU 1000a are analyzed to determine the loading state of a shopping basket near the store exit. Images from secondary camera 410a are used to determine the path of the shopping basket as it leaves the FOV of CVU 1000a (shown as a dashed line) and moves through the FOV of camera 410a (shown as a dash-dot line). Because the FOVs overlap at least partially, the anti-theft system 400 can have a high degree of confidence that the shopping basket identified by CVU 1000a is the same basket being tracked by camera 410a. Figure 6C An arrangement of the type shown can be advantageous because the image processing to determine the loading state of the cart performed by CVU 1000a is computationally more complex and processor-intensive than processing images from secondary camera 410a to determine the path of the basket towards the exit. Thus, CVU 1000a is primarily used for loading state determination (e.g., fully loaded, partially loaded, or empty), while secondary camera images are used for path determination, which is a less computationally demanding task.

[0138] Note that although three CVUs are shown in Figure 6A and Figure 6B in Figure 6CA CVU is shown, but this is for illustration only, and other numbers of CVUs (e.g., 2, 4, 5, 6, or more) can be used. Additionally, in some implementations, there is only a single CVU (e.g., having an image processor and an RF communication node), and one or more of the other shown CVUs can be replaced by a secondary camera. The CVUs can be placed at one, some, or all of the store exits, or additionally or alternatively, at other store locations (e.g., in high-value item areas, near payment points, etc.). The CTU can replace some or all of the CVUs. Many variations in the arrangement and orientation of the CVUs, CTUs, or secondary cameras can be envisioned. Also, these configurations of the CVUs (and cameras) are shown as examples to illustrate the various anti-theft situations and store-specific requirements that can be beneficially addressed by the various implementations of the anti-theft system 400, and are not intended to be limiting.

[0139] Various factors can affect the choice of the number and placement of CVUs, CTUs, or secondary cameras in the apparatus of a retail store. These factors can include the FOV of the cameras, CVUs, and CTUs, the ceiling height of the store (where the CVUs, CTUs, and cameras are typically mounted), the typical speed at which shopping baskets move in the area imaged by these components, the distance between the payment point or high-value items and the store exit, the need to maintain substantially continuous tracking of the shopping basket, and the time scale (e.g., about 100 ms) it takes for the system 400 to identify the shopping basket as a theft risk and take anti-theft actions (e.g., locking the shopping cart wheels).

[0140] Three-Dimensional (3D) Imaging

[0141] The 3D image of the shopping basket can provide more information for machine learning or computer vision classifiers to analyze, and can enable the anti-theft system to more accurately or robustly classify the loading state of the shopping basket. The 3D image can include a two-dimensional (2D) image plus distance or depth information in a direction perpendicular to the plane of the 2D image.

[0142] An empty shopping basket has a flat bottom, while at least partially loaded baskets will have items extending above the flat bottom. Thus, non-empty shopping baskets have a 3D topology significantly different from that of empty shopping baskets. This topology can be used at least in part to determine that the basket is non-empty, but can also provide information about the type of items in the shopping basket (e.g., the topology of baby diapers is different from that of wine due to their roughly cubic packaging and bottle-shaped packaging). Thus, in some implementations, the anti-theft system 400 includes sensors that provide depth information. As referenced Figure 4A above, such sensors 460 can include depth cameras, stereo pairs of cameras, ultrasonic sensors, time-of-flight sensors, lidar (scanning or non-scanning), millimeter-wave radars, etc.

[0143] For example, two cameras can be used to form a stereoscopic image of the shopping basket, and stereoscopic imaging technology can be used to obtain depth information. Since cameras, CVUs, etc. are often installed on the ceiling of a retail facility, and the height of the ceiling above the bottom of the shopping basket is in the range of about 3m to 5m, it is estimated that this pair of stereoscopic cameras can be placed about 20cm to 50cm apart to provide sufficient stereoscopic separation. A pair of stereoscopic cameras can be arranged within the housing of a CVU or CTU, or the cameras 410 of a CVU or CTU can be paired with a nearby secondary camera 410a to provide a stereoscopic depth sensing function.

[0144] A time-of-flight (ToF) sensor (sometimes called a time-of-flight array or non-scanning lidar) can be used with an imaging camera to provide a 3D image of the shopping basket. The ToF sensor can include a laser for emitting light pulses and a timing circuit for measuring the time between the emission and the light signal reflected back from an object. The measured time (and the speed of light) provides the distance to the object. An integrated ToF camera module that combines a ToF sensor and an imaging camera is available from Analog Devices, Inc. (Norwood, MA). Analog Devices also offers the ADDI9036 CCD ToF signal processor and the AD-96TOF1-EBZ 3D ToF development platform.

[0145] In some implementations, additionally or alternatively to an optical ToF sensor, an acoustic ultrasonic sensor can be used to provide distance / depth information. For example, an ultrasonic ranging sensor with a high frequency (e.g., greater than 100 kHz) can be aimed with the line of sight of an imaging camera. The ultrasonic sensor can have a relatively narrow field of view (providing sufficient resolution for the objects in the basket) and a high enough pulse repetition rate (providing sufficient temporal resolution due to the movement of the shopping basket). The high-frequency structure of the reflected ultrasonic pulses is different when scattered from the bottom of the shopping basket (usually metal or plastic mesh) compared to when scattered from the surfaces of the goods in a non-empty shopping basket. The ultrasonic signal can be used to simply detect whether there are items in the shopping basket or (with a sufficiently narrow FOV) to identify the depth profile of the items in the shopping basket.

[0146] In some implementations, radar sensors can be used to provide depth sensing capabilities. An example of a radar sensor is the RETINA 4D image radar available from Smart Radar System (Gyeonggi-do, Korea), which provides 3D position information and velocity. The RETINA radar can generate a 4D point cloud, and the point cloud can be analyzed by a neural network to identify objects within the point cloud.

[0147] 3D images can be used to train machine learning or computer vision models, and the additional depth information (compared to 2D images) can help provide a more accurate or robust estimate of the loading state of the shopping basket. Furthermore, because different types of items will have different depth features, the machine learning or computer vision model can learn to distinguish between different types of items and can be configured to incorporate this type of information into the loading state (e.g., a higher loading state for a basket containing wine bottles than for a basket containing produce).

[0148] Identifying Children in Shopping Cart

[0149] As reference Figure 2B As described above, the shopping cart 30 may include a child seat 1620 in which a shopper can place a child. The child may be placed in the shopping basket 205 itself. A thief may use a child to hide items, distract attention, or disguise his or her intentions (e.g., a parent with a child is less likely to be considered a thief). Therefore, some implementations of the anti-theft system 400 may be configured to determine whether a child (including an infant) is present in the shopping cart.

[0150] Images (2D or 3D) obtained by the system 400 (e.g., from a CVU, CTU, or secondary camera) can be analyzed to make this determination. Items placed in a cart (in a shopping basket 205 or child seat 1620) tend not to move relative to the cart, whereas children do tend to move relative to the cart (e.g., move their heads, arms, or legs, rock their bodies, etc.). Thus, a discriminant of whether an object in the cart is a child is whether the object moves relative to the cart. A time series of images (2D or 3D) can be analyzed to determine which objects, if any, move relative to the translational or rotational motion of the cart itself. For example, the position of an object relative to a fixed location in the cart (e.g., relative to the handle 1610, relative to the sidewalls of the basket, etc.) can be compared over the duration of the time series to distinguish whether the object is moving relative to the cart and, therefore, whether the object is likely to be a child. In the case of 3D images, depth information can provide a discriminant for children because the depth features of a child (head, arms, torso, legs) are different than typical retail store merchandise.

[0151] Example Path and Potential Theft of Shopping Basket

[0152] Figure 7Schematically shows an example of the path taken by a shopping basket near the entrance / exit of a retail store. The empty shopping baskets shown are not hatched, and the at least partially loaded baskets are shown hatched. In this illustrative example, the shopping basket is part of shopping carts 30a - 30e, but this is for illustration only and not a limitation. The symbols on the path taken by the shopping basket (shopping cart in this example) indicate the possibility of push-out theft, as will be further described below. The path taken by shopping carts 30a - 30e can be determined by the CVU as described above. For example, Figures 6A - 6C the configuration of the CVU shown in can be adapted to image the store entrance / exit, the areas where high-value items are located, and the areas where payment points are located.

[0153] In Figure 7 the paths of carts 30a - 30e are annotated with symbols that graphically display attributes and theft possibilities. Circles are used for incoming shopping carts, with an empty circle indicating an empty shopping cart and a solid circle indicating an at least partially loaded shopping cart. Squares are used to indicate outgoing shopping carts, with an empty square indicating an empty shopping cart and a solid square indicating an at least partially loaded shopping cart. A departing cart with a high theft potential is shown as a solid four-pointed star.

[0154] Cart 30a is entering the store and is determined to be unloaded. The path of cart 30a is marked with an empty circle. Cart 30b also enters the store, and the CVU determines that the cart is at least partially loaded. This could be due to a child or a handbag in the open child seat, or due to items stored outside the store that the shopper has placed in the shopping basket. Carts 30d and 30e are leaving the store. Cart 30e is unloaded and has no theft potential.

[0155] The cart 30d is loaded, but since it is approaching the exit from the direction of the payment point, there may be no possibility of theft for the cart 30d. To further confirm the payment status of the shopping cart 30d, the anti-theft system 400 can query the payment point (e.g., the cash register 34 or the mobile payment point 35) to determine whether the cart 30d has actually passed through an active checkout lane (e.g., spent enough time on the lane to indicate payment), or used a mobile payment application to pay for the items in the cart basket. If so, the system 400 can issue a leave permission to the cart 30d. In an implementation using a checkout barrier (CB), the command can be to unlock the CB to allow leaving. If there is no further payment mark, the cart 30d may attempt to simulate payment by coming from the direction of the payment point, or quickly pass through the payment point without spending enough time to indicate that payment has occurred, and the system 400 can issue an anti-theft command to the transceiver of the cart 30d. Assuming the cart 30d is coming from the direction of the payment point, this command can be a warning command (instead of a locking command). In an implementation using a checkout barrier (CB), the command can be to keep the CB locked to prevent the cart 30d from leaving. The anti-theft system 40 can issue a command to the store personnel to arrive at the associated CB or exit to determine whether the shopper has actually paid for the goods.

[0156] It is determined that the cart 30c has been loaded and is approaching the exit from the direction where the high-value items are located rather than from the direction of the payment point. The cart 30c represents an increased likelihood of theft and is marked with a solid star. The anti-theft system 400 can query the payment point to determine whether the cart 30c has passed through the payment point previously. If so, the cart 30c may have subsequently entered the high-value item area and placed high-value items in the shopping cart basket without paying. The system 400 can refuse to issue a leave permission to the cart 30c (or refuse to open the checkout barrier), but instead issue a locking command to the shopping cart 30d or an alarm to the store personnel to try to prevent the goods from being stolen from the store.

[0157] The foregoing examples are intended to illustrate rather than limit the logic that can be performed by the implementation of the anti-theft system 400 using the image processing capabilities of the CVU.

[0158] Although Figure 7 the symbols shown in Figure 7 are generally intended for description, in some implementations, the images (e.g., often videos) acquired by the CVU can be overlaid (or annotated) with markings similar to those shown in Figure 7the symbols shown in

[0159] Example Machine Learning Techniques for Anti-Theft Systems

[0160] In some implementations, the CVU 1000 (e.g., the image processor 420) implements one or more object recognizers that can crawl through the received data (e.g., the acquisition of an image) and recognize or draw points, tag the image, attach semantic information to the object (e.g., shopping cart, motorized mobility cart, shopping basket, empty, loaded, etc.), and so on.

[0161] The object recognizer can recognize shopping baskets, shopping carts, motorized shopping carts or mobility carts, items and merchandise within the shopping basket (which can include a lower basket at the bottom of the shopping cart), the presence of an object in the child seat of the cart (e.g., a child or a handbag), markers associated with the user pushing or operating the cart (e.g., a particular style of shirt (e.g., plaid or striped), particular pants, skirt, jacket or hat), user characteristics (e.g., facial features, body features), etc. The object recognizer can recognize store personnel, for example, by recognizing that the person is wearing a store uniform, identifying tags, etc. In some implementations, for privacy purposes, the object recognizer does not recognize facial or body features.

[0162] The object recognizer can recognize entrances / exits, checkout aisles or other objects in the store. One or more object recognizers can be specialized to recognize objects with certain characteristics. For example, one object recognizer can be used to recognize shopping baskets, while another object recognizer can be used to identify items or merchandise within the shopping basket, and another object recognizer can be used to recognize characteristics of the user associated with the shopping basket, and so on.

[0163] By analyzing multiple sequential images (e.g., frames from a video), the object recognizer can determine the path of the basket when the shopping basket enters or leaves the store. In some cases, the object recognizer can classify shopping cart behavior, such as entering or exiting, rather than determining the path (or in addition to determining the path).

[0164] Object recognition can be performed using a variety of computer vision techniques. One or more computer vision algorithms can be used to perform these tasks. Non-limiting examples of computer vision algorithms include: Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoint (FREAK), Viola-Jones algorithm, Eigenface method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean Shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), Bundle Adjustment, Adaptive Thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), Histogram of Feature Points, various machine learning algorithms (e.g., Support Vector Machine, Relevance Vector Machine, k-Nearest Neighbor Clustering algorithm, Naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.), and so on.

[0165] Object recognition can additionally or alternatively be performed by various machine learning algorithms. After training, the machine learning algorithms can be stored by the CVU 1000 (e.g., the image processor 420). Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., Ordinary Least Squares regression, Logistic Regression (e.g., Logit model)), instance-based algorithms (e.g., Learning Vector Quantization), decision tree algorithms (e.g., Classification and Regression Trees), Bayesian algorithms (e.g., Naive Bayes), clustering algorithms (e.g., k-Means clustering), association rule learning algorithms (e.g., Apriori algorithm), artificial neural network algorithms (e.g., Perceptron), deep learning algorithms (e.g., Deep Boltzmann Machine or deep neural network), dimensionality reduction algorithms (e.g., Principal Component Analysis), ensemble algorithms (e.g., Stacking generalization), and / or other machine learning algorithms.

[0166] The machine learning model can include neural networks, such as convolutional neural networks, recurrent or cyclic neural networks, stacked autoencoders, etc. The neural network can include a deep neural network having many layers (e.g., greater than 3 layers, 5 layers, 10 layers or more). The neural network can include convolutional layers, pooling layers, fully connected layers, classifier layers (e.g., soft-max), activation functions (e.g., Rectified Linear Unit), loss layers, etc. Supervised or unsupervised learning techniques can be used to learn the weights in the neural network.

[0167] Separate machine learning models can be customized for individual applications or installations. For example, the CVU 1000 is capable of storing a default model for analyzing shopping basket images. The default model can be used as a starting point for generating additional models specific to the conditions at the CVU installation. For example, when installed in a specific store with specific exits, the object recognizer can learn the specific characteristics of the exits that the CVU is monitoring, and the specific characteristics of the shopping baskets, shopping carts, etc. used in that specific retail store. For example, the CVU can use supervised or unsupervised training techniques applied to images acquired after installation to update computer vision, machine learning, or object recognition algorithms. Thus, the default computer vision, machine learning, or object recognizer can be specific to the particular environment in which it analyzes images. The computer vision, machine learning, or object recognizer can continue to learn over time and can become more efficient and accurate in its object recognition tasks.

[0168] In some implementations, machine learning techniques can use TensorFlow TM or TensorFLow Lite (available from www.tensorflow.org) or use Torch TM (available from torch.ch, or the Python implementation, PyTorch TM , available from pytorch.org), each of which provides an open-source software library for programming machine learning applications, such as image recognition using neural networks. The neural network can include a convolutional neural network (CNN) and can be a deep network (e.g., an artificial neural network including 3 or more layers, where each layer is trained to extract one or more features of an image). In some implementations, one or more fully connected (layers), support vector machines (SVMs), softmax layers, or other types of classification layers can follow the convolutional layer. For example, the output of the neural network can be a classification, such as the load state of a shopping basket. As described herein, the load state can indicate that the shopping basket is loaded or unloaded, or is empty, partially loaded, or fully loaded, a fraction, a numerical range, etc. As another example, the output of the neural network can be a classification: the shopping basket is entering the store (e.g., this does not indicate a potential theft event) or leaving the store (e.g., this indicates a potential theft event).

[0169] Machine learning techniques can be trained via supervised or unsupervised learning methods. For example, training images of shopping baskets entering or leaving a store exit can be obtained (e.g., from a CVU, CTU, or store monitoring system) and classified with a load state, such as empty, partially loaded with goods, or fully loaded with goods. In some cases, the training images are segmented into regions that include the front of the shopping basket and do not include the child seat 1620 (see Figure 2B)。This can advantageously reduce training or classification problems, as machine learning techniques do not need to process images of children (or handbags or other non-merchandise items) that may be included in the child seat 1620.

[0170] Semantic information such as the type of shopping basket (e.g., human-powered cart, motorized cart, hand basket), the shopping basket entering or leaving the store, the presence of high-value items in the basket, etc. can also be categorized in the training images. In some implementations, the training images are manually annotated / classified. This training data can be learned by machine learning techniques (e.g., a convolutional neural network with a fully connected layer classifier) on how to analyze and classify new images. The training can be performed by the cloud platform 470 described with reference to Figure 4B FIG.

[0171] During the use of the anti-theft system 400, images of the shopping basket can continue to be obtained and classified / annotated, and these images can be used for further training or updating of the machine learning techniques. Images of the shopping cart path can be examined or analyzed (e.g., see Figure 7 ) to improve the ambiguous classification or recognition of loaded or unloaded carts. Cart position or path data obtained using, for example, RF technology can be compared with position or path data determined by image processing techniques to improve the position or path determination of the system (e.g., using machine learning training methods).

[0172] Thus, the machine learning techniques can be specialized over time for the actual retail store environment in which the anti-theft system 400 is installed, which can advantageously lead to improved accuracy, efficiency, or performance in recognizing potential theft situations.

[0173] Example Processing Pipelines and Workflows for Anti-Theft Systems

[0174] Figure 8 FIG. shows an example of a processing pipeline for training a machine learning (ML) model. This pipeline can be, for example, with reference to FIGS. 1, Figure 4A and Figure 4BThe cloud platform 470 of the described anti-theft system 400 executes. The pipeline receives image data collected from multiple CVUs, CTUs, secondary cameras, store surveillance systems, etc. The image data can be from a single retail facility or multiple retail facilities. The pipeline can be accessed or controlled via an ML training application (app) 800, which can be a web-based interface to the cloud platform 470. For example, a system administrator can use the ML training app to start training an ML model, look up information about the CVU status (e.g., from a database 810 of CVU information), and communicate the updated ML model to a specific CVU or CVUs. Since the received image data can be from one or more specific stores, the trained ML model can be customized for that (or those) specific store. For example, some stores may use shopping baskets with plastic nets (instead of wire meshes), and the ML model can be trained on images from those stores to not only better recognize baskets with plastic net structures but also better recognize the store merchandise placed in such baskets.

[0175] At Figure 8 Point 1 of the illustrated pipeline, the image data set is uploaded from one or more CVUs to the cloud platform 470 via the WAN gateway 465. At point 2, the notification platform receives the new image data set. At point 3, the image data set is prepared and submitted for data labeling (which can be performed by a human classifier). For example, shopping baskets can be identified in the images, and the shopping baskets can be classified with reference to the loading status. At point 4, the labeled imaging data set can be used to train an ML model. The training can be used to generate a new ML model or update an existing ML model.

[0176] At point 5 of the pipeline, the ML app 800 can be used to select CVUs with labeled data for training or provide other control instructions to the cloud platform 470. At point 6, an instance of ML training can be initiated and executed by a computing engine in the cloud platform 470. The computing engine can train a new ML model or an updated ML model, and at point 7, the trained model can be saved. At point 8, the CVUs can be notified that a new or updated ML model is available, and at point 9 of the pipeline, the trained model can be downloaded by the CVUs.

[0177] Figure 9 An example of a processing pipeline for analyzing images obtained from an anti-theft system is schematically shown. The pipeline can be referenced by Figure 4BExecute using the described cloud platform 470. The pipeline receives image data collected from multiple CVUs, CTUs, secondary cameras, store surveillance systems, etc. The image data can be from a single retail facility or multiple retail facilities. As described above, the image data generally includes video of a possible theft event in a retail store. At points 1a and 1b of the pipeline, image data from a CVU, CTU, secondary camera, or other store surveillance system is periodically uploaded to the cloud platform 470 via, for example, the WAN gateway 465 and stored in cloud storage. At point 1c, the load status detection event determined by the CVU can additionally or alternatively be uploaded to the cloud platform 470. The load status detection event can include a timestamp of the event and the determined load status (e.g., empty, partially loaded, fully loaded, etc.) of the shopping basket involved in the event. The image data can include annotations, such as bounding boxes around the shopping basket or cart involved in the event, around the goods in the cart, around the customer using the shopping basket, etc. At point 2, the cloud platform 470 can be notified of a new upload.

[0178] At points 3a and 3b of the pipeline, the cloud platform 470 can analyze the image data to determine, for example, image metadata and store the metadata in a cloud database 810 (e.g., a Structured Query Language (SQL) database). The metadata can include inferred metadata determined from CVU detection events. The inferred metadata can include, for example, the location of the shopping basket in the image frame, the time of the event, how much time it took to perform the load status determination, a confidence value related to the confidence that the ML model can correctly infer the load status from the image data, the ML model weights and parameters used in the image analysis, etc. The metadata can also include image metadata that associates the image data of the theft event obtained from the CTU, secondary camera, or store surveillance video (which may not be relevant to the load status determination) with the image data of the event obtained from the CVU (which will be associated with the load status determination). The image metadata can include information about the correlation between the image data from the CTU, secondary camera, or store surveillance video and the image data from the CVU.

[0179] Figure 10 An example of a processing pipeline for real-time event detection or live streaming from an anti-theft system is schematically shown. The processing pipeline can be used to provide reference Figure 4B The described real-time event detection or live stream 476. The processing pipeline can be managed using the ML app 800 described by reference Figure 8 Since the real-time streaming of image data may utilize a large amount of bandwidth of the WAN gateway 465, the live streaming function is only activated when needed, for example, by a system administrator or an authorized on-site service personnel or retail store personnel to perform diagnostics, testing, installation, or maintenance of the anti-theft system 400.

[0180] At point 1 of the pipeline, live detection is enabled, and at point 2, a live stream server is started. At point 3, image data from the CVU, CTU, secondary cameras, or store surveillance system is streamed via the WAN gateway 465 to the cloud platform 470. The ML app 800 can include a stream player 840 that can display the streamed image data. In various implementations, the WAN gateway 465 can implement one or more streaming protocols, such as, Real-Time Streaming Protocol (RTSP), HTTP Live Streaming (HLS), or Real-Time Messaging Protocol (RTMP).

[0181] Figure 11 An example of a pipeline for business intelligence (BI) analysis of image data from the anti-theft system 400 is schematically shown. At point 1 of the pipeline, an ETL (Extract, Transform, Load) process can be used to access the database 810 and run the required BI queries on the image data and metadata stored by the cloud platform 470. At point 2 of the pipeline, the image data can be downloaded from the cloud platform to the stream viewer 840, which can be a component of the ML app 800 for interacting with the cloud platform 470. As described above, the cloud platform 470 can store image data (e.g., video) of possible checkout theft events in a retail store. The image data can include a video 852 of a checkout theft event obtained from the CTU, secondary cameras, or store surveillance video (which may not be relevant to load status determination) and a video 854 of a checkout theft event obtained from the CVU (which will be associated with load status determination, such as whether the shopping basket is empty, partially loaded, or full). Authorized store personnel or system administrators can view the checkout theft videos, run analyses on the image data and metadata, etc. For example, as shown in the example of Figure 11 , the surveillance video 852 shows that a checkout theft is occurring, and the associated CVU video of the event identifies the shopping basket as full. This example represents a successful checkout theft detection by the anti-theft system (indicated by the checkmark in Figure 11 ). The BI queries can include information about system-generated false positives (e.g., innocent events misidentified as theft) or false negatives (e.g., checkout thefts misidentified as innocent), statistical information about frequency, time of occurrence, or the estimated loss amount of checkout thefts, etc.

[0182] Referring to Figures 8 to 11 the example pipeline described is intended to be illustrative and not exclusive. The cloud platform 470 can be configured to perform the functions of one, some, or all of these pipelines in various implementations.

[0183] Figure 12FIG. 0 schematically shows an example of a processing pipeline in the CVU 1000. The processing pipeline can be executed at least in part by the processor 420 of the CVU. Many functions of the CVU 1000 have been described above and will not be repeated here. In this example pipeline, the CVU is configured to provide a loading state of the shopping cart 30, which includes a shopping basket 205 for storing goods. In this example, for illustrative purposes, the loading state is empty or full, but as described herein, a more general loading state can be provided. The processor 420 is programmed to execute a classification engine 1202 and a detection engine 1204.

[0184] At point 1a of the pipeline, image data (e.g., video of a monitored area of a store) is sent to the classification engine 1202, which executes an ML rollout classification model. The classification model can be trained to identify the presence (or absence) of a cart in the image and the location of the cart in the image. The cart classification model can segment the image to identify the shopping basket portion of the cart. Since the image typically includes things other than the cart, the classification model can also classify the image as to whether it contains a person (e.g., a shopper pushing the cart or a store employee or a child in the cart) or other living objects (e.g., service animals). At point 2 of the pipeline, the classified image (and the classification metadata determined by the classification engine) can be stored in the cart image dataset. The classified image can be annotated with a bounding box around the objects classified in the image (e.g., cart, basket, shopper, etc.). In some cases, images that do not contain a shopping cart are not stored, which helps reduce memory usage.

[0185] At point 1b of the pipeline, the image data is conveyed to the detection engine 1204, which executes an ML detection model to determine the loading state of the cart (or basket). The ML classification model can be different from the ML detection model, and the ML detection model can advantageously allow each of these ML models to be trained for its respective task. In other implementations, the ML classification model and the ML detection model are part of the same ML model, and since there is an overlap between the classification and detection tasks, it can advantageously allow the ML model to be trained in an integrated manner. In some implementations, the classification engine 1202 is executed before the detection engine 1204. If the classification engine 1202 determines that there is no cart in the image, the detection engine 1204 may not be executed, which advantageously saves power and processing cycles and improves efficiency. In other implementations, the classification engine 1202 acts as a preprocessor and only executes the detection engine 1204 when a cart is detected in the image. This also advantageously saves power and processing cycles and improves efficiency.

[0186] At point 3 of the pipeline, the loading state (e.g., "FULL" or "EMPTY" in this example) is communicated to the door manager 450 (e.g., as described with reference to Figure 1A , Figure 1B and Figure 4A ) to take appropriate anti-theft actions on the cart. For example, if the detection engine determines that the cart is FULL, the enabled lock state can be communicated to the door manager 450, and the door manager 450 will communicate a lock command to the intelligent lock wheel 215 of the cart. If the detection engine determines that the cart is EMPTY, the disabled lock state can be communicated to the door manager 450, and the door manager 450 may not take any action to actuate the wheel lock or may communicate an unlock command to the intelligent lock wheel 215 of the cart.

[0187] At point 4 of the pipeline, detection metadata (e.g., loading state) can be communicated to the cloud platform 470.

[0188] Figure 12 The processing pipeline schematically shown in Figures 8 to 11 can be used in various implementations of the CVU 1000, as described in the pipeline schematically shown in

[0189] Additional Aspects

[0190] Aspect 1. An anti-theft system includes: a computer vision unit (CVU) configured to image an area of a facility, the CVU including: a camera; a radio frequency (RF) communication node; and an image processor; a manually propelled wheeled cart including: a basket configured to hold merchandise; wheels including brakes configured to prohibit movement of the cart when the brakes are actuated; and an RF cart transceiver configured to communicate with the RF communication node and the brakes of the CVU, wherein the image processor is programmed to analyze an image of the facility area obtained by the camera to: determine that the basket of the cart is at least partially loaded with merchandise; and determine that the cart is attempting to leave the area of the facility, wherein the RF communication node is configured to: communicate a command to the RF cart transceiver to actuate the brakes of the wheels.

[0191] Aspect 2. The anti-theft system according to aspect 1, wherein the CVU is further configured to: communicate with a payment point of the facility; and receive an indication that the merchandise in the basket of the cart has not been paid for from the payment point, wherein the indication is received before communicating a command to the RF cart transceiver to actuate the brakes of the wheels.

[0192] Aspect 3. The anti-theft system according to aspect 1 or aspect 2, wherein the image processor is programmed to apply a neural network to the image obtained by the camera.

[0193] Aspect 4. The anti-theft system according to any one of aspects 1 to 3, wherein the image processor is further programmed to determine the path of the trolley in the area of the facility.

[0194] Aspect 5. The anti-theft system according to any one of aspects 1 to 4, wherein the CVU is further programmed to store the image of the area in a remote non-transitory computer storage medium.

[0195] Aspect 6. The anti-theft system according to any one of aspects 1 to 5, wherein the camera, the RF communication node and the image processor are arranged in a housing configured to be mounted to a structure in the facility.

[0196] Aspect 7. The anti-theft system according to any one of aspects 1 to 5, wherein the camera and the RF communication node are arranged in a housing configured to be mounted to a structure in the facility, and the image processor is arranged away from the housing.

[0197] Aspect 8. The anti-theft system according to any one of aspects 1 to 7, wherein the facility includes a retail store and the manually propelled trolley includes a shopping trolley.

[0198] Aspect 9. The anti-theft system according to aspect 8, wherein the area includes a store entrance, a store exit, a checkout aisle, a payment point or an area for storing high-value goods.

[0199] Aspect 10. A method for reducing theft of goods in a retail store, the method comprising: under the control of an anti-theft system including computer hardware: obtaining an image of an area of the retail store; identifying from the image whether a shopping basket exists in the area; determining from the image a loading state indicating whether the shopping basket is at least partially loaded with goods; receiving payment information indicating whether the goods in the shopping basket have been paid for; and communicating an anti-theft command at least in part based on the loading state and the payment information.

[0200] Aspect 11. The method according to aspect 10, wherein the identification or the determination operation is performed using a neural network.

[0201] Aspect 12. The method according to aspect 10 or aspect 11, further comprising determining from the image the path of the shopping basket in the area.

[0202] Aspect 13. The method according to any one of aspects 10 to 12, wherein receiving the payment information includes: obtaining a second image of the payment point; from the second image, determining whether the shopping basket has passed through the payment point, whether it has spent more than a threshold time near the payment point, whether it has interacted with a store attendant, or whether it has accessed a payment system at the payment point.

[0203] Aspect 14. The method according to any one of aspects 10 to 13 further includes determining a unicast address associated with a radio frequency (RF) receiver, wherein the radio frequency (RF) receiver is associated with the shopping basket.

[0204] Aspect 15. The method according to any one of aspects 10 to 14, wherein communicating the anti-theft command includes: communicating with a transceiver associated with the shopping basket, communicating with a checkout barrier, communicating with a brake associated with a wheel to which the shopping basket is associated, or communicating with a video surveillance system of the retail store.

[0205] Aspect 16. The method according to any one of aspects 10 to 15, wherein the anti-theft command includes a command for locking or braking a wheel associated with the shopping basket, a command for activating an alarm or warning, or a command to store personnel that theft is occurring.

[0206] Aspect 17. The method according to any one of aspects 10 to 16, wherein the shopping basket is associated with a wheeled, human-propelled shopping cart.

[0207] Aspect 18. The method according to aspect 17, wherein the shopping cart includes wheels having brakes, and the anti-theft command includes a command to actuate the brakes.

[0208] Aspect 19. The method according to any one of aspects 10 to 16, wherein the shopping basket is associated with a handheld shopping basket.

[0209] Aspect 20. The method according to any one of aspects 10 to 19 further includes: classifying images of areas of a retail store to label the shopping basket or the loading state of the shopping basket to provide a set of training images; and training a machine learning algorithm using the set of training images.

[0210] Aspect 21. An anti-theft system for a retail facility, the anti-theft system including: a camera configured to image an area of the retail facility; a hardware circuit in communication with the camera, the hardware circuit configured to: analyze an image of the area obtained by the camera with a machine learning (ML) classification model to determine the presence of a shopping basket in the image; in response to determining the presence of the shopping basket in the image, analyze the image of the area with an ML detection model to determine the loading state of the shopping basket in the image; and communicate a signal related to the loading state of the shopping basket to a shopping basket enclosure system.

[0211] Aspect 22. The anti-theft system according to aspect 21, wherein the camera includes a plurality of cameras.

[0212] Aspect 23. The anti-theft system according to aspect 22, wherein the plurality of cameras are configured to obtain stereoscopic images of the area.

[0213] Aspect 24. The anti-theft system according to any one of aspects 21 to 23, wherein the plurality of cameras includes a first camera having a first field of view (FOV) and a second camera having a second FOV, and the first FOV and the second FOV at least partially overlap.

[0214] Aspect 25. The anti-theft system according to aspect 24, wherein an image of the area analyzed by the hardware circuit using the ML classification model and the ML detection model is acquired by the first camera, and the hardware circuit is further configured to analyze an image obtained by the second camera to determine a path of the shopping basket passing through the area.

[0215] Aspect 26. The anti-theft system according to any one of aspects 21 to 25, wherein the camera includes a polarizing lens or a polarizing filter.

[0216] Aspect 27. The anti-theft system according to any one of aspects 21 to 26, wherein the system is configured to obtain depth information from a depth camera, a lidar sensor, or an optical or ultrasonic time-of-flight sensor.

[0217] Aspect 28. The anti-theft system according to aspect 27, wherein the hardware circuit is configured to analyze the depth information.

[0218] Aspect 29. The anti-theft system according to any one of aspects 21 to 28, wherein the ML classification model and the ML detection model are different ML models.

[0219] Aspect 30. The anti-theft system according to any one of aspects 21 to 29, wherein the loaded state includes a numerical range, a rank, or a score.

[0220] Aspect 31. The anti-theft system according to any one of aspects 21 to 30, wherein the loaded state includes empty, at least partially full, or full.

[0221] Aspect 32. The anti-theft system according to any one of aspects 21 to 32, wherein the shopping basket is part of a wheeled shopping cart, and the wheeled shopping cart has wheels that include brakes and are configured to receive a braking signal from a basket enclosure system.

[0222] Aspect 33. The anti-theft system according to any one of aspects 21 to 32, wherein the basket enclosure system is separate from the anti-theft system.

[0223] Aspect 34. The anti-theft system according to any one of aspects 21 to 33, wherein the hardware circuit is configured to receive payment information indicating whether the goods in the shopping basket have been paid for.

[0224] Aspect 35. The anti-theft system according to aspect 34, wherein the signal associated with the loaded state is further based on the payment information.

[0225] Aspect 36. An anti-theft system includes: a computer vision unit (CVU) configured to image an area of a facility, the CVU including: a camera; a radio frequency (RF) communication node; and an image processor; and an access point device configured to communicate with the RF communication node of the CVU and with an RF transceiver in a human-powered wheeled cart, the wheeled cart including: a basket configured to hold merchandise; wheels including brakes configured to prohibit movement of the cart when the brakes are actuated, the brakes communicating with the RF transceiver, wherein the image processor is programmed to analyze an image of the facility area obtained by the camera to: determine the loading state of the basket of the wheeled cart; and determine that the cart is attempting to leave the area of the facility, and wherein the RF communication node of the CVU is configured to: convey a potential theft command to the access point.

[0226] Aspect 37. The anti-theft system according to aspect 36, wherein the loading state includes a classification, a numerical range, or a score.

[0227] Aspect 38. The anti-theft system according to aspect 36 or aspect 37, wherein the classification includes an indication that the basket is empty, or an indication that the basket is partially loaded, or an indication that the basket is fully loaded.

[0228] Aspect 39. The anti-theft system according to any one of aspects 36 to 38, wherein the system is configured to: communicate with a payment point of the facility; and receive an indication from the payment point that the merchandise in the basket of the cart has not been paid for.

[0229] Aspect 40. The anti-theft system according to any one of aspects 36 to 39, wherein the image processor is programmed to apply a neural network to the image obtained by the camera.

[0230] Aspect 41. The anti-theft system according to any one of aspects 36 to 40, wherein the image processor is further programmed to determine the path of the cart in the area of the facility.

[0231] Aspect 42. The anti-theft system according to any one of aspects 36 to 41, wherein the facility includes a retail store and the human-powered cart includes a shopping cart.

[0232] Aspect 43. The anti-theft system according to aspect 42, wherein the area includes a store entrance, a store exit, a checkout aisle, a payment point, or an area for storing high-value merchandise.

[0233] Aspect 44. The anti-theft system according to any one of aspects 36 to 43, wherein the camera includes a depth camera or a stereo camera.

[0234] Aspect 45. The anti-theft system according to any one of aspects 36 to 44, wherein in response to receiving a potential theft command, the access point is configured to transmit an anti-theft signal to the RF transceiver of the cart.

[0235] Aspect 46. The anti-theft system according to aspect 45, wherein the anti-theft signal includes a signal for actuating the brake.

[0236] Other Information

[0237] The various illustrative logical blocks, modules, and processes described herein can be implemented or performed by a machine, such as a computer, a processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination designed to perform the functions described herein. The processor can be a microprocessor, a controller, a microcontroller, a state machine, a graphics processor, a tensor processor, combinations thereof, etc. The processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors or processor cores, one or more graphics or stream processors, a combination of one or more microprocessors and a DSP, or any other such configuration.

[0238] In addition, certain implementations of the object localization system of the present disclosure are sufficiently complex mathematically, computationally, or technically, e.g., due to the amount or complexity of the computations involved (e.g., analyzing image data using computer vision or machine learning) or to provide results substantially in real time (e.g., determining whether a shopping basket near the store exit is at least partially loaded) (e.g., so that anti-theft actions can be performed before or simultaneously with the shopping basket approaching the store exit), that dedicated hardware (e.g., an FPGA or ASIC) or one or more physical computing devices (with appropriate executable instructions) may be required to perform the functions.

[0239] The blocks or states of the processes described herein can be directly included in hardware, embodied in software modules stored in a non-transitory memory and executed by a hardware processor, or embodied in a combination of both. For example, each of the processes described above can also be included in a software module (stored in a non-transitory memory) executed by one or more machines such as a computer or a computer processor and be fully automated by the software module. The module can reside in a non-transitory computer-readable medium such as, for example, RAM, flash memory, ROM, EPROM, EEPROM, registers, a hard disk, an optical disk, a memory capable of storing firmware, or any other form of computer-readable medium (e.g., a storage medium). The computer-readable medium can be coupled to the processor such that the processor can read information from the computer-readable medium and write information to the computer-readable medium. Alternatively, the computer-readable medium can be integrated into the processor. The processor and the computer-readable medium can be in an ASIC. The computer-readable medium can include non-transient data memories (e.g., a hard disk, non-volatile memory, etc.).

[0240] Processes, methods, and systems can be implemented in a network (or distributed) computing environment. For example, a central control unit or a base station or other components of a cart enclosure system can be implemented in a distributed networked computing environment. The network environment includes enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourcing computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network, a terrestrial or satellite network, or any other type of communication network.

[0241] According to examples, certain actions, events, or functions of any of the processes or methods described herein can be executed in a different order, can be added, combined, or completely omitted. Thus, in certain examples or implementations, not all of the described actions or events are necessary for the practice of the process. Additionally, in certain examples or implementations, actions or events can be executed simultaneously rather than sequentially, for example, by multi-threading, interrupt handling, or via multiple processors or processor cores. In any apparatus, system, or method, no element or action is necessary or indispensable for all examples or implementations, and the disclosed apparatus, systems, and methods can be arranged differently than shown or described.

[0242] Unless otherwise expressly stated or otherwise understood in context, conditional language used herein, such as "able to", "may", "can", "for example", etc., generally aims to convey that certain examples or implementations include certain features, elements, and / or states, while other examples or implementations do not include certain features, elements, and / or states. Thus, such conditional language generally does not imply that one or more examples or implementations require features, elements, and / or states in any way, or that one or more examples or implementations necessarily include logic for determining whether such features, elements, and / or states are included in or will be performed in any particular example or implementation with or without author input or prompting. The terms "comprising", "having", etc. are synonyms and are used inclusively in an open-ended manner and do not exclude other elements, features, acts, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than its exclusive sense), so when used, for example, to connect a series of elements, the term "or" means one, some, or all of the elements in the list.

[0243] Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is generally understood in the context in which it is used to mean that the clause, term, etc. can be X, Y, or Z. Thus, such conjunctive language generally does not imply that certain examples or implementations require the presence of each of at least one of X, at least one of Y, and at least one of Z. In this specification and the appended claims, the articles "a" or "the" when referring to an element mean one or more elements unless the context clearly dictates otherwise.

[0244] Although the foregoing detailed description has shown, described, and pointed out novel features applied to various examples or implementations, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated logical blocks, modules, and processes may be made without departing from the spirit of the present disclosure. As will be recognized, since some features may be used or implemented separately from other features, certain examples of the invention described herein may be implemented in forms that do not provide all of the features and benefits set forth herein.

Claims

1. A theft prevention system, comprising: A computer vision unit configured to image an area of ​​a facility, the computer vision unit comprising: a camera mounted at a height h0 above the floor of the facility; a radio frequency communication node; and image processor; and A human-propelled wheeled cart comprising: A height h, wherein the height h is less than the height h0; A basket configured to hold merchandise; a wheel including a brake configured to inhibit movement of the wheeled cart when the brake is actuated; and a radio frequency cart transceiver configured to communicate with the radio frequency communication node of the computer vision unit and the actuator, wherein the computer vision unit is further configured to associate the wheeled cart with a specific unicast address, wherein the image processor is programmed to analyze images of the area of ​​the facility acquired by the camera to: determining that the basket of the wheeled cart is at least partially loaded with merchandise; and determining that the wheeled cart is attempting to leave the area of ​​the facility, wherein, in response to determining that the basket of the wheeled cart is at least partially loaded with merchandise and determining that the wheeled cart is attempting to leave the area of ​​the facility, the radio frequency communication node is configured to: A command is communicated to the radio frequency cart transceiver of the wheeled cart using the specific unicast address to actuate the brake of the wheel.

2. The anti-theft system of claim 1, wherein the computer vision unit is further configured to: communicating with a payment point of said facility; receiving an indication from the payment point that merchandise in the basket of the wheeled cart has not been paid for, in, The indication is received prior to communicating a command to the radio frequency cart transceiver to actuate the brake of the wheel.

3. The anti-theft system of claim 1 , wherein the image processor is programmed to apply a neural network to images obtained by the camera.

4. The anti-theft system of claim 1, wherein the image processor is further programmed to determine a path of the wheeled cart within the area of ​​the facility.

5. The anti-theft system of claim 1, wherein the computer vision unit is further programmed to store the image of the area in a remote non-transitory computer storage medium.

6. The anti-theft system of claim 1, wherein the camera, the radio frequency communication node, and the image processor are disposed in a housing configured to be mounted to a structure in the facility.

7. The anti-theft system of claim 1, wherein the camera and the radio frequency communication node are disposed in a housing, the housing being configured to be mounted to a structure in the facility, and the image processor is disposed remote from the housing.

8. The anti-theft system of claim 1, wherein the facility comprises a retail store and the wheeled cart comprises a shopping cart.

9. The anti-theft system of claim 8, wherein the area includes a store entrance, a store exit, a checkout lane, a payment point, or an area where high-value goods are stored.

10. A method for reducing theft of merchandise in a retail store, the method comprising: Under the control of an anti-theft system including computer hardware: obtaining an image of an area of ​​the retail store using a camera mounted at a height h0 above a floor of the retail store; Identify from the image whether there is a wheeled cart in the area, the wheeled cart having a height h, the height h being less than the height h0; associating the wheeled cart with a specific unicast address; determining from the image a loading status indicating whether a shopping basket of the wheeled cart is at least partially loaded with merchandise; receiving payment information indicating whether payment has been made for items in the shopping basket; as well as Based at least in part on the loading status and the payment information, a theft prevention command is communicated to the wheeled cart using the specific unicast address.

11. The method of claim 10, wherein the identifying or the determining is performed using a neural network.

12. The method of claim 10, further comprising determining from the image a path of the shopping basket in the area.

13. The method of claim 10, wherein receiving payment information comprises: obtaining a second image of the payment point; From the second image, it is determined whether the shopping basket has passed the payment point, spent more than a threshold time in the vicinity of the payment point, interacted with a store attendant, or accessed a payment system at the payment point.

14. The method of claim 10, wherein: The unicast address is determined to be associated with a radio frequency receiver, wherein the radio frequency receiver is associated with the shopping basket.

15. The method of claim 10, wherein communicating the anti-theft command comprises communicating with a transceiver associated with the shopping basket, communicating with a checkout barrier, communicating with brakes associated with wheels associated with the shopping basket, or communicating with a video surveillance system of the retail store.

16. The method of claim 10, wherein the anti-theft command comprises a command to lock or brake wheels associated with the shopping basket, a command to activate an alarm or warning, or a command to store personnel that a theft is occurring.

17. The method of claim 10, wherein the shopping basket is associated with a wheeled, human-propelled shopping cart.

18. The method of claim 17, wherein the shopping cart includes wheels having brakes, and the anti-theft command includes a command to actuate the brakes.

19. The method of claim 10, further comprising: classifying images of the area of ​​the retail store to annotate shopping baskets or loading states of the shopping baskets to provide a set of training images; as well as A machine learning algorithm is trained using the set of training images.

Images