Refined region of interest within image

Abstract

Techniques for determining a refined region of interest are described. In an embodiment, the refined region of interest may be associated with a radio frequency (RF) device. In some embodiments, a method may include obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein an RF device is associated with the target object; obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, where the region of interest is based at least in part on the state information, the object information, or a combination thereof.

Description

[0001] Related applications

[0002] This application claims the benefit of Greek application No. 20230100385, filed on May 12, 2023, entitled “REFINED REGIONS OF INTERESTWITHIN AN IMAGE”, which has been assigned to the assignee of this application and is incorporated herein by reference in its entirety. Background Technology 1. TECHNICAL FIELD

[0004] This disclosure relates generally to the field of context-based device management, and more specifically to identifying refined regions of interest associated with radio frequency (RF) devices.

[0005] 2. DESCRIPTION OF RELATED ART

[0006] A region of interest (ROI) within an image can indicate the location or possible location of an object or part of an object within the image. ROI can be indicated in an image using, for example, a bounding box around an object. Determining ROI can be useful in various scenarios, such as visual monitoring or supervision in industrial environments (e.g., inventory management or item detection in a warehouse with objects (boxes, shelves, aisles, etc.)) or various other environments such as outdoor spaces or indoor environments (e.g., homes, offices). Summary of the Invention

[0007] In some aspects of the invention, a method for determining a refined region of interest associated with a mobile communication device is disclosed. In some embodiments, the method may include: obtaining sensor data from one or more sensors corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtaining state information associated with the RF device, object information associated with the target object, or a combination thereof, from the RF device or a storage device; and performing an action with respect to the target object based on the region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, wherein the region of interest is at least partially based on the state information, the object information, or the combination thereof.

[0008] In some aspects of the application, a network apparatus is disclosed. In some embodiments, the network apparatus can comprise: one or more communication interfaces configured for data communication with one or more sensors and a storage device; one or more memories; and one or more processors communicatively coupled to the one or more communication interfaces and the one or more memories and configured to: obtain, from the one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtain, from the RF device or the storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and perform an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0009] In some aspects of the application, a non-transitory computer-readable apparatus is disclosed. In some embodiments, the non-transitory computer-readable apparatus can comprise a storage medium comprising a plurality of instructions configured, upon execution by one or more processors, to cause a computerized apparatus to: obtain, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtain, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and perform an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0010] In some aspects of the application, an apparatus is disclosed. In some embodiments, the apparatus can comprise: means for obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; means for obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and means for performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0011] This overview is intended to provide a summary of subject matter of embodiments to be described in the present disclosure. It is not intended to be a key or essential feature of the claimed subject matter, nor does it delineate the scope of what will be claimed. The subject matter should be understood in reference to the entire specification, all or part of any or all drawings, and each claim. The foregoing and other features and examples are described in more detail in the following specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0012] FIG. 1 is an illustration of a positioning system according to an embodiment.

[0013] FIG. 2A is an illustration of an example indoor environment with one or more target objects.

[0014] FIG. 2B is an illustration of an example indoor environment of FIG. 2A

[0015] FIG. 3A is an illustration of an example image of a target object with an RF device attached, the example image having a region of interest around the target object, according to embodiments herein.

[0016] FIG. 3B is an illustration of an example image of a target object with an RF device attached of FIG. 3A

[0017] FIG. 4A is an illustration of a visual prior constructed based on a region of interest within an example image.

[0018] FIG. 4B is an illustration of a more informative visual prior constructed based on a refined region of interest within an example image.

[0019] FIG. 5A is an example image of a target object (user) with a region of interest and a refined region of interest, the refined region of interest determined according to embodiments herein.

[0020] FIG. 5B and FIG. 5C are example images of a target object (user) using an RF device (e.g., a UE), each example image representing a different operating condition, each example image having a region of interest and a refined region of interest, the refined region of interest determined according to embodiments herein.

[0021] FIG. 5D ​​are example images of target objects (users) using RF devices (e.g., UEs) under different operating conditions, the example images having regions of interest and different refined regions of interest, each refined region of interest determined according to different operating conditions and determined according to the implementations herein.

[0022] FIG. 6A and FIG. 6B Example images of target objects arranged according to some aspects of the target objects, each target object having a region of interest associated therewith, are shown.

[0023] FIG. 6C is an example image of a target object having a plurality of possible refined regions of interest identified according to the implementations herein. FIG. 6A Example images of target objects arranged according to some aspects of the target objects, each target object having a region of interest associated therewith, are shown.

[0024] FIG. 7 is a call flow diagram illustrating a process for determining a refined region of interest according to some implementations.

[0025] FIG. 8 is a flow diagram of a method of determining a refined region of interest according to some implementations.

[0026] FIG. 9 is a block diagram of an embodiment of a computer system that can be utilized in implementations as described herein.

[0027] Like reference symbols in the various drawings indicate like elements. In addition, multiple instances of an element can be indicated by adding a dash and a second number to the first number, e.g., 110-1, 110-2, 110-3, etc., or 110a, 110b, 110c, etc. When only the first number is used to refer to such an element, it is understood that any instance of that element (e.g., element 110 in the previous example would refer to elements 110-1, 110-2, and 110-3 or to elements 110a, 110b, and 110c) is intended. DETAILED DESCRIPTION

[0028] The following description is directed to certain implementations for the purposes of describing the innovative aspects of the various embodiments. However, a person having ordinary skill in the art will readily recognize that the teachings herein can be applied in a multitude of different ways. The described implementations can be implemented in any device, system or network that is capable of transmitting and receiving radio frequency (RF) signals according to any communication standard, such as any of the following: Institute of Electrical and Electronics Engineers (IEEE) 802.15.4 standard for Ultra-Wide-Band (UWB), IEEE 802.11 standards (including IEEE 802.11-2016 and IEEE P802.11ax, which is identified as High Efficiency Wireless ®(Technology standards), Bluetooth ® Standard, Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Global System for Mobile Communications (GSM), GSM / General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Terrestrial Trunking Radio (TETRA), Wideband CDMA (W-CDMA), Evolved Data Optimized (EV-DO), 1xEV-DO, EV-DO Revision A, EV-DO Revision B, High Rate Packet Data (HRPD), High Speed ​​Packet Access (HSPA), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Evolved High Speed ​​Packet Access (HSPA+), Long Term Evolution (LTE), Advanced Mobile Phone Systems (AMPS), or other known signals used for communication within wireless, cellular, or Internet of Things (IoT) networks, such as systems utilizing technologies of 3G, 4G, 5G, 6G, or further implementations thereof.

[0029] As used herein, an "RF signal" includes electromagnetic waves that transmit information through the space between a transmitter (or transmitting device) and a receiver (or receiving device). As used herein, a transmitter may send a single "RF signal" or multiple "RF signals" to a receiver. However, due to the propagation characteristics of individual RF signals through multiple channels or paths, a receiver may receive multiple "RF signals" corresponding to each transmitted RF signal.

[0030] Additionally, unless otherwise specified, references to “reference signal,” “location reference signal,” “reference signal for location,” etc., may refer to signals used for locating user equipment (UE). As described in more detail herein, such signals may include any of a variety of signal types, but are not necessarily limited to the Location Reference Signal (PRS) as defined in the relevant radio standards.

[0031] Furthermore, unless otherwise specified, the term "location" as used herein can refer to absolute location determination, relative location determination, ranging, or a combination thereof. For the purpose of location or sensing services, such location may include and / or be based on timing, angle, phase, or power measurements or combinations thereof (which may include RF sensing measurements).

[0032] Various aspects generally relate to determining refined regions of interest, where a region of interest represents an area within an image (optical image, radio frequency (RF) representation, RF image, etc.) that is associated with a location (or likely location) of an RF device such as a mobile communication device (e.g., a smartphone or other type of UE) or a radio frequency identification (RFID) tag. A typical region of interest can be a pixel location on an image that identifies an approximate location of an RF device or a target object, but a refined region of interest can be determined based on additional contextual information (e.g., information about the RF device or information about the target object). In some approaches, a region of interest can be progressively refined such that it provides more informative visual information.

[0033] Particular aspects of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages. A refined region of interest can be more relevant, more accurate, and more targeted compared to a typical region of interest, resulting in more enhanced and more useful results in downstream applications and actions such as localization or visualization of a target object or a portion thereof (as well as other applications and actions as mentioned elsewhere below). Additional contextual information can also be relatively easily obtained. Such information can include, for example, a mode or activity of the RF device, a type of target object to which the RF device is attached, an application or action to be performed with the refined region of interest, just to name a few.

[0034] Additional details will follow the initial description of the related systems and techniques.

[0035] FIG. 1This is a simplified illustration of a positioning system 100 according to an embodiment, in which a radio frequency (RF) device 105, a location server 160, and / or other components of the positioning system 100 may use the techniques provided herein to determine a refined region of interest within an image and / or perform actions with respect to a target object. In various embodiments, the RF device 105 may include a wireless or mobile communication device, such as a UE (e.g., a smartphone), a radio frequency identification (RFID) tag (passive or active), or another device capable of wirelessly communicating with or with an access point described below. The techniques described herein may be implemented by one or more components of the positioning system 100. The positioning system 100 may include: the RF device 105; one or more satellites 110 (also referred to as spacecraft (SV)), which may include Global Navigation Satellite System (GNSS) satellites (e.g., satellites of Global Positioning System (GPS), GLONASS, Galileo, BeiDou, etc.) and / or non-terrestrial network (NTN) satellites; a base station 120; an access point (AP) 130; a location server 160; a network 170; and an external client 180. Generally speaking, the positioning system 100 can estimate the location of the RF device 105 based on the known locations of RF signals received by and / or transmitted from the RF device 105 and other components that transmit and / or receive RF signals (e.g., GNSS satellite 110, base station 120, AP 130).

[0036] It should be pointed out that, FIG. 1 Only generalized examples of various components are provided, where any or all of the components may be utilized as appropriate, and each component may be repeated as needed. Specifically, although only one RF device 105 is illustrated, it should be understood that many RF devices (e.g., hundreds, thousands, millions, etc.) may utilize the positioning system 100. Similarly, the positioning system 100 may include more than FIG. 1 The illustrated number of base stations 120 and / or access points 130 may vary. The illustrated connections connecting the various components in the positioning system 100 include data and signaling connections, which may include additional (intermediate) components, direct or indirect physical and / or wireless connections and / or additional networks. Furthermore, the components may be rearranged, combined, separated, replaced, and / or omitted depending on desired functionality. In some embodiments, for example, an external client 180 may be directly connected to the location server 160. Those skilled in the art will recognize numerous modifications to the illustrated components.

[0037] Depending on the desired functionality, network 170 can comprise any of a variety of wireless and / or wired networks. Network 170 can for example comprise any combination of public and / or private networks, local and / or wide-area networks, and the like. Moreover, network 170 can utilize one or more wired and / or wireless communication technologies. In some embodiments, network 170 can comprise, for example, a cellular or other mobile network, a wireless local area network (WLAN), a wireless wide-area network (WW AN), and / or the Internet. Examples of network 170 include a Long-Term Evolution (LTE) wireless network, a Fifth Generation (5G) wireless network (also referred to as a New Radio (NR) wireless network or a 5G NR wireless network), a Wi-Fi WLAN, and the Internet. LTE, 5G, and NR are wireless technologies defined or being defined by the Third Generation Partnership Project (3GPP). Network 170 can also comprise more than one network and / or more than one type of network.

[0038] Base stations 120 and access points (APs) 130 can be communicatively coupled to network 170. In some embodiments, base stations 120 can be owned, maintained, and / or operated by a cellular network provider, and can employ any of a variety of wireless technologies, as described below. Depending on the technology of network 170, base stations 120 can comprise a Node B, an evolved Node B (eNodeB or eNB), a transceiver base station (BTS), a radio base station (RBS), an NR Node B (gNB), a next generation eNB (ng-eNB), and the like. In the case that network 170 is a 5G network, base stations 120 as gNBs or ng-eNBs can be part of a Next Generation Radio Access Network (NG-RAN) that can be connected to a 5G Core Network (5GC). In view of Open Radio Access Network (O-RAN) and / or Virtualized Radio Access Network (V-RAN or vRAN) in 5G or higher generation networks, functionality performed by base stations 120 in earlier networks (e.g., 3G and 4G) can be split into different functional components (e.g., Radio Units (RUs), Distributed Units (DUs), and Central Units (CUs)) and layers (e.g., LI / L2 / L3) that can be executed on different devices at different locations connected, for example, via fronthaul, midhaul, and backhaul connections. As referred to herein, a “base station” (or ng-eNB, gNB, etc.) can comprise any or all of these functional components. For example, AP 130 can comprise a Wi-Fi AP or Bluetooth AP, and can be owned, maintained, and / or operated by a different entity than the cellular network provider that owns, maintains, and / or operates base stations 120. ®The APs can be APs without cellular capabilities (e.g., Wi-Fi APs), or APs with cellular capabilities (e.g., 4G LTE and / or 5G NR). Thus, the RF device 105 can communicate and receive information with network-connected devices, such as the location server 160, by accessing the network 170 via the base stations 120 using the first communication links 133. Additionally or alternatively, because the APs 130 can also be communicatively coupled with the network 170, the RF device 105 can communicate with network- and internet-connected devices, including the location server 160, using the second communication links 135 or via one or more other mobile devices 145.

[0039] As used herein, the term “base station” can generally refer to a single physical transmission point that can be located at the base station 120 or multiple co-located physical transmission points. A transmission reception point (TRP) (also referred to as a transmission / reception point) corresponds to this type of transmission point, and the term “TRP” can be used interchangeably herein with the terms “gNB,” “ng-eNB,” and “base station.” In some cases, the base station 120 can include multiple TRPs— e.g., where each TRP is associated with a different antenna or different antenna array of the base station 120. As used herein, the transmission functionality of a TRP can be performed with a transmission point (TP), and / or the reception functionality of a TRP can be performed by a reception point (RP), which can be physically separate or distinct from the TP. That is, a TRP can include both a TP and a RP. A physical transmission point can include an antenna array of the base station 120 (e.g., as in a multiple-input multiple-output (MIMO) system and / or where the base station employs beamforming). The term “base station” can additionally refer to multiple non-co-located physical transmission points, which can be a distributed antenna system (DAS) (a network of spatially separated antennas connected to a common source via a transmission medium) or a remote radio head (RRH) (a remote base station connected to a serving base station).

[0040] As used herein, the term “cell” can generally refer to a logical communication entity that enables communication to and / or from the base station 120 and can be associated with an identifier (e.g., a physical cell identifier (PCID), a virtual cell identifier (VCID)) used to distinguish neighboring cells operating via the same or different carriers. In some examples, a carrier can support multiple cells, and different cells can be configured according to different protocol types (e.g., machine type communication (MTC), narrowband Internet of Things (NB-IoT), enhanced mobile broadband (eMBB), or other protocols) that can provide access for different types of devices. In some cases, the term “cell” can refer to a portion of a geographic coverage area (e.g., a sector) over which a logical entity operates.

[0041] Satellites 110 can be used to locate RF device 105 in one or more manners. For example, satellites 110 (also referred to as space vehicles (SVs)) can be part of a global navigation satellite system (GNSS), such as the Global Positioning System (GPS), GLONASS, Galileo, or Beidou. Positioning using RF signals from GNSS satellites can include measuring multiple GNSS signals at a GNSS receiver of RF device 105 to perform code-based and / or carrier-based positioning, which can be highly accurate. Additionally or alternatively, satellites 110 can be used for NTN-based positioning, where satellites 110 can operate functionally as TRPs (or TPs) of a network (e.g., an LTE and / or NR network) and can be communicatively coupled with network 170. Specifically, reference signals (e.g., PRSs) transmitted for NTN-based positioning by satellites 110 can be similar to those transmitted by base stations 120 and can be coordinated by location server 160. In some embodiments, satellites 110 for NTN-based positioning can be different from those for GNSS-based positioning. In some embodiments, NTN nodes can include non-terrestrial vehicles, such as airplanes, balloons, drones, etc., which can be in addition to or instead of NTN satellites.

[0042] Location server 160 can include servers and / or other computing devices configured to determine an estimated position of RF device 105 and / or to provide data (e.g., “assistance data”) to RF device 105 to facilitate position measurements and / or position determinations by RF device 105. According to some embodiments, location server 160 can include a home Secure User Plane Location (SUPL) Location Platform (H-SLP), which can support a SUPL User Plane (UP) positioning solution defined by the Open Mobile Alliance (OMA) and can support location services for RF device 105 based on subscription information for RF device 105 stored in location server 160. In some embodiments, location server 160 can include a Discovery SLP (D-SLP) or an Emergency SLP (E-SLP). Location server 160 can also include an Enhanced Serving Mobile Location Center (E-SMLC) that supports positioning of RF device 105 using a Control Plane (CP) positioning solution for LTE radio access for RF device 105. Location server 160 can also include a Location Management Function (LMF) that supports positioning of RF device 105 using a Control Plane (CP) positioning solution for NR or LTE radio access for RF device 105.

[0043] In a CP positioning solution, from the perspective of the network 170, signaling for controlling and managing positioning of the RF devices 105 can use existing network interfaces and protocols and be exchanged as signaling between elements of the network 170 and with the RF devices 105. In a UP positioning solution, from the perspective of the network 170, signaling for controlling and managing positioning of the RF devices 105 can be exchanged as data (e.g., data transmitted using Internet Protocol (IP) and / or Transmission Control Protocol (TCP)) between the location server 160 and the RF devices 105.

[0044] As previously noted (and discussed in greater detail below), an estimated position of the RF device 105 can be based on measurements of RF signals transmitted from and / or received by the RF device 105. In particular, these measurements can provide information about the relative distance and / or angle of the RF device 105 to one or more components in the positioning system 100 (e.g., GNSS satellites 110, APs 130, base stations 120). An estimated position of the RF device 105 can be geometrically estimated (e.g., using multilateration and / or multilateration) based on the distance and / or angle measurements along with known positions of the one or more components.

[0045] Although terrestrial components, such as APs 130 and base stations 120, can be fixed, embodiments are not so limited. Mobile components can be used. For example, in some embodiments, a position of the RF device 105 can be estimated based at least in part on measurements of RF signals 140 communicated between the RF device 105 and one or more other mobile devices 145, which can be mobile or fixed. As illustrated, the other mobile devices can include, for example, a mobile phone 145-1, a vehicle 145-2, a static communication / positioning device 145-3, or other static and / or mobile devices or combinations thereof capable of providing wireless signals for positioning the RF device 105. Wireless signals from the mobile devices 145 for positioning of the RF device 105 can include RF signals using, for example, Bluetooth ® (including Bluetooth Low Energy (BLE)), IEEE 802.1 lx (e.g., Wi-Fi ® ), Ultra-Wide Band (UWB), IEEE 802.15x, or combinations thereof. The mobile devices 145 can additionally or alternatively use non-RF wireless signals for positioning of the RF device 105, such as infrared signals or other optical techniques.

[0046] Mobile devices 145 can include other RF devices (e.g., UEs) that are communicatively coupled with a cellular network or other mobile network (e.g., network 170). When one or more other mobile devices 145 (including RF devices) are used in a positioning determination for a particular RF device 105, the RF device 105 for which positioning is to be determined can be referred to as a “target RF device,” and each of the other mobile devices 145 used can be referred to as an “anchor RF device.” To make a positioning determination for a target RF device, the respective positions of one or more anchor RF devices can be known and / or determined jointly with the target RF device. Direct communication between one or more other mobile devices 145 and the RF device 105 can include sidelink and / or similar device-to-device (D2D) communication techniques. Sidelink is a form of D2D communication under the cellular-based LTE and NR standards defined by 3GPP. UWB can be one such technique through which positioning of a target device (e.g., RF device 105) can be facilitated using measurements from one or more anchor devices (e.g., mobile devices 145).

[0047] According to some embodiments, such as when the RF device 105 includes and / or is incorporated into a vehicle, one form of D2D communication used by the RF device 105 can include vehicle-to-everything (V2X) communication. V2X is a communication standard for vehicles to exchange information about the traffic environment with related entities. V2X can include vehicle-to-vehicle (V2V) communication between V2X-capable vehicles, vehicle-to-infrastructure (V2I) communication between vehicles and infrastructure-based devices (often referred to as road-side units (RSUs)), vehicle-to-person (V2P) communication between vehicles and nearby people (pedestrians, bicyclists, and other road users), etc. Furthermore, V2X can use any of a variety of wireless RF communication technologies. For example, cellular V2X (CV2X) is a form of V2X that uses cellular-based communication in a direct communication mode defined by 3GPP, such as LTE (4G), NR (5G), and / or other cellular technologies. FIG. 1The illustrated RF device 105 can correspond to a component or device on a vehicle, RSU, or other V2X entity used to communicate V2X messages. In implementations in which V2X is used, the static communication / positioning device 145-3 (which can correspond to an RSU) and / or the vehicle 145-2 can thus communicate with the RF device 105 and can be used to determine a position of the RF device 105 using techniques similar to those used by the base stations 120 and / or APs 130 (e.g., using multilateration and / or multilateration). It can be further noted that, according to some implementations, the mobile device 145 (which can include a V2X device), the base stations 120, and / or the APs 130 can be used together (e.g., in a WW AN positioning solution) to determine a position of the RF device 105.

[0048] The estimated position of the RF device 105 can be used in a variety of applications, such as to assist a user of the RF device 105 in direction finding or navigation or to assist another user (e.g., associated with the external client 180) in locating the RF device 105. A "position" is also referred to herein as a "location estimate," an "estimated location," a "location," a "fix," a "position estimate," a "position fix," an "estimated position," a "location fix," or a "fix." The process of determining a position can be referred to as "positioning," "position determination," "location determination," etc. A position of the RF device 105 can include an absolute position of the RF device 105 (e.g., latitude and longitude and possibly altitude) or a relative position of the RF device 105 (e.g., a position expressed as a distance north or south, east or west, and possibly up or down from some other known fixed position (including, e.g., a position of a base station 120 or an AP 130) or some other position such as a position of the RF device 105 at some known prior time or a position of a mobile device 145 (e.g., another RF device) at some known prior time). A position can be specified as a geodetic position comprising coordinates, which can be absolute (e.g., latitude, longitude, and optionally altitude), relative (e.g., relative to some known absolute position), or local (e.g., X, Y, and optionally Z coordinates according to a coordinate system defined with respect to a local region such as a factory, warehouse, university campus, shopping center, stadium, or convention center). A position can instead be a civic position, and can then include one or more of a street address (e.g., including a country, state, county, city, road and / or street name or label, and / or a road or street number) and / or a label or name of a place, building, portion of a building, floor of a building, and / or room within a building, etc. A position can also include an indication of uncertainty or error, such as a distance within which the position is expected to be in error, possibly in both horizontal and vertical dimensions, or an indication of an area or volume (e.g., a circle or an ellipse) within which the RF device 105 is expected to be located with some level of confidence (e.g., 95% confidence).

[0049] The external client 180 can be a web server or remote application that can have some association with the RF device 105 (e.g., accessible by a user of the RF device 105), or can be a server, application, or computer system that provides location services to some other user or users, which can include obtaining and providing a location of the RF device 105 (e.g., to enable services such as friend or relative visitation or child or pet location). Additionally or alternatively, the external client 180 can obtain a location of the RF device 105 and provide the location to an emergency service provider, government agency, or the like.

[0050] In some implementations, the example positioning system 100 can be implemented using a wireless communication network, such as an LTE-based or 5G NR-based network.

[0051] Refining a region of interest

[0052] FIG. 2A is an illustration of an example indoor environment 210 having one or more target objects 212a-212e. Such target objects can be visible and in the line of sight of one or more image capture devices (e.g., cameras 214a-214d) positioned in the example indoor environment 210. In some cases, the image capture devices can include one or more radio frequency (RF) sensors configured to detect RF signals and cause generation of RF representations or RF images. In some cases, multiple cameras and / or RF sensors can be used depending on the application or level of visual information required.

[0053] In some implementations, the example indoor environment 210 can further include one or more access points 216a-216c. The access points 216a-216c can be configured to wirelessly communicate with the one or more image capture devices and / or one or more RF devices 218a-218f. In some implementations, the access points 216a-216c can receive image data from the one or more image capture devices and communicate the image data to a network (e.g., a server); or the access points 216a-216c can receive requests from a network (e.g., a server) to obtain information about the RF devices 218a-218f or the target objects 212a-212e, as will be discussed further elsewhere.

[0054] In some scenarios, RF devices 218a-218f can be associated with target objects 212a-212e. In some cases, the association of an RF device with a target object can include co-location with the target object (e.g., a person). In some cases, the RF device and the target can not be co-located (e.g., located in different rooms or zones). In different implementations, the RF devices can be wireless or mobile communication devices such as UEs (e.g., smartphones), RFID tags (passive or active), or another device capable of wireless communication (e.g., with access points 216a-216c, base stations of a wireless communication network, another RF device (e.g., an RF reader), or each other). In this context, association can refer to the RF device being attached to the target object, in contact with the target object, within the target object (e.g., a box), in communication with the target object (e.g., between a user and a UE placed away from the user), within a particular distance from the target object, or combinations thereof. These are illustrative examples, and other types of associations between RF devices and target objects can occur.

[0055] In FIG. 2A the illustrated example, RF device “A” 218a is associated with target object 212d, RF device “B” 218b is associated with target object 212e, RF devices “C” 218c, “D” 218d, and “E” 218e are associated with target object 212c, RF device “F” 218f is associated with target object 212b, and RF device “G” 218g is associated with target object 212a. As can be seen, multiple RF devices can be associated with a given target object. A given RF device can also be associated with multiple target objects; for example, an RF device can be associated with a shelf of boxes. In some cases, multiple possible locations of at least one RF device can be associated with a target object (e.g., at least one RF device within a box and not visible in the image). As will be described in further detail elsewhere, such possible locations can have respective probabilities or likelihoods associated therewith.

[0056] FIG. 2B is FIG. 2A an illustration of an example image 220 of the example indoor environment 210. In some embodiments, one or more images of the example indoor environment 210 can be obtained by cameras 214a-214d, resulting in image 220 representing the example indoor environment 210. In some approaches, image 220 can be derived from one image obtained by one or more of cameras 214a-214d, or image 220 can be derived from multiple images combined into a composite image. Other approaches known in the relevant art can be used to obtain image 220 of the example indoor environment 210.

[0057] In FIG. 2B the illustrated example, image 220 can includeFIG. 2A A top-down view of the target objects 212a-212e is shown, including object representations 222a-222e corresponding to the target objects 212a-212e. The image 220 can be a two-dimensional representation of an example indoor environment 210 in an x-y plane. A region of interest 224 can be shown around the target objects. In this example, the region of interest 224 can be a bounding box indicating the pixel location of the target object 222a.

[0058] In some implementations, a region of interest can be generated to identify a portion of an image where an RF device can be present. For example, a bounding box can be obtained via object detection configured to identify a particular type of object. Further, a region of interest can be associated across images, which can allow for identifying the same region of interest in multiple images, or a region of interest can be associated with an RF device, which can allow for determining which RF device is contained in a given region of interest.

[0059] Each region of interest can contain one or more RF devices. However, the actual location of the RF devices is generally unknown because the RF devices are not directly detected. In other words, the RF devices can be anywhere within the corresponding region of interest. Although the RF devices can be anywhere within a given region of interest, the region of interest can be used as a supporting basis for various applications or actions, such as determining the localization of the RF devices and / or the target objects associated with the RF devices. Another such application or action is the visualization of the localization (e.g., in three-dimensional coordinates) of the target objects and / or the RF devices, as will be discussed in further detail below in terms of a “visual localization prior” or “visual prior”.

[0060] In some implementations, the region of interest 224 can be determined and placed on the image 220 based at least on imaging techniques applied to the objects, such as feature extraction, pattern recognition, feature classification, etc. In some implementations, at least one RF device associated with a target object can be used to determine and / or select the region of interest. For example, based on information of the location of the RF devices, it can be determined that the region of interest is around objects with similar locations. In some implementations, camera parameters and / or orientation of the camera can be considered in determining the region of interest 224.

[0061] The region of interest 224 can be used to determine the localization of the target object 222a. However, as will be discussed below, some scenarios can benefit from a more refined region of interest, such as a bounding box or any boundary shape that more precisely indicates the location or possible locations of the RF devices or the target objects, which can enable more precise, more relevant, and more informative localization or other downstream applications or actions that utilize the refined region of interest. The refined region of interest can take any polygonal or freeform shape as needed. More will be discussed about FIG. 3A through FIG. 6BThe difference between a typical region of interest and a refined region of interest is described.

[0062] FIG. 3A is an illustration of an example image 310 of a target object 312 with an RF device 314 attached, having a region of interest 316 around the target object 312. As shown in the example image 310, the target object 312 can be a box. As noted above, the RF device 314 can be a wireless or mobile communication device, such as a smartphone or RFID tag, and can be an example of one of the RF devices 218a-218f. The region of interest 316, which in this case is a bounding box, can be visually determined from the example image 310 using one or more of the methods mentioned above. However, the RF device 314 can or can not be visible in the example image 310 (e.g., if the RF device is inside the box).

[0063] The region of interest 316 visually indicates the location of the target object 312 and the RF device 314 on the example image 310. Using the example image 310 (or in some cases, one or more images similar to the example image 310 in addition), a “visual localization prior” or “visual prior” can be constructed and generated. In this context, a “visual localization prior” or “visual prior” can refer to a representation or set of parameters of a target object or RF device associated with a target object in a different space than the image used to construct the representation. For example, a region in a three-dimensional coordinate system can be determined using one or more two-dimensional images, information contained in one or more images (e.g., a region of interest), state information related to the RF device, information related to the target object (e.g., object type), prior knowledge of the location of the RF device, other information obtainable from the RF device or elsewhere such as a network, or a combination thereof. The visual prior in a three-dimensional coordinate system can be constructed, for example, by back-projection of one or more regions of interest defined on one or more two-dimensional images. The more informative the visual prior, the more accurate the location or localization of the RF device or target object can be considered. With a more informative visual prior and more accurate location information (e.g., a small three-dimensional region in which the RF device or target object can be located), downstream applications or actions using the visual prior can be more effective. For example, more accurate or more granular monitoring of objects can be performed (e.g., by accurately isolating each item in a crowded group of items in an industrial environment, or determining a more accurate location of a UE relative to a user). By refining the region of interest according to the embodiments described herein, the visual prior can be made more informative. FIG. 4A and FIG. 4B An example of a building informative visual prior is illustrated.

[0064] An informationally-rich region of interest is characterized by three principles: (1) the region of interest contains the RF device. (2) Smaller regions of interest are generally more useful and generally yield more informationally-rich visual information. (3) The visual priors defined by the region of interest can be located on the image in which the region of interest is identified. In some cases, a region of interest can be further refined by incorporating additional contextual information related to the state, activity, object type, and / or location of the RF device and / or target object. In some cases, a region of interest can be further refined by post-processing the region of interest initially determined using the aforementioned additional contextual information.

[0065] Such additional contextual information can be defined as any “domain knowledge” or information related to the RF device or target object. A target object can refer to any object, item, or user associated with the RF device. A target object can be used to obtain a region of interest for an informationally-rich visual prior (and satisfy the above three principles). Such additional domain information can include a variety of information that can be categorized into two broad groups: state information related to the RF device, and object information related to a target object (e.g., a target object associated with the RF device).

[0066] More specifically, in various embodiments, examples of state information can include or be based on at least one of:

[0067] Communication information. In some implementations, communication information can include a call state indicating whether the RF device (e.g., a smartphone or another UE) is in an active call. In some implementations, communication information can include a communication state indicating whether the RF device is communicating with another device (e.g., via an access point 216a-216c, a base station of a wireless communication network, or a device-to-device communication with another RF device). In some implementations, as some examples, communication information can include a connectivity state indicating whether the RF device is actively connected to a short-range wireless network (e.g., with a Bluetooth device) or is using a short-range communication protocol (e.g., a near-field communication (NFC) protocol).

[0068] Operation information. In some implementations, as some examples, operation information can include an active state indicating whether the RF device is locked or unlocked (e.g., whether use of the UE is authenticated or whether some or most functionality of the UE is available for use by a user), whether and / or how the RF device is being actively used (e.g., reading, typing, or scrolling is detected on the UE), or whether the RF device (e.g., a UE) is in speakerphone mode.

[0069] Motion information. In some implementations, motion information can include a motion state indicating whether the RF device is in motion. For example, a UE or mobile communication device (e.g., a smartphone) can be on a user who is moving. The motion state can be determined according to various methods (e.g., using a gyroscope, or determining a difference in location derived using the above-mentioned UE-based or UE-assisted positioning methods).

[0070] Distance information. In some implementations, distance information can be relative to a target object, and can include a distance or range of the RF device relative to the target object and / or a proximity state indicating whether the distance is within or outside a distance threshold. In some cases, the distance can be known and stored on a network (e.g., a server or storage such as a database). An industrial setting, warehouse, or other environment with standard procedures can be an example of a scenario with standardized or known placement of objects or distances between objects. In some cases, the distance can be visually determined using an image such as image 220 (e.g., referencing known pixel-to-pixel distances). In some cases, the distance can be determined using RF signals transmitted by the RF device and reflected from the target object (e.g., using RTT). In some scenarios, distance information can also be calculated in a virtual reality (VR) setting, an augmented reality (AR) setting, or a mixed reality (MR) setting using the above-discussed parameters (e.g., known data, known reference distances).

[0071] Location information. In some implementations, location information can indicate a geodetic, relative, or local location of the RF device as described above. Location information can be used not only to refine a region of interest, but also to determine the aforementioned distance information.

[0072] Placement information of the RF device. In some implementations, placement information can be relative to a target object, which in some cases can be determined based on the aforementioned distance information. In some implementations, placement information can indicate one or more possible locations where the RF device can be placed. For example, the RF device can be inside a box and located at one of the eight corners of the box.

[0073] Placement type. In some implementations, a placement type can relate to a placement of the RF device relative to the target object. Using the above example of an RF device that can be located at one of the eight corners of a box, the placement type can be “corner.” This information can indicate that the RF device can be in a corner of an object that does not have eight different corners (a flat box with four corner placement, a table with four corners, a triangular container, etc.). Another example is that the placement type is an orientation or position relative to the target object. For example, the RF device can be behind or below the target object; in front of or above the target object; inside or outside the target object; or visible, not visible, or partially visible.

[0074] In various implementations, examples of object information can include or be based on at least one of:

[0075] Object type. In some implementations, an object type can be a type of the target object. The object type can include various descriptors of aspects of the target object that the device is associated with: size, dimensions, weight, color, category, whether the target object is a container, etc. The objects can be organized in some structured way, such as by one of the descriptors listed above.

[0076] Placement information of the target object. In some implementations, placement information can indicate a placement or position of the target object relative to other objects or the target object. For example, the placement information can indicate that the target object is at a particular shelf but not another shelf, or in a particular warehouse but not another warehouse. In some implementations, the placement information can be based on the object type discussed above. For example, objects of a particular size range (or range of other aspects such as weight) can be organized at a particular location, such as a bottom shelf for heavier items. In some implementations, the placement information can indicate a likely positioning or possible positioning of the RF device on the target object, or a placement type. Examples of the foregoing can include which corner the RF device is at or can be at, or whether the RF device is placed at a corner, side, or inside of the target object (e.g., a box). This placement information can be related to or overlap with the placement information and placement type discussed above.

[0077] In some embodiments, contextual information such as state information and object information can be obtained (e.g., by a server) from a storage device (e.g., a database) or an RF device (e.g., by request or periodically received). Where at least some of the aforementioned contextual information can be used to provide additional information about the activity, state, object type, location, etc. of the RF device and / or target object, the region of interest can be refined by incorporating this information to determine a refined region of interest. In some implementations, the refined region of interest can be further refined using different portions of the same contextual information, state information, and / or object information or newly obtained contextual information. That is, the initial region of interest can be refined more than once.

[0078] The aforementioned state information and object information are illustrative examples. Other types of state information (e.g., information related to, about, associated with the RF device) and / or object information (e.g., information related to, about, associated with the target object) of the RF device can be used with similar effect.

[0079] Reference is now made to FIG. 3B , FIG. 3A An illustration of an example image 350 of a target object 312 with an RF device 314 attached thereto, the example image 350 having a refined region of interest 356 determined in accordance with embodiments herein. In this example, the refined region of interest 356 is a bounding box that is significantly smaller than the region of interest 316 of FIG. 3A It is possible that the refined region of interest 356 has been determined using at least some of the contextual information discussed above. The RF device 314 can still not be visible in the example image 350. However, the refined region of interest 356 can be more accurate with respect to the location (or likely location) of the RF device 314.

[0080] FIG. 4A An illustration of a visual prior 401 constructed based on regions of interest 404a, 404b within example images 402a, 402b. In this example, multiple regions of interest 404a, 404b can be determined. Each region of interest 404a, 404b can indicate a pixel location of an RF device 406 and / or a target object (not shown), and can be determined within the example images 402a, 402b based on imaging techniques, location information of the RF device 406, and / or camera parameters, as discussed with respect to FIG. 2BThe example images 402a, 402b can or can not actually show the RF device 406, although indications 406a, 406b of where the RF device 406 can be are shown in the example images 402a, 402b for illustrative purposes. According to some approaches, the regions of interest 404a, 404b can be back-projected to determine the visual prior 401, which can be a three-dimensional region representing the location of the RF device 406 and / or the target object. In some particular scenarios, the visual prior 401 can be a box-shaped region containing the target object, which can be a box in this case.

[0081] FIG. 4B An illustration of a more informative visual prior 451 that is constructed based on refined regions of interest 454a, 454b within the example images 452a, 452b. As can be seen, the refined regions of interest 454a, 454b have one or more smaller dimensions compared to the regions of interest 404a, 404b, and the more informative visual prior 451 can be a three-dimensional region having one or more smaller dimensions compared to the visual prior 401. According to some approaches, the refined regions of interest 454a, 454b can be back-projected to determine the more informative visual prior 451. The example images 402a, 402b can or can not actually show the RF device 406, although indications 456a, 456b of where the RF device 406 can be are shown in the example images 452a, 452b for illustrative purposes.

[0082] The more informative visual prior 451 can indicate a more accurate position or location of the RF device 406, while the visual prior 401 indicates a relatively more general position of the RF device 406 and / or the target object. In some particular scenarios, the visual prior 401 can be a box-shaped region containing a portion of the RF device and the target object, rather than the entire target object (e.g., the entire box). The visual prior 451 can be used to directly infer the location of the RF device. This can be useful in situations where a more specific location of the target object is desired (e.g., multiple boxes are crowded together or a specific portion of the target object is desired to be located).

[0083] In some implementations, the determination or verification of the location or position of an RF device and / or a target object (or a portion thereof) can be based on visual priors and RF signals using RF transmitters and RF sensors. For example, RF-based positioning estimation can be obtained, for instance, via round-trip signal propagation delay (RTT). As another example, RF signals can also be used to generate RF images or RF representations, in which case additional regions of interest can be determined. In some cases, one or more measurements, such as range or angle (e.g., angle of arrival (AoA) or angle of departure (AoD)) estimates relative to the RF device (coinciding with the region of interest), can be obtained, in which case one or more measurements can be used in conjunction with visual priors to determine or verify the location of the RF device and / or the target object.

[0084] In some implementations, fewer or additional images may be obtained, for example, from different or the same camera or RF sensor (compared to, for example, the two example images 452a, 452b). Using one or more additional images, one or more corresponding refined regions of interest may be determined for a given RF device, for example, based on contextual information of the type discussed above. The additional refined regions of interest may produce even more informative visual priors.

[0085] Furthermore, the association of regions of interest (ROIs) across images and / or across RF devices can advantageously benefit from more informative, more precise ROIs and more informative visual priors. For example, in crowded scenes with many objects or users, the association of ROIs with RF devices can significantly benefit from smaller ROIs that minimize overlap and capture as much relevant information as possible while minimizing redundant information. [The following is a separate section:] ... FIG. 5A through FIG. 5D Describe exemplary scenarios involving users.

[0086] FIG. 5Ais an example image 510 of a target object (user 502), the example image 510 having a region of interest 504 and a refined region of interest 506a, the refined region of interest 506a determined in accordance with the embodiments herein. In some embodiments, the refined region of interest 506a can be determined based on additional context information as discussed elsewhere herein. In some cases, the additional context information can have certain assumptions. For example, a user typically carries an RF device (e.g., a UE such as a smart phone) near the upper part of the body (e.g., close to the ear, in a pocket, in a wallet). Then, using the context information and the assumption that the focus is on the upper part of the body of the user 502, the region of interest 504 (which can be an initial region of interest without considering the context information) can be refined to determine the refined region of interest 506a. Notably, the region of interest 504 and the refined region of interest 506a can have at least one different dimension. In this example, the refined region of interest 506a can have a smaller height and focus on the upper part of the user's body. With additional context information, the width can also be reduced. However, in some scenarios, one dimension of the region of interest can be enlarged instead of being reduced, in which case the total area of the refined region of interest is still smaller than the initial region of interest.

[0087] FIG. 5B and FIG. 5C are example images 520, 530 of a target object (user 502) using an RF device (e.g., UE 508), each example image 520, 530 representing a different operating condition, each example image 520, 530 having a region of interest 504 and a refined region of interest 506b, 506c, the refined region of interest 506b, 506c determined in accordance with the embodiments herein.

[0088] In the example of FIG. 5B , when determining the refined region of interest 506b, context information such as communication information (e.g., call status) and / or operating information (e.g., lock status) can be considered. For example, if the UE 508 is not in an active call and / or is locked, this information can imply that the UE 508 is not in use by the user 502. A user typically carries a UE on his body (e.g., a pocket or a wallet as noted above). This is an assumption that can provide additional information. The region of interest 504 can be refined to focus on the middle part of the body using the communication information, the operating information, and / or the assumption. As such, the refined region of interest 506b can be determined on the example image 520.

[0089] In contrast, with reference to FIG. 5Cexample, based on the call state and / or the lock state mentioned above), the context information can imply that the UE 508 is being used by the user 502. Most users hold the UE near their head, even when the UE is in speakerphone mode. In some scenarios, the UE can be a pair of earbuds that are fitted in the user's ears. These are assumptions that can provide additional information. Using the above context information and / or assumptions, the region of interest 504 can be refined to focus on the top of the body (e.g., the head of the user 502). Thus, a refined region of interest 506c can be determined on the example image 530. However, in some scenarios, one or more dimensions of the region of interest can be enlarged instead of being reduced, in which case the total area of the refined region of interest is still smaller than the initial region of interest, or in some cases larger than the initial region of interest.

[0090] FIG. 5D is an example image 540 of a target object (the user 502) using an RF device (e.g., the UE 508) under different operating conditions, the example image having a region of interest 504 and different refined regions of interest 506d, 506e, each refined region of interest 506d, 506e being determined according to different operating conditions and determined according to the embodiments herein.

[0091] If the UE 508 is in a call in speakerphone mode (determined using, for example, the call state), connected to a Bluetooth device (e.g., an external speaker, earbuds) (determined using, for example, the connectivity state), and / or in a different motion state than the person (e.g., walking while the UE 508 is stationary) (determined using, for example, the stationary state of the motion state), this information can indicate that the UE 508 is likely placed near the person (e.g., on a table) rather than directly on the person. Based on such context information, the region of interest 504 can be refined to incorporate the surrounding environment, resulting in a refined region of interest 506d. The refined region of interest 506d includes the UE 508, the user 502, and the table. Depending on the application or action (e.g., which object to locate), either or both of the user 502 or the table can be considered the target object.

[0092] In some cases, additional context information can be used. If distance information, such as the range between the user 502 and the UE 508, is available, the region of interest (the initial region of interest 504 or the refined region of interest 506d) can be further refined to a refined region of interest 506e based on the distance information and / or the previously listed context information used to determine the refined region of interest 506d.

[0093] In some implementations, a visual search (e.g., using the aforementioned imaging techniques) can occur based on proximity to an RF device (e.g., UE 508). In an example method, a visual search can start at an initial location in an image, such as example image 540. Examples of an initial location can include a predetermined location (e.g., the center of the image), the location of a salient object (e.g., user 502 or a table or another object, or a portion thereof), or an arbitrary location (e.g., randomly selected). The visual search can start at the initial location with a relatively narrow window or box, and expand gradually until one or more relevant target objects are detected. The relevance of a target object and the range of the visual search can be based on contextual information as discussed above. The greater the proximity, the smaller the range the visual search can expand. In some cases, if available, proximity or distance information, such as the range between user 502 and UE 508, can be used. For example, knowing the distance between a target object (e.g., user 502) and a table or RF device (e.g., UE 508) can provide an expectation that the region of interest should be expanded at least that distance in order to, for example, encompass user 502 and the table in a refined region of interest 506d. FIG. 5D

[0094] Thus, determination of a refined region of interest can be achieved in conjunction with domain knowledge and contextual information, which can be used to determine visual priors and / or perform an application or action on a target object, such as more accurate localization. Another example scenario involving warehouse monitoring of objects such as boxes is described below.

[0095] FIG. 6A is an example image 610 of target objects arranged according to some aspects of target objects 602, each target object 602 having a region of interest 604 associated therewith. For simplicity, not all target objects and not all regions of interest are labeled in FIG. 6A In some implementations, each target object 602 is associated with at least one RF device (not shown). In some examples, the RF device can be a type of device that can be detected from a distance, such as an RFID tag associated with a unique address, which would enable a monitoring system or device (e.g., an RF reader) to identify the RFID tag and thus the associated target object. Other types of RF devices that can be used in such a scenario include NFC chips.

[0096] In practice, warehouse objects (e.g., boxes) are often organized and stored in some structured manner by type, size, weight, geographic destination, etc. For example, a plurality of boxes can be organized according to weight. As FIG. 6A ​In the example, boxes in the set of boxes 620 can have lighter weights (e.g., within a certain range or below a certain threshold) and can be placed on the top-most shelf. Boxes in the set of boxes 630 can have greater weights than the boxes in the set of boxes 620 and can be placed on the middle shelf. Boxes in the set of boxes 640 can have greater weights than the boxes in the set of boxes 630 (and boxes 620) and can be placed on the floor or bottom shelf. In this example, weight does not necessarily correlate with size, so some smaller boxes (shown near the bottom right corner of the example image 610) can still be placed on the bottom shelf instead of the middle or top shelf.

[0097] A regular object detector performing object detection in the example image 610 can detect the target objects 602 and produce a single region of interest 604 per box, such as the bounding box represented in solid lines. In a tightly packed or cluttered layout (e.g., multiple boxes adjacent to each other and / or stacked on top of each other), the object detector can produce only a single region of interest (e.g., the bounding box 644) per aisle or shelf.

[0098] However, some embodiments of the present disclosure can consider additional contextual information, such as object information, e.g., object type. If knowledge of the object type (e.g., weight) is available, e.g., based on a unique ID or address of an RF device associated with (e.g., inside of) the target objects 602, this knowledge can be used to select only the relevant regions of interest 646 represented in solid lines, for simplicity, in the example image 610'. FIG. 6B Three of these relevant regions of interest are labeled in the example image 610' to indicate the target objects with the highest weight range.

[0099] Furthermore, the associated RF devices can be placed on specific locations of the objects. In some implementations, the specific locations can be associated with identifiers of the RF devices. For example, a device ID of an RF device placed in one corner of a box can be different from a device ID of an RF device placed in another corner of the box, e.g., the top of the box versus the bottom of the box. However, the exact location can be unknown. For example, the state information related to the RF device can indicate a “corner” type of placement, leaving the RF device possibly in one of the eight corners of the box.

[0100] FIG. 6C is the example image 610' with the regions of interest 656, 658 identified (e.g., narrowed down using contextual information) according to embodiments herein. FIG. 6AExample image 650 of target object 652. An RF device (e.g., an RFID tag) may be placed at one of the corners of target object 652. Possible refined regions of interest 656, 658 may be generated as bounding boxes around the corners of the object using status information (e.g., placement type "corner"). This reduces the search space for downstream applications or actions (such as positioning) compared to using a less refined region of interest 654, especially in tightly packed or cluttered layouts. By defining refined regions of interest (such as region of interest 654) at the corners of target object 652 rather than around the entire target object 652, identifying target object 652 allows for more efficient and accurate differentiation of which RF device is on which target object, and thus allows for more accurate identification of target object 652.

[0101] In some implementations, a probability or likelihood or confidence parameter or similar metric may be associated with each refined region of interest 656, 658. In some implementations, such a confidence parameter may be determined based on additional contextual information and / or assumptions. For example, in addition to weight, orientation information may be used to assign a higher probability to certain corners. If the target object is in an upright orientation (detectable using various information such as “this side up” labels or arrows), it may indicate that the RF device is attached to one of the corners based on standard procedures at the warehouse. In certain cases, the confidence parameter may be represented on the image or stored as a percentage (e.g., 25%, 0.89) or a heatmap indicating probability or likelihood. Based on the confidence parameter of each of the refined regions of interest 656, 658, the refined region of interest 658 with the highest probability can be determined. In some implementations, this refined region of interest 658 with the highest probability can be used for downstream applications or actions, such as the positioning of the RF device and / or the target object 652.

[0102] In some implementations, once the confidence parameter reaches a predetermined threshold, further collection of contextual information is unnecessary. This is because a sufficiently high confidence parameter can be considered sufficient to determine the most likely refined region of interest 658, or downstream applications or actions do not require the level of accuracy achievable using additional contextual information. Stopping the collection of additional contextual information saves time and energy associated with making requests, receiving data, and processing.

[0103] Method

[0104] FIG. 7 This is a call flowchart 700 illustrating a process for determining a refined region of interest according to some implementation schemes. It is used for execution... FIG. 7The functional structures illustrated by one or more of the operations shown can be executed by hardware and / or software components of a computer system or network device (e.g., a server). Components of such a computer system or network device may include, for example, one or more transceivers, memory, one or more processors, and / or computer-readable means including a storage medium storing computer-readable and / or computer-executable instructions configured to cause one or more processors or computerized means to perform the operations indicated by the boxes or arrows below when executed by one or more processors. FIG. 9 Example components of a computerized device (e.g., a server) are illustrated herein, and these example components are described in more detail below. It should also be noted that... FIG. 7 The operations can be performed in any suitable order, and do not have to be... FIG. 7 The order in which it is described. Furthermore, FIG. 7 The process shown may include more FIG. 7 The number of operations depicted is more or less to determine the refined region of interest.

[0105] Call flowchart 700 begins at box 702, which represents a computerized device capable of performing the operations described below. In some embodiments, the device may be a server, such as a location server. Location server 160 and external client 180 may be examples of this device. According to some embodiments, a device performing one or more of the operations described below will be referred to as a server. However, the operations are not limited to being performed by a server; the operations may be performed by other devices with substantially equivalent effects. In some embodiments, an RF device or UE may perform at least some of the operations, provided there are sufficient computing resources.

[0106] In some privacy-preserving implementations, operations can be performed at one or more sensors (e.g., cameras or other image capture devices, RF sensors) where sensor data corresponding to a scene associated with at least a portion of a target object can be captured. Raw image data and / or optical or RF images representing the image data may not be transmitted to a server, and regions of interest can be determined and generated locally at the sensor. In such implementations, the server may transmit necessary information to the sensor for the generation and refinement of the region of interest.

[0107] In some embodiments, at step 704, the server can obtain context information, e.g., state information and / or object information, from a database. In some cases, the information can be received from a remote storage device. In some cases, the information can be received from a local storage device (e.g., at the server). The received state information and / or object information can include historical information and / or recent information obtained over a period of time. The historical information can include, for example, a previous set of states (e.g., one or more of the previous states) of the state information and / or object information. The historical information can also include a length of the period of time over which the previous set of states was obtained or recorded. Multiple instances or records of the state or object information can indicate changes in the state over time. In an illustrative, non-limiting example, if motion information indicates that the UE is moving inconsistently but moving at an average of 30 miles per hour over a 15 minute period (with motion information obtained every minute), the motion information can indicate that the UE is being driven or is inside a vehicle. In an outdoor monitoring scenario, this historical information can indicate that the region of interest is more likely to be a vehicle or a part of a vehicle. In another illustrative example, placement information of a target object can vary over time. For example, at a first time, a target object (or a candidate for the target object) can be placed at a first distance from another object. At a second time, the target object and the other object can be placed at a second distance (different from the first distance) apart from each other. This historical object information can include a series of distances that vary one or more times over time, which can indicate a change in location of the target object from one shelf to another, from a user’s hand to a table, etc. This historical information can assist in identifying the RF device or the target object, determining a more refined region of interest around the RF device or the target object, or even predicting where to perform a visual search to detect the RF device or the target object over time.

[0108] Depending on the application and configuration, the database can be accessible from multiple storage devices, whether at the server and / or remote databases. That is, some of the contextual information can be stored at the server and already known. For example, in an industrial monitoring scenario, some of the contextual information can be stored on the server or local database. Some of the information stored locally can include the ID of the RF device, object information (e.g., size, weight, color, type), the relative placement of the target object and / or RF device. As such, the server can retrieve the domain knowledge and information by looking up the RF device ID (e.g., in a lookup table or data structure). In some scenarios or monitoring applications, the contextual information can be dynamic, changing over time (e.g., activity status). Accordingly, the server can communicate with the database, RF device, or UE to obtain the contextual information from a remote database that acquires the contextual information over time. As will be discussed below with respect to step 716, the server can also receive additional contextual information on demand from elsewhere.

[0109] At step 706, the server can obtain image data. In some embodiments, the image data can be received from one or more sensors or image capture devices (e.g., cameras, RF sensors, or a combination thereof). The one or more images and / or RF representations or RF images can represent the image data. The one or more images can be images of at least the target object. The one or more RF devices can be associated with the target object. Examples of RF devices can include UEs, mobile communication devices, and RFID tags. Example images 220, example images 350, example images 452a, 452b, example images 510, 520, 530, 540, example images 610, and example images 650 can be examples of images obtained from one or more sensors.

[0110] Not all domain knowledge and information is necessary, particularly in public monitoring settings where user privacy considerations should be adhered to. Moreover, if the region of interest is very small (e.g., a distant target), the additional contextual information can bring little improvement. Accordingly, even though more contextual information can be used or available, not all of the contextual information can be used when the refined region of interest is initially generated or received at block 710. However, the refined region of interest will be more relevant and targeted to the context compared to a region of interest that does not use any contextual information, such as region of interest 316, resulting in a more accurate and more useful region of interest for downstream applications or actions.

[0111] To this end, the server can use a combined incremental approach with respect to contextual information in determining the refined region of interest. The server can (1) create and maintain a priority list of contextual information, and / or (2) determine whether to seek more contextual information based on reliability requirements of downstream applications or actions.

[0112] In some embodiments, at block 708, the server can create, obtain, and / or maintain a priority list of contextual information. The priority list can include a list of contextual information to request or obtain, ranked in order of priority. The priority assigned to the contextual information can be based on a combination of criteria such as availability, quantity, or quality of the information; the application or action that will be used with the information; or security or user privacy considerations. In some implementations, when determining the refined region of interest, higher priority information can be given a higher weight or higher probability. As an example, the refined region of interest 658 can have a higher confidence parameter based on a higher priority assigned to it, where the higher priority can be assigned based on a greater availability of information related to the refined region of interest 658.

[0113] At block 710, the server can determine and generate or receive (e.g., from a database referenced with respect to step 704 or another device) the refined region of interest according to one or more methods and embodiments described herein. In some embodiments, generating the refined region of interest can use state information and / or object information obtained via step 704. In some implementations, the state information and / or object information used can be according to the priority list of block 708. For example, the top three pieces of contextual information can be used.

[0114] In some embodiments, as mentioned above, the server can determine whether to seek more contextual information depending on the current state of the region of interest. The server can use reliability requirements of the final application or action as a criterion to determine this by checking whether the current region of interest meets such reliability requirements. For example, the server can determine that a downstream application or action, such as localization, requires a certain level of accuracy, reliability, precision, etc. of the region of interest.

[0115] At block 712, the server can determine whether the refined region of interest satisfies the application requirements or action requirements. The aforementioned reliability requirements can be part of the application requirements. For example, the refined region of interest can be used to locate a target object (e.g., a body part of a user holding the UE) as a downstream application or action. The refined region of interest can be considered to satisfy the application requirements or action requirements if at least one dimension of the refined region of interest is smaller than the region of interest that would occur without refinement, or if multiple possible regions of interest are identified with corresponding confidence metrics (e.g., indicating refinement of a single bounding box), or if the area of the generated bounding box is smaller than the area of a known bounding box (e.g., previously generated), or multiple determined refined regions of interest do not deviate from each other (indicating precision and reliability). These are illustrative examples. Various criteria can be used to determine that the refined region of interest satisfies the application requirements or action requirements.

[0116] If the refined region of interest is considered insufficient to satisfy the application requirements or action requirements (e.g., the server is unable to determine an accurate (or sufficiently accurate) location of the target object), at block 714, the server can further determine whether the end of the priority list has been reached.

[0117] If no, at step 716, the server can request additional knowledge or information for the current priority list item. This is part of the incremental approach mentioned above. In some cases, a request can be made to an RF device 718 (mobile communication device, UE, active RFID tag, etc.). In some cases, a request can be made to one or more sensors for context information (relative position, motion state, proximity state, distance, range, etc.) obtained from optical images. At step 719, the requested information can be transmitted to the server.

[0118] If the end of the priority list has been reached, or if the refined region of interest satisfies the application requirements or action requirements, the server can proceed to a final determination of the refinement of the region of interest at block 720. If additional context information is obtained, the additional context information can be used to further refine the region of interest. In some cases, if further context information cannot be gathered via step 716, no further refinement can be produced. The final determined refined region of interest can be the same as the refined region of interest generated at block 710, or other post-processing procedures can be performed (e.g., from multiple possible refined regions of interest such as FIG. 6CThe highest probability of refinement (such as those regions of interest shown) is identified in the highest-level refinement, filtering, cropping, and image segmentation. In some cases, if refinement is not possible at boxes 710 and 720 (e.g., because the context information is not available locally, in a database, or from the RF device or UE), then a "legacy" region of interest that has not yet undergone refinement using context information can be determined at box 720.

[0119] Optionally, at box 722, an application or action (also referred to herein as a downstream application or action) can be performed. For example, an application or action can be performed with respect to a target object or RF device. Examples of executable downstream applications or actions include the localization of at least a portion of a target object (e.g., a portion of a user using the UE during a call, an environment where the user is using a UE utilizing speaker mode, a box in a warehouse), the generation of visual priors (a three-dimensional representation of the target object, which itself can be used, for example, for the localization of the target object), sensor or camera calibration (in which the sensor or camera uses bounding boxes or shapes around a known object (e.g., RF device, UE) to determine inherent parameters (where the better or more refined the bounding boxes, the better the calibration)), and RF-assisted object detection (in which contextual information (e.g., for a machine learning model, such as a deep neural network model) provides accurate reference or training data to generate more relevant bounding boxes, which can then be used for the downstream applications or actions mentioned above).

[0120] FIG. 8 This is a flowchart of method 800 for determining a refined region of interest based on some implementation schemes. It is used for execution. FIG. 8 The functional structures illustrated in one or more boxes shown can be implemented by hardware and / or software components of a computer system or network device (e.g., a server). Components of such a computer system or network device may include, for example, one or more transceivers, one or more processors, and / or computer-readable means including a storage medium storing computer-readable and / or computer-executable instructions configured to cause one or more processors or the computer system or network device to perform the operations indicated by the boxes below when executed by one or more processors. FIG. 9 Example components of a computer system or network device are illustrated herein, and these example components are described in more detail below. It should also be noted that... FIG. 8 The operations can be performed in any suitable order, and do not have to be... FIG. 8 The order in which it is described. Furthermore, FIG. 8 The process shown may include more FIG. 8 The number of operations depicted is more or less to determine the refined region of interest.

[0121] At block 810, the method 800 can include obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, where the RF device is associated with the target object. The sensor data can include, for example, image data from one or more cameras and / or RF sensing data from one or more RF sensors. The image data can be represented by one or more images (e.g., optical images). The RF sensing data can be represented by one or more RF images or RF representations.

[0122] In some embodiments, obtaining the sensor data can include receiving, from one or more cameras, image data associated with at least a portion of the target object, receiving, from one or more RF sensors, RF sensing data associated with at least a portion of the target object, or a combination thereof. The sensor information can be from a scene such as an industrial environment (e.g., a warehouse), a public environment, a private indoor environment, etc. In some embodiments, the RF device can be a wireless or mobile communication device (e.g., a UE such as a smartphone, a tablet computer, a smartwatch) or an RFID tag (active or passive) that can be attached to, held by, in proximity to, or otherwise associated with the target object. The target object can be an inanimate object (e.g., a box), a moving object (e.g., a car), a user, or a portion thereof (e.g., a user’s upper body). In some embodiments, the RF device can include a mobile communication device or an RFID tag.

[0123] Components for performing the functionality at block 810 can include the communication subsystem 930, the wireless communication interface 933, the wireless antenna 950, and / or other components of the server, as FIG. 9 illustrated.

[0124] At block 820, the method 800 can include obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof. In some embodiments, the state information related to the RF device can include current state information including a current state of the state information, historical state information including one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information can further include one or more types of state information, where the one or more types of state information include a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof. In some embodiments, the object information related to the target object can include current object information including a current state of the object information, historical object information including one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information can further include one or more types of object information, where the one or more types of object information include a type of the target object, a location of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of location of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof. These categories of state information and object information are described in greater detail elsewhere herein.

[0125] In some embodiments, obtaining the state information, the object information, or a combination thereof can include receiving at least a portion of the state information, at least a portion of the object information, or a combination thereof from the RF device; accessing at least a portion of the state information, at least a portion of the object information, or a combination thereof at the storage device; or a combination thereof.

[0126] In some embodiments, the method 800 can further include requesting, from the RF device, additional state information related to the RF device, additional object information related to the target object, or a combination thereof.

[0127] Components for performing the functionality at block 820 can include the bus 905, the one or more non-transitory storage devices 925, the communication subsystem 930, the wireless communication interface 933, the wireless antenna 950, and / or other components of the server, as FIG. 9 illustrated.

[0128] At block 830, the method 800 can include performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to at least a portion of the RF device and the target object, where the region of interest is based at least in part on the state information, the object information, or a combination thereof. In some cases, the region of interest can be displayable and visible on a rendered image representation (e.g., optical image, RF radio image, RF representation) based on the sensor data.

[0129] In some embodiments, performing an action with respect to the target object can include localizing the target object based on the region of interest. In some embodiments, the action can include generating visual priors (visual localization priors), sensor or camera calibration, RF-aided object detection, providing reference data, or training a machine learning model, as discussed above with respect to block 722. In some embodiments, the action can include notifying the RF device to obtain additional sensor data (e.g., image data, RF sensor data), which in some example cases can be based on boundary information (e.g., on the edges of the image data). This can be in addition to or in lieu of step 716 for the current priority list item, resulting in additional incremental information, where obtaining additional image data can be used to detect, identify, or generate the target object or a refined region of interest.

[0130] In some embodiments, the region of interest can include a region associated with at least a portion of the target object identified based on the state information related to the RF device.

[0131] In some embodiments, the method 800 can further include determining respective likelihoods associated with a plurality of possible placements of the RF device, where the region of interest includes a portion of the plurality of possible placements of the RF device determined based on the respective likelihoods. FIG. 6C The example image 650 can be an illustrative example of the plurality of possible placements of the RF device.

[0132] In some embodiments, the method 800 can further include generating or receiving a priority list including a priority associated with each of: the state information related to the RF device, the object information related to the target object, or a combination thereof; where the region of interest can be determined based on the priority list.

[0133] In some embodiments, the method 800 can further include requesting, from the RF device, additional state information related to the RF device, additional object information related to the target object, or a combination thereof.

[0134] The means for performing the functionality at block 830 can include the bus 905, the processor 910, the working memory 935, one or more applications 945, and / or other components of the server as FIG. 9 illustrated.

[0135] Apparatus

[0136] FIG. 9 is a block diagram of an embodiment of a computer system 900 that can be used in whole or in part to provide one or more network components (e.g., a server, a client device, etc.) as described in the embodiments herein. FIG. 1the location server 160 or external client 180). It should be noted that FIG. 9 Only intended to provide a generalized illustration of various components, any or all of which can be utilized as appropriate. Therefore, FIG. 9 The generalized illustration of the individual system elements can be implemented in a relatively decentralized or relatively more integrated manner. Also, FIG. 9 The illustrated components can be localized to a single device and / or distributed among various networked devices that can be disposed at different geographic locations.

[0137] The computer system 900 is shown comprising hardware elements that can be electrically coupled via a bus 905 (or can otherwise be in communication as appropriate). The hardware elements can include a processing unit(s) 910 which can include without limitation one or more general-purpose processors, one or more special-purpose processors (such as digital signal processing chips, graphics acceleration processing units, and / or the like), and / or other processing structure, which can be configured to perform one or more of the methods described herein. The computer system 900 also can include one or more input devices 915, which can include without limitation a mouse, a keyboard, a camera, a microphone, and / or the like; and one or more output devices 920, which can include without limitation a display device, a printer, and / or the like.

[0138] The computer system 900 can also include (and / or be in communication with) one or more non-transitory storage devices 925, which can include, without limitation, local and / or network accessible storage, and / or can include, without limitation, a disk drive, a drive array, an optical storage device, solid-state storage device such as a RAM and / or ROM, which can be programmable, and / or flash-updateable, and / or the like. Such storage devices can be configured to implement any appropriate data stores, including without limitation, various file systems, database structures, and / or the like. Such data stores can include databases and / or other data structures that are used to store and manage messages and / or other information to be communicated to one or more devices via a hub, as described herein.

[0139] The computer system 900 can also include a communications subsystem 930, which can include both wireless communication techniques managed and controlled by a wireless communication interface 933, as well as wired techniques (such as Ethernet, coaxial communications, Universal Serial Bus (USB), and the like). The wireless communication interface 933 can include one or more wireless transceivers, which can transmit and receive wireless signals 955 (e.g., signals according to 5G NR or LTE) via a wireless antenna 950. Thus, the communications subsystem 930 can include modems, network cards (wireless or wired), infrared communication devices, wireless communication devices and / or chipsets, and the like, which can enable the computer system 900 to communicate over any or all of the communications networks described herein with any of the devices on the respective networks, including user equipment (UE), base stations and / or other TRPs, and / or any other electronic devices described herein. Thus, the communications subsystem 930 can be used to receive and transmit data as described in the implementations herein.

[0140] In many embodiments, the computer system 900 will also include a working memory 935, which can include RAM or ROM devices, as described above. Software elements, shown as being located within the working memory 935, can include an operating system 940, device drivers, executable libraries, and / or other code such as one or more applications 945, which can include the computer programs provided by various embodiments, and / or can be designed to implement methods, and / or configure systems provided by other embodiments, as described herein. By way of example, one or more processes described with respect to the methods discussed above can be implemented as code and / or instructions executable by a computer (and / or a processor within a computer); in an aspect, such code and / or instructions can be used to configure and / or adapt a general purpose computer (or other device) to perform one or more operations in accordance with the described methods.

[0141] A set of these instructions and / or code might be stored on a non-transitory computer-readable storage medium, such as the storage device(s) 925 described above. In some cases, the storage medium might be incorporated within a computer system, such as computer system 900. In other implementations, the storage medium might be separate from a computer system (e.g., a removable medium, such as a compact disc), and / or provided in an installation package, such that the storage medium can be used to program, configure and / or adapt a general purpose computer with the instructions / code stored thereon. These instructions might take the form of executable code, which is executable by the computer system 900 and / or might take the form of source and / or installed code, which, upon compilation and / or installation on the computer system 900 (e.g., using any of a variety of generally available compilers, installation programs, compression / decompression utilities, etc.), then takes the form of executable code.

[0142] It will be apparent to those skilled in the art that substantial variations can be made in form, detail, and use of the application without departing from the spirit of the application. For example, customization hardware might also be used, and / or particular elements might be implemented in hardware, software (including portable software, such as applets, etc.), or both. Further, connection to other computing devices such as network input / output devices can be employed.

[0143] With reference to the appended drawings, components of the present application can include a memory that can include a non-transitory machine-readable medium. The terms "machine-readable medium" and "computer-readable medium" as used herein, refer to any storage medium that participates in providing instructions to a processor for execution. In the embodiments provided above, various machine-readable media might be involved in storing and / or carrying the instructions / code used to provide the processor(s) with the instructions / code for execution. Additionally or alternatively, the instructions / code might be transferred from one machine-readable medium to another machine-readable medium (e.g., from a storage medium to one or more dynamic storage devices and / or a communications interface). When introduced into the computer system, the resulting utilizations of the instructions / code by that processor would transform the general, non- specifically tailored computer system into a specially tailored computer system designed to provide the particular functionality taught by the instructions / code. The terms "machine-readable medium" and "computer-readable medium" also encompass all types of storage media that are used to store data that is thereafter to be transferred to another storage medium or to a dynamic storage device such as a computer system memory or a communications interface. The terms "machine-readable medium" and "computer-readable medium" also encompass transmission media such as carrier waves that have been modulated to carry code and / or data.

[0144] The methods, systems, and devices discussed herein are examples. Various implementations can omit, substitute, or add various procedures or components as appropriate. For instance, features described with respect to certain implementations can be combined in various other implementations. Different aspects and elements of the implementations can be combined in a similar manner. Also, techniques described herein can be employed in hardware and / or software having a programmable and / or fixed functionality. Again, numerous elements are illustrative only and are not limiting of the scope of the disclosure.

[0145] It proves convenient at times, principally for reasons of common usage, to refer to such signals as bits, information, values, elements, symbols, characters, variables, numbers, numerals, or the like. It should be understood, however, that all of these or similar terms are to be associated with appropriate physical quantities and are merely convenient labels. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout this specification discussions utilizing terms such as "processing," "computing," "calculating," "determining," "ascertaining," "identifying," "associating," "measuring," "performing," or the like can refer to actions or processes of a specific apparatus, such as a special purpose computer or a similar special purpose electronic computing device. In the context of this specification, therefore, a special purpose computer or a similar special purpose electronic computing device is capable of manipulating or transforming signals, typically represented as physical electronic, electrical, or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the special purpose computer or similar special purpose electronic computing device.

[0146] As used herein, the terms "and" and "or" can include a variety of meanings that also is expected to depend at least in part upon the context in which such terms are used. Typically, "or" if used to associate a list, such as A, B, or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular or can be used to describe some combination of features, structures, or characteristics. However, it should be noted that this is merely an illustrative example and claimed subject matter is not limited to this example. Furthermore, the term "at least one of" if used to associate a list, such as A, B, or C, can be interpreted in the

[0147] Having described several embodiments, various modifications, alternative constructions, and equivalents can be used without departing from the scope of the disclosure. For example, the above elements can merely be components of a larger system, wherein other rules can take precedence over, or otherwise modify, the application of the various embodiments. Additionally, a number of steps can be undertaken before, during, or after the above elements are considered. Accordingly, the above description does not limit the scope of the disclosure.

[0148] In view of this description, embodiments can include different combinations of features. Specific examples of embodiments are described in the numbered clauses below:

[0149] Clause 1. A method of determining a refined region of interest associated with a radio frequency (RF) device, the method comprising: obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least the portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0150] Clause 2. The method of clause 1, wherein performing the action with respect to the target object comprises localizing the target object based on the region of interest.

[0151] Clause 3. The method of any one of clauses 1-2, wherein the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; and obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

[0152] Clause 4. The method of any one of clauses 1-3, wherein the state information related to the RF device comprises: current state information comprising a current state of the state information, historical state information comprising one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information further comprises one or more types of the state information, wherein the one or more types of the state information comprise: a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information with respect to the target object, or a combination thereof.

[0153] Clause 5. The method of any one of clauses 1-4, wherein the region of interest comprises a region associated with the at least the portion of the target object identified based on the state information related to the RF device.

[0154] Clause 6. The method of any one of clauses 1-5, wherein the object information related to the target object comprises: current object information comprising a current state of the object information, historical object information comprising one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information further comprises one or more types of the object information, wherein the one or more types of the object information comprises: a type of the target object, a location of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of location of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

[0155] Clause 7. The method of any one of clauses 1-6, the method further comprising determining respective likelihoods associated with a plurality of possible placements of the RF device; wherein the region of interest comprises a portion of the plurality of possible placements of the RF device determined based on the respective likelihoods.

[0156] Clause 8. The method of any one of clauses 1-7, wherein obtaining the state information, the object information, or the combination thereof comprises: receiving at least a portion of the state information, at least a portion of the object information, or a combination thereof from the RF device; accessing at least a portion of the state information, at least a portion of the object information, or a combination thereof at the storage device; or a combination thereof.

[0157] Clause 9. The method of any one of clauses 1-8, the method further comprising generating or receiving a priority list comprising a priority associated with each of: the state information related to the RF device, the object information related to the target object, or the combination thereof; wherein the region of interest is determined based on the priority list.

[0158] Clause 10. The method of any one of clauses 1-9, the method further comprising requesting, from the RF device, additional state information related to the RF device, additional object information related to the target object, or a combination thereof.

[0159] Clause 11. A network device comprising: one or more communication interfaces configured for data communication with one or more sensors and a storage device; one or more memories; and one or more processors communicatively coupled to the one or more communication interfaces and the one or more memories and configured to: obtain, from the one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtain, from the RF device or the storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and perform an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least the portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0160] Clause 12. The network device of clause 11, wherein performing the action with respect to the target object comprises localizing the target object based on the region of interest.

[0161] Clause 13. The network device of any one of clauses 11 to 12, wherein the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; and obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

[0162] Clause 14. The network device of any one of clauses 11 to 13, wherein the state information related to the RF device comprises: current state information comprising a current state of the state information, historical state information comprising one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information further comprises one or more types of the state information, wherein the one or more types of the state information comprise: a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof.

[0163] Clause 15. The network device of any one of clauses 11 to 14, wherein the region of interest comprises a region associated with at least the portion of the target object identified based on the state information related to the RF device.

[0164] Clause 16. The network device of any one of clauses 11-15, wherein the object information related to the target object comprises: current object information comprising a current state of the object information, historical object information comprising one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information further comprises one or more types of the object information, wherein the one or more types of the object information comprise: a type of the target object, a location of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of location of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

[0165] Clause 17. The network device of any one of clauses 11-16, wherein the one or more processors are further configured to determine respective likelihoods associated with a plurality of possible placements of the RF device; wherein the region of interest comprises a portion of the plurality of possible placements of the RF device determined based on the respective likelihoods.

[0166] Clause 18. The network device of any one of clauses 11-17, wherein obtaining the state information, the object information, or the combination thereof comprises: receiving at least a portion of the state information, at least a portion of the object information, or a combination thereof from the RF device; accessing at least a portion of the state information, at least a portion of the object information, or a combination thereof at the storage device; or a combination thereof.

[0167] Clause 19. The network device of any one of clauses 11-18, wherein the one or more processors are further configured to generate or receive a priority list comprising a priority associated with each of: the state information related to the RF device, the object information related to the target object, or the combination thereof; wherein the region of interest is determined based on the priority list.

[0168] Clause 20. The network device of any one of clauses 11-19, wherein the one or more processors are further configured to request, from the RF device, additional state information related to the RF device, additional object information related to the target object, or a combination thereof.

[0169] Clause 21. A non-transitory computer-readable device comprising a storage medium including a plurality of instructions configured, upon execution by one or more processors, to cause a computerized device to: obtain, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtain, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and perform an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least the portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0170] Clause 22. The non-transitory computer-readable device of clause 21, wherein performing the action with respect to the target object comprises localizing the target object based on the region of interest.

[0171] Clause 23. The non-transitory computer-readable device of any one of clauses 21-22, wherein the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

[0172] Clause 24. The non-transitory computer-readable device of any one of clauses 21-23, wherein the state information related to the RF device comprises: current state information comprising a current state of the state information, historical state information comprising one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information further comprises one or more types of the state information, wherein the one or more types of the state information comprise: a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof.

[0173] Clause 25. The non-transitory computer-readable device of any of clauses 21-24, wherein the object information related to the target object comprises: current object information comprising a current state of the object information, historical object information comprising one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information further comprises one or more types of the object information, wherein the one or more types of the object information comprise: a type of the target object, a position of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of position of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

[0174] Clause 26. The non-transitory computer-readable device of any of clauses 21-25, wherein obtaining the state information, the object information, or the combination thereof comprises: receiving at least a portion of the state information, at least a portion of the object information, or a combination thereof from the RF device; accessing at least a portion of the state information, at least a portion of the object information, or a combination thereof at the storage device; or a combination thereof.

[0175] Clause 27. An apparatus comprising: means for obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; means for obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and means for performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least the portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

[0176] Clause 28. The apparatus of clause 27, wherein the means for performing the action with respect to the target object comprises means for localizing the target object based on the region of interest.

[0177] Clause 29. The apparatus of any of clauses 27-28, wherein the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; and the means for obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

[0178] Clause 30. The apparatus of any of clauses 27-29, wherein the state information related to the RF device comprises: a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof; and the object information related to the target object comprises: a type of the target object, a location of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of location of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

Claims

1. A method of determining a refined region of interest associated with a radio frequency (RF) device, the method comprising: obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least the portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

2. The method of claim 1, wherein performing the action with respect to the target object comprises localizing the target object based on the region of interest.

3. The method of claim 1, wherein: the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; and obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

4. The method of claim 1, wherein: the state information related to the RF device comprises current state information comprising a current state of the state information, historical state information comprising one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information further comprises one or more types of the state information, wherein the one or more types of the state information comprise a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof.

5. The method of claim 4, wherein the region of interest comprises a region associated with at least the portion of the target object identified based on the state information related to the RF device.

6. The method of claim 1, wherein: the object information related to the target object comprises current object information comprising a current state of the object information, historical object information comprising one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information further comprises one or more types of the object information, wherein the one or more types of the object information comprise a type of the target object, a position of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of position of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

7. The method of claim 6, further comprising determining respective likelihoods associated with a plurality of possible placements of the RF device; wherein the region of interest comprises a portion of the plurality of possible placements of the RF device determined based on the respective likelihoods.

8. The method of claim 1, wherein obtaining the state information, the object information, or the combination thereof comprises: receiving, from the RF device, at least a portion of the state information, at least a portion of the object information, or a combination thereof; accessing, at the storage device, at least a portion of the state information, at least a portion of the object information, or a combination thereof; or a combination thereof.

9. The method of claim 1, further comprising generating or receiving a priority list comprising a priority associated with each of: the state information related to the RF device, the object information related to the target object, or the combination thereof; wherein the region of interest is determined based on the priority list.

10. The method of claim 1, further comprising requesting, from the RF device, additional state information related to the RF device, additional object information related to the target object, or a combination thereof.

11. A network device comprising: one or more communication interfaces configured for data communication with one or more sensors and a storage device; one or more memories; and one or more processors communicatively coupled to the one or more communication interfaces and the one or more memories and configured to: obtain, from the one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtain, from the RF device or the storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and perform an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least the portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

12. The network device of claim 11, wherein performing the action with respect to the target object comprises localizing the target object based on the region of interest.

13. The network device of claim 11, wherein: the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; and obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

14. The network device of claim 11, wherein the status information related to the RF device comprises: current state information including a current state of the state information, historical state information including one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information further includes one or more types of the state information, wherein the one or more types of the state information include: a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof.

15. The network device of claim 14, wherein the region of interest includes a region associated with the at least the portion of the target object identified based on the state information related to the RF device.

16. The network device of claim 11, wherein: the object information related to the target object includes: current object information including a current state of the object information, historical object information including one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information further includes one or more types of the object information, wherein the one or more types of the object information include: a type of the target object, a location of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of location of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

17. The network device of claim 16, wherein the one or more processors are further configured to determine respective likelihoods associated with a plurality of possible placements of the RF device; wherein the region of interest includes a portion of the plurality of possible placements of the RF device determined based on the respective likelihoods.

18. The network device of claim 11, wherein obtaining the state information, the object information, or the combination thereof includes: receiving at least a portion of the state information, at least a portion of the object information, or a combination thereof from the RF device; accessing at least a portion of the state information, at least a portion of the object information, or a combination thereof at the storage device; or a combination thereof.

19. The network device of claim 11, wherein the one or more processors are further configured to generate or receive a priority list including a priority associated with each of: the state information related to the RF device, the object information related to the target object, or the combination thereof; wherein the region of interest is determined based on the priority list.

20. The network device of claim 11, wherein the one or more processors are further configured to request, from the RF device, additional state information related to the RF device, additional object information related to the target object, or a combination thereof.

21. A non-transitory computer-readable device comprising a storage medium comprising a plurality of instructions configured, upon execution by one or more processors, to cause a computerized device to: obtain, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; obtain, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and perform an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

22. The non-transitory computer-readable device of claim 21, wherein performing the action with respect to the target object comprises localizing the target object based on the region of interest.

23. The non-transitory computer-readable device of claim 21, wherein: the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

24. The non-transitory computer readable device of claim 21, wherein the status information related to the RF device comprises: the state information comprises current state information comprising a current state of the state information, historical state information comprising one or more states of the state information obtained prior to the current state, or a combination thereof; and the state information further comprises one or more types of the state information, wherein the one or more types of the state information comprise: a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof.

25. The non-transitory computer-readable device of claim 21, wherein: the object information related to the target object comprises: current object information comprising a current state of the object information, historical object information comprising one or more states of the object information obtained prior to the current state, or a combination thereof; and the object information further comprises one or more types of the object information, wherein the one or more types of the object information comprise: a type of the target object, a position of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of position of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

26. The non-transitory computer-readable device of claim 21, wherein obtaining the state information, the object information, or the combination thereof comprises: receiving at least a portion of the state information, at least a portion of the object information, or a combination thereof from the RF device; accessing at least a portion of the state information, at least a portion of the object information, or a combination thereof at the storage device; or a combination thereof.

27. An apparatus comprising: means for obtaining, from one or more sensors, sensor data corresponding to a scene associated with at least a portion of a target object, wherein the RF device is associated with the target object; means for obtaining, from the RF device or a storage device, state information related to the RF device, object information related to the target object, or a combination thereof; and means for performing an action with respect to the target object based on a region of interest associated with the sensor data, the region of interest corresponding to the RF device and at least a portion of the target object, wherein the region of interest is based at least in part on the state information, the object information, or the combination thereof.

28. The apparatus of claim 27, wherein the means for performing the action with respect to the target object comprises means for localizing the target object based on the region of interest.

29. The apparatus of claim 27, wherein: the one or more sensors comprise one or more cameras, one or more radio frequency (RF) sensors, or a combination thereof; and the means for obtaining the sensor data comprises receiving, from the one or more cameras, image data associated with at least the portion of the target object, receiving, from the one or more RF sensors, RF sensing data associated with at least the portion of the target object, or a combination thereof.

30. The apparatus of claim 27, wherein: the state information related to the RF device comprises a communication state of the RF device, operational information of the RF device, a connectivity state of the RF device, motion information of the RF device, distance information relative to the target object, or a combination thereof; and the object information related to the target object comprises a type of the target object, a position of the target object relative to one or more other objects, a placement of the RF device relative to the target object, a type of position of the RF device relative to the target object, one or more possible placements of the RF device, or a combination thereof.

Images