Privacy-preserving joint object association across images and devices using RF-based position information
The integration of RF-based position estimates and a layered security paradigm in visual-based positioning systems addresses privacy and security concerns by allowing joint object association across images without sharing raw visual data, enhancing both security and functionality.
Patent Information
- Application Number
- PCT/US2024/018961
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-03-07
- Publication Date
- 2024-11-07
AI Technical Summary
Current visual-based positioning systems face challenges in maintaining privacy and security, particularly in joint object association across images captured by different cameras, as they often require sharing raw visual features, which can be compromised by unauthorized access and violate privacy concerns.
The implementation of a privacy-preserving transformation that uses RF-based position estimates as proxies for joint association, allowing the server to associate RF devices across images without requiring complete visual information, and a layered security paradigm that gradually reveals more visual information only when necessary for accurate association.
This approach enhances security and privacy by enabling joint object association across images while minimizing the exchange of sensitive visual data, ensuring privacy preservation and maintaining system functionality without relying on raw visual features.
Smart Images

Figure US2024018961_07112024_PF_FP_ABST
Abstract
Description
Qualcomm Ref. No.2301671WO 1 PRIVACY-PRESERVING JOINT OBJECT ASSOCIATION ACROSS IMAGES AND DEVICES USING RF-BASED POSITION INFORMATION CROSS-REFERENCE TO RELATED APPLICATION
[0001] This application claims the benefit of Greek Application Serial No. 20230100364, entitled “PRIVACY-PRESERVING JOINT OBJECT ASSOCIATION ACROSS IMAGES AND DEVICES USING RF-BASED POSITION INFORMATION” and filed on May 4, 2023, which is expressly incorporated by reference herein in its entirety. TECHNICAL FIELD
[0002] The present disclosure relates generally to communication systems, and more particularly, to a wireless communication involving visual positioning. INTRODUCTION
[0003] Wireless communication systems are widely deployed to provide various telecommunication services such as telephony, video, data, messaging, and broadcasts. Typical wireless communication systems may employ multiple-access technologies capable of supporting communication with multiple users by sharing available system resources. Examples of such multiple-access technologies include code division multiple access (CDMA) systems, time division multiple access (TDMA) systems, frequency division multiple access (FDMA) systems, orthogonal frequency division multiple access (OFDMA) systems, single-carrier frequency division multiple access (SC-FDMA) systems, and time division synchronous code division multiple access (TD-SCDMA) systems.
[0004] These multiple access technologies have been adopted in various telecommunication standards to provide a common protocol that enables different wireless devices to communicate on a municipal, national, regional, and even global level. An example telecommunication standard is 5G New Radio (NR). 5G NR is part of a continuous mobile broadband evolution promulgated by Third Generation Partnership Project (3GPP) to meet new requirements associated with latency, reliability, security, scalability (e.g., with Internet of Things (IoT)), and other requirements. 5G NR includes services associated with enhanced mobile broadband (eMBB), massive machine type communications (mMTC), and ultra-reliable low latency 129025-1887WO01Qualcomm Ref. No.2301671WO 2 communications (URLLC). Some aspects of 5G NR may be based on the 4G Long Term Evolution (LTE) standard. There exists a need for further improvements in 5G NR technology. These improvements may also be applicable to other multi-access technologies and the telecommunication standards that employ these technologies. BRIEF SUMMARY
[0005] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects. This summary neither identifies key or critical elements of all aspects nor delineates the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.
[0006] In an aspect of the disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus transmits, to a set of cameras, a request to provide a region of interest (RoI) descriptor for radio frequency (RF) devices based on visual information captured by the set of cameras. The apparatus receives, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices. The apparatus associates multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0007] In an aspect of the disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus receives a set of images captured by a set of cameras, where the set of images are associated with a set of RF devices. The apparatus associates each RF device in the set of RF devices with a list of regions of interest (RoIs) across the set of images captured by the set of cameras. The apparatus generates a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs. The apparatus estimates a visual position for the set of RF devices based on the set of point clouds.
[0008] To the accomplishment of the foregoing and related ends, the one or more aspects may include the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are indicative, however, 129025-1887WO01Qualcomm Ref. No.2301671WO 3 of but a few of the various ways in which the principles of various aspects may be employed. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] FIG.1 is a diagram illustrating an example of a wireless communications system and an access network.
[0010] FIG. 2A is a diagram illustrating an example of a first frame, in accordance with various aspects of the present disclosure.
[0011] FIG. 2B is a diagram illustrating an example of downlink (DL) channels within a subframe, in accordance with various aspects of the present disclosure.
[0012] FIG.2C is a diagram illustrating an example of a second frame, in accordance with various aspects of the present disclosure.
[0013] FIG. 2D is a diagram illustrating an example of uplink (UL) channels within a subframe, in accordance with various aspects of the present disclosure.
[0014] FIG.3 is a diagram illustrating an example of a base station and user equipment (UE) in an access network.
[0015] FIG. 4 is a diagram illustrating an example of a UE positioning based on reference signal measurements.
[0016] FIG.5 is a diagram illustrating an example of visual-based positioning in accordance with various aspects of the present disclosure.
[0017] FIG. 6 is a diagram illustrating an example projective transformation in accordance with various aspects of the present disclosure.
[0018] FIG.7 is a diagram illustrating an example warehouse scenario where multiple objects are localized or tracked using visual-based positioning in accordance with various aspects of the present disclosure.
[0019] FIG. 8 is a diagram illustrating an example functional blocks and related flows of localizing object(s) in the warehouse in accordance with various aspects of the present disclosure.
[0020] FIG.9 is a diagram illustration an example of a privacy-preserving transformation in accordance with various aspects of the present disclosure.
[0021] FIG.10 is a diagram illustration an example of images with visual features removed in accordance with various aspects of the present disclosure. 129025-1887WO01Qualcomm Ref. No.2301671WO 4
[0022] FIG. 11 is a diagram illustration an example region of interest (RoI) in accordance with various aspects of the present disclosure.
[0023] FIG. 12 is a diagram illustrating an example of estimating the locations of radio frequency (RF) devices and associating RF devices across images based on their estimated locations in accordance with various aspects of the present disclosure.
[0024] FIG.13 is a diagram illustrating an example of associating one or more RF devices to a bounding box centroid in accordance with various aspects of the present disclosure.
[0025] FIG.14 is a diagram illustrating an example of a multiple-tier security paradigm for joint association in accordance with various aspects of the present disclosure.
[0026] FIG. 15 is a communication flow illustrating an example message flow of a server performing privacy-preserving joint object association across images and RF devices with multiple-tier security paradigm in accordance with various aspects of the present disclosure.
[0027] FIG.16 is a diagram illustrating an example scenario where RF devices are not visible to a camera in accordance with various aspects of the present disclosure.
[0028] FIG. 17A is a diagram illustrating an example of visual-based position priors in a single-view system in accordance with various aspects of the present disclosure.
[0029] FIG. 17B is a diagram illustrating an example of visual-based position priors in a multiple-view system in accordance with various aspects of the present disclosure.
[0030] FIG.18 is a diagram illustrating an example of visual-based position priors associated with a single-view system in accordance with various aspects of the present disclosure.
[0031] FIG.19 is a diagram illustrating an example of visual-based position priors associated with a multiple-view system in accordance with various aspects of the present disclosure.
[0032] FIG.20 is a diagram illustrating an example sampling-based learning of vision-based position priors in accordance with various aspects of the present disclosure.
[0033] FIG. 21 is a diagram illustrating an example of point clouds in a three-dimensional (3D) space in accordance with various aspects of the present disclosure.
[0034] FIG.22A is a diagram illustrating an example back-projection of pixel samples for a single-view system in accordance with various aspects of the present disclosure.
[0035] FIG.22B is a diagram illustrating an example back-projection of pixel samples for a multiple-view system in accordance with various aspects of the present disclosure. 129025-1887WO01Qualcomm Ref. No.2301671WO 5
[0036] FIG. 23 is a communication flow illustrating an example message flow of a server implemented with a visual position prior estimation mode in accordance with various aspects of the present disclosure.
[0037] FIG.24 is a communication flow illustrating an example logic of the visual position prior estimation in accordance with various aspects of the present disclosure.
[0038] FIG.25 is a flowchart of a method of wireless communication.
[0039] FIG.26 is a flowchart of a method of wireless communication.
[0040] FIG. 27 is a diagram illustrating an example of a hardware implementation for an example network entity.
[0041] FIG.28 is a flowchart of a method of wireless communication.
[0042] FIG.29 is a flowchart of a method of wireless communication.
[0043] FIG. 30 is a diagram illustrating an example of a hardware implementation for an example network entity. DETAILED DESCRIPTION
[0044] Aspects presented herein may improve the security and privacy for visual-based positioning. Aspects presented herein may enable a server (e.g., a local server, a fusion center, an external cloud server, etc.) to associate one or more objects across images captured by different cameras (which may be referred to as joint object association or joint association across images) while applying privacy-preserving transformation (which may be referred to as privacy-preservation) to the images. For example, in one aspect of the present disclosure, a server may be configured to use radio frequency (RF)-based position estimates of the RF device(s) contained in a set of region of interests (RoIs) / objects to perform joint association across images and RF devices. This may alleviate the specification for additional visual features and descriptors and instead use RF position estimates (e.g., pixel location(s) of their projections onto the image) as proxies for performing the joint association. In another aspect of the present disclosure, a layered security paradigm is introduced which provides multiple different security levels, where at each level gradually more visual information is revealed for the purpose of improving the joint association at the server. For example, if a server determines that additional visual information is specified to perform the joint association, the server may request one or more cameras to provide additional visual information gradually until the server is able to perform the joint association. 129025-1887WO01Qualcomm Ref. No.2301671WO 6 This may enable the server to achieve joint association without all / complete visual information (e.g., without specifying raw images).
[0045] In another aspect of the present disclosure, instead of using visual-based positioning estimates (which may also be referred to as visual point estimates), a server may be configured to estimate visual prior distributions for the positions of the RF devices using the RoIs on the images. For example, in one aspect of the present disclosure, a sampling-based visual position prior estimation procedure is provided which may enable a server / camera to estimate the location of one or more RF devices that are not visible to the camera (e.g., not within the camera’s FOV).
[0046] The detailed description set forth below in connection with the drawings describes various configurations and does not represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, these concepts may be practiced without these specific details. In some instances, well known structures and components are shown in block diagram form in order to avoid obscuring such concepts.
[0047] Several aspects of telecommunication systems are presented with reference to various apparatus and methods. These apparatus and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as “elements”). These elements may be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.
[0048] By way of example, an element, or any portion of an element, or any combination of elements may be implemented as a “processing system” that includes one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. One or more processors in the processing system may execute software. Software, 129025-1887WO01Qualcomm Ref. No.2301671WO 7 whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise, shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, or any combination thereof.
[0049] Accordingly, in one or more example aspects, implementations, and / or use cases, the functions described may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computer. By way of example, such computer-readable media can include a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the types of computer- readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.
[0050] While aspects, implementations, and / or use cases are described in this application by illustration to some examples, additional or different aspects, implementations and / or use cases may come about in many different arrangements and scenarios. Aspects, implementations, and / or use cases described herein may be implemented across many differing platform types, devices, systems, shapes, sizes, and packaging arrangements. For example, aspects, implementations, and / or use cases may come about via integrated chip implementations and other non-module-component based devices (e.g., end-user devices, vehicles, communication devices, computing devices, industrial equipment, retail / purchasing devices, medical devices, artificial intelligence (AI)-enabled devices, etc.). While some examples may or may not be specifically directed to use cases or applications, a wide assortment of applicability of described examples may occur. Aspects, implementations, and / or use cases may range a spectrum from chip-level or modular components to non-modular, non-chip- level implementations and further to aggregate, distributed, or original equipment manufacturer (OEM) devices or systems incorporating one or more techniques herein. In some practical settings, devices incorporating described aspects and features may also include additional components and features for implementation and practice of 129025-1887WO01Qualcomm Ref. No.2301671WO 8 claimed and described aspect. For example, transmission and reception of wireless signals necessarily includes a number of components for analog and digital purposes (e.g., hardware components including antenna, RF-chains, power amplifiers, modulators, buffer, processor(s), interleaver, adders / summers, etc.). Techniques described herein may be practiced in a wide variety of devices, chip-level components, systems, distributed arrangements, aggregated or disaggregated components, end-user devices, etc. of varying sizes, shapes, and constitution.
[0051] Deployment of communication systems, such as 5G NR systems, may be arranged in multiple manners with various components or constituent parts. In a 5G NR system, or network, a network node, a network entity, a mobility element of a network, a radio access network (RAN) node, a core network node, a network element, or a network equipment, such as a base station (BS), or one or more units (or one or more components) performing base station functionality, may be implemented in an aggregated or disaggregated architecture. For example, a BS (such as a Node B (NB), evolved NB (eNB), NR BS, 5G NB, access point (AP), a transmission reception point (TRP), or a cell, etc.) may be implemented as an aggregated base station (also known as a standalone BS or a monolithic BS) or a disaggregated base station.
[0052] An aggregated base station may be configured to utilize a radio protocol stack that is physically or logically integrated within a single RAN node. A disaggregated base station may be configured to utilize a protocol stack that is physically or logically distributed among two or more units (such as one or more central or centralized units (CUs), one or more distributed units (DUs), or one or more radio units (RUs)). In some aspects, a CU may be implemented within a RAN node, and one or more DUs may be co-located with the CU, or alternatively, may be geographically or virtually distributed throughout one or multiple other RAN nodes. The DUs may be implemented to communicate with one or more RUs. Each of the CU, DU and RU can be implemented as virtual units, i.e., a virtual central unit (VCU), a virtual distributed unit (VDU), or a virtual radio unit (VRU).
[0053] Base station operation or network design may consider aggregation characteristics of base station functionality. For example, disaggregated base stations may be utilized in an integrated access backhaul (IAB) network, an open radio access network (O- RAN (such as the network configuration sponsored by the O-RAN Alliance)), or a virtualized radio access network (vRAN, also known as a cloud radio access network (C-RAN)). Disaggregation may include distributing functionality across two or more 129025-1887WO01Qualcomm Ref. No.2301671WO 9 units at various physical locations, as well as distributing functionality for at least one unit virtually, which can enable flexibility in network design. The various units of the disaggregated base station, or disaggregated RAN architecture, can be configured for wired or wireless communication with at least one other unit.
[0054] FIG.1 is a diagram 100 illustrating an example of a wireless communications system and an access network. The illustrated wireless communications system includes a disaggregated base station architecture. The disaggregated base station architecture may include one or more CUs 110 that can communicate directly with a core network 120 via a backhaul link, or indirectly with the core network 120 through one or more disaggregated base station units (such as a Near-Real Time (Near-RT) RAN Intelligent Controller (RIC) 125 via an E2 link, or a Non-Real Time (Non-RT) RIC 115 associated with a Service Management and Orchestration (SMO) Framework 105, or both). A CU 110 may communicate with one or more DUs 130 via respective midhaul links, such as an F1 interface. The DUs 130 may communicate with one or more RUs 140 via respective fronthaul links. The RUs 140 may communicate with respective UEs 104 via one or more radio frequency (RF) access links. In some implementations, the UE 104 may be simultaneously served by multiple RUs 140.
[0055] Each of the units, i.e., the CUs 110, the DUs 130, the RUs 140, as well as the Near- RT RICs 125, the Non-RT RICs 115, and the SMO Framework 105, may include one or more interfaces or be coupled to one or more interfaces configured to receive or to transmit signals, data, or information (collectively, signals) via a wired or wireless transmission medium. Each of the units, or an associated processor or controller providing instructions to the communication interfaces of the units, can be configured to communicate with one or more of the other units via the transmission medium. For example, the units can include a wired interface configured to receive or to transmit signals over a wired transmission medium to one or more of the other units. Additionally, the units can include a wireless interface, which may include a receiver, a transmitter, or a transceiver (such as an RF transceiver), configured to receive or to transmit signals, or both, over a wireless transmission medium to one or more of the other units.
[0056] In some aspects, the CU 110 may host one or more higher layer control functions. Such control functions can include radio resource control (RRC), packet data convergence protocol (PDCP), service data adaptation protocol (SDAP), or the like. Each control function can be implemented with an interface configured to 129025-1887WO01Qualcomm Ref. No.2301671WO 10 communicate signals with other control functions hosted by the CU 110. The CU 110 may be configured to handle user plane functionality (i.e., Central Unit – User Plane (CU-UP)), control plane functionality (i.e., Central Unit – Control Plane (CU-CP)), or a combination thereof. In some implementations, the CU 110 can be logically split into one or more CU-UP units and one or more CU-CP units. The CU-UP unit can communicate bidirectionally with the CU-CP unit via an interface, such as an E1 interface when implemented in an O-RAN configuration. The CU 110 can be implemented to communicate with the DU 130, as necessary, for network control and signaling.
[0057] The DU 130 may correspond to a logical unit that includes one or more base station functions to control the operation of one or more RUs 140. In some aspects, the DU 130 may host one or more of a radio link control (RLC) layer, a medium access control (MAC) layer, and one or more high physical (PHY) layers (such as modules for forward error correction (FEC) encoding and decoding, scrambling, modulation, demodulation, or the like) depending, at least in part, on a functional split, such as those defined by 3GPP. In some aspects, the DU 130 may further host one or more low PHY layers. Each layer (or module) can be implemented with an interface configured to communicate signals with other layers (and modules) hosted by the DU 130, or with the control functions hosted by the CU 110.
[0058] Lower-layer functionality can be implemented by one or more RUs 140. In some deployments, an RU 140, controlled by a DU 130, may correspond to a logical node that hosts RF processing functions, or low-PHY layer functions (such as performing fast Fourier transform (FFT), inverse FFT (iFFT), digital beamforming, physical random access channel (PRACH) extraction and filtering, or the like), or both, based at least in part on the functional split, such as a lower layer functional split. In such an architecture, the RU(s) 140 can be implemented to handle over the air (OTA) communication with one or more UEs 104. In some implementations, real-time and non-real-time aspects of control and user plane communication with the RU(s) 140 can be controlled by the corresponding DU 130. In some scenarios, this configuration can enable the DU(s) 130 and the CU 110 to be implemented in a cloud-based RAN architecture, such as a vRAN architecture.
[0059] The SMO Framework 105 may be configured to support RAN deployment and provisioning of non-virtualized and virtualized network elements. For non-virtualized network elements, the SMO Framework 105 may be configured to support the 129025-1887WO01Qualcomm Ref. No.2301671WO 11 deployment of dedicated physical resources for RAN coverage requirements that may be managed via an operations and maintenance interface (such as an O1 interface). For virtualized network elements, the SMO Framework 105 may be configured to interact with a cloud computing platform (such as an open cloud (O-Cloud) 190) to perform network element life cycle management (such as to instantiate virtualized network elements) via a cloud computing platform interface (such as an O2 interface). Such virtualized network elements can include, but are not limited to, CUs 110, DUs 130, RUs 140 and Near-RT RICs 125. In some implementations, the SMO Framework 105 can communicate with a hardware aspect of a 4G RAN, such as an open eNB (O- eNB) 111, via an O1 interface. Additionally, in some implementations, the SMO Framework 105 can communicate directly with one or more RUs 140 via an O1 interface. The SMO Framework 105 also may include a Non-RT RIC 115 configured to support functionality of the SMO Framework 105.
[0060] The Non-RT RIC 115 may be configured to include a logical function that enables non-real-time control and optimization of RAN elements and resources, artificial intelligence (AI) / machine learning (ML) (AI / ML) workflows including model training and updates, or policy-based guidance of applications / features in the Near- RT RIC 125. The Non-RT RIC 115 may be coupled to or communicate with (such as via an A1 interface) the Near-RT RIC 125. The Near-RT RIC 125 may be configured to include a logical function that enables near-real-time control and optimization of RAN elements and resources via data collection and actions over an interface (such as via an E2 interface) connecting one or more CUs 110, one or more DUs 130, or both, as well as an O-eNB, with the Near-RT RIC 125.
[0061] In some implementations, to generate AI / ML models to be deployed in the Near-RT RIC 125, the Non-RT RIC 115 may receive parameters or external enrichment information from external servers. Such information may be utilized by the Near-RT RIC 125 and may be received at the SMO Framework 105 or the Non-RT RIC 115 from non-network data sources or from network functions. In some examples, the Non-RT RIC 115 or the Near-RT RIC 125 may be configured to tune RAN behavior or performance. For example, the Non-RT RIC 115 may monitor long-term trends and patterns for performance and employ AI / ML models to perform corrective actions through the SMO Framework 105 (such as reconfiguration via O1) or via creation of RAN management policies (such as A1 policies). 129025-1887WO01Qualcomm Ref. No.2301671WO 12
[0062] At least one of the CU 110, the DU 130, and the RU 140 may be referred to as a base station 102. Accordingly, a base station 102 may include one or more of the CU 110, the DU 130, and the RU 140 (each component indicated with dotted lines to signify that each component may or may not be included in the base station 102). The base station 102 provides an access point to the core network 120 for a UE 104. The base station 102 may include macrocells (high power cellular base station) and / or small cells (low power cellular base station). The small cells include femtocells, picocells, and microcells. A network that includes both small cell and macrocells may be known as a heterogeneous network. A heterogeneous network may also include Home Evolved Node Bs (eNBs) (HeNBs), which may provide service to a restricted group known as a closed subscriber group (CSG). The communication links between the RUs 140 and the UEs 104 may include uplink (UL) (also referred to as reverse link) transmissions from a UE 104 to an RU 140 and / or downlink (DL) (also referred to as forward link) transmissions from an RU 140 to a UE 104. The communication links may use multiple-input and multiple-output (MIMO) antenna technology, including spatial multiplexing, beamforming, and / or transmit diversity. The communication links may be through one or more carriers. The base station 102 / UEs 104 may use spectrum up to Y MHz (e.g., 5, 10, 15, 20, 100, 400, etc. MHz) bandwidth per carrier allocated in a carrier aggregation of up to a total of Yx MHz (x component carriers) used for transmission in each direction. The carriers may or may not be adjacent to each other. Allocation of carriers may be asymmetric with respect to DL and UL (e.g., more or fewer carriers may be allocated for DL than for UL). The component carriers may include a primary component carrier and one or more secondary component carriers. A primary component carrier may be referred to as a primary cell (PCell) and a secondary component carrier may be referred to as a secondary cell (SCell).
[0063] Certain UEs 104 may communicate with each other using device-to-device (D2D) communication link 158. The D2D communication link 158 may use the DL / UL wireless wide area network (WWAN) spectrum. The D2D communication link 158 may use one or more sidelink channels, such as a physical sidelink broadcast channel (PSBCH), a physical sidelink discovery channel (PSDCH), a physical sidelink shared channel (PSSCH), and a physical sidelink control channel (PSCCH). D2D communication may be through a variety of wireless D2D communications systems, such as for example, Bluetooth®, Wi-Fi® based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard, LTE, or NR. 129025-1887WO01Qualcomm Ref. No.2301671WO 13
[0064] The wireless communications system may further include a Wi-Fi AP 150 in communication with UEs 104 (also referred to as Wi-Fi stations (STAs)) via communication link 154, e.g., in a 5 GHz unlicensed frequency spectrum or the like. When communicating in an unlicensed frequency spectrum, the UEs 104 / AP 150 may perform a clear channel assessment (CCA) prior to communicating in order to determine whether the channel is available.
[0065] The electromagnetic spectrum is often subdivided, based on frequency / wavelength, into various classes, bands, channels, etc. In 5G NR, two initial operating bands have been identified as frequency range designations FR1 (410 MHz – 7.125 GHz) and FR2 (24.25 GHz – 52.6 GHz). Although a portion of FR1 is greater than 6 GHz, FR1 is often referred to (interchangeably) as a “sub-6 GHz” band in various documents and articles. A similar nomenclature issue sometimes occurs with regard to FR2, which is often referred to (interchangeably) as a “millimeter wave” band in documents and articles, despite being different from the extremely high frequency (EHF) band (30 GHz – 300 GHz) which is identified by the International Telecommunications Union (ITU) as a “millimeter wave” band.
[0066] The frequencies between FR1 and FR2 are often referred to as mid-band frequencies. Recent 5G NR studies have identified an operating band for these mid-band frequencies as frequency range designation FR3 (7.125 GHz – 24.25 GHz). Frequency bands falling within FR3 may inherit FR1 characteristics and / or FR2 characteristics, and thus may effectively extend features of FR1 and / or FR2 into mid- band frequencies. In addition, higher frequency bands are currently being explored to extend 5G NR operation beyond 52.6 GHz. For example, three higher operating bands have been identified as frequency range designations FR2-2 (52.6 GHz – 71 GHz), FR4 (71 GHz – 114.25 GHz), and FR5 (114.25 GHz – 300 GHz). Each of these higher frequency bands falls within the EHF band.
[0067] With the above aspects in mind, unless specifically stated otherwise, the term “sub-6 GHz” or the like if used herein may broadly represent frequencies that may be less than 6 GHz, may be within FR1, or may include mid-band frequencies. Further, unless specifically stated otherwise, the term “millimeter wave” or the like if used herein may broadly represent frequencies that may include mid-band frequencies, may be within FR2, FR4, FR2-2, and / or FR5, or may be within the EHF band.
[0068] The base station 102 and the UE 104 may each include a plurality of antennas, such as antenna elements, antenna panels, and / or antenna arrays to facilitate beamforming. 129025-1887WO01Qualcomm Ref. No.2301671WO 14 The base station 102 may transmit a beamformed signal 182 to the UE 104 in one or more transmit directions. The UE 104 may receive the beamformed signal from the base station 102 in one or more receive directions. The UE 104 may also transmit a beamformed signal 184 to the base station 102 in one or more transmit directions. The base station 102 may receive the beamformed signal from the UE 104 in one or more receive directions. The base station 102 / UE 104 may perform beam training to determine the best receive and transmit directions for each of the base station 102 / UE 104. The transmit and receive directions for the base station 102 may or may not be the same. The transmit and receive directions for the UE 104 may or may not be the same.
[0069] The base station 102 may include and / or be referred to as a gNB, Node B, eNB, an access point, a base transceiver station, a radio base station, a radio transceiver, a transceiver function, a basic service set (BSS), an extended service set (ESS), a TRP, network node, network entity, network equipment, or some other suitable terminology. The base station 102 can be implemented as an integrated access and backhaul (IAB) node, a relay node, a sidelink node, an aggregated (monolithic) base station with a baseband unit (BBU) (including a CU and a DU) and an RU, or as a disaggregated base station including one or more of a CU, a DU, and / or an RU. The set of base stations, which may include disaggregated base stations and / or aggregated base stations, may be referred to as next generation (NG) RAN (NG-RAN).
[0070] The core network 120 may include an Access and Mobility Management Function (AMF) 161, a Session Management Function (SMF) 162, a User Plane Function (UPF) 163, a Unified Data Management (UDM) 164, one or more location servers 168, and other functional entities. The AMF 161 is the control node that processes the signaling between the UEs 104 and the core network 120. The AMF 161 supports registration management, connection management, mobility management, and other functions. The SMF 162 supports session management and other functions. The UPF 163 supports packet routing, packet forwarding, and other functions. The UDM 164 supports the generation of authentication and key agreement (AKA) credentials, user identification handling, access authorization, and subscription management. The one or more location servers 168 are illustrated as including a Gateway Mobile Location Center (GMLC) 165 and a Location Management Function (LMF) 166. However, generally, the one or more location servers 168 may include one or more location / positioning servers, which may include one or more of the GMLC 165, the 129025-1887WO01Qualcomm Ref. No.2301671WO 15 LMF 166, a position determination entity (PDE), a serving mobile location center (SMLC), a mobile positioning center (MPC), or the like. The GMLC 165 and the LMF 166 support UE location services. The GMLC 165 provides an interface for clients / applications (e.g., emergency services) for accessing UE positioning information. The LMF 166 receives measurements and assistance information from the NG-RAN and the UE 104 via the AMF 161 to compute the position of the UE 104. The NG-RAN may utilize one or more positioning methods in order to determine the position of the UE 104. Positioning the UE 104 may involve signal measurements, a position estimate, and an optional velocity computation based on the measurements. The signal measurements may be made by the UE 104 and / or the base station 102 serving the UE 104. The signals measured may be based on one or more of a satellite positioning system (SPS) 170 (e.g., one or more of a Global Navigation Satellite System (GNSS), global position system (GPS), non-terrestrial network (NTN), or other satellite position / location system), LTE signals, wireless local area network (WLAN) signals, Bluetooth® signals, a terrestrial beacon system (TBS), sensor-based information (e.g., barometric pressure sensor, motion sensor), NR enhanced cell ID (NR E-CID) methods, NR signals (e.g., multi-round trip time (Multi-RTT), DL angle- of-departure (DL-AoD), DL time difference of arrival (DL-TDOA), UL time difference of arrival (UL-TDOA), and UL angle-of-arrival (UL-AoA) positioning), and / or other systems / signals / sensors.
[0071] Examples of UEs 104 include a cellular phone, a smart phone, a session initiation protocol (SIP) phone, a laptop, a personal digital assistant (PDA), a satellite radio, a global positioning system, a multimedia device, a video device, a digital audio player (e.g., MP3 player), a camera, a game console, a tablet, a smart device, a wearable device, a vehicle, an electric meter, a gas pump, a large or small kitchen appliance, a healthcare device, an implant, a sensor / actuator, a display, or any other similar functioning device. Some of the UEs 104 may be referred to as IoT devices (e.g., parking meter, gas pump, toaster, vehicles, heart monitor, etc.). The UE 104 may also be referred to as a station, a mobile station, a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communications device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable terminology. In some scenarios, the term UE may also apply to one or more companion devices such as in 129025-1887WO01Qualcomm Ref. No.2301671WO 16 a device constellation arrangement. One or more of these devices may collectively access the network and / or individually access the network.
[0072] Referring again to FIG.1, in certain aspects, a server (e.g., a network entity within the core network 120, a local server, a cloud server, one or more location servers 168, etc.) may include a visual positioning component 197 / 199 that may be configured to transmit, to a set of cameras, a request to provide an RoI descriptor for RF devices based on visual information captured by the set of cameras; receive, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices; and associate multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras. In certain aspects, the visual positioning component 197 may be configured to receive a set of images captured by a set of cameras, where the set of images are associated with a set of RF devices; associate each RF device in the set of RF devices with a list of RoIs across the set of images captured by the set of cameras; generate a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs; and estimate a visual position for the set of RF devices based on the set of point clouds. In certain aspects, the UE 104 may include a camera component 198 that may be configured to capture images for the server.
[0073] FIG.2A is a diagram 200 illustrating an example of a first subframe within a 5G NR frame structure. FIG. 2B is a diagram 230 illustrating an example of DL channels within a 5G NR subframe. FIG. 2C is a diagram 250 illustrating an example of a second subframe within a 5G NR frame structure. FIG. 2D is a diagram 280 illustrating an example of UL channels within a 5G NR subframe. The 5G NR frame structure may be frequency division duplexed (FDD) in which for a particular set of subcarriers (carrier system bandwidth), subframes within the set of subcarriers are dedicated for either DL or UL, or may be time division duplexed (TDD) in which for a particular set of subcarriers (carrier system bandwidth), subframes within the set of subcarriers are dedicated for both DL and UL. In the examples provided by FIGs.2A, 2C, the 5G NR frame structure is assumed to be TDD, with subframe 4 being configured with slot format 28 (with mostly DL), where D is DL, U is UL, and F is flexible for use between DL / UL, and subframe 3 being configured with slot format 1 (with all UL). While subframes 3, 4 are shown with slot formats 1, 28, respectively, any particular subframe may be configured with any of the various available slot 129025-1887WO01Qualcomm Ref. No.2301671WO 17 formats 0-61. Slot formats 0, 1 are all DL, UL, respectively. Other slot formats 2-61 include a mix of DL, UL, and flexible symbols. UEs are configured with the slot format (dynamically through DL control information (DCI), or semi- statically / statically through radio resource control (RRC) signaling) through a received slot format indicator (SFI). Note that the description infra applies also to a 5G NR frame structure that is TDD.
[0074] FIGs.2A-2D illustrate a frame structure, and the aspects of the present disclosure may be applicable to other wireless communication technologies, which may have a different frame structure and / or different channels. A frame (10 ms) may be divided into 10 equally sized subframes (1 ms). Each subframe may include one or more time slots. Subframes may also include mini-slots, which may include 7, 4, or 2 symbols. Each slot may include 14 or 12 symbols, depending on whether the cyclic prefix (CP) is normal or extended. For normal CP, each slot may include 14 symbols, and for extended CP, each slot may include 12 symbols. The symbols on DL may be CP orthogonal frequency division multiplexing (OFDM) (CP-OFDM) symbols. The symbols on UL may be CP-OFDM symbols (for high throughput scenarios) or discrete Fourier transform (DFT) spread OFDM (DFT-s-OFDM) symbols (for power limited scenarios; limited to a single stream transmission). The number of slots within a subframe is based on the CP and the numerology. The numerology defines the subcarrier spacing (SCS) (see Table 1). The symbol length / duration may scale with 1 / SCS.Table 1: Numerology, SCS, and CP 129025-1887WO01Qualcomm Ref. No.2301671WO 18
[0075] For normal CP (14 symbols / slot), different numerologies µ 0 to 4 allow for 1, 2, 4, 8, and 16 slots, respectively, per subframe. For extended CP, the numerology 2 allows for 4 slots per subframe. Accordingly, for normal CP and numerology µ, there are 14 symbols / slot and 2µ slots / subframe. The subcarrier spacing may be equal to 2 ^^^^ ∗15 kHz, where ^^^^ is the numerology 0 to 4. As such, the numerology µ=0 has asubcarrier spacing of 15 kHz and the numerology µ=4 has a subcarrier spacing of 240 kHz. The symbol length / duration is inversely related to the subcarrier spacing. FIGs. 2A-2D provide an example of normal CP with 14 symbols per slot and numerology µ=2 with 4 slots per subframe. The slot duration is 0.25 ms, the subcarrier spacing is 60 kHz, and the symbol duration is approximately 16.67 μs. Within a set of frames, there may be one or more different bandwidth parts (BWPs) (see FIG. 2B) that are frequency division multiplexed. Each BWP may have a particular numerology and CP (normal or extended).
[0076] A resource grid may be used to represent the frame structure. Each time slot includes a resource block (RB) (also referred to as physical RBs (PRBs)) that extends 12 consecutive subcarriers. The resource grid is divided into multiple resource elements (REs). The number of bits carried by each RE depends on the modulation scheme.
[0077] As illustrated in FIG.2A, some of the REs carry reference (pilot) signals (RS) for the UE. The RS may include demodulation RS (DM-RS) (indicated as R for one particular configuration, but other DM-RS configurations are possible) and channel state information reference signals (CSI-RS) for channel estimation at the UE. The RS may also include beam measurement RS (BRS), beam refinement RS (BRRS), and phase tracking RS (PT-RS).
[0078] FIG.2B illustrates an example of various DL channels within a subframe of a frame. The physical downlink control channel (PDCCH) carries DCI within one or more control channel elements (CCEs) (e.g., 1, 2, 4, 8, or 16 CCEs), each CCE including six RE groups (REGs), each REG including 12 consecutive REs in an OFDM symbol of an RB. A PDCCH within one BWP may be referred to as a control resource set (CORESET). A UE is configured to monitor PDCCH candidates in a PDCCH search space (e.g., common search space, UE-specific search space) during PDCCH monitoring occasions on the CORESET, where the PDCCH candidates have different DCI formats and different aggregation levels. Additional BWPs may be located at greater and / or lower frequencies across the channel bandwidth. A primary synchronization signal (PSS) may be within symbol 2 of particular subframes of a 129025-1887WO01Qualcomm Ref. No.2301671WO 19 frame. The PSS is used by a UE 104 to determine subframe / symbol timing and a physical layer identity. A secondary synchronization signal (SSS) may be within symbol 4 of particular subframes of a frame. The SSS is used by a UE to determine a physical layer cell identity group number and radio frame timing. Based on the physical layer identity and the physical layer cell identity group number, the UE can determine a physical cell identifier (PCI). Based on the PCI, the UE can determine the locations of the DM-RS. The physical broadcast channel (PBCH), which carries a master information block (MIB), may be logically grouped with the PSS and SSS to form a synchronization signal (SS) / PBCH block (also referred to as SS block (SSB)). The MIB provides a number of RBs in the system bandwidth and a system frame number (SFN). The physical downlink shared channel (PDSCH) carries user data, broadcast system information not transmitted through the PBCH such as system information blocks (SIBs), and paging messages.
[0079] As illustrated in FIG. 2C, some of the REs carry DM-RS (indicated as R for one particular configuration, but other DM-RS configurations are possible) for channel estimation at the base station. The UE may transmit DM-RS for the physical uplink control channel (PUCCH) and DM-RS for the physical uplink shared channel (PUSCH). The PUSCH DM-RS may be transmitted in the first one or two symbols of the PUSCH. The PUCCH DM-RS may be transmitted in different configurations depending on whether short or long PUCCHs are transmitted and depending on the particular PUCCH format used. The UE may transmit sounding reference signals (SRS). The SRS may be transmitted in the last symbol of a subframe. The SRS may have a comb structure, and a UE may transmit SRS on one of the combs. The SRS may be used by a base station for channel quality estimation to enable frequency- dependent scheduling on the UL.
[0080] FIG.2D illustrates an example of various UL channels within a subframe of a frame. The PUCCH may be located as indicated in one configuration. The PUCCH carries uplink control information (UCI), such as scheduling requests, a channel quality indicator (CQI), a precoding matrix indicator (PMI), a rank indicator (RI), and hybrid automatic repeat request (HARQ) acknowledgment (ACK) (HARQ-ACK) feedback (i.e., one or more HARQ ACK bits indicating one or more ACK and / or negative ACK (NACK)). The PUSCH carries data, and may additionally be used to carry a buffer status report (BSR), a power headroom report (PHR), and / or UCI. 129025-1887WO01Qualcomm Ref. No.2301671WO 20
[0081] FIG.3 is a block diagram of a base station 310 in communication with a UE 350 in an access network. In the DL, Internet protocol (IP) packets may be provided to a controller / processor 375. The controller / processor 375 implements layer 3 and layer 2 functionality. Layer 3 includes a radio resource control (RRC) layer, and layer 2 includes a service data adaptation protocol (SDAP) layer, a packet data convergence protocol (PDCP) layer, a radio link control (RLC) layer, and a medium access control (MAC) layer. The controller / processor 375 provides RRC layer functionality associated with broadcasting of system information (e.g., MIB, SIBs), RRC connection control (e.g., RRC connection paging, RRC connection establishment, RRC connection modification, and RRC connection release), inter radio access technology (RAT) mobility, and measurement configuration for UE measurement reporting; PDCP layer functionality associated with header compression / decompression, security (ciphering, deciphering, integrity protection, integrity verification), and handover support functions; RLC layer functionality associated with the transfer of upper layer packet data units (PDUs), error correction through ARQ, concatenation, segmentation, and reassembly of RLC service data units (SDUs), re-segmentation of RLC data PDUs, and reordering of RLC data PDUs; and MAC layer functionality associated with mapping between logical channels and transport channels, multiplexing of MAC SDUs onto transport blocks (TBs), demultiplexing of MAC SDUs from TBs, scheduling information reporting, error correction through HARQ, priority handling, and logical channel prioritization.
[0082] The transmit (TX) processor 316 and the receive (RX) processor 370 implement layer 1 functionality associated with various signal processing functions. Layer 1, which includes a physical (PHY) layer, may include error detection on the transport channels, forward error correction (FEC) coding / decoding of the transport channels, interleaving, rate matching, mapping onto physical channels, modulation / demodulation of physical channels, and MIMO antenna processing. The TX processor 316 handles mapping to signal constellations based on various modulation schemes (e.g., binary phase-shift keying (BPSK), quadrature phase-shift keying (QPSK), M-phase-shift keying (M-PSK), M-quadrature amplitude modulation (M-QAM)). The coded and modulated symbols may then be split into parallel streams. Each stream may then be mapped to an OFDM subcarrier, multiplexed with a reference signal (e.g., pilot) in the time and / or frequency-domain, and then combined together using an Inverse Fast Fourier Transform (IFFT) to produce a 129025-1887WO01Qualcomm Ref. No.2301671WO 21 physical channel carrying a time-domain OFDM symbol stream. The OFDM stream is spatially precoded to produce multiple spatial streams. Channel estimates from a channel estimator 374 may be used to determine the coding and modulation scheme, as well as for spatial processing. The channel estimate may be derived from a reference signal and / or channel condition feedback transmitted by the UE 350. Each spatial stream may then be provided to a different antenna 320 via a separate transmitter 318Tx. Each transmitter 318Tx may modulate a radio frequency (RF) carrier with a respective spatial stream for transmission.
[0083] At the UE 350, each receiver 354Rx receives a signal through its respective antenna 352. Each receiver 354Rx recovers information modulated onto an RF carrier and provides the information to the receive (RX) processor 356. The TX processor 368 and the RX processor 356 implement layer 1 functionality associated with various signal processing functions. The RX processor 356 may perform spatial processing on the information to recover any spatial streams destined for the UE 350. If multiple spatial streams are destined for the UE 350, they may be combined by the RX processor 356 into a single OFDM symbol stream. The RX processor 356 then converts the OFDM symbol stream from the time-domain to the frequency-domain using a Fast Fourier Transform (FFT). The frequency-domain signal includes a separate OFDM symbol stream for each subcarrier of the OFDM signal. The symbols on each subcarrier, and the reference signal, are recovered and demodulated by determining the most likely signal constellation points transmitted by the base station 310. These soft decisions may be based on channel estimates computed by the channel estimator 358. The soft decisions are then decoded and deinterleaved to recover the data and control signals that were originally transmitted by the base station 310 on the physical channel. The data and control signals are then provided to the controller / processor 359, which implements layer 3 and layer 2 functionality.
[0084] The controller / processor 359 can be associated with a memory 360 that stores program codes and data. The memory 360 may be referred to as a computer-readable medium. In the UL, the controller / processor 359 provides demultiplexing between transport and logical channels, packet reassembly, deciphering, header decompression, and control signal processing to recover IP packets. The controller / processor 359 is also responsible for error detection using an ACK and / or NACK protocol to support HARQ operations. 129025-1887WO01Qualcomm Ref. No.2301671WO 22
[0085] Similar to the functionality described in connection with the DL transmission by the base station 310, the controller / processor 359 provides RRC layer functionality associated with system information (e.g., MIB, SIBs) acquisition, RRC connections, and measurement reporting; PDCP layer functionality associated with header compression / decompression, and security (ciphering, deciphering, integrity protection, integrity verification); RLC layer functionality associated with the transfer of upper layer PDUs, error correction through ARQ, concatenation, segmentation, and reassembly of RLC SDUs, re-segmentation of RLC data PDUs, and reordering of RLC data PDUs; and MAC layer functionality associated with mapping between logical channels and transport channels, multiplexing of MAC SDUs onto TBs, demultiplexing of MAC SDUs from TBs, scheduling information reporting, error correction through HARQ, priority handling, and logical channel prioritization.
[0086] Channel estimates derived by a channel estimator 358 from a reference signal or feedback transmitted by the base station 310 may be used by the TX processor 368 to select the appropriate coding and modulation schemes, and to facilitate spatial processing. The spatial streams generated by the TX processor 368 may be provided to different antenna 352 via separate transmitters 354Tx. Each transmitter 354Tx may modulate an RF carrier with a respective spatial stream for transmission.
[0087] The UL transmission is processed at the base station 310 in a manner similar to that described in connection with the receiver function at the UE 350. Each receiver 318Rx receives a signal through its respective antenna 320. Each receiver 318Rx recovers information modulated onto an RF carrier and provides the information to a RX processor 370.
[0088] The controller / processor 375 can be associated with a memory 376 that stores program codes and data. The memory 376 may be referred to as a computer-readable medium. In the UL, the controller / processor 375 provides demultiplexing between transport and logical channels, packet reassembly, deciphering, header decompression, control signal processing to recover IP packets. The controller / processor 375 is also responsible for error detection using an ACK and / or NACK protocol to support HARQ operations.
[0089] At least one of the TX processor 368, the RX processor 356, and the controller / processor 359 may be configured to perform aspects in connection with the camera component 198 of FIG.1. 129025-1887WO01Qualcomm Ref. No.2301671WO 23
[0090] At least one of the TX processor 316, the RX processor 370, and the controller / processor 375 may be configured to perform aspects in connection with the visual positioning component 199 of FIG.1.
[0091] FIG.4 is a diagram 400 illustrating an example of a UE positioning based on reference signal measurements (which may also be referred to as “network-based positioning”) in accordance with various aspects of the present disclosure. The UE 404 may transmit UL SRS 412 at time TSRS_TX and receive DL positioning reference signals (PRS) (DL PRS) 410 at time TPRS_RX. The TRP 406 may receive the UL SRS 412 at time TSRS_RX and transmit the DL PRS 410 at time TPRS_TX. The UE 404 may receive the DL PRS 410 before transmitting the UL SRS 412, or may transmit the UL SRS 412 before receiving the DL PRS 410. In both cases, a positioning server (e.g., location server(s) 168) or the UE 404 may determine the RTT 414 based on ||TSRS_RX – TPRS_TX| – |TSRS_TX– TPRS_RX||. Accordingly, multi-RTT positioning may make use of the UE Rx-Tx time difference measurements (i.e., |TSRS_TX – TPRS_RX|) and DL PRS reference signal received power (RSRP) (DL PRS-RSRP) of downlink signals received from multiple TRPs 402, 406 and measured by the UE 404, and the measured TRP Rx-Tx time difference measurements (i.e., |TSRS_RX – TPRS_TX|) and UL SRS-RSRP at multiple TRPs 402, 406 of uplink signals transmitted from UE 404. The UE 404 measures the UE Rx-Tx time difference measurements (and / or DL PRS-RSRP of the received signals) using assistance data received from the positioning server, and the TRPs 402, 406 measure the gNB Rx-Tx time difference measurements (and / or UL SRS-RSRP of the received signals) using assistance data received from the positioning server. The measurements may be used at the positioning server or the UE 404 to determine the RTT, which is used to estimate the location of the UE 404. Other methods are possible for determining the RTT, such as for example using DL-TDOA and / or UL-TDOA measurements.
[0092] PRSs may be defined for network-based positioning (e.g., NR positioning) to enable UEs to detect and measure more neighbor transmission and reception points (TRPs), where multiple configurations are supported to enable a variety of deployments (e.g., indoor, outdoor, sub-6, mmW, etc.). To support PRS beam operation, beam sweeping may also be configured for PRS. The UL positioning reference signal may be based on sounding reference signals (SRSs) with enhancements / adjustments for positioning purposes. In some examples, UL-PRS may be referred to as “SRS for positioning,” 129025-1887WO01Qualcomm Ref. No.2301671WO 24 and a new Information Element (IE) may be configured for SRS for positioning in RRC signaling.
[0093] DL PRS-RSRP may be defined as the linear average over the power contributions (in [W]) of the resource elements of the antenna port(s) that carry DL PRS reference signals configured for RSRP measurements within the considered measurement frequency bandwidth. In some examples, for FR1, the reference point for the DL PRS- RSRP may be the antenna connector of the UE. For FR2, DL PRS-RSRP may be measured based on the combined signal from antenna elements corresponding to a given receiver branch. For FR1 and FR2, if receiver diversity is in use by the UE, the reported DL PRS-RSRP value may not be lower than the corresponding DL PRS- RSRP of any of the individual receiver branches. Similarly, UL SRS-RSRP may be defined as linear average of the power contributions (in [W]) of the resource elements carrying sounding reference signals (SRS). UL SRS-RSRP may be measured over the configured resource elements within the considered measurement frequency bandwidth in the configured measurement time occasions. In some examples, for FR1, the reference point for the UL SRS-RSRP may be the antenna connector of the base station (e.g., gNB). For FR2, UL SRS-RSRP may be measured based on the combined signal from antenna elements corresponding to a given receiver branch. For FR1 and FR2, if receiver diversity is in use by the base station, the reported UL SRS- RSRP value may not be lower than the corresponding UL SRS-RSRP of any of the individual receiver branches.
[0094] PRS-path RSRP (PRS-RSRPP) may be defined as the power of the linear average of the channel response at the i-th path delay of the resource elements that carry DL PRS signal configured for the measurement, where DL PRS-RSRPP for the 1st path delay is the power contribution corresponding to the first detected path in time. In some examples, PRS path Phase measurement may refer to the phase associated with an i- th path of the channel derived using a PRS resource.
[0095] DL-AoD positioning may make use of the measured DL PRS-RSRP of downlink signals received from multiple TRPs 402, 406 at the UE 404. The UE 404 measures the DL PRS-RSRP of the received signals using assistance data received from the positioning server, and the resulting measurements are used along with the azimuth angle of departure (A-AoD), the zenith angle of departure (Z-AoD), and other configuration information to locate the UE 404 in relation to the neighboring TRPs 402, 406. 129025-1887WO01Qualcomm Ref. No.2301671WO 25
[0096] DL-TDOA positioning may make use of the DL reference signal time difference (RSTD) (and / or DL PRS-RSRP) of downlink signals received from multiple TRPs 402, 406 at the UE 404. The UE 404 measures the DL RSTD (and / or DL PRS-RSRP) of the received signals using assistance data received from the positioning server, and the resulting measurements are used along with other configuration information to locate the UE 404 in relation to the neighboring TRPs 402, 406.
[0097] UL-TDOA positioning may make use of the UL relative time of arrival (RTOA) (and / or UL SRS-RSRP) at multiple TRPs 402, 406 of uplink signals transmitted from UE 404. The TRPs 402, 406 measure the UL-RTOA (and / or UL SRS-RSRP) of the received signals using assistance data received from the positioning server, and the resulting measurements are used along with other configuration information to estimate the location of the UE 404.
[0098] UL-AoA positioning may make use of the measured azimuth angle of arrival (A-AoA) and zenith angle of arrival (Z-AoA) at multiple TRPs 402, 406 of uplink signals transmitted from the UE 404. The TRPs 402, 406 measure the A-AoA and the Z-AoA of the received signals using assistance data received from the positioning server, and the resulting measurements are used along with other configuration information to estimate the location of the UE 404. For purposes of the present disclosure, a positioning operation in which measurements are provided by a UE to a base station / positioning entity / server to be used in the computation of the UE’s position may be described as “UE-assisted,” “UE-assisted positioning,” and / or “UE-assisted position calculation,” while a positioning operation in which a UE measures and computes its own position may be described as “UE-based,” “UE-based positioning,” and / or “UE-based position calculation.”
[0099] Additional positioning methods may be used for estimating the location of the UE 404, such as for example, UE-side UL-AoD and / or DL-AoA. Note that data / measurements from various technologies may be combined in various ways to increase accuracy, to determine and / or to enhance certainty, to supplement / complement measurements, and / or to substitute / provide for missing information.
[0100] Note that the terms “positioning reference signal” and “PRS” generally refer to specific reference signals that are used for positioning in NR and LTE systems. However, as used herein, the terms “positioning reference signal” and “PRS” may also refer to any type of reference signal that can be used for positioning, such as but 129025-1887WO01Qualcomm Ref. No.2301671WO 26 not limited to, PRS as defined in LTE and NR, TRS, PTRS, CRS, CSI-RS, DMRS, PSS, SSS, SSB, SRS, UL-PRS, etc. In addition, the terms “positioning reference signal” and “PRS” may refer to downlink or uplink positioning reference signals, unless otherwise indicated by the context. To further distinguish the type of PRS, a downlink positioning reference signal may be referred to as a “DL PRS,” and an uplink positioning reference signal (e.g., an SRS-for-positioning, PTRS) may be referred to as an “UL-PRS.” In addition, for signals that may be transmitted in both the uplink and downlink (e.g., DMRS, PTRS), the signals may be prepended with “UL” or “DL” to distinguish the direction. For example, “UL-DMRS” may be differentiated from “DL-DMRS.” In addition, the term “location” and “position” may be used interchangeably throughout the specification, which may refer to a particular geographical or relative place.
[0101] In addition to Global Navigation Satellite Systems (GNSS)-based positioning and network-based positioning (e.g., as described in connection with FIG. 4), various visual-based positioning has also been developed to provide alternative / additional positioning mechanisms / modes. Visual-based positioning, which may also be referred to as “visual positioning,” “vision-based positioning,” “vision-positioning,” “camera- based positioning,” and / or “camera-based visual positioning,” is a positioning mechanism / mode that uses images captured by at least one camera to determine the location of a target (e.g., a UE, one or more objects that are in the field-of-view (FOV) of the at least one camera, etc.). For example, images captured by a camera in a warehouse may be used for calculating / estimating the location of inventories in the warehouse.
[0102] FIG. 5 is a diagram 500 illustrating an example of visual-based positioning in accordance with various aspects of the present disclosure. Visual-based positioning may offer high-accuracy position estimation, where coordinates (e.g., two- dimensional (2D) / three-dimensional (3D) coordinates, latitude and longitude coordinates, etc.) of points of interests (e.g., positions of UEs, objects in a warehouse, RF devices used for tracking / identifying inventories, etc.) may be computed from the corresponding pixels of a 2D image using (the inverse of) a projective transformation that maps world points onto the 2D image. For example, as shown at 506, a projective transformation may enable coordinates of an object 504 to be computed from the corresponding pixel of the object 504 on a 2D image 502, where the projective transformation may map the world points of the object 504 onto the 2D image 502. In 129025-1887WO01Qualcomm Ref. No.2301671WO 27 some scenarios, the (accuracy and reliability of) projective transformation may depend on the camera pose (e.g., the position and the orientation of a camera) and a set of internal camera parameters (e.g., focal length, resolution, image size, etc.). Thus, knowing the (inverse of) projective transformation may be a key for reliable / accurate localization of points / objects of interest in the FOV of a camera.
[0103] For purposes of the present disclosure, an RF device may refer to any device with the capability of transmitting RF signals, such that the device may be used for positioning. An RF-based location estimate may refer to an estimation of an RF device’s location using at least one RF-related technology, such as Bluetooth®, Wi-Fi®, RF sensing, and / or ultrawide band (UWB), etc. A location estimate may also refer to an estimated / approximated location of an RF device, an RF device is within a threshold distance / radius, and / or the accuracy of the positioning is within certain degree threshold (e.g., within certain degree of accuracy), etc.
[0104] In one example, the projective transformation may be learned by a device (e.g., a server, a UE, a TRP, a network node, etc.) based on a process called camera calibration. During camera calibration, training data may be provided to the device which matches pairs of 2D / 3D world points of one or more objects with known coordinates and the pixel locations of their corresponding image projections. For example, scene features may be extracted from an image, followed by matching with features obtained from scene representation (e.g., such as a 3D model). In some scenarios, the camera calibration process may not be reliable if the extracted features are not descriptive (e.g., in an indoor environment, under low lighting, at nighttime, etc.) or the scene representation is absent (e.g., the 3D model is not available for location or access to external server for 3D model retrieval is not possible, etc.). Also, in some use-cases, the projective transform may change frequently due to mobility or sporadic internal camera adjustments (examples include mobile smartphones, virtual reality (VR) glasses, unmanned aerial vehicles (UAVs), self-driving cars etc.).
[0105] FIG. 6 is a diagram 600 illustrating an example projective transformation in accordance with various aspects of the present disclosure. In one example, parameters that may be used for determining the projection of world points of one or more objects onto an image plane 602 may include external parameters such as the camera pose (e.g., the position and the orientation of a camera) with regard to the world coordinate system (WCS) and internal parameters such as the focal length, size, resolution, skew, and / or distortion, etc. of the camera. 129025-1887WO01Qualcomm Ref. No.2301671WO 28
[0106] The projective transformation may be a linear operation in homogenous coordinates, represented by a calibration matrix ^^^^ of size 3×4:where ^^^^Wmay the world point of an object (e.g., the WCS of the object) and ^^^^∗may indicate pixel point of the object on the image plane 602, such as shown at 604. This may be valid up to a scale in homogenous coordinates. If the pixel coordinates of the world point ^^^^Wis represented by(^^^^, ^^^^), using the homogeneous vector ^^^^∗, the following may be obtained: ^^^^ = ^^^^∗[1] / ^^^^, ^^^^ = ^^^^∗[2] / ^^^^.
[0107] In some scenarios, camera calibration may be performed via geometric error minimization. For example, given a dataset ( ^^^^) of matched training points:the camera calibration matrix ^^^^ may be found as:where ^^^^(∙,∙) may indicate the Euclidean distance between the non-homogenous pixel coordinates and the projections of the world points on the image plane 602. The minimization may be iterative, where a suitable initialization for ^^^^ may be given by a direct linear transform (DLT) (which may be a standard procedure). The training data may be normalized first (e.g., zero mean and fixed standard deviation), where the normalization may be performed on pixels and world coordinates separately. In some examples, a random sample consensus (RANSAC) (an iterative method for estimating a mathematical model from a data set that contains outliers) may be used to select a subset of training points in ^^^^ that produce well-conditioned / best / most suitable estimate of ^^^^.
[0108] As shown at 606, once ^^^^ is known, the projections of the world coordinates on a ground plane (e.g., ^^^^ = 0) may be computed from relevant pixels on the image plane 602. For example, the column corresponding to the coordinate may be removed to obtain square matrix ^^^^^^^^which can be inverted: 129025-1887WO01Qualcomm Ref. No.2301671WO 29Then, the ground plane coordinates of pixel(^^^^, ^^^^)may be obtained based on: ^^�^^W = ^�^^^W[1] / s, ^�^^^W = ^�^^^W[2] / s
[0109] FIG. 7 is a diagram 700 illustrating an example warehouse scenario where multiple objects are localized or tracked using visual-based positioning in accordance with various aspects of the present disclosure. As shown at 720, a warehouse may include a plurality of objects (e.g., inventories, packages, and / or living things, etc.), such as a first object 708, a second object 710, and a third object 712, etc. Each object may be equipped / attached with one or more RF devices 714, such as radio-frequency identification (RFID) tags or Wi-Fi® modules, which may be used for identifying the associated object (e.g., based on their corresponding IDs) and / or for estimating a coarse location of the object (e.g., using RF-based positioning). For purposes of the present disclosure, an RF device may refer to any device with the capability of transmitting RF signals, such that the device may be used for positioning. Coarse location of an object may refer to an estimated / approximated location of the object, the object is within a threshold distance / radius, and / or the accuracy of the positioning is within certain degree threshold (e.g., within certain degree of accuracy), etc.
[0110] For example, the first object 708 may include three RF devices 714 (e.g., RF devices A, B, and C), the second object 710 may include two RF devices 714 (e.g., RF devices D and E), and the third object 712 may include one RF device 714 (e.g., RF device F), etc. The warehouse may be equipped with a plurality of access points (APs) 716, which may be used for communicating with the RF devices 714, such as transmitting RF signals to the RF devices 714 and / or receiving RF signals (e.g., reflected RF signals) from the RF devices 714. Then, based on the information carried by the RF signals and / or the time of flight (ToF) / round trip time (RTT) of the RF signals, a server may be able to obtain or determine information associated with a particular object. For example, an RF signal from an RF device 714 may provide a unique 129025-1887WO01Qualcomm Ref. No.2301671WO 30 identification (ID) that enables the server to identify the object (e.g., a specified merchandize, a specified equipment, a specified animal, a specified person, etc.) to which the RF device 714 is attached. In some implementations, based on the ToF / RTT of the RF signal(s), the server may also be able to estimate the location of the object. The RF devices 714 may be placed anywhere on or inside the objects (e.g., placed inside / attached outside of boxes, carried by humans such as warehouse workers, etc.).
[0111] As shown at 722, multiple cameras may also be installed in the warehouse for monitoring the objects in the warehouse. For example, a first camera 702 may be installed on one side of the warehouse and facing the center of the warehouse, a second camera 704 may be installed on another side of the warehouse and also facing the center of the warehouse, and a third camera 706 may be installed near the ceiling of the warehouse and looking down, etc. As such, each camera may be able to capture a view or an image of one or more objects in the warehouse, such as shown at 724. Based on such a setup, the location of specific objects may be localized through fusion of multiple RF and vision-based modalities. For example, as described in connection with FIGs.5 and 6, a fusion-based positioning framework may typically involve steps such as camera calibration, target association across images and RF devices, visual position prior estimation, and / or inference through fusion with RF measurements, etc.
[0112] FIG.8 is a diagram 800 illustrating an example functional blocks and related flows of localizing object(s) in the warehouse in accordance with various aspects of the present disclosure. As shown at 802, upon a request from a server (e.g., a local server, a fusion center, etc.), the first camera 702, the second camera 704, and the third camera 706 may send their captured images (e.g., images captured within their FOVs) to the server for processing, where the server may be able to identify an object and calculate its location based on the images, such as described in connection with FIGs.5 and 6. In some implementations, as shown at 804, the server may also maintain a connection to an external cloud system (e.g., a software as a service (SaaS) system), which may be used to perform heavy weight image processing if specified (e.g., if the local server does not have sufficient processing capabilities).
[0113] While the setup described in connection with FIGs. 7 and 8 may enable a server to locate one or more objects in an environment based on visual positioning, there are also various weak points with regards to the security and privacy for such setup, such as between cameras and the local server (e.g., the fusion center), and / or between the 129025-1887WO01Qualcomm Ref. No.2301671WO 31 local server and the external server (e.g., the cloud server), etc. For example, as shown at 802, as the local server may specify images captured by cameras to be sent to the local server, an unauthorized entity (e.g., a hacker, a malicious device, etc.) may try to intercept and hack the communications between the cameras and the local to obtain images captured by the cameras (e.g., to learn what objects are inside the warehouse or for other illegal purposes, etc.). In addition, even if the local network (e.g., network between the cameras and the local server) is secure (e.g., wired), there may also be aspect of consent and privacy for certain individuals. For example, people working in the warehouse may not be willing to allow processing of their visual features even at the local site. Similarly, the unauthorized entity may also try to gain an unauthorized access to the local server and / or the cloud / external server to obtain this information (e.g., images captured by the camera).
[0114] FIG.9 is a diagram 900 illustration an example of a privacy-preserving transformation in accordance with various aspects of the present disclosure. In some examples, to preserve the privacy, a privacy-preserving transformation (which may also be referred to as privacy-preservation in some examples) may be applied to raw images captured by a camera such that any potentially compromising visual features may be removed from the images. For example, as shown at 902, visual features may be removed from images captured by the first camera 702, the second camera 704, and the third camera 706, such that the server (e.g., the local server and / or the external server) may obtain non-visual features of one or more objects from the cameras. For example, non-visual features may include description of an object (e.g., a car, a television, a shoe box, etc.), the bounding box of the object (e.g., 10 cm x 6 cm), the rough location of the object, etc. For purposes of the present disclosure, a bounding box may refer to an imaginary rectangle that serves as a point of reference for object detection and creates a collision box for that object in projects on image processing.
[0115] However, as shown by a diagram 1000 of FIG. 10, removing visual features from images for visual identification may have implications on key steps in the visual positioning processing chain that rely on visual features. For instance, to utilize the benefits of multiple view positioning (e.g., visual positioning using more than one camera), the server / fusion center may be specified to associate objects across multiple images (e.g., associate the object 708 across images captured by the first camera 702, the second camera 704, the third camera 706, etc.). As this association is typically achieved by matching visual features (e.g., the colors, shapes, contours, etc.) of the 129025-1887WO01Qualcomm Ref. No.2301671WO 32 objects across different images, it may be difficult or even impossible to achieve the association on images that undergo the privacy-preservation.
[0116] Aspects presented herein may improve the security and privacy for visual-based positioning. Aspects presented herein may enable a server (e.g., a local server, a fusion center, an external cloud server, etc.) to associate one or more objects across images captured by different cameras (which may be referred to as joint object association or joint association across images) while applying privacy-preserving transformation to the images. For example, in one aspect of the present disclosure, a server may be configured to use RF-based position estimates of the RF device(s) contained in a set of region of interests (RoIs) / objects to perform joint association across images and across RF devices. This alleviates the specification for additional visual features and descriptors and instead uses RF position estimates (e.g., pixel location(s) of their projections onto the image) as proxies for performing the joint association. In another aspect of the present disclosure, a layered security paradigm is introduced which provides multiple different security levels, where at each level gradually more visual information is revealed for the purpose of improving the joint association at the server. For example, if a server determines that additional information from the images is specified to perform the joint association, the server may request one or more cameras to provide additional visual information gradually until the server is able to perform the joint association. This may enable the server to achieve joint association without all visual information (e.g., without specifying raw images).
[0117] For purposes of the present disclosure, as shown by a diagram 1100 of FIG. 11, a region of interest (RoI) may refer to a set of pixel locations on an image corresponding to a region in the world or in a specified area, or a region in the World Coordinate System (WCS), where one or more RF devices are physically located. Depending on implementations, there may be two key aspects with regards to the RoI: RoI generation and RoI association. RoI generation involves identifying the part of an image where one or more RF devices might be present. For example, a bounding box obtained via an object detector may be trained to identify specific types of objects / targets. Depending on how a RoI is actually defined in an implementation and what method is used to compute the RoI, the RoI may be identified via few parameters that uniquely determine its location on an image. For instance, a bounding box may be defined via the width, the height, and the centroid (e.g., three parameters). As such, the RoI on an image may be defined through the location of the pixels. 129025-1887WO01Qualcomm Ref. No.2301671WO 33
[0118] RoI association may include two types of associations. The first type of association is association across images, which deals with identifying the same RoI in multiple images (e.g., captured by multiple camera systems). In some examples, this may be achieved by matching visual features across images (which may also be referred to as visual features matching). For example, referring back to FIG. 7, the server may perform RoI association for the first object 708 across the first camera 702, the second camera 704, and the third camera 706, such that RoIs (e.g., a set of pixels) corresponding to the first object 708 on images captured by different cameras are associated with (linked to) the same object (i.e., the first object 708). In one example, the association may be based on the visual feature(s) associated with the first object 708, such as its color, shape, contour, and / or size, etc.
[0119] The second type of association is association with RF devices, which may also be referred to as RoI-RF matching. Association with RF devices determines which RF device(s) are contained in a given RoI. Typically, RF-based location features may be used for the association with RF devices. For example, bipartite graph matching may be used between RF-based position estimates of RF devices and centroids of RoIs. For example, referring back to FIG.7, after the RoI for the first object 708 is being identified, as the first object 708 is attached with three RF devices 714 (e.g., RF devices A, B, and C), the server may associate this three RF devices 714 with the identified RoI.
[0120] In some scenarios, a fusion engine (e.g., an engine run by a fusion center) may technically just specify the image RoIs of relevant object(s) / target(s) for performing visual positioning or for positioning purposes. For example, as described in connection with FIGs.5 and 6, if a camera is calibrated (e.g., the camera matrices are known), an image RoI may be back-projected into a three-dimensional (3D) space to find a corresponding region. Therefore, extracting just image pixel locations and transmitting them over to a server (e.g., a fusion center) without additional image / visual features may provide an effective way to mitigate potential security breaches and ensure privacy. For example, if each camera is configured to run an RoI generation algorithm (e.g., a standard object detector which may be downloaded from the server or an external cloud server) after taking an image, and send the obtained RoIs to the server as pixel location descriptors, the level of privacy-preservation may be very high as no additional visual information is exchanged (apart from information relevant for positioning). As discussed in connection with FIGs.9 and 10, removal of 129025-1887WO01Qualcomm Ref. No.2301671WO 34 visual information may have implications on various key processes related for visual positioning such as the association of object(s) / target(s) across images in multiple cameras / views systems. For purposes of the present disclosure, a camera matrix may refer to a matrix that describes a mapping of a pinhole camera from 3D points in the world to 2D points in an image.
[0121] To remedy this and enable seamless execution of all key functions for visual positioning, aspects presented herein use the RF-based position estimates of the RF devices contained in each of the RoIs to perform joint association across images and across RF devices. This alleviates the specification for additional visual features and descriptors and instead uses the RF position estimates (i.e., the pixel locations of their projections onto the image) as proxies. Aspects presented herein further introduce layered security paradigm that considers multiple different security levels. At each level gradually more visual information is revealed for the purpose of improved association at the fusion center. Depending on the case at hand, the fusion center may decide that additional information from the images is specified to perform the association and may send a request to the camera(s) to send the specified additional information.
[0122] In one aspect of the present disclosure, a server (e.g., a fusion center) may be configured to perform joint association for objects / RF devices across images captured by different cameras based on the estimated position(s) of RF device(s) attached to the objects (which may also be referred to as RF-based position information of the objects / targets). For example, as described in connection with FIGs. 5 and 6, if the calibration matrices for a camera is known, the coordinates of an RF device location may be projected onto the image(s) captured by the camera.
[0123] FIG.12 is a diagram 1200 illustrating an example of estimating the locations of RF devices and associating RF devices across images based on their estimated locations in accordance with various aspects of the present disclosure. As shown by the diagram 1200, a set of access points (APs) 716 may be installed around the first object 708, the second object 710, and the third object 712, such as described in connection with FIG.7. Then, using RF-based positioning, a server may estimate the location of each RF device 714. For example, the sever may estimate that RF device A is at a first set of coordinates (X, Y), and the RF device B is at a second set of coordinates (M, N), etc. Then, as described in connection with FIGs.5 and 6, as calibration matrices for the first camera 702, the second camera 704, and the third camera 706 are known, the 129025-1887WO01Qualcomm Ref. No.2301671WO 35 server (or a fusion center) may back-project the locations (e.g., apply inverse transformation) of the RF devices 714 to the corresponding images captured by each camera, such that pixel location(s) of an RF device 714 may be identified on views / images captured on various cameras.
[0124] In one example, the corresponding pixel locations of RF devices or the object(s) they are attaching to may be associated with bounding box centroids within a given image, such as by using a nearest neighbor type of approach. These associations may then be used to associate bounding boxes across images. For purposes of the present disclosure, a centroid may refer to a geometric center of a given image or image plane. For example, the centroid of an image may be a fixed point located at the intersection of all of the hyperplanes of symmetry within that image.
[0125] FIG. 13 is a diagram 1300 illustrating an example of associating one or more RF devices to a bounding box centroid in accordance with various aspects of the present disclosure. As shown at 1302, the corresponding pixel locations of RF devices 714 or the object(s) they are attaching to may be associated with bounding box centroids within a given image. For example, the pixel locations of RF devices A, B, and C may be associated with a bounding box centroid 1304 (e.g., based on using nearest neighbor type of approach), the pixel locations of RF devices E and D may be associated with a bounding box centroid 1306, and the pixel location of RF device F may be associated with a bounding box centroid 1308, etc. In most implementations, these centroids may be the minimum specified information for performing the joint association (without visual features). As such, a more accurate association may be achieved if more information is available such as the actual bounding box (e.g., its size and boundary), the silhouette, and / or visual descriptors associated with the objects. In addition, as each RF device 714 is likely associated with a unique ID, aspects presented herein may also enable the server to associate a target object with its corresponding RF device(s). For example, the RF devices A, B, and C may include unique ID(s) that are associated with the first object 708 which they are attaching to.
[0126] Aspects described in connection with FIGs.13 and 14 may enable cameras to provide RoIs associated with RF device(s) and / or their corresponding object(s) instead of full visual information to a server (e.g., a local, a fusion center, an external cloud server, etc.), where the server may perform joint association across the images and RF devices while maintaining privacy-preservation. 129025-1887WO01Qualcomm Ref. No.2301671WO 36
[0127] While estimated location(s) of RF device(s) may be used for joint association of RF devices / objects across images, in some scenarios, a server may still specify additional information from a camera if location information provided by the camera is not sufficient enough for the server to perform the joint association or to achieve the joint association with a specified confidence level. As such, in another aspect of the present disclosure, the server may also be configured to apply a layered security mechanism / paradigm to the joint association of object(s) / RF device(s) across images captured by different cameras.
[0128] As described in connection with FIGs. 12 and 13, assuming each camera is implemented with an image processing algorithm that is capable of identifying RoIs associated with one or more RF devices, such as by using standard object detector(s) which may be downloaded from the server for instance. In some examples, such an image processing algorithm may also be referred to as an RoI generation algorithm. By using an RoI generation algorithm as well as the original image, a camera may extract several relevant RoI descriptors such as RoI centroid, RoI bounding box, RoI shape / silhouette as well as a variety of visual feature descriptors (e.g., discussed in connection with FIG.14 below). Then, the camera may be configured to provide each of these pieces of information gradually to increase the association / matching success at the expense of reduced privacy.
[0129] FIG.14 is a diagram 1400 illustrating an example of a multiple-tier security paradigm for joint association in accordance with various aspects of the present disclosure. In one example, a server may be configured to apply a four (4)-tier security paradigm to the joint association of RF devices and / or their associated objects across images. As shown at 1402, at a first / highest level (e.g., level 0) of the security paradigm, the third camera 706 may be configured to provide the server with just the pixel centroids of the first object 708, the second object 710, and the third object 712 (e.g., the RoI descriptors include least visual information). In some examples, the pixel centroids may also be referred to as RoI centroids. However, if the server is unable to perform the joint association for at least one of the object(s) / RF devices, the server may request the third camera 706 to provide additional information for the first object 708, the second object 710, and / or the third object 712, or to just provide additional information related to the object(s) which the server is unable to perform the joint association. 129025-1887WO01Qualcomm Ref. No.2301671WO 37
[0130] As shown at 1404, in response to the server’s (first) request for additional information, the camera 706 may apply a second / next level (e.g., level 1) of the security paradigm, where third camera 706 may be configured to provide the server with pixel centroids of the first object 708, the second object 710, and the third object 712 and also their corresponding bounding boxes (e.g., the RoI descriptors include more / additional visual information). In some examples, the bounding boxes may also be referred to as RoI bounding boxes. As such, the server may perform the joint association for the objects / RF devices based on both their pixel centroids and the bounding boxes. This may improve the likelihood of success for the joint association, but at the expense of reduced privacy as additional (visual) information may be revealed. However, if the server is still unable to perform the joint association for at least one of the object(s) / RF devices, the server may again request the third camera 706 to provide additional information for the first object 708, the second object 710, and / or the third object 712.
[0131] As shown at 1406, in response to the server’s (second) request for additional information, the camera 706 may apply a third / next level (e.g., level 2) of the security paradigm, where third camera 706 may be configured to provide the server with pixel centroids of the first object 708, the second object 710, and the third object 712 and also their corresponding bounding boxes as well as shape silhouettes (e.g., the RoI descriptors include more / additional visual information). In some examples, the shape silhouettes may also be referred to as RoI shape / silhouettes, which shows the overall contour / outline of an object. As such, the server may perform the joint association for the objects / RF devices based on their pixel centroids, bounding boxes, and / or shape silhouettes. This may further improve the likelihood of success for the joint association, but at the expense of further reduced privacy as more (visual) information is revealed. Similarly, if the server is still unable to perform the joint association for at least one of the object(s) / RF devices, the server may again request the third camera 706 to provide additional information for the first object 708, the second object 710, and / or the third object 712.
[0132] As shown at 1408, in response to the server’s (third) request for additional information, the camera 706 may apply a fourth / lowest level (e.g., level 3) of the security paradigm, where third camera 706 may be configured to provide the server with actual visual feature descriptors from the RoIs (e.g., colors, shapes, patterns, etc.) of the first object 708, the second object 710, and the third object 712 in addition to their corresponding bounding boxes and silhouettes. As such, the server may perform 129025-1887WO01Qualcomm Ref. No.2301671WO 38 the joint association for the objects / RF devices based on their visual feature descriptors, bounding boxes, and / or shape silhouettes. This likely will provide the server with sufficient information to successfully perform the joint association, but privacy preservation may also be the lowest / minimum compared to other levels of the security paradigm. In other words, level 0 may have the highest security / privacy protection, closely followed by level 1 and then level 2, whereas level 3 has the lowest security / privacy protection.
[0133] In some implementations, the server may be configured to always begin with lowest level (e.g., level 0) and will consider upper levels / tiers when some RoI association(s) fail, or when their association confidence is lower than a predefined confidence threshold. In some implementations, when the server requests a camera to increase the level of the security paradigm (e.g., to provide more information about the RF devices / objects), such request may not pertain to the entire image but just to a specific, difficult region. Hence even in case when the security is lowered, this lowering may be localized with the rest of the image maintaining higher security levels. For example, referring to FIG.14, if the server is able to perform joint association for the first object 708 and the second object 710 based on their pixel centroids but is unable to perform joint association for the third object 712 based on its pixel centroid, the server may request the third camera 706 to increase the level / tier for regions specific to the third objects. Then, the third camera 706 may just reveal the bounding box, shape silhouettes, and / or visual feature descriptors associated with the third object 712 depending on the level of security paradigm.
[0134] FIG. 15 is a communication flow 1500 illustrating an example message flow of a server performing privacy-preserving joint object association across images and RF devices with multiple-tier security paradigm in accordance with various aspects of the present disclosure. The numberings associated with the communication flow 1500 do not specify a particular temporal order and are merely used as references for the communication flow 1500. In one example, as illustrated by FIG. 7, a server 1502 (e.g., a local server, a fusion center, an external cloud server, etc.) may communicate with a set of cameras 1504 (e.g., the first camera 702, the second camera 704, and / or the third camera 706, etc.) for localizing one or more objects (e.g., the first object 708, the second object 710, and / or the third object 712, etc.) that are equipped with one or more RF devices (e.g., RF devices 714, RFID tags, Wi-Fi® modules, Bluetooth modules, tracking modules, etc.). 129025-1887WO01Qualcomm Ref. No.2301671WO 39
[0135] The server 1502 may be implemented with a joint association procedure for associating objects / RF devices across images from multiple cameras. In some examples, the server 1502 may also have the capability to perform other positioning- related functions, such as estimating the locations of RF devices using RF-based positioning as described in connection with FIG.12. The joint association procedure may typically be a key functional block in a visual positioning chain and may be configured not to initiate on its own. The set of cameras 1504 may include a networked camera system that is connected to the server 1502 via wired or wireless connection(s), where each camera in the set of cameras 1504 may be characterized with a corresponding projective transformation matrix, such as described in connection with FIGs. 5 and 6. The project transformation matrix may include parameters of the camera, such as its zoom-in, zoom-out, focal length, etc. In some examples, the set of cameras 1504 may also record any change(s) in their internal and external parameters locally and send them to the server 1502. For example, if the focal length of a camera changes, the camera may notify the server 1502 regarding the change, such that the server 1502 may make corresponding adjustment(s) associated with the camera.
[0136] At 1510, the server 1502 may message / ping the set of cameras 1504 (which may be relevant camera(s) specifically selected to detect object(s) in a specified region, such as based on their FOVs) to request relevant visual information for one or more objects and / or for a specified region.
[0137] At 1512, the set of cameras 1504 may generate RoI for their captured images, such as described in connection with FIG.11. As discussed in connection with FIG.14, the set of cameras 1504 may begin at a highest level (e.g., level 0) of the privacy / security paradigm, where the set of cameras 1504 may just provide pixel centroids for one or more objects. Then, at 1514, the set of cameras 1504 may transmit the generated RoIs (e.g., level 0 centroids) of the one or more objects to the server 1502.
[0138] At 1516, based on the RoIs of the one or more objects, the server 1502 may perform joint association of relevant objects across images captured by the set of cameras 1504, such as described in connection with FIGs.8 to 10. In one example, the server 1502 may perform the joint association using a predefined algorithm, such as a bipartite matching algorithm / mechanism which may also indicate a confidence level for each of the decisions (e.g., for each of the RoIs). 129025-1887WO01Qualcomm Ref. No.2301671WO 40
[0139] At 1518, the server 1502 may compare the confidence level for each of the RoIs with a specific confidence threshold. Then, at 1520, if the confidence level for one or more RoIs is lower than the confidence threshold, the server 1502 may determine whether the privacy / security paradigm has reached its final tier (e.g., lowest privacy / security level, level 3, etc.).
[0140] If the privacy / security paradigm has not reached its final tier, as shown at 1522, the server 1502 may message / ping the set of cameras 1504 (or just camera(s) in the set of cameras 1504 that do not meet the confidence threshold) to move up to the next privacy / security paradigm level, such as from level 0 to level 1, from level 1 to level 2, or from level 2 to level 3, etc. On the other hand, if the privacy / security paradigm has reached its final tier (e.g., the lowest privacy security level, level 3, etc.), as shown at 1524, the server 1502 may terminate the joint association procedure. In some implementations, the server 1502 may also request the set of cameras 1504 to provide raw images instead (e.g., privacy-preservation is not applied to the images).
[0141] At 1526, based on the request from server 1502 to move up to the next privacy / security paradigm level, the set of cameras 1504 may transmit RoIs with the increased / requested privacy / security paradigm level, such as described in connection with FIG. 14. In some examples, the set of cameras 1504 may transmit just corresponding information for the RoIs that do not meet the joint association confidence level instead of all RoIs to maintain the high security / privacy for RoIs that meet the joint association confidence level. This exchange (e.g., the steps described between 1516 to 1526) may be repeated back and forth until the confidence levels are satisfied for the one or more objects or when the lowest security level (e.g., level 3) has been reached.
[0142] FIG.16 is a diagram 1600 illustrating an example scenario where RF devices are not visible to a camera in accordance with various aspects of the present disclosure. As described in connection with FIGs. 5 and 6, visual-based positioning estimates obtained with calibrated cameras may have a very high accuracy. If a target RF device is within a camera’s field-of-view (FOV) and is visible on the image, the visual-based positioning estimates may be an order of magnitude more precise than RF-based positioning estimates alone. However, as shown by the diagram 1600, in certain monitoring applications such as industrial / warehouse monitoring applications, the target RF devices may not be directly visible in the camera’s FOV. For example, consider a warehouse 1602 with multiple objects 1604 (e.g., inventories, displays, 129025-1887WO01Qualcomm Ref. No.2301671WO 41 products, etc.) where each object may be equipped / attached with one or more RF devices 1606, such as RFID tags or Wi-Fi® / Bluetooth® modules, etc. Multiple cameras may be installed in the warehouse 1602 (e.g., near the ceiling / room corners and looking down, etc.).
[0143] As shown at 1608, the RF devices 1606 may be placed anywhere on or inside the objects 1604 (e.g., boxes, shelfs, aisles, etc.), where at least some of the objects 1604 may be under a pile or on the shelfs across the aisles (thus, occluding each other either partially or completely). As such, in many scenarios, the RF devices may not be directly visible to the cameras, and visual-based positioning may not be applicable for these non-visible RF devices. In other words, the server / cameras may not be able to perform direct visual positioning for these non-visible RF devices.
[0144] As described in connection with FIG. 11, a server / camera may be configured to identify an RoI on the image that contains the pixel locations of RF device(s) (e.g., the RF devices 1606). Thus, in another aspect of the present disclosure, instead of using visual-based positioning estimates (which may also be referred to as visual point estimates), a server may be configured to estimate visual prior distributions for the positions of the RF devices using the RoIs on the images. For example, in one aspect of the present disclosure, a sampling-based visual position prior estimation procedure is provided which may enable a server / camera to estimate the location of one or more RF devices that are not visible to the camera (e.g., not within the camera’s FOV).
[0145] FIGs.17A and 17B are diagrams 1700A and 1700B illustrating an example of visual- based position priors in a single-view system and in a multiple-view system, respectively, in accordance with various aspects of the present disclosure. While an RoI may contain one or more RF devices, the actual pixel location(s) of the RF device(s) may not be known to a server / camera. In other words, the RF device(s) may be anywhere in the RoI. As discussed previously, each RoI may correspond to a specific world region in a two-dimensional (2D) or three-dimensional (3D) space, and an RoI and its corresponding world region may be related through projective transformation, such as described in connection with FIGs.5 and 6.
[0146] For example, as shown by the diagram 1700A, in a single-view system (e.g., a system with one camera), the world region corresponding to a given RoI may lie on a 2D plane (e.g., the ground plane). On the other hand, as shown by the diagram 1700B, in a multiple-view system (e.g., a system with multiple cameras), the world region corresponding to a given RoI may be a 3D object. In some scenarios, the uncertainty 129025-1887WO01Qualcomm Ref. No.2301671WO 42 regarding the pixel location(s) of RF device(s) may be modelled using a probability distribution defined over the RoI in an image, where this probability distribution may be referred to as “image-domain position prior” in some examples. The image-domain position prior may induce a distribution defined over the corresponding world region through the projective transformation, where this prior may be referred to as “visual position prior” or “visual-based position prior.” For purposes of the present disclosure, a prior probability distribution (or simply a prior) of an uncertain quantity may refer to an assumed probability distribution before some evidence is taken into account.
[0147] In one aspect of the present disclosure, a server / camera may apply a sampling-based approach for estimating visual position priors. For generality, aspects presented herein are illustrated based on a multiple-view system, such that the visual position prior is defined over a 3D space, such as shown by FIG.17B. Aspects presented herein may also apply to a single-view system, where the visual position prior is defined over a 2D space instead, such as shown by FIG.17A.
[0148] In one aspect, initially, RoIs from images captured by different cameras that correspond to the same world region may be identified through an association module. For example, as described in connection with FIGs. 8 to 10, a server may perform joint association across images (e.g., RoIs) and RF devices using RF-based location information of the RF devices (e.g., using RF-based position estimates). Then, sampling distributions may be selected as proxies to the image-domain position priors in each image. Similarly, parametric or non-parametric model distribution may be selected as a proxy to the visual position prior. In some scenarios, the choice of the sampling distribution and the visual position prior model may depend on the number of RF devices in the RoIs. If just one RoI is presented, the sampling distribution may be unimodal or uniform. On the other hand, if multiple RF devices are presented, a suitable multimodal distribution may be selected / used instead (e.g., a Gaussian mixture model). After the RoIs are sampled, a number of pixel locations may be drawn from each RoI. The pixel location samples may then be back-projected into the world coordinate system, where these back-projections may form a point cloud in the world coordinate system in which the RF devices are located. Then, a visual position prior model may be fitted to the point cloud (discussed in details below).
[0149] FIG. 18 is a diagram 1800 illustrating an example of visual-based position priors associated with a single-view system in accordance with various aspects of the present 129025-1887WO01Qualcomm Ref. No.2301671WO 43 disclosure. As shown at 1802, an RoI (e.g., a bounding box) on an image may be denoted by ℛ∗. The projective transformation between world coordinates and the image pixels may be denoted by ℋ. When a camera is calibrated, then ℋ is known. As shown at 1804, the world region ℛWcorresponding to ℛ∗may be determined as ℛW= ℋ−1(ℛ∗). However, the image location ^^^^∗= ℋ( ^^^^W) of an RF device’s true position ^^^^Wmay not be known (as it may not be visible to camera(s)). As such, the image location of the RF device ^^^^∗may be anywhere in ℛ∗, following a distribution ^^^^( ^^^^∗;ℛ∗). Then, the pixel location distribution may induce a visual position prior ^^^^(^^^^W;ℛW)= ^^^^(^^^^W;ℛ∗,ℋ)through the transformation ℛW = ℋ−1(ℛ∗).
[0150] FIG. 19 is a diagram 1900 illustrating an example of visual-based position priors associated with a multiple-view system in accordance with various aspects of the present disclosure. The principles may remain the same as the single-view system described in connection with FIG. 18. However, the world region ℛWmay be a 3D object instead of a 2D plane. Under the multiple-view system, the image locations ^�^^^∗( ^^^^)of the RF device true position ^^^^Win view ^^^^ may not be known. As such, the image locations of the RF device ^�^^^∗( ^^^^)may be anywhere in ℛ∗( ^^^^), following distribution ^^^^� ^�^^^∗( ^^^^)|ℛ∗( ^^^^)�. Then, the pixel location distribution may induce a visual prior ^^^^(^�^^^W|ℛW)in the WCS through the back-projection.
[0151] FIG.20 is a diagram 2000 illustrating an example sampling-based learning of vision- based position priors in accordance with various aspects of the present disclosure. As shown at 2002, the bounding box ℛ∗may first be determined. Then, a sampling distribution ^^^^( ^^^^∗) defined over ℛ∗may be selected. For instance, if just one RF device is presented in ℛ∗, ^^^^( ^^^^∗) may be unimodal or uniform over the bounding box ℛ∗. On the other hand, if multiple ( ^^^^) RF devices are presented in ℛ∗, ^^^^( ^^^^∗) may be multimodal, such as a Gaussian mixture model (GMM) for instance. Samples ^^^^∗^^^^~ ^^^^( ^^^^∗), ^^^^ =… , ^^^^ (pixel locations) may then be drawn. As shown at 2004, the ^^^^ samples may then be back-projected ^^^^W, ^^^^= ℋ−1( ^^^^∗^^^^), ^^^^ = 1, … , ^^^^. If just one camera is available (e.g., for a single-view system), the back-projections may form a 2D shape, whereas if multiple cameras are available (e.g., for a multiple-view system), point clouds may be formed in a 3D space, such as described in connection with FIGs. 17A and 17B. The ^^^^ back-projections ^^^^W, ^^^^, ^^^^ = 1, … , ^^^^ may then be used to estimate the visual prior using either parametric or non-parametric model (e.g., histogram). 129025-1887WO01Qualcomm Ref. No.2301671WO 44
[0152] FIG. 21 is a diagram 2100 illustrating an example of point clouds in a 3D space in accordance with various aspects of the present disclosure. As shown by FIG.7, three objects (e.g., the first object 708, the second object 710, and the third object 712) may be scattered around the floor of a warehouse and three cameras (e.g., the first camera 702, the second camera 704, and the third camera 706) are mounted around the warehouse. As shown at 2102, when all three objects appear in various FOV(s) (e.g., image plane) of the camera(s), the point clouds for the three objects may be obtained based on aspects described in connection with FIGs.18 to 20.
[0153] FIG.22A is a diagram 2200A illustrating an example back-projection of pixel samples for a single-view system in accordance with various aspects of the present disclosure. Note sample index ^^^^ may be avoided for notation simplicity. While an RF device’s true position ^^^^Wmay not be recovered from ( ^^^^, ^^^^) unambiguously, its projection on world planes such as ground plane ^^^^ = 0 may be found instead, such that:^^�^^W = ^�^^^W[1] / ^�^^^W[3], ^^�^^W = ^�^^^W[2] / ^�^^^W[3].
[0154] FIG.22B is a diagram 2200B illustrating an example back-projection of pixel samples for a multiple-view system in accordance with various aspects of the present disclosure. Note sample index ^^^^ may be avoided for notation simplicity. Let ^^^^ denote the set of|^^^^|camera views (FOVs) that a target appears in. The target projection in each view may be governed by:The following may be obtained when solving forand rearranging with regards to ^^^^W: 129025-1887WO01Qualcomm Ref. No.2301671WO 45Multiple view visual positioning entails solving the homogenous system, where ^^^^ is2| ^^^^| × 4 matrix obtained by stackingfor all ^^^^ ∈ ^^^^:^^^^ ^^^^W= ^^^^
[0155] FIG. 23 is a communication flow 2300 illustrating an example message flow of a server implemented with a visual position prior estimation mode in accordance with various aspects of the present disclosure. The numberings associated with the communication flow 2300 do not specify a particular temporal order and are merely used as references for the communication flow 2300. In one example, as illustrated by FIG.7, a server 2302 (e.g., a local server, a fusion center, an external cloud server, etc.) may communicate with a set of cameras 2304 (e.g., the first camera 702, the second camera 704, and / or the third camera 706, etc.) for localizing one or more objects (e.g., the first object 708, the second object 710, and / or the third object 712, etc.) that are equipped with one or more RF devices (e.g., RF devices 714, RFID tags, Wi-Fi® modules, Bluetooth® modules, tracking modules, etc.). The set of cameras 2304 may be a networked camera system connected to the server 2302 via wired or wireless connection, and each camera in the set of cameras 2304 may be characterized with a corresponding projective transformation matrix. The set of cameras 2304 may also record any change in their internal and external parameters locally and send them over to the server 2302.
[0156] Initially, the server 2302 may initiate a visual position prior estimation procedure. For example, as shown at 2310, the procedure initiation may be triggered periodically or on-demand by an application (e.g., a worker in a warehouse requesting position estimate for an item associated with specific RF device ID(s)).
[0157] At 2312, the server 2302 may message / ping the set of cameras 2304 and to request for updates on their internal and / or external parameters (e.g., their focal lengths, camera sizes, resolution, pixels, orientation, etc.). Based on the request, at 2314, the set of cameras 2304 may transmit corresponding updates on their internal / external parameters (if any).
[0158] At 2316, the server 2302 may use the updates from the set of cameras 2304 to update their corresponding projective transformation matrices, such as described in 129025-1887WO01Qualcomm Ref. No.2301671WO 46 connection with FIGs. 5 and 6. For instance, the orientations and / or internal parameters of a camera such as its focal length may have changed.
[0159] At 2318, after the camera matrices are up-to-date, the server 2302 may determine or obtain the RF-based location information for a set of relevant RF devices, such as described in connection with FIGs.12 and 13. In some examples, the set of relevant RF devices may include RF devices that are attached to object(s) to be detected / located. Then, based on the (estimated) locations of the set of relevant RF devices, the server 2302 may determine which camera(s)’ FOVs the corresponding objects are in, and the server 2302 may message / ping the corresponding cameras for requesting RoIs, such as described in connection with FIG.11.
[0160] At 2320, the server 2302 may perform joint association across images and target RF devices. In some implementations, the server 2302 may also (optionally) be configured to perform the privacy-preserving mechanism described in connection with FIG.15, where the set of cameras 2304 may initially send just RoI descriptors following a layered security paradigm (e.g., as shown at 2322 and 2324). Then, the server 2302 may perform the joint association based on the RoI descriptors.
[0161] At 2326, after the joint association process is finished, the server 2302 may execute the logic of the visual position prior estimation (referring to FIG.24 below).
[0162] FIG. 24 is a communication flow 2400 illustrating an example logic of the visual position prior estimation in accordance with various aspects of the present disclosure. The numberings associated with the communication flow 2400 do not specify a particular temporal order and are merely used as references for the communication flow 2400.
[0163] At 2402, an RoI generation module (e.g., a standard object detector) may be implemented at each camera of the set of cameras 2304, where the RoI generation module may generate a number of relevant RoIs (e.g., centroids, bounding boxes, etc.) for each image captured by the camera.
[0164] At 2404, these relevant RoIs may then be transmitted / handed over to an RoI association module together with the corresponding camera calibration matrices and / or the RF-based position information of the target RF devices contained in the RoIs. As described in connection with FIG. 15, the RoI association module may associate each target RF device with a list of RoIs and the corresponding images / cameras in which the list of RoIs appear. 129025-1887WO01Qualcomm Ref. No.2301671WO 47
[0165] At 2406, the associated RoIs may be transmitted / handed over to an RoI sampling module, which takes each of these lists of RoIs and produces RoI samples from the list of RoIs, such as described in connection with FIG.20.
[0166] At 2408, the RoI samples are then transmitted / handed over to a back-projection module to form points clouds in the world coordinate system, such as described in connection with FIGs.18 and 19.
[0167] At 2410, the visual position priors may be fitted to the point clouds for each target RF device.
[0168] Aspects presented herein may improve the security and privacy for visual-based positioning. Aspects presented herein may enable a server (e.g., a local server, a fusion center, an external cloud server, etc.) to associate one or more objects across images captured by different cameras (which may be referred to as joint object association or joint association across images) while applying privacy-preserving transformation (which may be referred to as privacy-preservation) to the images. For example, in a given warehouse location having objects equipped with RF devices, it is suitable / desirable to localize objects through fusion of multiple RF and vision-based modalities. To preserve privacy, a privacy-preserving transformation is applied to raw images such that any potentially compromising visual features are removed from the images. Matching visual features across images for removal of such features is challenging as that may have implications on processes related to visual positioning (for example, association of targets across multiple view systems). In one aspect, RF- based position estimates of the RF devices included in each of a region of interest (in an image) to perform joint association across images and across RF devices, thus alleviating the need for additional visual features and descriptors. In another aspect, a layered security paradigm is proposed wherein decision is made at a Fusion Center if any additional information is needed to perform the association across different images.
[0169] Vision-based position estimates obtained with calibrated cameras provide good accuracy. However, target RF devices might not be directly visible in a camera FOV, thus impacting accuracy. In a region of interest (ROI) of a given image, actual pixel location of an RF device is unknown, and the uncertainty regarding the pixel location of the RF device is modeled using a probability distribution function Image Domain Position Prior). The Image Domain Position Prior induces a distribution defined over the corresponding world region through projective transformation (Visual Position 129025-1887WO01Qualcomm Ref. No.2301671WO 48 Prior). In one aspect, a sampling-based method is proposed for estimating visual priors. Fusion Center implements the Visual Position Prior estimation.
[0170] FIG.25 is a flowchart 2500 of a method of wireless communication. The method may be performed by a network entity (e.g., the server 1502, 2302; the network entity 2760). The method may enable network entity e.g., (a server, a local server, a fusion center, an external cloud server, etc.) to perform joint object association across images and RF devices using RF-based position information while applying privacy- preservation.
[0171] At 2504, the network entity may transmit, to a set of cameras, a request to provide an RoI descriptor for RF devices based on visual information captured by the set of cameras, such as described in connection with FIG.15. For example, at 1510, a server 1502 may request a set of cameras 1504 to provide RoIs for one or more RF devices based on visual information captured by the set of cameras 1504. The transmission of the request may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27.
[0172] At 2506, the network entity may receive, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices, such as described in connection with FIG. 15. For example, at 1512, the server 1502 may receive a set of RoI descriptors for a set of RF devices from each camera in the set of cameras 1504. The reception of the first set of RoI descriptors may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27.
[0173] At 2508, the network entity may associate multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras, such as described in connection with FIG. 15. For example, at 1516, the server 1502 may perform joint association of relevant objects, which includes associating multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras 1504 based on the RoI descriptors for the set of RF devices. The association of multiple RF devices with their corresponding RoI descriptors may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27. 129025-1887WO01Qualcomm Ref. No.2301671WO 49
[0174] In one example, to associate the multiple RF devices with their corresponding RoI descriptors across the images captured by the set of cameras, the network entity may identify at least one common RoI descriptor from the images captured by the set of cameras for each RF device of the multiple RF devices, or determine which RF devices in the multiple RF devices correspond to a particular RoI descriptor based on their estimated positions.
[0175] In another example, each RoI descriptor in the first set of ROI descriptors may provide a pixel location of an RF device on an image corresponding to a region where the RF device is physically located.
[0176] In another example, the network entity may receive a request to perform a joint association for the set of cameras, such as described in connection with FIG.15. For example, an application may request the server 1502 to provide joint association for the set of cameras 1504. The reception of the request may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27.
[0177] In another example, the network entity may determine that a confidence level for an association between an RoI descriptor and at least one RF device is below a confidence threshold; transmit, to at least one camera in the set of cameras, a second request to provide a second set of RoI descriptors for the set of RF devices, where the second set of RoI descriptors provide an increased amount of visual features than the first set of RoI descriptors; and receive, from the at least one camera in the set of cameras, the second set of RoI descriptors for the set of RF devices, such as described in connection with FIG.15. For example, at 1518, the server 1502 may find RoIs with association confidence level lower than a confidence threshold. At 1520, if such RoIs are found, the server 1502 may determine whether the upper tier has been reached. Then, at 1522, if the upper tier has not been reached, the server 1502, the server 1502 may request the set of cameras 1504 (or one or more camera(s)) within the set of cameras 1504) to provided RoIs with an increased tier. At 1526, the server 1502 may receive RoI descriptors from the set of cameras 1504 based on the requested tier. The determination of the confidence level, the transmission of the second request, and / or the reception of the second set of RoI descriptors may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG. 27. In some implementations, the visual features may include: RoI centroids for the set of RF devices, RoI bounding boxes for 129025-1887WO01Qualcomm Ref. No.2301671WO 50 the set of RF devices, RoI shapes for the set of RF devices, RoI silhouettes for the set of RF devices, colors for the set of RF devices, patterns for the set of RF devices, or a combination thereof.
[0178] In another example, the network entity may obtain locations of the multiple RF devices based on RF-based positioning. In some implementations, to associate the multiple RF devices with their corresponding RoI descriptors, the network entity may associate the locations of the multiple RF devices with their corresponding RoI descriptors.
[0179] In another example, the network entity may output an indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning. In some implementations, to output the indication of the association, the network entity may transmit, to a cloud server, the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning, or the network entity may store the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning.
[0180] In another example, each RF device in the multiple RF devices may be attached to an object, and the network entity may determine a location of the object using vision- based positioning based on the association between the multiple RF devices and their corresponding RoI descriptors.
[0181] FIG.26 is a flowchart 2600 of a method of wireless communication. The method may be performed by a network entity (e.g., the server 1502, 2302; the network entity 2760). The method may enable network entity e.g., (a server, a local server, a fusion center, an external cloud server, etc.) to perform joint object association across images and RF devices using RF-based position information while applying privacy- preservation.
[0182] At 2604, the network entity may transmit, to a set of cameras, a request to provide an RoI descriptor for RF devices based on visual information captured by the set of cameras, such as described in connection with FIG.15. For example, at 1510, a server 1502 may request a set of cameras 1504 to provide RoIs for one or more RF devices based on visual information captured by the set of cameras 1504. The transmission of the request may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27. 129025-1887WO01Qualcomm Ref. No.2301671WO 51
[0183] At 2606, the network entity may receive, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices, such as described in connection with FIG. 15. For example, at 1512, the server 1502 may receive a set of RoI descriptors for a set of RF devices from each camera in the set of cameras 1504. The reception of the first set of RoI descriptors may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27.
[0184] At 2608, the network entity may associate multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras, such as described in connection with FIG. 15. For example, at 1516, the server 1502 may perform joint association of relevant objects, which includes associating multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras 1504 based on the RoI descriptors for the set of RF devices. The association of multiple RF devices with their corresponding RoI descriptors may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27.
[0185] In one example, to associate the multiple RF devices with their corresponding RoI descriptors across the images captured by the set of cameras, the network entity may identify at least one common RoI descriptor from the images captured by the set of cameras for each RF device of the multiple RF devices, or determine which RF devices in the multiple RF devices correspond to a particular RoI descriptor based on their estimated positions.
[0186] In another example, each RoI descriptor in the first set of ROI descriptors may provide a pixel location of an RF device on an image corresponding to a region where the RF device is physically located.
[0187] In another example, at 2602, the network entity may receive a request to perform a joint association for the set of cameras, such as described in connection with FIG.15. For example, an application may request the server 1502 to provide joint association for the set of cameras 1504. The reception of the request may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG.27. 129025-1887WO01Qualcomm Ref. No.2301671WO 52
[0188] In another example, at 2610, the network entity may determine that a confidence level for an association between an RoI descriptor and at least one RF device is below a confidence threshold; transmit, to at least one camera in the set of cameras, a second request to provide a second set of RoI descriptors for the set of RF devices, where the second set of RoI descriptors provide an increased amount of visual features than the first set of RoI descriptors; and receive, from the at least one camera in the set of cameras, the second set of RoI descriptors for the set of RF devices, such as described in connection with FIG.15. For example, at 1518, the server 1502 may find RoIs with association confidence level lower than a confidence threshold. At 1520, if such RoIs are found, the server 1502 may determine whether the upper tier has been reached. Then, at 1522, if the upper tier has not been reached, the server 1502, the server 1502 may request the set of cameras 1504 (or one or more camera(s)) within the set of cameras 1504) to provided RoIs with an increased tier. At 1526, the server 1502 may receive RoI descriptors from the set of cameras 1504 based on the requested tier. The determination of the confidence level, the transmission of the second request, and / or the reception of the second set of RoI descriptors may be performed by, e.g., the visual positioning component 197, the network processor 2712, and / or the network interface 2780 of the network entity 2760 in FIG. 27. In some implementations, the visual features may include: RoI centroids for the set of RF devices, RoI bounding boxes for the set of RF devices, RoI shapes for the set of RF devices, RoI silhouettes for the set of RF devices, colors for the set of RF devices, patterns for the set of RF devices, or a combination thereof.
[0189] In another example, the network entity may obtain locations of the multiple RF devices based on RF-based positioning. In some implementations, to associate the multiple RF devices with their corresponding RoI descriptors, the network entity may associate the locations of the multiple RF devices with their corresponding RoI descriptors.
[0190] In another example, the network entity may output an indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning. In some implementations, to output the indication of the association, the network entity may transmit, to a cloud server, the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning, or the network entity may store the indication 129025-1887WO01Qualcomm Ref. No.2301671WO 53 of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning.
[0191] In another example, each RF device in the multiple RF devices may be attached to an object, and the network entity may determine a location of the object using vision- based positioning based on the association between the multiple RF devices and their corresponding RoI descriptors.
[0192] FIG.27 is a diagram 2700 illustrating an example of a hardware implementation for a network entity 2760. In one example, the network entity 2760 may be within the core network 120. The network entity 2760 may include a network processor 2712. The network processor 2712 may include on-chip memory 2712'. In some aspects, the network entity 2760 may further include additional memory modules 2714. The network entity 2760 communicates via the network interface 2780 directly (e.g., backhaul link) or indirectly (e.g., through a RIC) with the CU 2702. The on-chip memory 2712' and the additional memory modules 2714 may each be considered a computer-readable medium / memory. Each computer-readable medium / memory may be non-transitory. The processor 2712 is responsible for general processing, including the execution of software stored on the computer-readable medium / memory. The software, when executed by the corresponding processor(s) causes the processor(s) to perform the various functions described supra. The computer-readable medium / memory may also be used for storing data that is manipulated by the processor(s) when executing software.
[0193] As discussed supra, the visual positioning component 197 may be configured to transmit, to a set of cameras, a request to provide an RoI descriptor for RF devices based on visual information captured by the set of cameras. The visual positioning component 197 may also be configured to receive, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices. The visual positioning component 197 may also be configured to associate multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras. The visual positioning component 197 may be within the processor 2712. The visual positioning component 197 may be one or more hardware components specifically configured to carry out the stated processes / algorithm, implemented by one or more processors configured to perform the stated processes / algorithm, stored within a computer-readable medium for implementation 129025-1887WO01Qualcomm Ref. No.2301671WO 54 by one or more processors, or some combination thereof. The network entity 2760 may include a variety of components configured for various functions. In one configuration, the network entity 2760 may include means for transmitting, to a set of cameras, a request to provide an RoI descriptor for RF devices based on visual information captured by the set of cameras. The network entity 2760 may further include means for receiving, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices. The network entity 2760 may further include means for associating multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0194] In one configuration, the means for associating the multiple RF devices with their corresponding RoI descriptors across the images captured by the set of cameras include configuring the network entity 2760 to identify at least one common RoI descriptor from the images captured by the set of cameras for each RF device of the multiple RF devices, or determine which RF devices in the multiple RF devices correspond to a particular RoI descriptor based on their estimated positions.
[0195] In another configuration, each RoI descriptor in the first set of ROI descriptors may provide a pixel location of an RF device on an image corresponding to a region where the RF device is physically located.
[0196] In another configuration, the network entity 2760 may further include means for receiving a request to perform a joint association for the set of cameras.
[0197] In another configuration, the network entity 2760 may further include means for determining that a confidence level for an association between an RoI descriptor and at least one RF device is below a confidence threshold; means for transmitting, to at least one camera in the set of cameras, a second request to provide a second set of RoI descriptors for the set of RF devices, where the second set of RoI descriptors provide an increased amount of visual features than the first set of RoI descriptors; and means for receiving, from the at least one camera in the set of cameras, the second set of RoI descriptors for the set of RF devices. In some implementations, the visual features may include: RoI centroids for the set of RF devices, RoI bounding boxes for the set of RF devices, RoI shapes for the set of RF devices, RoI silhouettes for the set of RF devices, colors for the set of RF devices, patterns for the set of RF devices, or a combination thereof. 129025-1887WO01Qualcomm Ref. No.2301671WO 55
[0198] In another configuration, the network entity 2760 may further include means for obtaining locations of the multiple RF devices based on RF-based positioning. In some implementations, the means for associating the multiple RF devices with their corresponding RoI descriptors may include configuring the network entity 2760 to associate the locations of the multiple RF devices with their corresponding RoI descriptors.
[0199] In another configuration, the network entity 2760 may further include means for outputting an indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning. In some implementations, the means for outputting the indication of the association may include configuring the network entity to transmit, to a cloud server, the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning, or to store the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning.
[0200] In another configuration, each RF device in the multiple RF devices may be attached to an object, and the network entity 2760 may further include means for determining a location of the object using vision-based positioning based on the association between the multiple RF devices and their corresponding RoI descriptors.
[0201] The means may be the visual positioning component 197 of the network entity 2760 configured to perform the functions recited by the means.
[0202] FIG.28 is a flowchart 2800 of a method of wireless communication. The method may be performed by a network entity (e.g., the server 1502, 2302; the network entity 3060). The method may enable network entity e.g., (a server, a local server, a fusion center, an external cloud server, etc.) to estimate visual position prior for a set of RF devices that are not visible to one or more camera.
[0203] At 2808, the network entity may receive a set of images captured by a set of cameras, where the set of images are associated with a set of RF devices, such as described in connection with FIGs.23 and 24. For example, the server 2302 may receive a set of images captured by the set of cameras 2304, where the set of images are associated with a set of RF devices (e.g., the set of images may include object(s) that contain the set of RF devices). The reception of the set of images may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30. 129025-1887WO01Qualcomm Ref. No.2301671WO 56
[0204] At 2810, the network entity may associate each RF device in the set of RF devices with a list of RoIs across the set of images captured by the set of cameras, such as described in connection with FIGs.23 and 24. For example, at 2320 of FIG.23, the server 2302 may perform joint association of targets across images and RF devices, which may include associating each of the RF devices with a list of RoIs across the set of images captured by the set of cameras 2304. The association of each RF device in the set of RF devices with a list of RoIs may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30.
[0205] At 2812, the network entity may generate a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs, such as described in connection with FIGs.23 and 24. For example, at 2408 of FIG. 24, the RoI samples may be transmitted / handed over to a back-projection module to form points clouds in the world coordinate system. The generation of the set of point clouds may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG. 30.
[0206] At 2814, the network entity may estimate a visual position for the set of RF devices based on the set of point clouds, such as described in connection with FIGs.23 and 24. For example, at 2410 of FIG.24, the server 2302 may fit the visual position priors to the point clouds for each target RF device. The estimation of the visual position may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG. 30.
[0207] In one example, the network entity may receive a request to locate the set of RF devices, where estimating the visual position for the set of RF devices is based on the request to locate the set of RF devices, such as described in connection with FIGs.23 and 24. For example, at 2310 of FIG.23, the server 2302 may initiate a visual position prior estimation procedure, where the procedure initiation may be triggered periodically or on-demand by an application (e.g., a worker in a warehouse requesting position estimate for an item associated with specific RF device ID(s)). The reception of the request may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30. 129025-1887WO01Qualcomm Ref. No.2301671WO 57
[0208] In another example, the network entity may estimate coarse locations of the set of RF devices, and select the set of cameras based on the estimated coarse locations of the set of RF devices, such as described in connection with FIGs.23 and 24. For example, at 2318 of FIG. 23, after the camera matrices are up-to-date, the server 2302 may determine or obtain the RF-based location information for a set of relevant RF devices. Then, based on the (estimated) locations of the set of relevant RF devices, the server 2302 may determine which camera(s)’ FOVs the corresponding objects are in, and the server 2302 may message / ping the corresponding cameras for requesting RoIs. The estimation of the coarse locations of the set of RF devices and / or the selection of the set of cameras may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG. 30. In some implementations, to estimate the coarse locations of the set of RF devices, the network entity may obtain RF-based location information for the set of RF devices.
[0209] In another example, the network entity may update a set of camera matrices for the set of cameras, such as described in connection with FIGs.23 and 24. For example, at 2316 of FIG.23, the server 2302 may use the updates from the set of cameras 2304 to update their corresponding projective transformation matrices. The update of the set of camera matrices may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30. In some implementations, to update the set of camera matrices for the set of cameras, the network entity may transmit, to each camera in the set of cameras, a request for an update to at least one parameter associated with each camera in the set of cameras; receive, from each camera in the set of cameras, the update to the at least one parameter associated with each camera in the set of cameras; and update the set of camera matrices for each camera in the set of cameras based on its corresponding update. In some implementations, the at least one parameter may include: at least one internal parameter, at least one external parameter, or a combination thereof.
[0210] In another example, the network entity may determine a more precise location for each RF device in the set of RF devices based on the estimated visual position.
[0211] In another example, the network entity may output an indication of the estimated visual position. In some implementations, to output the indication of the estimated 129025-1887WO01Qualcomm Ref. No.2301671WO 58 visual position, the network entity may transmit the indication of the estimated visual position, or storing the indication of the estimated visual position.
[0212] In another example, the set of images associated with the set of RF devices may include one or more images of at least one object to which at least one RF device in the set of RF devices is attached. In some implementations, the at least one object is in a FOV of at least one camera in the set of cameras, and where the at least one RF device is not in the FOV of the at least one camera.
[0213] In another example, the set of images associated with the set of RF devices may correspond to a set of RoI descriptors for the set of RF devices. Then, to associate each RF device in the set of RF devices with the list of RoIs across the set of images captured by the set of cameras, the network entity may associate one or more RF devices with their corresponding RoI descriptors across the set of images captured by the set of cameras based on the set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0214] In another example, each RoI in the list of ROIs may provide a pixel location of an RF device in the set of RF devices on an image corresponding to a region where the RF device is physically located.
[0215] In another example, the estimated visual position provides a distribution or a probability of a likely location of an RF device in the set of RF devices.
[0216] FIG.29 is a flowchart 2900 of a method of wireless communication. The method may be performed by a network entity (e.g., the server 1502, 2302; the network entity 3060). The method may enable network entity e.g., (a server, a local server, a fusion center, an external cloud server, etc.) to estimate visual position prior for a set of RF devices that are not visible to one or more camera.
[0217] At 2908, the network entity may receive a set of images captured by a set of cameras, where the set of images are associated with a set of RF devices, such as described in connection with FIGs.23 and 24. For example, the server 2302 may receive a set of images captured by the set of cameras 2304, where the set of images are associated with a set of RF devices (e.g., the set of images may include object(s) that contain the set of RF devices). The reception of the set of images may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30.
[0218] At 2910, the network entity may associate each RF device in the set of RF devices with a list of RoIs across the set of images captured by the set of cameras, such as 129025-1887WO01Qualcomm Ref. No.2301671WO 59 described in connection with FIGs.23 and 24. For example, at 2320 of FIG.23, the server 2302 may perform joint association of targets across images and RF devices, which may include associating each of the RF devices with a list of RoIs across the set of images captured by the set of cameras 2304. The association of each RF device in the set of RF devices with a list of RoIs may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30.
[0219] At 2912, the network entity may generate a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs, such as described in connection with FIGs.23 and 24. For example, at 2408 of FIG. 24, the RoI samples may be transmitted / handed over to a back-projection module to form points clouds in the world coordinate system. The generation of the set of point clouds may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG. 30.
[0220] At 2914, the network entity may estimate a visual position for the set of RF devices based on the set of point clouds, such as described in connection with FIGs.23 and 24. For example, at 2410 of FIG.24, the server 2302 may fit the visual position priors to the point clouds for each target RF device. The estimation of the visual position may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG. 30.
[0221] In one example, at 2902, the network entity may receive a request to locate the set of RF devices, where estimating the visual position for the set of RF devices is based on the request to locate the set of RF devices, such as described in connection with FIGs. 23 and 24. For example, at 2310 of FIG. 23, the server 2302 may initiate a visual position prior estimation procedure, where the procedure initiation may be triggered periodically or on-demand by an application (e.g., a worker in a warehouse requesting position estimate for an item associated with specific RF device ID(s)). The reception of the request may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30.
[0222] In another example, at 2904, the network entity may estimate coarse locations of the set of RF devices, and select the set of cameras based on the estimated coarse locations 129025-1887WO01Qualcomm Ref. No.2301671WO 60 of the set of RF devices, such as described in connection with FIGs.23 and 24. For example, at 2318 of FIG.23, after the camera matrices are up-to-date, the server 2302 may determine or obtain the RF-based location information for a set of relevant RF devices. Then, based on the (estimated) locations of the set of relevant RF devices, the server 2302 may determine which camera(s)’ FOVs the corresponding objects are in, and the server 2302 may message / ping the corresponding cameras for requesting RoIs. The estimation of the coarse locations of the set of RF devices and / or the selection of the set of cameras may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG. 30. In some implementations, to estimate the coarse locations of the set of RF devices, the network entity may obtain RF-based location information for the set of RF devices.
[0223] In another example, at 2906, the network entity may update a set of camera matrices for the set of cameras, such as described in connection with FIGs. 23 and 24. For example, at 2316 of FIG. 23, the server 2302 may use the updates from the set of cameras 2304 to update their corresponding projective transformation matrices. The update of the set of camera matrices may be performed by, e.g., the visual positioning component 197, the network processor 3012, and / or the network interface 3080 of the network entity 3060 in FIG.30. In some implementations, to update the set of camera matrices for the set of cameras, the network entity may transmit, to each camera in the set of cameras, a request for an update to at least one parameter associated with each camera in the set of cameras; receive, from each camera in the set of cameras, the update to the at least one parameter associated with each camera in the set of cameras; and update the set of camera matrices for each camera in the set of cameras based on its corresponding update. In some implementations, the at least one parameter may include: at least one internal parameter, at least one external parameter, or a combination thereof.
[0224] In another example, the network entity may determine a more precise location for each RF device in the set of RF devices based on the estimated visual position.
[0225] In another example, the network entity may output an indication of the estimated visual position. In some implementations, to output the indication of the estimated visual position, the network entity may transmit the indication of the estimated visual position, or storing the indication of the estimated visual position. 129025-1887WO01Qualcomm Ref. No.2301671WO 61
[0226] In another example, the set of images associated with the set of RF devices may include one or more images of at least one object to which at least one RF device in the set of RF devices is attached. In some implementations, the at least one object is in a FOV of at least one camera in the set of cameras, and where the at least one RF device is not in the FOV of the at least one camera.
[0227] In another example, the set of images associated with the set of RF devices may correspond to a set of RoI descriptors for the set of RF devices. Then, to associate each RF device in the set of RF devices with the list of RoIs across the set of images captured by the set of cameras, the network entity may associate one or more RF devices with their corresponding RoI descriptors across the set of images captured by the set of cameras based on the set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0228] In another example, each RoI in the list of ROIs may provide a pixel location of an RF device in the set of RF devices on an image corresponding to a region where the RF device is physically located.
[0229] In another example, the estimated visual position provides a distribution or a probability of a likely location of an RF device in the set of RF devices.
[0230] FIG.30 is a diagram 3000 illustrating an example of a hardware implementation for a network entity 3060. In one example, the network entity 3060 may be within the core network 120. The network entity 3060 may include a network processor 3012. The network processor 3012 may include on-chip memory 3012'. In some aspects, the network entity 3060 may further include additional memory modules 3014. The network entity 3060 communicates via the network interface 3080 directly (e.g., backhaul link) or indirectly (e.g., through a RIC) with the CU 3002. The on-chip memory 3012' and the additional memory modules 3014 may each be considered a computer-readable medium / memory. Each computer-readable medium / memory may be non-transitory. The processor 3012 is responsible for general processing, including the execution of software stored on the computer-readable medium / memory. The software, when executed by the corresponding processor(s) causes the processor(s) to perform the various functions described supra. The computer-readable medium / memory may also be used for storing data that is manipulated by the processor(s) when executing software.
[0231] As discussed supra, the visual positioning component 197 may be configured to receive a set of images captured by a set of cameras, where the set of images are 129025-1887WO01Qualcomm Ref. No.2301671WO 62 associated with a set of RF devices. The visual positioning component 197 may also be configured to associate each RF device in the set of RF devices with a list of RoIs across the set of images captured by the set of cameras. The visual positioning component 197 may also be configured to generate a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs. The visual positioning component 197 may also be configured to estimate a visual position for the set of RF devices based on the set of point clouds. The visual positioning component 197 may be within the processor 3012. The visual positioning component 197 may be one or more hardware components specifically configured to carry out the stated processes / algorithm, implemented by one or more processors configured to perform the stated processes / algorithm, stored within a computer-readable medium for implementation by one or more processors, or some combination thereof. The network entity 3060 may include a variety of components configured for various functions. In one configuration, the network entity 3060 may include means for receiving a set of images captured by a set of cameras, where the set of images are associated with a set of RF devices. The network entity 3060 may further include means for associating each RF device in the set of RF devices with a list of RoIs across the set of images captured by the set of cameras. The network entity 3060 may further include means for generating a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs. The network entity 3060 may further include means for estimating a visual position for the set of RF devices based on the set of point clouds.
[0232] In one configuration, the network entity 3060 may further include means for receiving a request to locate the set of RF devices, where estimating the visual position for the set of RF devices is based on the request to locate the set of RF devices.
[0233] In another configuration, the network entity 3060 may further include means for estimating coarse locations of the set of RF devices, and means for selecting the set of cameras based on the estimated coarse locations of the set of RF devices. In some implementations, the means for estimating the coarse locations of the set of RF devices may include configuring the network entity 3060 to obtain RF-based location information for the set of RF devices.
[0234] In another configuration, the network entity 3060 may further include means for updating a set of camera matrices for the set of cameras. In some implementations, the means for updating the set of camera matrices for the set of cameras may include 129025-1887WO01Qualcomm Ref. No.2301671WO 63 configuring the network entity 3060 to transmit, to each camera in the set of cameras, a request for an update to at least one parameter associated with each camera in the set of cameras; receive, from each camera in the set of cameras, the update to the at least one parameter associated with each camera in the set of cameras; and update the set of camera matrices for each camera in the set of cameras based on its corresponding update. In some implementations, the at least one parameter may include: at least one internal parameter, at least one external parameter, or a combination thereof.
[0235] In another configuration, the network entity 3060 may further include means for determining a more precise location for each RF device in the set of RF devices based on the estimated visual position.
[0236] In another configuration, the network entity 3060 may further include means for outputting an indication of the estimated visual position. In some implementations, the means for outputting the indication of the estimated visual position may include configuring the network entity 3060 to transmit the indication of the estimated visual position, or to store the indication of the estimated visual position.
[0237] In another configuration, the set of images associated with the set of RF devices may include one or more images of at least one object to which at least one RF device in the set of RF devices is attached. In some implementations, the at least one object is in a FOV of at least one camera in the set of cameras, and where the at least one RF device is not in the FOV of the at least one camera.
[0238] In another configuration, the set of images associated with the set of RF devices may correspond to a set of RoI descriptors for the set of RF devices. Then, the means for associating each RF device in the set of RF devices with the list of RoIs across the set of images captured by the set of cameras may include configuring the network entity 3060 to associate one or more RF devices with their corresponding RoI descriptors across the set of images captured by the set of cameras based on the set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0239] In another configuration, each RoI in the list of ROIs may provide a pixel location of an RF device in the set of RF devices on an image corresponding to a region where the RF device is physically located.
[0240] In another configuration, the estimated visual position provides a distribution or a probability of a likely location of an RF device in the set of RF devices. 129025-1887WO01Qualcomm Ref. No.2301671WO 64
[0241] The means may be the visual positioning component 197 of the network entity 3060 configured to perform the functions recited by the means.
[0242] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not limited to the specific order or hierarchy presented.
[0243] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not limited to the aspects described herein, but are to be accorded the full scope consistent with the language claims. Reference to an element in the singular does not mean “one and only one” unless specifically so stated, but rather “one or more.” Terms such as “if,” “when,” and “while” do not imply an immediate temporal relationship or reaction. That is, these phrases, e.g., “when,” do not imply an immediate action in response to or during the occurrence of an action, but simply imply that if a condition is met then an action will occur, but without requiring a specific or immediate time constraint for the action to occur. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof” may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. Sets should be interpreted as a set of elements where the elements number one or more. Accordingly, for a set of X, X would include one or more elements. If a first apparatus receives data from or transmits data to a second apparatus, the data may be received / transmitted directly 129025-1887WO01Qualcomm Ref. No.2301671WO 65 between the first and second apparatuses, or indirectly between the first and second apparatuses through a set of apparatuses. A device configured to “output” data, such as a transmission, signal, or message, may transmit the data, for example with a transceiver, or may send the data to a device that transmits the data. A device configured to “obtain” data, such as a transmission, signal, or message, may receive, for example with a transceiver, or may obtain the data from a device that receives the data. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are encompassed by the claims. Moreover, nothing disclosed herein is dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. The words “module,” “mechanism,” “element,” “device,” and the like may not be a substitute for the word “means.” As such, no claim element is to be construed as a means plus function unless the element is expressly recited using the phrase “means for.”
[0244] As used herein, the phrase “based on” shall not be construed as a reference to a closed set of information, one or more conditions, one or more factors, or the like. In other words, the phrase “based on A” (where “A” may be information, a condition, a factor, or the like) shall be construed as “based at least on A” unless specifically recited differently.
[0245] The following aspects are illustrative only and may be combined with other aspects or teachings described herein, without limitation.
[0246] Aspect 1 is a method of wireless communication at a network entity, including: transmitting, to a set of cameras, a request to provide a region of interest (RoI) descriptor for radio frequency (RF) devices based on visual information captured by the set of cameras; receiving, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices; and associating multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0247] Aspect 2 is the method of aspect 1, where associating the multiple RF devices with their corresponding RoI descriptors across the images captured by the set of cameras includes at least one of: identifying at least one common RoI descriptor from the images captured by the set of cameras for each RF device of the multiple RF devices, 129025-1887WO01Qualcomm Ref. No.2301671WO 66 or determining which RF devices in the multiple RF devices correspond to a particular RoI descriptor based on their estimated positions.
[0248] Aspect 3 is the method of aspect 2, where each RoI descriptor in the first set of ROI descriptors provides a pixel location of an RF device on an image corresponding to a region where the RF device is physically located.
[0249] Aspect 4 is the method of any of aspects 1 to 3, further including: receiving a second request to perform a joint association for the set of cameras.
[0250] Aspect 5 is the method of any of aspects 1 to 4, further including: determining that a confidence level for an association between an RoI descriptor and at least one RF device is below a confidence threshold; transmitting, to at least one camera in the set of cameras, a second request to provide a second set of RoI descriptors for the set of RF devices, where the second set of RoI descriptors provide an increased amount of visual features than the first set of RoI descriptors; and receiving, from the at least one camera in the set of cameras, the second set of RoI descriptors for the set of RF devices.
[0251] Aspect 6 is the method of any of aspects 1 to 5, where the visual features include: RoI centroids for the set of RF devices, RoI bounding boxes for the set of RF devices, RoI shapes for the set of RF devices, RoI silhouettes for the set of RF devices, colors for the set of RF devices, patterns for the set of RF devices, or a combination thereof.
[0252] Aspect 7 is the method of any of aspects 1 to 6, further including: obtaining locations of the multiple RF devices based on RF-based positioning.
[0253] Aspect 8 is the method of any of aspects 1 to 7, where associating the multiple RF devices with their corresponding RoI descriptors includes: associating the locations of the multiple RF devices with their corresponding RoI descriptors.
[0254] Aspect 9 is the method of any of aspects 1 to 8, further including: outputting an indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning.
[0255] Aspect 10 is the method of any of aspects 1 to 9, where outputting the indication of the association includes: transmitting, to a cloud server, the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning; or storing the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning. 129025-1887WO01Qualcomm Ref. No.2301671WO 67
[0256] Aspect 11 is the method of any of aspects 1 to 10, where each RF device in the multiple RF devices is attached to an object, the method further including: determining a location of the object using vision-based positioning based on the association between the multiple RF devices and their corresponding RoI descriptors.
[0257] Aspect 12 is an apparatus for wireless communication at a first network node, including: a memory; and at least one processor coupled to the memory and, based at least in part on information stored in the memory, the at least one processor is configured to implement any of aspects 1 to 11.
[0258] Aspect 13 is the apparatus of aspect 12, further including at least one of a transceiver or an antenna coupled to the at least one processor.
[0259] Aspect 14 is an apparatus for wireless communication including means for implementing any of aspects 1 to 11.
[0260] Aspect 15 is a computer-readable medium (e.g., a non-transitory computer-readable medium) storing computer executable code, where the code when executed by a processor causes the processor to implement any of aspects 1 to 11.
[0261] Aspect 16 is a method of wireless communication at a network entity, including: receiving a set of images captured by a set of cameras, where the set of images are associated with a set of radio frequency (RF) devices; associating each RF device in the set of RF devices with a list of regions of interest (RoIs) across the set of images captured by the set of cameras; generating a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs; and estimating a visual position for the set of RF devices based on the set of point clouds.
[0262] Aspect 17 is the method of aspect 16, further including: receiving a request to locate the set of RF devices, where estimating the visual position for the set of RF devices is based on the request to locate the set of RF devices.
[0263] Aspect 18 is the method of aspect 16 or 17, further including: estimating coarse locations of the set of RF devices; and selecting the set of cameras based on the estimated coarse locations of the set of RF devices.
[0264] Aspect 19 is the method of any of aspects 16 to 18, where estimating the coarse locations of the set of RF devices includes: obtaining RF-based location information for the set of RF devices.
[0265] Aspect 20 is the method of any of aspects 16 to 19, further including: updating a set of camera matrices for the set of cameras. 129025-1887WO01Qualcomm Ref. No.2301671WO 68
[0266] Aspect 21 is the method of any of aspects 16 to 20, where updating the set of camera matrices for the set of cameras includes: transmitting, to each camera in the set of cameras, a request for an update to at least one parameter associated with each camera in the set of cameras; receiving, from each camera in the set of cameras, the update to the at least one parameter associated with each camera in the set of cameras; and updating the set of camera matrices for each camera in the set of cameras based on its corresponding update.
[0267] Aspect 22 is the method of any of aspects 16 to 21, where the at least one parameter includes: at least one internal parameter, at least one external parameter, or a combination thereof.
[0268] Aspect 23 is the method of any of aspects 16 to 22, further including: determining a more precise location for each RF device in the set of RF devices based on the estimated visual position.
[0269] Aspect 24 is the method of any of aspects 16 to 23, further including: outputting an indication of the estimated visual position.
[0270] Aspect 25 is the method of any of aspects 16 to 24, where outputting the indication of the estimated visual position includes: transmitting the indication of the estimated visual position; or storing the indication of the estimated visual position.
[0271] Aspect 26 is the method of any of aspects 16 to 25, where the set of images associated with the set of RF devices includes one or more images of at least one object to which at least one RF device in the set of RF devices is attached.
[0272] Aspect 27 is the method of any of aspects 16 to 26, where the at least one object is in a field-of-view (FOV) of at least one camera in the set of cameras, and where the at least one RF device is not in the FOV of the at least one camera.
[0273] Aspect 28 is the method of any of aspects 16 to 27, where the set of images associated with the set of RF devices corresponds to a set of RoI descriptors for the set of RF devices, where associating each RF device in the set of RF devices with the list of RoIs across the set of images captured by the set of cameras includes: associating one or more RF devices with their corresponding RoI descriptors across the set of images captured by the set of cameras based on the set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
[0274] Aspect 29 is the method of any of aspects 16 to 28, where each RoI in the list of ROIs provides a pixel location of an RF device in the set of RF devices on an image corresponding to a region where the RF device is physically located. 129025-1887WO01Qualcomm Ref. No.2301671WO 69
[0275] Aspect 30 is the method of any of aspects 16 to 29, where the estimated visual position provides a distribution or a probability of a likely location of an RF device in the set of RF devices.
[0276] Aspect 31 is an apparatus for wireless communication at a network entity, including: a memory; and at least one processor coupled to the memory and, based at least in part on information stored in the memory, the at least one processor is configured to implement any of aspects 16 to 30.
[0277] Aspect 32 is the apparatus of aspect 31, further including at least one of a transceiver or an antenna coupled to the at least one processor.
[0278] Aspect 33 is an apparatus for wireless communication including means for implementing any of aspects 16 to 30.
[0279] Aspect 34 is a computer-readable medium (e.g., a non-transitory computer-readable medium) storing computer executable code, where the code when executed by a processor causes the processor to implement any of aspects 16 to 30. 129025-1887WO01
Claims
Qualcomm Ref. No.2301671WO 70 CLAIMS WHAT IS CLAIMED IS:
1. An apparatus for wireless communication at a network entity, comprising: a memory; and at least one processor coupled to the memory and, based at least in part on information stored in the memory, the at least one processor is configured to: transmit, to a set of cameras, a request to provide a region of interest (RoI) descriptor for radio frequency (RF) devices based on visual information captured by the set of cameras; receive, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices; and associate multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
2. The apparatus of claim 1, wherein to associate the multiple RF devices with their corresponding RoI descriptors across the images captured by the set of cameras, the at least one processor is configured to: identify at least one common RoI descriptor from the images captured by the set of cameras for each RF device of the multiple RF devices, or determine which RF devices in the multiple RF devices correspond to a particular RoI descriptor based on their estimated positions.
3. The apparatus of claim 1, wherein each RoI descriptor in the first set of ROI descriptors provides a pixel location of an RF device on an image corresponding to a region where the RF device is physically located.
4. The apparatus of claim 1, wherein the at least one processor is further configured to: receive a second request to perform a joint association for the set of cameras.
5. The apparatus of claim 1, wherein the at least one processor is further configured to: 129025-1887WO01Qualcomm Ref. No.2301671WO 71 determine that a confidence level for an association between an RoI descriptor and at least one RF device is below a confidence threshold; transmit, to at least one camera in the set of cameras, a second request to provide a second set of RoI descriptors for the set of RF devices, wherein the second set of RoI descriptors provide an increased amount of visual features than the first set of RoI descriptors; and receive, from the at least one camera in the set of cameras, the second set of RoI descriptors for the set of RF devices.
6. The apparatus of claim 5, wherein the visual features include: RoI centroids for the set of RF devices, RoI bounding boxes for the set of RF devices, RoI shapes for the set of RF devices, RoI silhouettes for the set of RF devices, colors for the set of RF devices, patterns for the set of RF devices, or a combination thereof.
7. The apparatus of claim 1, wherein the at least one processor is further configured to: obtain locations of the multiple RF devices based on RF-based positioning.
8. The apparatus of claim 7, wherein to associate the multiple RF devices with their corresponding RoI descriptors, the at least one processor is configured to: associate the locations of the multiple RF devices with their corresponding RoI descriptors.
9. The apparatus of claim 1, wherein the at least one processor is further configured to: output an indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting vision-based positioning.
10. The apparatus of claim 9, wherein to output the indication of the association, the at least one processor is configured to: 129025-1887WO01Qualcomm Ref. No.2301671WO 72 transmit, to a cloud server, the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting the vision-based positioning; or store the indication of the association between the multiple RF devices and their corresponding RoI descriptors for assisting the vision-based positioning.
11. The apparatus of claim 1, wherein each RF device in the multiple RF devices is attached to an object, wherein the at least one processor is further configured to: determine a location of the object using vision-based positioning based on the association between the multiple RF devices and their corresponding RoI descriptors.
12. The apparatus of claim 1, further comprising a transceiver coupled to the at least one processor, wherein to transmit the request, the at least one processor is configured to transmit, via the transceiver, the request, and wherein to receive the first set of RoI descriptors, the at least one processor is configured to receive, via the transceiver, the first set of RoI descriptors.
13. A method of wireless communication at a network entity, comprising: transmitting, to a set of cameras, a request to provide a region of interest (RoI) descriptor for radio frequency (RF) devices based on visual information captured by the set of cameras; receiving, from each camera in the set of cameras, a first set of RoI descriptors for a set of RF devices; and associating multiple RF devices with their corresponding RoI descriptors across images captured by the set of cameras based on the first set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
14. An apparatus for wireless communication at a network entity, comprising: a memory; and at least one processor coupled to the memory and, based at least in part on information stored in the memory, the at least one processor is configured to: receive a set of images captured by a set of cameras, wherein the set of images are associated with a set of radio frequency (RF) devices; 129025-1887WO01Qualcomm Ref. No.2301671WO 73 associate each RF device in the set of RF devices with a list of regions of interest (RoIs) across the set of images captured by the set of cameras; generate a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs; and estimate a visual position for the set of RF devices based on the set of point clouds.
15. The apparatus of claim 14, wherein the at least one processor is further configured to: receive a request to locate the set of RF devices, wherein estimating the visual position for the set of RF devices is based on the request to locate the set of RF devices.
16. The apparatus of claim 14, wherein the at least one processor is further configured to: estimate coarse locations of the set of RF devices; and select the set of cameras based on the estimated coarse locations of the set of RF devices.
17. The apparatus of claim 16, wherein to estimate the coarse locations of the set of RF devices, the at least one processor is configured to: obtain RF-based location information for the set of RF devices.
18. The apparatus of claim 14, wherein the at least one processor is further configured to: update a set of camera matrices for the set of cameras.
19. The apparatus of claim 18, wherein to update the set of camera matrices for the set of cameras, the at least one processor is configured to: transmit, to each camera in the set of cameras, a request for an update to at least one parameter associated with each camera in the set of cameras; receive, from each camera in the set of cameras, the update to the at least one parameter associated with each camera in the set of cameras; and update the set of camera matrices for each camera in the set of cameras based on its corresponding update. 129025-1887WO01Qualcomm Ref. No.2301671WO 74 20. The apparatus of claim 19, wherein the at least one parameter includes: at least one internal parameter, at least one external parameter, or a combination thereof.
21. The apparatus of claim 14, wherein the at least one processor is further configured to: determine a more precise location for each RF device in the set of RF devices based on the estimated visual position.
22. The apparatus of claim 14, wherein the at least one processor is further configured to: output an indication of the estimated visual position.
23. The apparatus of claim 22, wherein to output the indication of the estimated visual position, the at least one processor is configured to: transmit the indication of the estimated visual position; or store the indication of the estimated visual position.
24. The apparatus of claim 14, wherein the set of images associated with the set of RF devices includes one or more images of at least one object to which at least one RF device in the set of RF devices is attached.
25. The apparatus of claim 24, wherein the at least one object is in a field-of-view (FOV) of at least one camera in the set of cameras, and wherein the at least one RF device is not in the FOV of the at least one camera.
26. The apparatus of claim 14, wherein the set of images associated with the set of RF devices corresponds to a set of RoI descriptors for the set of RF devices, wherein to associate each RF device in the set of RF devices with the list of RoIs across the set of images captured by the set of cameras, the at least one processor is configured to: 129025-1887WO01Qualcomm Ref. No.2301671WO 75 associate one or more RF devices with their corresponding RoI descriptors across the set of images captured by the set of cameras based on the set of RoI descriptors for the set of RF devices received from each camera in the set of cameras.
27. The apparatus of claim 14, wherein each RoI in the list of ROIs provides a pixel location of an RF device in the set of RF devices on an image corresponding to a region where the RF device is physically located.
28. The apparatus of claim 14, wherein the estimated visual position provides a distribution or a probability of a likely location of an RF device in the set of RF devices.
29. The apparatus of claim 14, further comprising a transceiver coupled to the at least one processor, wherein to receive the set of images captured by the set of cameras, the at least one processor is configured to receive, via the transceiver, the set of images captured by the set of cameras.
30. A method of wireless communication at a network entity, comprising: receiving a set of images captured by a set of cameras, wherein the set of images are associated with a set of radio frequency (RF) devices; associating each RF device in the set of RF devices with a list of regions of interest (RoIs) across the set of images captured by the set of cameras; generating a set of point clouds for the set of RF devices based on the association between the set of RF devices and their corresponding RoIs; and estimating a visual position for the set of RF devices based on the set of point clouds. 129025-1887WO01
Citation Information
Patent Citations
Track detection method and robot
CN108491758A
Methods and systems for camera 3D pose determination
EP3903285A1
Homography error correction using sensor locations
US20210124953A1
System and method of personalized navigation inside a business enterprise
WO2021080933A1
