Object Tracking System

Abstract

To provide an object-tracking system capable of controlling a camera of a partner in environments where the GPS or the like are unavailable.SOLUTION: A tracking system is designed for environments where the global positioning system (GPS), radio frequency (RF), and / or cellular communication signals are unavailable. The system uses camera-captured images of the surrounding environment in conjunction with inertial measurements to perform visual and / or conventional odometry. An object detection algorithm and / or tracking scheme are used to detect objects within the captured images so as to help determine a user position relative to the objects. The detector architecture allows target (and / or object) independent camera detection and / or tracking that is easily configurable and / or reconfigurable depending on a type of the object to be detected and / or tracked.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an object tracking system, and more particularly to an object tracking system designed for environments where the Global Positioning System (GPS), radio frequency (RF), and / or mobile communication signals are not available.

Background Art

[0002] Humans have a basic desire for order and often seek to know their past, present, and future positions. For military, security, and / or investigative personnel, this basic desire for location is particularly important when first operating in an area. Without known landmarks and / or infrastructure options for positioning, the location and / or direction can become difficult to determine, leading to increased stress levels and / or the risk of other potential dangers.

[0003] Existing tracking systems, including wearable tracking systems, typically rely on the Global Positioning System (GPS), pre-deployed radio frequency (RF) infrastructure, and / or other positioning infrastructure. For example, in urban, indoor, and / or underground environments, there is a risk that GPS may not be available (i.e., a GPS-denied environment). When GPS becomes unavailable, significant problems can arise in positioning for such tracking systems. Various RF-based tracking systems have been developed for tracking the position of people and objects indoors (e.g., using cell sites and Wi-Fi signals), but such tracking systems tend to rely on pre-deployed and / or potentially expensive RF infrastructure. Therefore, there is a need for an object tracking system designed for environments where GPS, RF, and / or mobile communication signals are not available.

Summary of the Invention

[0004] The present disclosure relates to an object tracking system, including those designed for environments where GPS, RF, and / or mobile communication signals are not available.

[0005] According to a first aspect, an object tracking system includes a camera configured to capture images of a surrounding environment according to a first camera configuration, the camera configured to employ a second camera configuration, and a computer operably connected to the camera, the computer configured to process images from the camera, detect a movable object in the images using a detection algorithm selected from a library of detection algorithms, estimate a current position of the movable object, estimate a current position of a user relative to the current position of the movable object, predict a future position of the movable object, and determine the second camera configuration based at least in part on the future position of the movable object.

[0006] In certain aspects, the object tracking system further comprises an inertial measurement unit (IMU), in which case the computer is configured to measure at least one of the angular velocity or linear acceleration of the user.

[0007] In certain aspects, a computer is configured to: (1) estimate a current position of a user based at least in part on the IMU measurements, and (2) predict a future position of a movable object based at least in part on the IMU measurements.

[0008] In certain aspects, the computer is configured to generate or update a map that reflects the current position of the movable object and the current position of the user relative to the movable object.

[0009] In certain aspects, the camera is connected to or integrated into a wearable coupled to the user.

[0010] In certain aspects, at least one of the current position of the movable object, the current position of the user, or the future position of the movable object is determined using a Kalman filter.

[0011] In certain embodiments, the computer is operably connected to a Global Positioning System (GPS). In such a case, the computer is configured to determine the user's current position relative to a moving object within a GPS denial environment.

[0012] According to a second aspect, a positioning system includes a camera oriented according to a current pan, tilt, and / or zoom (PTZ) configuration and configured to capture an image while oriented according to the current PTZ configuration, a processor configured to process the image using computer vision methods, a controller configured to receive the current PTZ configuration from the camera, generate a new PTZ configuration, and communicate the new PTZ configuration to the camera, a detector configured to detect a moving object in the image, where the moving object is detected using a detection algorithm selected from a library of bounding boxes and object detection algorithms and the selection is based on the type of object detected and the detection algorithm is deactivated if the detected type of object no longer matches, and a state estimator configured to store the user's current estimated position and calculate a new estimated position of the user based on the type of object, the estimated position of the moving object, and a stored map, where the stored map includes the estimated position of the moving object relative to the user's current estimated position.

[0013] In certain embodiments, the camera is connected to or integrated with a wearable attached to the user.

[0014] In certain embodiments, the controller generates a new PTZ configuration based at least in part on at least one of the type of object detected, the user's newly estimated position, or information shared by an external device.

[0015] In certain embodiments, the camera is an omnidirectional camera.

[0016] In certain embodiments, the positioning system further comprises a second camera configured to capture an image.

[0017] In certain embodiments, the positioning system further comprises an inertial measurement unit (IMU).

[0018] In certain embodiments, the state estimator uses at least partially odometry measurements from an odometer to calculate a new estimated position of the user.

[0019] In certain embodiments, the state estimator uses a Kalman filter.

[0020] In certain embodiments, the positioning system further comprises an interface configured to receive user input. In that case, the input is used to help determine the type of object being detected.

[0021] According to a third aspect, a method for visually localizing an individual object comprises capturing an image with a camera using a first pan, tilt, and / or zoom (PTZ) configuration, processing the image to determine an appropriate detection algorithm, selecting an appropriate detection algorithm from a library of detection algorithms, detecting an object in the image using a detection algorithm that surrounds the object with a bounding box, determining whether the object is moving or stationary, in response to a determination that the object is stationary, estimating the position of the object relative to one of a user or another object, estimating the position using a Kalman filter and inertial measurements from an inertial measurement unit (IMU), storing the position of the object in a map memory, determining a second PTZ configuration, and orienting the camera according to the second PTZ configuration.

[0022] In certain embodiments, computer vision is used in at least one of the steps of processing an image, selecting an appropriate detection algorithm, detecting an object in the image, and determining whether the object is moving or stationary.

[0023] In certain embodiments, the camera comprises a plurality of cameras that provide omnidirectional coverage.

[0024] In certain embodiments, the method further comprises sharing at least one of the estimated position and / or map information with an external device.

[0025] The above or other objectives, features, and advantages of the devices, systems, and methods described in this specification will be readily understood from the following description of their specific embodiments. In that case, the accompanying drawings are shown, and like numbers refer to like structures. The drawings are not necessarily drawn to scale, and instead, emphasis is placed on showing the principles of the devices, systems, and methods described in this specification.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Best Mode for Carrying Out the Invention

[0027] Preferred embodiments of the present disclosure will be described below in this specification with reference to the accompanying drawings. The components in the drawings are not necessarily drawn to an exact scale; rather, emphasis is placed on clearly showing the principles of the present invention. For example, the size of an element may be emphasized for clarity and convenience of explanation. Further, where possible, the same reference numerals are used throughout the drawings to refer to the same or similar parts of one embodiment. In the following description, well-known functions and explanations are not described in detail because they may obscure the present disclosure with unnecessary details. No word in the specification should be construed as indicating an element that is not claimed as essential to the practice of an embodiment. In this application, the following terms and definitions apply.

[0028] As used herein, the terms "about" and "approximately" mean appropriately close to the value or range of values when used to modify or explain a value (or range of values). Thus, the embodiments described herein should not be limited only to the recited values and ranges of values, but rather should include deviations that can function appropriately.

[0029] As used herein, the term "and / or" means any one or more of the items in a list joined by "and / or". As an example, "x and / or y" means any one of the three-element set {(x), (y), (x, y)}. In other words, "x and / or y" means "one or both of x and y". As another example, "x, y, and / or z" means any one of the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y, and / or y" means "one or more of x, y, and z".

[0030] As used herein, the terms "circuit" and / or "electrical circuit" refer to physical electronic components (i.e., hardware), such as analog and / or digital components, power and / or control elements, and / or microprocessors, as well as any software and / or firmware ("code") that constitutes the hardware, is executed by the hardware, and / or may otherwise be related to the hardware.

[0031] As used herein, the terms "communicate" and "communicating" refer to (1) transmitting or otherwise conveying data from a source to a destination, and / or (2) delivering data to be conveyed to a destination over a communication medium, system, channel, network, device, wire, cable, fiber, circuit, and / or link.

[0032] As used herein, "connected," "connected to," and "connected with" each mean a structural and / or electrical connection, which may be attached, linked, coupled, joined, fastened, linked, and / or otherwise fixed. As used herein, the term "attach" means being linked, coupled, connected, joined, fastened, linked, and / or otherwise fixed. As used herein, the term "coupled" means being attached, linked, connected, joined, fastened, linked, and / or otherwise fixed.

[0033] As used herein, the term "database" means an organized body of related data, regardless of the manner in which the data or organized body of data is represented. For example, the organized body of related data may take one or more forms of data represented in a table, map, grid, packet, datagram, frame, file, e-mail, message, document, report, list, or any other form.

[0034] As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "e.g." and "for example" highlight a list of one or more non-limiting examples, instances, or illustrations.

[0035] As used herein, the term "memory" means computer hardware or circuitry for storing information for use by a processor and / or other digital devices. The memory can be any suitable type of computer memory, such as, for example, read-only memory (ROM), random access memory (RAM), cache memory, compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), computer-readable media, or any other type of electronic storage media.

[0036] As used herein, the term "network" includes all types of networks, including the Internet, as well as interconnected networks, and is not limited to any particular network or interconnected network.

[0037] As used herein, "operatively connected" means that several elements or assemblies are connected together such that a first element / assembly changes from one state (and / or configuration, orientation, position, etc.) to another state, and a second element / assembly operatively connected to the first element / assembly also changes from one state (and / or configuration, orientation, position, etc.) to another state. Note that it is possible for a first element to be "operatively connected" to a second element while the second element is not "operatively connected" to the first element.

[0038] As used herein, the term "processor" refers to processor devices, apparatus, programs, circuits, components, systems, and subsystems, whether implemented in hardware, explicitly embodied in software, or both, and whether programmable or not. As used herein, the term "processor" includes, but is not limited to, one or more computing devices, hardwired circuits, signal modifying devices and systems, devices and machines for controlling systems, central processing units, programmable devices and systems, field programmable gate arrays, application specific integrated circuits, systems on a chip, systems with discrete elements and / or circuits, state machines, virtual machines, data processors, processing facilities, and any combination thereof. For example, a processor can be any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an application specific integrated circuit (ASIC), or the like. A processor can be connected to or integrated with a memory device.

[0039] Disclosed herein are object tracking systems, such as object-agnostic tracking systems, that can help locate an individual (e.g., a user who may be wearing a wearable) or an object (e.g., a movable object). The object tracking system can also relate the location of an individual or object to the location(s) of other individuals or objects to help the individual or object navigate within an unknown, unspecified, complex environment. In other words, as described more fully below, the object tracking system can provide detection and processing of moving and / or stationary objects (e.g., relative to a tracked object or person) to facilitate navigation and mapping within GPS-denied, RF-denied, or other track-denied environments, thereby providing an estimate (e.g., localization) of the location of the individual (e.g., a user) or object (e.g., a vehicle, equipment, etc.) based on surrounding objects.

[0040] Vision-based position tracking can work effectively even in the absence of GPS and / or any deployed RF infrastructure. For example, vision-based position tracking provides the ability to locate itself (e.g., a person or object with an optical element attached) and generate maps in a manner similar to humans. Research on human tracking that does not rely on GPS has led to algorithmic solutions based on highly accurate visual inertial odometry measurements, but these algorithmic solutions are post-processed and often limited to single-core computer processing units (CPUs). However, pan, tilt, and zoom (PTZ) network cameras can help address the problem of position tracking, for example, within a GPS-denied environment. One tracking algorithm may be associated with commercial off-the-shelf (COTS) PTZ networked cameras. Most PTZ camera tracking algorithms are originally tied to control schemes implemented to detect and / or track objects. The compound error associated with object detection can lead to inaccuracies in tracking. This can limit the number of objects that a target tracking system can accurately track or require the entire system to be retrained to accurately track different types of objects. Additionally, the algorithms may not be structured for rapid adaptation to existing target tracking platforms.

[0041] To address at least one of the above, the present disclosure presents an object tracking system configured to track the position and / or location of a user (or object) without the need for GPS and / or RF deployable infrastructure. The object tracking system can operate with high precision, such as within 0.2% of the total travel distance, in a GPS-denied and / or RF-denied environment by developing more robust hardware and portable algorithms to utilize a general-purpose graphics processing unit (GPGPU) architecture. A GPGPU refers to a graphics processing unit (GPU) configured to execute non-specialized computations that can normally be performed by a CPU.

[0042] The objective of the object tracking system is to facilitate object-independent PTZ tracking that is easily configurable for the type of object or person being tracked and highly extensible to other object domains with little effort on the part of the user. In certain aspects, the object tracking system may be configured to support the definition of parameterized, general-purpose object detectors within a set of standardized software modules trained by an artificial neural network. This system architecture design can maximize the scalability of the architecture across all detection domains. The object tracking system can further include a library of object detectors that can be easily tailored for various use cases with a reconfigurable design that employs only the necessary algorithms and modules, while also allowing for rapid activation or deactivation of algorithms as needed.

[0043] Accordingly, the present disclosure describes a complete, extensible, and reconfigurable object-independent system for controlling networked PTZ cameras for object tracking and / or navigation operations. Thus, an object tracking system can provide a complete solution that uses one or more wearable sensors and computers, such as a portable device, enabled by developing GPGPU computer vision algorithms for use on a computer. In certain aspects, the object tracking system may be wearable and unobtrusive, whereby wearable (e.g., clothing) can be combined with one or more small and / or unobtrusive cameras and computers. The object tracking system can be configured, among other things, as follows: namely, (1) achieve an accuracy of less than 5 meters with a worst-case performance of less than 20 meters over a 2-hour endurance mission, (2) process 500,000 floating-point operations per second (FLOPS) via a portable user device to store information on a mapped area, including a local storage area of more than 100 GB, (3) transfer information using a local communication infrastructure such as Wi-Fi, Bluetooth, or a mobile communication network, (4) detect a position using an image from a camera instead of relying on (a) using a deployable GPS / RF infrastructure or (b) pre-surveying an area, (5) provide output data that is consistent with local command and control (C2) mapping tools such as a cursor over a target, and (6) operate within a position drift of up to 2 hours and within the limits of the data storage area in a GPS-denied environment due to an initial loss of GPS or a similar accuracy correction.

[0044] 1 illustrates components of an object tracking system 100. As shown, object tracking system 100 may include a control system 108 operably connected to one or more sensors, such as a camera 104, a temperature sensor 106, an inertial measurement unit (IMU) 110, a microphone (which may be integrated into camera 104, for example), etc. As shown, object tracking system 100 may include or be embodied in a wearable 102, such as a cap, hat, shirt, jacket, sweater, shoes, boots, gloves, skirt, pants, shorts, glasses, and / or any other suitable article of clothing, accessory, and / or other type of wearable.

[0045] In certain embodiments, the object tracking system 100 may employ commercially available, wearable, miniature surveillance cameras 104 that are unobtrusively embedded within the wearable 102. The cameras 104 may be operatively connected to a computer 112 via a control system 108 that may be integrated into the wearable 102. The object tracking system 100 may implement a multi-state constrained Kalman filter (MSCKF) to maintain navigation accuracy using one or more spy cameras 104 that perform simultaneous localization and mapping (SLAM) to improve the performance of the inertial measurement unit (IMU) 110 under GPS-denied conditions. During operation, the cameras 104 and / or computer 112 may serve to capture visual data (image data) and perform real-time visual odometry and / or tracking of identified objects in space, even under GPS-denied conditions, such as urban or underground environments.

[0046] The object tracking system 100 will be described in connection with a wearable 102 for primarily tracking a user (e.g., a person or an animal), but the object tracking system need not necessarily be embodied within the wearable. Rather, the object tracking system 100 can operate to facilitate the localization and / or navigation of virtually any movable object, including, for example, a transporter (e.g., an automobile, an aircraft, a ship, etc.), a facility, and other objects. For example, the object tracking system 100 can be integrated into a movable object or a transporter (e.g., as part of its control system or navigation system) to provide the disclosed features.

[0047] The control system 108 and one or more sensors may be attached to (and / or embedded within) the wearable 102 (shown as a cap). For example, the camera 104 may be embedded within the semi-rigid cap lining of the wearable 102. In certain aspects, the camera 104 can provide omnidirectional coverage when considered in combination. That is, the cameras 104, when combined, can have the ability to capture images and / or video from a substantially 360-degree area around a user wearing and / or operating the cameras 104 in each of the x, y, and z planes within a Cartesian coordinate system. Similarly, one or more temperature sensors 106 may also be attached to (and / or embedded within) the wearable 102. In one embodiment, one or more of the temperature sensors 106 may be configured to measure the coefficient of thermal expansion (CTE). In one embodiment, one or more of the temperature sensors 106 may comprise a thermistor, a thermostat, etc.

[0048] The object tracking system 100 may further include a computer 112 or may be operably connected to a computer 112. For example, the object tracking system 100 may include a transceiver 126 configured to communicate with the computer 112. The computer 112 may be located locally or remotely with respect to the control system 108 and / or the wearable 102. In certain embodiments, the computer 112 may be a commercial off-the-shelf (COTS) portable device such as a smartphone, a tablet computer, a personal digital assistant (PDA), a smartwatch, smart glasses, a laptop computer, a portable gaming device, and / or similar devices. However, the computer 112 may be a remote computer or a device equipped with another processor, including, for example, a stationary computer located at a command center. In certain embodiments, the computer 112 may comprise a customized device and / or a customized microchip. In certain embodiments, the object tracking system 100 may include a plurality of computers 112 or may be operably connected to a plurality of computers 112.

[0049] The computer 112 may include a processor 120, a display 116, one or more memory devices 122 (e.g., RAM, ROM, flash memory, etc.), a transceiver 124, and / or a user interface (UI) 114. The computer 112 may be configured to communicate directly with the control system 108 or via a communication network 118 (e.g., the Internet or another network). For example, the control system 108 may be configured to communicate with the computer 112 via the transceivers 124, 126 (e.g., wireless transceivers). The transceivers 124, 126 may be configured to communicate via one or more wireless standards such as Bluetooth (e.g., short wavelength, UHF radio waves within the 2.4 to 2.485 GHz ISM band), NFC, Wi-Fi (e.g., IEEE 802.11 standard). However, it is also contemplated that the computer 112 may be configured to communicate with the control system 108 via a wired connection.

[0050] In certain embodiments, display 116 may provide at least a portion of user interface 114. For example, display 116 may be configured as a touch screen display, and thus, user interface 114 is a touch screen digitizer covering an LCD display. In this embodiment, display 116 can display a graphical user interface (GUI) that can be selected via the touch screen. In other embodiments, user interface 114 may be a microphone that facilitates speech recognition technology, or may include such a microphone. One or more cameras 104, one or more temperature sensors 106, control system 108, and / or computer 112 may be operably connected by wires, cables, conductors, and / or other electrical means known to those skilled in the art. In certain embodiments, one or more cameras 104, one or more temperature sensors 106, control system 108, and / or computer 112 may be operably connected using wireless technology, such as via a cellular phone network (e.g., TDMA, GSM, and / or CDMA), Wi-Fi (e.g., 802.11a, b, g, n, ac), Bluetooth, near field communication (NFC), optical communication, wireless communication, and / or other suitable wireless communication techniques.

[0051] FIG. 2 shows an exemplary camera 104 that can be used within the object tracking system 100. In certain embodiments, camera 104 may comprise a small, unobtrusive surveillance camera that can be relatively easily hidden so as to be less conspicuous. In certain embodiments, camera 104 may comprise an optical sensor configured to capture photographs, videos, and / or audiovisual images. Camera 104 may be configured to operate in various modes, such as, for example, a normal mode, a night vision mode, a thermal mode, an infrared mode, etc. In certain embodiments, a user may be able to select an appropriate camera 104 mode via user interface 114. In certain embodiments, one or more cameras 104 may automatically detect the most appropriate mode for the environment, and either recommend the most appropriate mode (and / or one or more other modes) to the user or automatically switch to the most appropriate mode.

[0052] Camera 104 may be operably connected to camera module 200. Camera module 200 may support camera 104 and be able to supply electrical input and output (e.g., power and / or data such as streamed video) to or from camera 104. In certain embodiments, camera module 200 may be embodied as a circuit board. Camera module 200 may be operably connected to other components of object tracking system 100 via cable 202, thereby obviating the need for a transceiver and / or local battery. Cable 202 may be able to carry power, data, or both power and data. In certain embodiments, cable 202 may be omitted and data may be transmitted to / from control system 108 via transmitters, receivers, and / or transceivers integrated within camera module 200. Accordingly, camera module 200 may include or be connected to a wireless transceiver and / or a local battery for powering camera 104.

[0053] Camera 104 may be configured to send and receive information to / from control system 108 via camera module 200 and / or cable 202. For example, such information may include image data (such as a live video, a still image, etc.), instructions for capturing an image / live video, a notification that an image has been captured, image information, instructions for adopting a particular configuration (such as a pan, tilt, and / or zoom configuration), a notification that a particular configuration has been adopted, and / or other appropriate information, data, or instructions.

[0054] For example, camera 104 may be a pan, tilt, and zoom (PTZ) camera and may be configured to pan (and / or swivel, rotate, turn, twist, etc.) around the Y axis. In certain aspects, camera 104 may be configured to fully pan 360 degrees. In other aspects, camera 104 may be configured to pan less than 360 degrees, such as 270 degrees or 180 degrees. Camera 104 may be further configured to tilt (swivel, rotate, turn, twist, etc.) around the X axis. In certain aspects, camera 104 may be configured to fully tilt 360 degrees. In other aspects, camera 104 may be configured to tilt less than 360 degrees, such as 270 degrees or 180 degrees. In certain aspects, camera module 200 may risk interfering with image capture at a particular tilt angle. In certain aspects, camera 104 may be integrated or implemented into a control unit (such as for controlling pan, tilt, or zoom).

[0055] Camera 104 may be further configured to zoom in and zoom out using zoom lens 204. Thereby, zoom lens 204 is configured to change its focal length to enlarge (and / or expand) an image of a scene. In certain embodiments, zoom lens 204 may be an ultra-wide-angle lens such as, for example, a fish-eye lens. In certain embodiments, zoom lens 204 may comprise a 220-degree megapixel (MP) quality fish-eye lens, and camera 104 may comprise an 18-megapixel universal serial bus (USB) camera. In embodiments where multiple cameras are used, each camera 104 may have the same PTZ functionality or different PTZ functionality.

[0056] FIG. 3 shows various components of control system 108 with respect to other components of object tracking system 100. As shown, control system 108 generally comprises a processing circuit 300, an object detector 306, a detector library 308, a state estimator 310, and / or a data management unit 312. For example, data management unit 312 may be a data distribution service (DDS). For example, processing circuit 300 may comprise a graphics processing unit (GPU) 302 and a logic controller 304. Although shown as separate components, GPU 302 and logic controller 304 may be integrated within a single component such as a processor or a CPU. In certain embodiments, IMU 110 may be integrated with control system 108 (e.g., provided via a single substrate or chip).

[0057] During operation, the components of the object tracking system 100 move through the following process. That is, (1) obtaining image data of a scene and the current camera 104 configuration from the camera 104 (one or more), (2) processing the captured images by the processing circuit 300, (3) detecting objects in the scene image by the object detector 306, (4) filtering the found bounding boxes by the object detector 306 and / or the detector library 308, (5) estimating the state of the system from these bounding boxes by the state estimator 310, and (6) determining the control output (pan, tilt, zoom) sent back to the camera 104 by the processing circuit 300. Further, various types of information may be sent from the state estimator 310 and / or the processing circuit 300 to the data management unit 312. The data management unit 312 may communicate with the computer 112 via the transceivers 124, 126 (either wired or wirelessly via an antenna system). In certain aspects, the communication may be a process via an external interface layer 314 (e.g., a communication bus).

[0058] The control system 108 may be provided as a single microchip such as a system-on-chip (SoC) or system-on-board (SoB). For example, the GPU 302, the data management unit 312, the logic controller 304, the object detector 306, and / or the state estimator 310 may all be included within (or provided via) a single microchip. In one aspect, the detector library 308 may also be integrated within a single microchip. In certain aspects, the components of the control system 108 may be implemented within hardware, software, and / or a combination of the two. In certain aspects, the components of the control system 108 may be implemented across several microchips and / or other devices. The control system 108 is shown as a stand-alone component independent of the computer 112, but the control system 108 may be integrated with the computer 112 depending on the application.

[0059] During operation, the GPU 302 may be configured to process an image (i.e., image data) received from the camera 104. The GPU 302 may be operably connected to the camera 104 via a wired and / or wireless communication connection. In certain embodiments, the GPU 302 may be configured to implement real-time computer vision methods such as feature extraction. The GPU 302 may further or alternatively be configured to assist in the measurements by the visual odometer of the object tracking system 100. For example, the measurement method by the visual odometer may include an extended Kalman filter (EKF)-based algorithm such as a multiple state constraint Kalman filter (MSCKF). The measurement method by the visual odometer may be implemented at least in part using a vectorized computer programming language such as OpenCL.

[0060] In certain embodiments, the GPU 302 may be operably connected to a display (e.g., a local display or the display 116 of the computer 112), and / or a user interface (e.g., a local user interface or the user interface 114 of the computer 112). The communication may be facilitated via the data management unit 312 and / or the interface layer 314, or via other suitable means. The GPU 302 may be configured to draw image data (e.g., graphics, images, photos, and / or videos) on the display 116. For example, the GPU 302 may draw a map and / or relative positions of one or more users and / or one or more objects with respect to the user on the display 116. Examples of these are shown in FIGS. 6a and 6b. The GPU 302 may also be operably connected to the data management unit 312, the camera 104, and the logic controller 304. In certain embodiments, the GPU 302 may similarly be operably connected to other components.

[0061] The logic controller 304 may be configured to execute specific programmed processes and / or logical processes of the object tracking system 100, whether alone or in combination with other components of the object tracking system 100. In certain aspects, the logic controller 304 may be a processor such as a CPU. In certain aspects, for example, the logic controller 304 may be an octa-core CPU having four cores operating at 2.45 Hz and four cores operating at 1.9 GHz. As described above, the GPU 302 may be integrated with the logic controller 304.

[0062] The camera 104 may be configured to capture an image of the surrounding environment according to a plurality of camera configurations (e.g., a PTZ configuration) by adopting one or more camera configurations. For example, the camera 104 may capture a first image (or a first delivery video) using a first camera configuration and then may capture a second image (or a second delivery video) using a second camera configuration. The logic controller 304 may be configured to determine an appropriate second PTZ configuration (e.g., a new PTZ configuration) of the camera 104 according to a first PTZ configuration (e.g., a current or previous PTZ configuration). For example, the logic controller 304 may use information regarding the current PTZ camera 104 configuration when selecting the second PTZ configuration. The current PTZ camera configuration may be provided by the GPU 302. The GPU 302 may also help in determining. The logic controller 304 may also use information from the state estimator 310 for selection. For example, the logic controller 304 may use prediction and / or estimation information regarding not only the user of the object tracking system 100 and / or multiple users of other object tracking systems but also a specific object that has been detected and / or tracked in order to determine one or more new PTZ camera configurations. For example, the new PTZ configuration may correspond to a configuration that directs the camera 104 towards the center / centroid of a roughly estimated and / or predicted bounding box of an object being tracked within the scene of the image. The logic controller 304 (and / or the GPU 302) may be able to determine a new PTZ configuration for each camera 104. In some cases, the new PTZ configuration may be the same as or substantially similar to the current PTZ configuration.

[0063] The logic controller 304 may be configured to send other commands to the GPU 302 and / or the camera 104. For example, the logic controller 304 may send a similar command, such as in response to a command to immediately capture an image from a user via the user interface 114. In certain aspects, the logic controller 304 may send a command to capture an image to the GPU 302 and / or the camera 104 each time the camera 104 captures a new image. In certain aspects, the command to capture an image may be part of a new PTZ configuration. In certain aspects, the GPU 302 and / or the camera 104 may continuously capture images even if it does not receive a specific command from the logic controller 304. In certain aspects, the GPU 302 and / or the camera 104 may refrain from capturing an image if it does not receive a specific command. In some embodiments, the user may select, via the interface 114, whether the camera 104 should wait for a specific command before capturing an image or whether the camera 104 should continuously capture images.

[0064] The object detector 306 may be configured to detect objects within the scene of an image, such as an image captured by the camera 104. The object detector 306 may be a parameterized object detector. Thus, the object detector 306 can be adapted to a wide variety of domains. The object detector 306 can be implemented as hardware, software, or a combination thereof. In certain aspects, the object detector 306 may be classes and / or class instances, such as when implemented using an object-oriented programming language. In certain aspects, the object detector 306 may be implemented using OpenCL, C, C++, Java, Python, Perl, Pascal, and / or other applicable methods. The object detector 306 may be operably connected to the GPU 302 and / or the state estimator 310. The object detector 306 may further be communicable with the detector library 308. The connection between the object detector 306 and the detector library 308 may be via a human-machine interface. For example, the detector may be selected via a human-machine interface (e.g., the display 116 of the computer 112) and loaded from the detector library 308.

[0065] The detector library 308 can employ one or more algorithms (and / or methods, modules, etc.) for detecting objects in response to image data. The detector library 308 may further or alternatively comprise a collection of control schemes for tracking objects. Generally speaking, the detector library 308 can act as a collection of algorithms and / or libraries (e.g., a collection of known / learned images). The detector library 308 assists the object tracking system 100 in determining which objects within the scene are moving and which are not.

[0066] When the user (or object) and / or the camera 104 moves, the detector library 308 generates a map of the environment. As can be appreciated, the object tracking system 100 should distinguish which object is moving. For example, a signboard may be included in the category of being stationary, while a face may be included in the category of being moving. The object tracking system 100 can learn the attributes of stationary objects and / or can start from known attributes via the detector library 308. In other aspects, it is also contemplated that the object tracking system 100 can generate a library of images between two categories. One or more image processing techniques may be employed to identify objects within the images. For example, the one or more image processing techniques may include 2D and 3D object recognition, image segmentation, motion detection (e.g., single particle tracking), video tracking, optical flow method, 3D pose estimation device, and the like.

[0067] In certain aspects, the detection algorithm and the tracking control scheme may be linked and / or otherwise associated. In certain aspects, the detection algorithm and the tracking control scheme may be structured to be easily exchangeable within and / or out of the object detector 306 to conform to a particular modular format. In certain aspects, the detection algorithm and / or the tracking control scheme may be adapted for various uses in a reconfigurable design. In certain aspects, the detection algorithm and the tracking control scheme may be trained via machine learning by an artificial neural network. In one embodiment, a particular detection algorithm and / or tracking control scheme may be suitable for detecting and / or tracking a particular class, classification, type, variety, category, group, and / or class of objects better than those of other objects. The detector library 308 may be implemented in hardware and / or software. In certain aspects, the detector library 308 may comprise a database.

[0068] The object detector 306 may activate an appropriate detection algorithm and / or tracking scheme according to the object being detected and / or tracked, while deactivating an inappropriate detection algorithm and / or tracking scheme. In certain aspects, the object detector 306 can activate and / or deactivate a detection algorithm according to the class, classification, type, variety, category, group, and / or class of the object being detected and / or tracked. In certain aspects, the object detector 306 may activate an appropriate detection algorithm and / or tracking scheme, and / or deactivate an inappropriate detection algorithm and / or tracking scheme, according to a desired and / or selected use.

[0069] In certain aspects, in determining an appropriate detection algorithm and / or tracking scheme for activation and / or deactivation, the GPU 302 can provide pre-processed images from the camera 104 to the object detector 306 to assist the object detector 306. In certain aspects, in determining an appropriate detection algorithm and / or tracking scheme for activation and / or deactivation, a user can provide information via the user interface 114 to assist the object detector 306. For example, the user can input information regarding the surrounding environment, such as approximately its location, indoor, outdoor, urban, rural, elevated, underground, etc. This can assist the object detector 306 in eliminating detection algorithms and / or tracking schemes that are not very useful (e.g., a mountain detector / tracker in an underground urban environment, an elevator detector / tracker in an outdoor rural setting, etc.). In certain aspects, the object detector 306 can automatically detect aspects of the surrounding environment to activate and / or deactivate an appropriate detection algorithm and / or tracking scheme. When the detection of an object requires differentiation between the object and various environmental factors, features may be extracted from one or more images independent of the object's bounding box. Aspects of the environment such as foreground / background classification, environmental classification, lighting, etc. The architecture of the object detector 306 may be configured to enable a PTZ camera 104 object-independent target tracking system that is easily configurable for the type of object being tracked and highly extensible to other object domains with little work required on the part of the user.

[0070] In certain embodiments, the object detector 306 can use a bounding box to enclose an object within the scene during detection. In certain embodiments, the detector can use a centroid centered within the bounding box to assist in detecting and / or tracking the object. In certain embodiments, the object detector 306 can determine whether a detected object is moving independent of any movement by the user and / or the object tracking system 100. In certain embodiments, the object detector 306 can use information provided by the state estimator 310 to aid in determining whether an object is moving or stationary. For example, the camera 104 can be used to process an image sequence to identify objects via three-dimensional reconstruction techniques, such as optical flow. Optical flow can be used to determine patterns of apparent movement of objects, surfaces, and edges within a visual scene caused by relative motion between the observer and the scene (images).

[0071] In certain aspects, the object detector 306 can use acoustic information from the camera 104 in determining whether an object is moving or stationary. For example, changing amplitude of a particular sound and / or changing frequency of a particular sound can be interpreted as indicating movement. The object detector 306 can ignore objects (and / or corresponding bounding boxes and / or centroids) determined to be moving. For example, moving objects can be humans, vehicles, and / or animals. The object detector 306 can provide bounding box and / or centroid information corresponding to stationary objects to the state estimator 310. For example, stationary objects can include sign posts, landmarks, vending machines, exit / entrance doors, buildings, terrain, etc. In certain aspects, the object detector 306 can perform its operations in conjunction with (and / or with the assistance of) other components of the object tracking system 100, such as the logic controller 304, the GPU 302, the IMU 110, the data management unit 312, and / or the state estimator 310.

[0072] The IMU 110 may be configured to measure a specific force, angular rate, and / or magnetic field surrounding the user. The IMU 110 may further or alternatively measure the angular velocity, rotational speed, and / or linear acceleration of the user. The IMU 110 may include one or more of an accelerometer, a gyroscope, and / or a magnetometer. In certain embodiments, the IMU 110 may include a plurality of accelerometers, gyroscopes, and / or magnetometers.

[0073] The state estimator 310 may be configured to perform various operations. In certain aspects, the state estimator 310 can estimate and / or predict the current and / or future (one or more) positions (and / or (one or more) locations) of one or more objects detected and / or tracked by the camera 104 and / or the object detector 306. In certain aspects, the state estimator 310 can estimate and / or predict the current and / or future (one or more) positions (and / or (one or more) locations) of one or more users of the object tracking system 100. In certain aspects, the state estimator 310 can perform Simultaneous Localization and Mapping (SLAM) using one or more SLAM algorithms to estimate and / or predict the current and / or future (one or more) positions of objects and users within a local environment. In certain aspects, the state estimator 310 may employ measurements from a visual odometer using a Kalman filter, which aids in performing its prediction and / or estimation. In certain aspects, the Kalman filter may be an Extended Kalman Filter (EKF). In certain aspects, the state estimator 310 can also employ measurements from a conventional odometer using information provided by the IMU 110, which aids in performing its prediction and / or estimation. In one embodiment, drift may be prevalent in the measurements of the IMU 110, and the measurements from the visual odometer used by the state estimator 310 may help correct this drift. In one embodiment, the IMU 110 may be part of the computer 112. Information to and / or from the IMU 110 may be routed through the data management unit 312.

[0074] The state estimator 310 may use information from the object detector 306 and / or the IMU 110 in conjunction with a SLAM algorithm, odometer measurement, and / or visual odometer measurement to estimate and / or predict the current and / or future position(s) of the user and / or objects within the local environment, and may use this information to generate, maintain, and / or update a local map. The map may be stored in the memory device 122. In certain aspects, the map may be generated using map information obtained before tracking services (e.g., GPS, satellite, and / or cellular capability) are lost. The GPU 302 may be configured to render the map on the display 116 according to corresponding selections by the user via the user interface 114, one example of which is described in connection with FIGS. 6a and 6b.

[0075] The data management unit 312 may be configured to provide an interface between the object tracking system 100 and / or other systems and / or device components external to the object tracking system 100. For example, the data management unit 312 may be able to provide an interface between the GPU 302, the controller, the state estimator 310, and / or the object detector 306 and the memory device 122, the IMU 110, and / or the user interface 114. The data management unit 312 may also be able to provide an interface between the object tracking system 100 and a computer 112 (or another external computer such as a base station computer or a second computer 112). For example, the data management unit 312 may be able to assist in providing an interface between the object tracking system 100 and other users operating similar systems. In certain aspects, the data management unit 312 may be able to interact with an interface layer 314 to perform its operations. The interface layer 314 may include circuitry, software, ports, and / or protocols that are compatible with communicating with components of the object tracking system 100 and / or external devices of the object tracking system 100. For example, the data management unit 312 may include circuitry, software, ports, and / or protocols that enable wired and / or wireless communication, such as cable ports (e.g., HDMI, CAT5, CAT5e, CAT6, USB, etc.), wireless receivers, wireless transmitters, wireless transceivers, wireless communication protocols, Bluetooth circuitry (and / or corresponding protocols), NFC circuitry (and / or corresponding protocols), etc.

[0076] Figure 4 shows an exemplary method of operation 400 for the object tracking system 100. The example assumes that the object tracking system 100 is already involved, either manually or automatically, for example, when the computer 112 loses GPS, satellite, and / or mobile communication signals. The system starts at step 402, where the camera 104 captures image data representing an image of the scene of the environment. The image may be a photograph, a video, and / or a visual image. The image may actually be a plurality of images captured by a plurality of cameras 104 of the system. Each image may be analyzed together or independently of each other. The image may be captured while the camera is in a position and / or orientation corresponding to the current PTZ configuration. The image may be captured in response to specific instructions by the logic controller 304, implicit instructions by the logic controller 304, and / or user input. Prior to the capture of the image, the camera 104 and / or the GPU 302 may send a pre-processed version of the image to the object detector 306 to assist in activating and / or deactivating the detection algorithm and / or the tracking control scheme.

[0077] At step 404, the image is processed by the GPU 302. The GPU 302 may use feature extraction and / or other computer vision and / or image processing techniques to process the image. At step 406, the object detector 306 may deactivate one or more inappropriate detector algorithms and / or tracking control schemes. At step 408, the object detector 306 may activate one or more appropriate detector algorithms and / or tracking control schemes. At step 410, the object detector 306 may use the detector algorithm and / or the tracking control scheme to detect stationary objects in the captured image. Objects determined to be moving may be discarded by the object detector 306. A bounding box surrounding the object may be used when executing the detector algorithm and / or the tracking control scheme, as best shown in the captured image 500 of FIGS. 5A and 5B.

[0078] In step 412, a current position is estimated for the user and one or more objects detected in the captured image (e.g., captured image 500). The estimation of the user's current position may be based on one or more previous user and / or object position estimates and / or predictions, previously compiled map information, IMU 110 information, the current PTZ configuration of camera 104, the object(s) detected in the captured image, the position and / or estimated position of the object(s) in the captured image, and / or other information, in conjunction with SLAM, odometer measurement, and / or visual odometer measurement. The estimation of each object's current position may be based on one or more previous user and / or object position estimates and / or predictions, previously compiled map information, IMU 110 information, the current PTZ configuration of camera 104, the object(s) detected in the captured image, the position and / or estimated position of the object(s) in the captured image, and / or other information, in conjunction with SLAM, odometer measurement, and / or visual odometer measurement. In such cases, the estimate of the object's current position determined by the object detector may be fused with the object's estimated position from other visual odometry methods.

[0079] In step 414, new positions for the detected user and one or more objects within the captured image are predicted. The prediction of the new position of the user may be based on the estimation of the current position of the user, the estimation of the position of one or more current objects, the estimation and / or prediction of the position of one or more previous users and / or objects, previously edited map information, IMU110 information, the current PTZ configuration of camera 104, the detected (one or more) objects within the captured image, the position and / or estimated position of the (one or more) objects within the captured image, and / or other information, in conjunction with SLAM, measurements by an odometer, and / or measurement methods by a visual odometer. The prediction of the new position of each object may be based on the estimation of the current position of the user, the estimation of the position of one or more current objects, the estimation and / or prediction of the position of one or more previous users and / or objects, previously edited map information, IMU110 information, the current PTZ configuration of camera 104, the detected (one or more) objects within the captured image, the position and / or estimated position of the (one or more) objects within the captured image, and / or other information, in conjunction with SLAM, measurements by an odometer, and / or measurement methods by a visual odometer.

[0080] [[ID=http: / / www.example.com]] In step 416, the object tracking system 100 can communicate with other users and / or systems external to the object tracking system 100. The data management unit 312 and / or the interface layer 314 may help provide an interface between the object tracking system 100 and other users operating systems similar to the object tracking system 100 and / or other systems. Information may be communicated between the user's object tracking system 100 and other users and / or systems external to the object tracking system 100. For example, the information communicated may include the prediction of the new position of other users and / or objects, the estimation of the current position of other users and / or objects, the estimation and / or prediction of the position of one or more previous users and / or objects, previously edited map information, information related to external systems (e.g., IMU information, PTZ camera configuration, etc.), and / or other information.

[0081] In step 418, the map can be updated using the estimated current position of the user and / or one or more objects. The map may have been pre-generated when GPS, satellite, and / or mobile communication is still available, or may be newly generated by the object tracking system 100. The map may further be updated to include the information obtained in step 416. Exemplary maps are described in more detail in connection with FIGS. 6a and 6b.

[0082] In step 420, the camera configuration can be updated using the new PTZ configuration. Thereafter, the process may be repeated until it is manually terminated by the user or automatically terminated (such as when GPS, RF, and / or mobile communication is restored). In step 422, the process may be repeated with additional captured image data or may end in step 424 either (such as when resuming the tracking service or when the user terminates via the computer 112).

[0083] Although described in a particular order, the steps described in connection with FIG. 4 may overlap, occur in parallel, occur in a different order, and / or occur more than once. For example, steps 402 and 404 may overlap. Thus, while one image is being processed, other images may be captured, and / or while a portion of an image is being processed, that image may still be being captured. In one embodiment, a portion of an image may be pre-processed before image capture. In certain aspects, the order of steps 406 and 408 may be reversed, may overlap, and / or may be executed in parallel. In one embodiment, steps 414 and 412 may be reversed, may overlap, and / or may be executed in parallel. In one embodiment, step 418 may be executed before, in parallel with, overlapping with, and / or both before and after step 416.

[0084] The following example scenario illustrates how a user of object tracking system 100 might use object tracking system 100. The user enters an underground market and their cell phone loses GPS signal. Core software (e.g., its operating system) running on the user's computer 112 activates the camera 104 on the user's wearable 102 via control system 108 and begins locating and mapping the market.

[0085] As the camera 104 captures images, the image processing algorithms of the computer 112 track easily identified stationary objects across the images to align the scene. The algorithms distinguish significantly between moving and stationary objects, such as people, cars, or animals. This allows the algorithms to eliminate moving objects at an early stage. Stationary objects that are tracked include lettering on sign posts, landmarks, vending machines, exit and entrance doors, and common household items. The algorithms of the computer 112 perform SLAM to generate and / or store a map of the local area for future use while tracking the user's location on the map.

[0086] FIG. 5a illustrates an example image 500 that may be (e.g., was) captured and / or processed by the object tracking system 100 of FIG. 1, while FIG. 5b illustrates a magnified portion of the example image 500 of FIG. 5a. As best shown in FIG. 5b, bounding boxes 502, 504 can be used to track and / or process features of the image 500. In certain aspects, stationary and moving objects may be tracked differently relative to one another. For mapping purposes, movable objects may be ignored while stationary objects may be included in the map. For navigation purposes, both movable and stationary objects may be tracked to mitigate positional risk between a user and objects in an environment. As shown, stationary objects, such as text / letters on signs and / or the location of fixed objects in an area, are labeled using a solid bounding box 502, while moving (or movable) objects, such as pedestrians, are labeled using a dashed bounding box 504. Objects within each of the bounding boxes 502, 504 may be processed and / or identified (e.g., via the control system 108 and / or the computer 112). For example, optical character recognition (OCR) may be used to process the text within the solid bounding box 502. Similarly, facial recognition techniques may be used to identify people (or other characteristics of people, such as gender, age, ethnicity, etc.), such as individuals within the dashed bounding box 504.

[0087] Figures 6a and 6b show exemplary maps 602a, 602b that can be generated / updated by object tracking system 100 for display on a display (e.g., display 116). As shown, a user interface 114 may be provided (e.g., via a display 116 which may be a touch screen) to enable a user to manipulate the map via one or more functions such as zoom, pan, rotate, save, etc. Maps 602a, 602b can provide the relative positions of tracked objects or people 604 (e.g., a user) and / or one or more objects 606, 608 (e.g., movable object 606 and / or stationary object 608) in the environment, either in a two-dimensional (2D) space or a three-dimensional (3D) space (as shown). Objects 606, 608 may be identified and / or stored via the detector library 308 described above. In one embodiment, such as that shown in FIG. 6a, map 602a may employ a pre-generated map of an area (e.g., showing known streets, buildings, etc.) as a starting point, such as a map provided by a third-party mapping service using GPS, RF, and / or a mobile communication system. In such an embodiment, object tracking system 100 can update the pre-generated map to include information (e.g., object position, details, etc.) regarding the tracked object or person 604 and / or one or more tracked objects 606, 608, thereby resulting in map 602a. Map 602a may be updated based at least in part on the last known positions of the tracked object or person 604 and / or one or more tracked objects 606. Referring to FIG. 6b, another form of map 602b may be generated by object tracking system 100. Thereby, one or more objects 606 may be divided into categories (e.g., near, medium, or far) according to distance. In certain aspects, regardless of the form, maps 602a, 602b may be generated using data from multiple sensors (e.g., multiple cameras 104). The multiple sensors may be part of a single object tracking system 100 or multiple object tracking systems 100.Multiple object tracking systems 100 may be operatively connected to one another via one or more networks, such as communications network 118 .

[0088] It will be understood that aspects of the present disclosure may be implemented by hardware, software, and / or a combination thereof. The software may be stored in a non-transitory machine-readable (e.g., computer-readable) storage medium, such as an erasable or rewritable read-only memory (ROM), a memory, such as a random access memory (RAM), a memory chip, a memory device, or a memory integrated circuit (IC), or an optically or magnetically recordable non-transitory machine-readable, e.g., computer-readable storage medium, such as a compact disc (CD), a digital versatile disc (DVD), a magnetic disk, or a magnetic tape.

[0089] While the present method and / or system has been described with reference to specific embodiments, those skilled in the art will recognize that various modifications and equivalent substitutions may be made without departing from the scope of the present method and / or system. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the scope of the present disclosure. For example, the systems, blocks, and / or components of the disclosed embodiments may be combined, divided, rearranged, and / or otherwise modified. Therefore, the present method and / or system is not limited to the particular embodiments disclosed. Instead, the present method and / or system is intended to include all embodiments falling within the scope of the appended claims, both literally and under the doctrine of equivalents.

[0090] This disclosure includes subject matter described in the following clauses. Article 1. An object tracking system (100), comprising: A camera (104) configured to capture an image (500) of the surrounding environment according to a first camera (104) configuration, the camera (104) configured to employ a second camera (104) configuration, and A computer (112) operably connected to the camera (104), processing the image (500) from the camera (104), detecting moving objects (606, 608) (608) in the image (500) using a detection algorithm selected from a library of detection algorithms, estimating a current position of the moving object (606, 608) (608), estimating a current position of the user relative to the current position of the moving object (606, 608) (608), predicting a future position of the moving object (606, 608) (608), and An object tracking system (100) comprising a computer (112) configured to determine the second camera (104) configuration based at least in part on the future position of the moving object (606, 608) (608). Clause 2. The object tracking system (100) according to clause 1, further comprising an inertial measurement unit (IMU (110)), wherein the computer (112) is configured to measure at least one of the angular velocity or linear acceleration of the user. Clause 3. The computer (112) is configured to (1) estimate the current position of the user based at least in part on the measurements of the IMU (110), and (2) predict the future position of the moving object (606, 608) (608) based at least in part on the measurements of the IMU (110). The object tracking system (100) according to clause 2. Clause 4. The object tracking system (100) according to any one of clauses 1 to 3, wherein the computer (112) is configured to generate or update a map reflecting the current position of the movable object (606, 608) and the current position of the user with respect to the movable object (606, 608). Clause 5. The object tracking system (100) according to clause 2, wherein the camera (104) is connected or integrated to a wearable device coupled to the user. Clause 6. The object tracking system (100) according to any one of clauses 1 to 5, wherein at least one of the current position of the movable object (606, 608), the current position of the user, or the future position of the movable object (606, 608) is determined using a Kalman filter. Clause 7. The object tracking system (100) according to any one of clauses 1 to 6, wherein the computer (112) is operably connected to a global positioning system (GPS) and is configured to determine the current position of the user with respect to the movable object (606, 608) within a GPS-denied environment. Clause 8. A positioning system, A camera (104) oriented according to a current pan, tilt, and / or zoom (PTZ) configuration, the camera (104) being configured to capture an image (500) while oriented according to the current PTZ configuration, A processor configured to process the image (500) using a computer (112) vision method, A controller configured to receive the current PTZ configuration from the camera (104), generate a new PTZ configuration, and communicate the new PTZ configuration to the camera (104), a detector configured to detect movable objects (606, 608) (608) in the image (500), wherein the movable objects (606, 608) (608) are detected using a bounding box (502, 504) and a detection algorithm selected from a library of object (606, 608) detection algorithms, the selection being based on the type of object (606, 608) being detected, and configured to deactivate the detection algorithm if the type of object (606, 608) no longer matches the type of object (606, 608) being detected; 1. A positioning system comprising: a state estimator configured to store a current estimated position of a user and calculate a new estimated position of the user based on the object type, the estimated positions of the movable objects (606, 608)(608), and a stored map, the stored map including the estimated positions of the movable objects (606, 608)(608) relative to the current estimated position of the user. Article 9. 9. The positioning system of clause 8, wherein the camera (104) is connected to or integrated into a wearable coupled to the user. Article 10. 10. The positioning system of claim 8 or 9, wherein the controller generates a new PTZ configuration based at least in part on at least one of the type of object (606)(608) being detected, the new estimated position of the user, or information shared by an external device. Article 11. 11. The positioning system according to any one of clauses 8 to 10, wherein the camera (104) is an omnidirectional camera (104). Article 12. 12. The positioning system of any one of clauses 8 to 11, further comprising a second camera (104) configured to capture an image (500). Article 13. 13. The positioning system of any one of clauses 8 to 12, further comprising an inertial measurement unit (IMU (110)). Article 14. The positioning system according to any one of clauses 8 to 13, wherein the state estimator uses at least partially measurements by an odometer to calculate a new estimated position of the user. Clause 15. The positioning system according to any one of clauses 8 to 14, wherein the state estimator uses a Kalman filter. Clause 16. The positioning system according to any one of clauses 8 to 15, further comprising an interface configured to receive user input, the input being used to assist in determining the type of the detected object (606, 608). Clause 17. A method for visually localizing an individual object, comprising: capturing an image (500) by a camera (104) using a first pan, tilt, and / or zoom (PTZ) configuration; processing the image (500) to determine an appropriate detection algorithm; selecting the appropriate detection algorithm from a library of detection algorithms; detecting the object (606, 608) in the image (500) using the detection algorithm that surrounds the object (606, 608) with a bounding box (502, 504); determining whether the object (606, 608) is moving or stationary; in response to a determination that the object (606, 608) is stationary, estimating the position of the object (606, 608) relative to one of a user or another object (606, 608) using inertial measurement values from a Kalman filter and an inertial measurement unit (IMU (110)); storing the position of the object (606, 608) in a map memory; determining a second PTZ configuration; and orienting the camera (104) according to the second PTZ configuration. Clause 18. 18. The method of claim 17, wherein computer (112) vision is used in at least one of the steps of processing the image (500), selecting the appropriate detection algorithm, detecting objects (606, 608) in the image (500), and determining whether the objects (606, 608) are moving or stationary. Article 19. 19. The method of claim 17 or 18, wherein the camera (104) comprises a plurality of cameras (104), the plurality of cameras (104) providing omnidirectional coverage. Article 20. 20. The method of any one of clauses 17 to 19, further comprising sharing at least one of the estimated location and / or map information with an external device.

Claims

1. An object tracking system (100), A camera (104) connected or integrated to a wearable device coupled to a user, configured to capture an image (500) of the surrounding environment according to a first camera (104) configuration, and configured to employ a second camera (104) configuration, An inertial measurement unit (IMU) (110), and A computer (112) operably connected to the camera (104), Processing the image (500) from the camera (104), Using an object detector to detect moving objects (606, 608) (608) in the image (500) using a detection algorithm selected from a library of detection algorithms, the object detector being configured to activate an appropriate detection algorithm and deactivate an inappropriate algorithm based on aspects of the surrounding environment automatically detected by the object detector, the object detector being configured to surround the moving object with a bounding box while detecting the moving object, and configured to extract features independent of the bounding box from the image to differentiate between the moving object and the surrounding environment, the object detector being configured to use a dashed bounding box to surround a pedestrian, Processing the bounding box and using a face recognition technique to identify a person or features of a person within the dashed bounding box, Measuring, by the IMU (110), at least one of the angular velocity or linear acceleration of the user, Estimating the current position of the moving object (606, 608) (608), Estimating the current position of the user relative to the current position of the moving object (606, 608) (608) based at least in part on the measurements of the IMU (110), Predicting the future position of the moving object (606, 608) (608) based at least in part on the measurements of the IMU (110), and An object tracking system (100) comprising a computer (112) configured to determine the second camera (104) configuration based at least in part on the future position of the moving object (606, 608) (608).

2. The object tracking system (100) according to claim 1, wherein the computer (112) is configured to generate or update a map reflecting the current position of the movable object (606, 608) (608) and the current position of the user with respect to the movable object (606, 608) (608).

3. The object tracking system (100) according to claim 1 or 2, wherein at least one of the current position of the movable object (606, 608) (608), the current position of the user, or the future position of the movable object (606, 608) (608) is determined using a Kalman filter.

4. The object tracking system (100) according to any one of claims 1 to 3, wherein the computer (112) is operably connected to a global positioning system (GPS) and is configured to determine the current position of the user with respect to the movable object (606, 608) (608) within a GPS-denied environment.

Images