Tracking camera and operation thereof

Abstract

A tracking camera is disclosed that includes a mirror assembly and a sensor communicatively coupled to a controller. The controller receives data from the sensor and adjusts the mirror assembly based on the data. The sensor includes a time-of-flight sensor for determining distance.

Description

Technical Field

[0001] This disclosure relates to optical imaging apparatus, and more specifically, to a camera and camera assembly for tracking objects. Embodiments relate to methods and devices for tracking objects. Background Technology

[0002] Tracking cameras can track the movement of objects in space. For example, ball tracking systems (such as ball tracking cameras) are used in tennis and golf. Tracking moving objects with visual sensors (such as cameras) presents many technical challenges (such as the appropriate responsiveness and optical resolution of the tracking system / method). It may be desirable to design tracking devices and methods to track fast-moving / accelerating objects (specifically, small and / or low optical contrast objects) and to improve the optical images obtained with the tracking camera. Summary of the Invention

[0003] In view of the technical challenges in designing tracking devices and methods capable of rapidly tracking moving or accelerating objects, this document discloses a tracking camera according to an embodiment of the technical solution. Further advantages are provided by exemplary embodiments of the technical solution.

[0004] This document discloses a tracking camera comprising a mirror assembly, a camera sensor, and a time-of-flight (TOF) sensor. Each sensor is communicatively coupled to a controller. The controller receives data from the camera sensor and the TOF sensor and adjusts the mirror assembly based on that data.

[0005] This document discloses a method for operating a tracking camera, including receiving data from sensors and a camera sensor, determining an adjustment signal based on the data, sending the adjustment signal to a mirror assembly, and adjusting the mirror assembly according to the adjustment signal. A non-transitory computer-readable medium computer program having program code is disclosed, which, when executed on a processor, causes the method to be performed. Attached Figure Description

[0006] The following will describe some embodiments of the device and / or method by way of example and with reference to the accompanying drawings, wherein...

[0007] Figure 1 A tracking camera according to an embodiment described herein is shown;

[0008] Figure 2 A tracking camera according to an embodiment described herein is shown;

[0009] Figure 3 An azimuth array according to an embodiment described herein is shown;

[0010] Figure 4 A block diagram of a system, method, or apparatus for tracking according to embodiments described herein is shown;

[0011] Figure 5 A hybrid sensor according to an embodiment described herein is shown; and

[0012] Figure 6 A tracking camera according to an embodiment described herein is shown. Detailed Implementation

[0013] Various embodiments will now be described more fully with reference to the accompanying drawings, some of which illustrate specific embodiments. The drawings are not necessarily drawn to scale.

[0014] Figure 1 A tracking camera 100 according to an embodiment described herein is shown, the tracking camera 100 including the tracking camera shown in other figures. The tracking camera 100 includes a mirror assembly 130, a TOF sensor 110, a camera sensor 119, and a controller 150. The controller 150 can receive data 115a, 115b from the camera sensor 119 and the TOF sensor 110. The mirror assembly 130 can be adjusted by the controller 150 based on the data 115a, 115b. Figure 1 Camera sensor data 115a and TOF sensor data 115b are shown. For example, the TOF sensor can be a pixel array of up to about 800 × 600 pixels.

[0015] The tracking camera 100 can track an object of interest 190, or object 190b, such as a fast-moving object like a car or a ball. Figure 1 As shown, the object of interest 190 is in a first position at time t1, and the object of interest can be in a second position 190b at a later time t2. The controller 150 can adjust the positioning of the reflector assembly 130 so that the laser 112 is pointed at the object of interest 190.

[0016] For example, camera sensor 119 and / or TOF sensor 110 can both be communicatively coupled to controller 150 for data communication. Controller 150 can receive camera sensor data 115a from camera sensor 119 and can adjust mirror assembly 130 based on camera sensor data 115a. Camera sensor 119 can be an event camera sensor, a frame camera sensor, or a combination thereof. Event camera sensors can provide camera sensor data 115a quickly and allow for faster response from tracking camera 100. Event cameras can operate asynchronously. Event camera sensors can have a temporal resolution of hundreds of microseconds or even smaller (such as 20 microseconds), which can exceed the temporal resolution of conventional frame cameras. Event cameras may be required for faster tracking.

[0017] The tracking camera 100 described herein allows recording images, measuring trajectories, angular velocities, object rotation, vibrations, and combinations thereof.

[0018] The tracking camera 100 includes a mirror assembly 130. The mirror assembly 130 of the tracking camera 100 can guide light from a laser 112 (such as a laser for TOF determination). The laser 112 can illuminate the object of interest and provide a TOF signal to the TOF sensor 110. The laser 112 can be a pulsed laser for providing the TOF signal. The laser 112 can be controlled by a controller 150. For example, the laser pulses can be adjusted for duration, duty cycle, repetition rate, etc.

[0019] like Figure 1 As shown, for example, laser 112 can be pointed at object 190 of interest via the position / direction of mirror assembly 130. Beam splitter 113 can allow the optical axis of laser 112 to overlap with that of TOF sensor 110. Laser 112 and TOF sensor 110 can share an optical axis (e.g., an optical axis coaxial with the incident beam 141 of the laser). Figure 1 As shown, the incident beam 141 from laser 112 can be directed toward object of interest 190, for example, by adjusting the mirror assembly 130. For example, the mirror assembly 130, which can be dynamically adjusted by controller 150, can point laser 112 toward the object of interest.

[0020] Beam splitter 113 can intersect with the optical axis of tracking camera 100 and guide the light from laser 112 out of tracking camera 100.

[0021] The mirror assembly 130 may include at least one actuable mirror for adjustment. Adjustment may be based on an adjustment signal from the controller 150. The mirror of the mirror assembly 130 may include at least one of a galvanometer mirror, a microelectromechanical microscope (MEMS) mirror, and a piezoelectric microscope.

[0022] The controller 150 of the tracking camera 100 may include at least one of circuitry, computing devices, a CPU, a programmable field array, etc. The controller 150 may be programmed to perform part or all of the methods described herein (such as determining the mirror adjustment of the mirror assembly 130 based on data 115 from the camera sensor 119 and / or the TOF sensor 110), specifically, to be able to track the object of interest 190.

[0023] Data 115 may include camera sensor data 115a and / or TOF data 115b. For example, camera sensor data 115a may include event camera data. Camera sensor 119 may send camera sensor data 115a to controller 150. TOF sensor 110 may send TOF data 115b to controller 150. Controller 150 may be communicatively coupled to mirror assembly 130.

[0024] Figure 1 Lens assemblies 120A and 120B, which are part of the tracking camera 100, are also shown and can be used to collect light for the TOF sensor 110 and / or the camera sensor 119. Figure 1 The lens assembly is divided into two parts: a first sub-assembly 120A for the camera sensor 119 and a second sub-assembly 120B for the TOF sensor 110.

[0025] Tracking fast-moving objects and measuring their distances with a vision sensor that has a high level of detail can be challenging. This paper describes the use of an actuated mirror and a Time-of-Flight (TOF) camera. The device may also include an event camera. The distance to the object of interest can be measured using the actuated mirror of the mirror assembly to train a laser on the object of interest and use the laser for time-of-flight (TOF) data. An image camera may also be used. High spatial and temporal resolution visual information can be provided. Alternatively / otherwise, accurate orientation information of the object in 3D space (e.g., including position) may be possible.

[0026] Event cameras can be used to capture visual information of fast-moving objects with high spatial and temporal resolution. Time-of-Flight (TOF) cameras can be used to measure depth information of the same object.

[0027] Light rays from the object of interest can be guided to the event camera via a mirror assembly that can be positioned so that the object is within the camera's field of view. By processing the event camera's output at a high rate (or continuously), the movement of the object of interest can be observed, and the mirror's positioning can be adjusted accordingly, for example, to keep the object of interest within the camera's field of view.

[0028] Using a mirror to control the gaze of an event camera allows the use of lenses with narrow angles of view and enables the production of high-resolution images of the object. A laser beam with pulse intensity can be illuminated at the object of interest via the mirror, where the laser beam is scattered / reflected. For example, a portion of the scattered / reflected light can be collected by a TOF sensor along the same optical path via the mirror. The time interval between the moment the laser illumination light is emitted and the moment it is detected can be determined (e.g., determining the time of flight).

[0029] The distance between objects in a scene and the sensor can be calculated based on time-of-flight using the constant speed of light. For example, a laser scanner or a LiDAR (Light Detection and Ranging) system can focus a laser beam onto a single point. Specifically, using a laser makes excellent signal-to-noise ratio (SNR) possible. The tracking camera and method described in this paper can be used outdoors, even in bright sunlight. The tracking camera and method described in this paper can detect objects at longer distances of 50 meters and beyond. For example, spatial information of an object of interest can be determined by guiding a laser beam toward it. The focused illumination of a LiDAR system can be combined with single-lens and / or multi-lens capture by a TOF (Time-of-Flight) camera. For example, compared to raster scanning systems, the tracking camera of this paper allows for faster determination of the orientation information of an object of interest.

[0030] Figure 2 A tracking camera according to an embodiment described herein is illustrated, including a tracking camera described with reference to other accompanying drawings. The tracking camera 200 includes a mirror assembly 230, a Time-of-Flight (TOF) sensor 210, a camera sensor 219, and a controller 250. The controller 250 can receive data 215a, 215b from the camera sensor 219 and the TOF sensor 210. The mirror assembly 230 can be adjusted by the controller 250 based on the data 215a, 215b. Figure 2 Camera sensor data 215a and TOF sensor data 215b are shown.

[0031] The controller 250 can determine the distance to the object of interest 290 based on the TOF data sensor data 215b.

[0032] The tracking camera 200 can track an object of interest 290 or object 290b, such as a fast-moving object like a car or a ball. Figure 1 As shown, the object of interest 290 is in a first position at time t1, and the object of interest can be in a second position 290b at a later time t2. The controller 250 can adjust the positioning of the reflector assembly 230 so that the laser 212 is pointed at the object of interest 290.

[0033] For example, camera sensor 219 and / or TOF sensor 210 can be communicatively connected to controller 250 for data communication.

[0034] The tracking camera 200 includes a mirror assembly 230. The mirror assembly 230 of the tracking camera 200 can guide light from a laser 212 (such as a laser used for TOF determination). For example, a controller 250 can also be communicatively coupled to the mirror assembly 230 to send control signals for adjusting the positioning of the mirrors in the mirror assembly.

[0035] The tracking camera 200 may have an optical axis 201 shared by the TOF sensor 210 and the camera sensor 119. The optical axis 201 may extend beyond the tracking camera 200. The mirror assembly 230 may be adjusted to bring the optical axis 201 toward the object of interest 290. The incident laser beam 241 may propagate along the optical axis 201. The scattered portion 242 of the laser (e.g., the portion scattered away from the object of interest 290) may be detected by the TOF sensor 210. The lens assembly 220 may collect the scattered portion 242 detected by the TOF sensor 210.

[0036] exist Figure 2 In this system, the collected light is collected by lens assembly 220. The collected light is split by a first beam splitter 213a, which allows camera sensor 219 to collect a first portion of the collected light and allows TOF sensor 210 to collect a second portion of the collected light. A second beam splitter 213b allows laser 212 to be guided along the optical axis of tracking camera 200. Beam splitter 213a may intersect with the optical axis 201 of tracking camera 200. Beam splitter 213a can separate light from the field of view (e.g., the field of view including at least a portion of the object of interest 290) into each of camera sensor 219 and TOF sensor 210.

[0037] The camera sensor 219 can be sensitive to the visible light spectrum. For example, a tracking camera is configured such that the camera sensor 219 is sensitive to visible light up to about 750, 800, 850 or 900 nm.

[0038] Filter 211 can selectively transmit a laser wavelength to TOF sensor 210. For example, the laser wavelength can be an infrared wavelength, and filter 211 can be an infrared long-pass or band-pass filter. For example, filter 211 is a band-pass filter suitable for the laser wavelength. For example, the laser wavelength can be between 850 nm and 1650 nm (such as 850 nm or 940 nm). Beam splitter 213a can be used at least partially as filter 211, for example, by splitting the light by wavelength so that laser backscattering 242 is directed towards TOF sensor 210, and camera sensor 219 receives other light (such as visible light used to determine images and / or events). For example, beam splitter 213a can be a dichroic beam splitter, such as transmitting wavelengths of 800 nm and higher to TOF sensor 210, and transmitting the reflected lower wavelengths to camera sensor 219. A combination of filters can be used, such as providing a narrow wavelength band including the laser wavelength to the TOF sensor 210; for example, the bandwidth delivered to the TOF sensor 210 can be less than 50 nm, or less than 20 nm, or less than 10 nm, or less than 5 nm, or less than 2 nm. Providing a narrow bandwidth to the TOF sensor can increase the signal-to-noise ratio (SNR). Filter 211 may include a bandpass filter and / or a filter that blocks ambient light. Suppressing background light can increase the SNR.

[0039] For example, the beam splitter 213b can be placed at the pupil position of the lens inside the lens assembly. It can also be placed in front of or behind the lens assembly.

[0040] The TOF sensor 210 can be a Si-based detector. An infrared sensor (e.g., an InGaAs sensor) can be used as the TOF sensor 210. For example, it is advantageous to have the TOF sensor 210 in infrared light while the camera sensor 219 is in visible light. The TOF sensor 210 can operate as a single-pixel detector. Alternatively / otherwise, the TOF sensor 210 can be a single-pixel detector (such as a single photodiode or photomultiplier tube). To increase the signal-to-noise ratio, using a single-channel detection scheme for the TOF signal may be advantageous. Specifically, a single-channel TOF sensor 210 responding to electromagnetic radiation with wavelengths up to 1700 nm or possibly 2600 nm is considered. For example, the wavelength of a laser could be 1550 nm. A wavelength of 1550 nm is less harmful to the human eye.

[0041] The tracking camera 200 may have an optical axis 201 extending outward from the tracking camera 200 into the field of view, for example, into the object of interest 290 within the field of view. Based on the movement of the mirrors in the mirror assembly 230, the optical axis 201 (specifically, as...) Figure 2The portion of the tracking camera 200 shown is movable. In this embodiment, the optical axis 201 of the tracking camera 200 moves when the reflector assembly 230 is adjusted.

[0042] For example, controller 250 determines or effectively determines the target direction and sends an adjustment signal to mirror assembly 230, causing optical axis 201 to move toward the target direction (such as target direction 201b) and / or a second position. Optical axis 201 may be pointed toward object of interest 290. At another time t2, when object of interest is in the second position 290b or is expected to be in the second position 290b, mirror assembly 230 may be adjusted such that optical axis 201 extends to the second position 290b. Controller may determine adjustment and / or adjustment signals to point the laser toward object of interest (e.g., by actuating the mirrors of mirror assembly 230).

[0043] Figure 2 The tracking camera 200 can be compared to Figure 1 The design is more compact. A smaller form factor for the tracking camera 200 might be desirable. Figure 2 The lens assembly 220 can be used as an imaging lens for both the camera sensor 219 and the TOF sensor 210. For example, the lens assembly 220 can be configured to provide a relatively small field of view such that the image of the object of interest 290 almost fills the camera sensor 219.

[0044] The beam from laser 212 can be guided toward object 290 by beam splitter 213b and / or mirror assembly 230. Light source 212 can be communicatively coupled to a controller. Illuminating object 290 can facilitate TOF acquisition. Alternatively / specifically, illuminating object 290 can allow for easier pickup of the object from the environment if the light source is intensity modulated in a manner detectable by sensors 210, 219. In one implementation, the illuminated object 290 can be detected by camera sensor 219 by allowing at least a portion of the laser scattering 242 to reach camera sensor 219. Allowing some laser scattering 242 to reach camera sensor 219 can allow object 290 to be more easily identified and / or more easily tracked.

[0045] For example, camera sensor 219 could be an event camera that records events caused by changing lighting conditions. A modulated light source 212, such as a pulsed laser, could generate changing lighting conditions to trigger events on camera sensor 219. Alternatively / additionally, the intensity of scattered laser 242 (e.g., reflected) leaving object 290 of interest could be significantly brighter than the environment. Intensity-modulated (e.g., flash) laser 212 could be used to illuminate object 290 of interest and brighten it against the background / environment to facilitate identification / detection / tracking of object 290 of interest.

[0046] In this implementation, at least one of the sensors 210 and 219 can be time-synchronized with the laser 212 (specifically, the TOF sensor 210). The known position and / or time synchronization can be used to determine orientation information (such as an orientation array representing the object of interest 290).

[0047] like Figure 2 As shown, beam splitter 213a (shown as intersecting the optical path 201, for example, between sensors 210, 219 and the object of interest 290) allows the tracking camera 200 to collect light from the object of interest 290 and / or a field of view including at least a portion of the object of interest 290. Beam splitter 213a can separate the light from the object of interest 290, such that light reaches both camera sensor 219 and TOF sensor 210 simultaneously.

[0048] Beam splitter 213b can help connect laser 212 to the optical axis 201 of the tracking camera.

[0049] For example, camera sensor 219 can be a conventional camera sensor / frame camera sensor (such as a wide-angle camera), which can facilitate object recognition by collecting light from a region (especially a wide or variable region). Using camera sensor 219 with TOF sensor 210 can facilitate the determination of 3D orientation information of object of interest 290.

[0050] Figure 3 An azimuth array according to an embodiment described herein is shown, which can be combined with other embodiments described herein, particularly those shown in any other figures. The azimuth array 300 can be used to simulate an object of interest.

[0051] The azimuth array 300 may be based on data 115. The azimuth array 300 may correspond to an object of interest 190 (such as its position (x, y, z)). The azimuth array 300 may include any one of x, x', x'', and t. Herein, x may be a 1-, 2-, or 3-dimensional position; and correspondingly, x' and x' ...

[0052] Motion estimation algorithms can determine and / or predict azimuth array 300 and / or azimuth information. Motion estimation algorithms can help determine / predict the position of an object of interest in a time instance (e.g., current, t1, or t2).

[0053] Figure 4 A block diagram of a system, method, or apparatus for tracking according to an embodiment described herein is shown, which may be combined with other embodiments described herein (specifically, those shown in any other figures). Figure 4 A tracking camera 400 may be shown. An object of interest 490 may be in the line of sight and / or field of view of the tracking camera, specifically its sensor 410 and / or its mirror assembly 430. The mirror assembly 430 may guide light to the sensor 410 (e.g., a camera sensor and / or a TOF sensor).

[0054] The controller 450 can accumulate / receive data 415 from the sensor 410. The controller 450 can determine an adjustment signal 460 based on the data 415 and can send the adjustment signal 460 to the mirror assembly 430 to adjust the mirror assembly 430. The mirror assembly 430 can be adjusted such that the object of interest 490 remains within the line of sight of the camera sensor 410, for example, such that the image of the object of interest 490 overlaps with and / or is within the sensor 410 (e.g., at the center of the sensor 410).

[0055] Alternatively, the optical axis of the tracking camera can be moved so that it is pointed at the object of interest 490.

[0056] The controller 450 can receive / accumulate data 415 over time intervals. The controller 450 can use the data 415 to determine the adjustment signal 460. The adjustment signal 460 can be based on an azimuth array (x, x', x”, t).

[0057] The orientation array (x, x', x”, t) optionally includes at least one element relating to time, which may correspond to a previous, current, or future time. For example, the orientation array has elements corresponding to the instantaneous coordinates (x, y) of the object of interest and the time of the corresponding event from the event camera sensor. In another embodiment, the orientation array has elements corresponding to a determined average position (e.g., x, y, and z) and the average time of multiple events from the event camera sensor and / or TOF sensor used to determine the corresponding average position. In another embodiment, such as by interpolating and / or averaging (e.g., weighted averaging) z at time t using TOF sensor data acquired at a set of times near time t, the orientation array has elements corresponding to the instantaneous coordinates (x, y) of the object of interest determined from the event camera sensor and the instantaneous coordinate z determined from the TOF sensor data.

[0058] It is conceivable that at a future time t (e.g., when the mirror assembly 430 moves to the position encoded by the adjustment signal 460), the controller 450 determines the orientation array (x, x', x”, t) for the projected position of the object of interest 490. Due to the finite response time of the mirrors of the mirror assembly (430), this determination can take into account the amount of time spent adjusting the mirror assembly 430. Alternatively / additionally, the orientation array may include at least one element having a time corresponding to a previous time.

[0059] For example, controller 450 may (i) accumulate data 415 over a time interval, then (ii) determine a common time within that time interval, and (iii) estimate the azimuth array elements (x, x', x”, t) at that common time based on the accumulated data 415. Controller 450 may determine adjustment signal 460 based on the azimuth array elements (x, x', x”, tc) at the common time tc or some other time t0 (specifically, after the common time tc). Time t0 may correspond to the time after which the mirror is expected to reach a new position following the transmission of adjustment signal 460. Time t0 may be the time when controller 450 transmits adjustment signal 460 and / or receives data 415, which may be useful if the mirror adjustment is fast enough compared to the movement of object 490 of interest.

[0060] Steps (i), (ii), and (iii) above can be performed, and then (iv) the mirror assembly can be adjusted; and steps i to iv can be repeated at subsequent time intervals. These steps can be repeated to track the object.

[0061] For example, an azimuth array can be used to determine the trajectory of an object of interest at 190° and 290°. Or / otherwise, an azimuth array can be used to determine a modulating signal at 460°.

[0062] The controller 450 can adjust the accumulated set of events from the event camera sensor so that the accumulated events have a common time t. c Public time t c It can be used for motion compensation (e.g., motion compensation occurring during the duration of data accumulation). It can be used at common time t. c Synthesize image frames, such as by accumulating data 415 (and / or mirror position, for example using adjustment signal 460 or the history of feedback signals from the mirror assembly).

[0063] In this implementation, when data 415 is sent to controller 450, asynchronous data from the event camera sensor (e.g., sensor 410 including the event camera sensor) (e.g., asynchronous data of data 415) can be processed asynchronously. For example, multiple estimates at different time points (e.g., azimuth arrays or azimuth estimates) can be probabilistically combined, for example, using a recursive filter (such as an extended Kalman filter, which may also include a model of object dynamics). Asynchronous or asynchronous methods or control algorithms can be used to actuate the reflector (e.g., determine the adjustment signal 460).

[0064] The actuation and / or adjustment signal 460 can be based on the difference between the estimated position of the object in image coordinates (e.g., the position element corresponding to the azimuth array elements (such as x, y in the reference frame of the camera sensor)) and the center of the sensor 410. The actuation and / or adjustment signal 460 can be determined such that the object (e.g., repeatedly) is centered in the field of view of the sensor 410. Alternatively / addressably, the excitation and / or adjustment signal 460 can be determined to point the optical axis 201 of the tracking camera 200 toward the object of interest 290.

[0065] The controller 450 can determine an azimuth array (x, x', x") in any coordinate system and / or any coordinate system, such as a coordinate system with a tracking camera at the origin. The azimuth array may include coordinates related to the positioning of the mirrors of the mirror assembly 430, such as a mirror positioning array.

[0066] Alternatively / in addition, the adjustment signal 460 can be based on the mirror positioning array. It can be an array of prior and / or desired / target positioning of the mirrors of the mirror assembly 430. Mirror positioning array It can be determined based on data (based on data 415) and / or other determinations such as azimuth arrays and / or feedback signals from the mirror assembly indicating the position of the mirror.

[0067] According to the embodiments described herein, the tracking camera can be controlled / operated by a controller. This operation may include receiving data from at least one camera sensor; determining an adjustment signal based on the data; sending the adjustment signal to a mirror assembly; and adjusting the mirror assembly according to the adjustment signal. Operation of the tracking camera may also include collecting light and forming an image of the object of interest on the camera sensor. The collected light may pass through a lens assembly, which includes a pupil position near or within the mirror assembly and / or the lens assembly.

[0068] The controller 450 can estimate an azimuth array (x, x', x”, t) based on data 415. The azimuth array (x, x', x”, t) can correspond to the object of interest 190 (e.g., its position). The adjustment signal can be determined based on this azimuth array.

[0069] The communication connection between the controller 450 and the mirror assembly 430 can be bidirectional. For example, the controller 450 sends an adjustment signal 460 to the mirror assembly 430. Alternatively / additionally, the mirror assembly 430 and / or the positioning feedback system 439 (which may be part of the mirror assembly 430) sends a mirror assembly status 435 to the controller 450. The mirror assembly status 435 may be a position / positioning signal (such as data regarding the position and / or positioning of the mirrors of the mirror assembly 430, such as a mirror positioning array). and / or mirror positioning array (Data).

[0070] In embodiments that can be combined with any other embodiments described herein, the tracking camera 400 includes a positioning feedback system 439 communicatively coupled to the controller 150 for transmitting mirror assembly status (such as mirror position / location and / or mirror positioning array). The positioning feedback system 439 can transmit mirror position / location for at least one mirror of the mirror assembly (possibly up to each mirror of the mirror assembly), and optionally include time information such as timestamps.

[0071] For example, the positioning feedback system 439 can utilize capacitive sensing. The positioning feedback system 439 can provide data to the controller 450, which can be used to determine the position / or orientation of the object of interest 490, and / or the direction of the optical path leading to the tracking camera 400, and / or the position / or orientation of the illumination point of the field of view 480.

[0072] For example, controller 450 can send an adjustment signal that moves the optical axis of tracking camera 400 (which may extend to the center of sensor 410) based on the azimuth array (x, x', x”, t) to a target direction (e.g., also extending to the center of sensor 410). Or / otherwise, adjustment signal 460 may be based on a target direction, which may be determined based on data 415.

[0073] For example, controller 450 may send adjustment signal 460, causing the optical axis 201 of tracking camera 200 (which may extend to the center of sensor 410) to move in the target direction (e.g., also extending to the center of sensor 410) based on the azimuth array (x, x', x”, t). Or / otherwise, adjustment signal 460 may be based on target direction 202, which may be determined based on data 415.

[0074] The target orientation can be effectively determined such that the objects of interest 190, 290 overlap with or are within the field of view of at least one sensor 410 (e.g., centered therein). Alternatively / additionally, such as to generate a pattern of saccade motion around the objects of interest 190, 290, the controller 450 can determine multiple target orientations. For example, the controller 450 can determine multiple adjustment signals 460, such as based on the accumulation of data 415 (e.g., over the entire time interval). The multiple adjustment signals 460 can be sequentially sent to the mirror assembly 430, such that the optical axis 201 is sequentially moved to each of the multiple target orientations.

[0075] Multiple target directions and / or adjustment signals 460 can make the objects of interest 190, 290 overlap with or fall within the field of view of sensor 410 with each movement of optical axis 201 toward each target direction.

[0076] In one implementation, controller 450 asynchronously determines multiple target directions and / or adjustment signals 460 from data 415. The optical axis can be effectively moved away from objects of interest 190, 290, such as intermittently moving away from objects of interest 109, 290 (e.g., moving around objects of interest 190, 290). In another implementation, optical axis 201 is effectively moved alternately away from and towards objects of interest 190, 290. This movement allows for consideration of environmental effects (such as anticipated collisions) on the movement of objects of interest 190, 290, as well as environmental data.

[0077] In one implementation, the adjustment signal 460 may be sent from the controller 450 to the reflector assembly 430 at regular time intervals. The adjustment signal 460 may be determined by the controller 450 based on data 415 accumulated between adjustments.

[0078] The tracked object 490 can be illuminated / lit by a light source such as a laser. The light source can be communicatively coupled to a controller 450, which can control the modulation of the light source. The light source can be guided by a mirror assembly 430 (such as the same mirror used to direct light in the field of view to the camera sensor 410). Illuminating the tracked object 490 can provide a TOF signal. The light source may increase the signal: noise and allow for more precise tracking. Illuminating the tracked object 490 can allow tracking in dark conditions. Or / otherwise, specifically, if the light source is intensity modulated to trigger the camera sensor 410, illuminating the tracked object 490 can allow the tracked object to be more easily picked up from the environment.

[0079] Figure 5A hybrid sensor 500 according to an embodiment described herein is illustrated. Any tracking camera described herein may include a hybrid sensor 500, which may include a camera sensor 530 and a TOF sensor 540. The hybrid sensor 500 may include a TOF pixel 510 for TOF data (e.g., for determining TOF and / or distance), and a camera sensor pixel 520 for imaging and / or event data. Event data may be determined by an event pixel, which is a camera sensor pixel configured for event camera operation. Pixels 510 and 520 may generate data for transmission to a controller. Alternatively / additionally, camera sensor pixel 520 may be configured to generate image data for transmission to a controller, such as image frames. Alternatively, camera sensor pixels may be configured to generate event data for transmission to a controller.

[0080] For example, the hybrid sensor 500 includes TOF pixels; and, in addition to TOF pixels, the hybrid sensor 500 may also include event pixels and / or image pixels.

[0081] Figure 5 A coordinate system x, y, z is shown, which can be associated with the orientation array and / or orientation information described herein, regardless of the sensor type (camera sensor, TOF camera, or hybrid). Coordinates x and y can correspond to the rows and columns of sensor 500 and / or the x and y directions of the field of view of the object of interest. For example, coordinate z can be perpendicular to the plane of sensor 500, which connects the sensor (such as its center) to the center of the object of interest and / or the field of view. The distance to the object can be measured along z. When adjusting the mirror assembly, coordinates x, y, and / or z can be moved, for example, relative to another reference frame.

[0082] In the embodiment, using Figure 1 As shown, a first lens assembly 120A and an image sensor (e.g., camera sensor 119) capture images of the scene. A controller 150, which may include a processing unit, can detect an object of interest 190 within the scene at a certain time instance t1. The controller 150 can provide its control unit with information about the angular position of the object of interest 190, such as as seen from the lens. The controller 150 can adjust the mirror assembly 130 to track the position of the object of interest 190.

[0083] Laser 112 can emit a laser beam 141, which can be coupled to the optical axis of second lens 120B via beam splitter 113. Controller 150 can send control signals to mirror assembly 130, which can be positioned (e.g., by rotation) such that the incident laser beam 141 is guided toward and illuminates object 190.

[0084] The lateral displacement between the lens assembly 120A and the mirror assembly 130 is small compared to the distance between the mirror assembly 130 and the object of interest 190, to reduce errors in angular position determination. A portion 142 of the illumination light (e.g., incident light 141 from laser 112) can be scattered / reflected from the object of interest 190 and can propagate backward in the opposite direction of the laser beam, enter the lens 120B, and ultimately be captured by the TOF sensor 110. Another portion 143 of the illumination light can be collected by the lens 120A and detected by the camera sensor 119.

[0085] The controller 150 can process the TOF signal and calculate the depth information of the object of interest 190. At a later time instance t2, when the object of interest 190 has moved to another position 190b, the controller 150 detects the new position again. The controller 150 can then set the actuable mirror assembly 130 to the new position, so that the incident laser beam 141 illuminates the moved object of interest 190 again at a later time instance t2.

[0086] When the size and distance of the object of interest are predictable within a certain range, the field of view (FOV) of the lens assembly 120B can be adapted so that the image of the object of interest almost fills the entire pixel area of ​​the TOF sensor. This can result in a magnified depth map of the object of interest 190. The FOV of the lens assembly 120B can be set based on the desired depth and estimated size of the object of interest. For example, the FOV of a lens assembly used for tracking a golf ball at 50 meters can be configured to be narrower than that of a lens assembly configured to track a car at 3 to 10 meters.

[0087] Figure 6 A tracking camera according to an embodiment described herein is illustrated, including a tracking camera described with reference to other figures. Tracking camera 600 includes a mirror assembly 630, a sensor 610, and a controller 650. Controller 650 can receive data 615 from sensor 610. The sensor may include a TOF sensor and a camera sensor. For example, sensor 610 is a hybrid sensor, such as... Figure 5 The controller 650 can adjust the reflector assembly 630 based on data 615.

[0088] Laser 612 can be directed toward object of interest 690, for example, by actuation of mirror assembly 630. Beam splitter 613 can allow laser 612 to be guided along the optical axis 601 of tracking camera 600. For example, the optical axis 601 of tracking camera 600 can be moved to a target direction 601b, for example, by moving mirror assembly 630. For example, at a later time, object of interest may be at a second position 690b. Mirror assembly 630 can be actuated such that laser 612 is trained on object of interest 690. Focusing laser energy onto object of interest 690 is advantageous, such as providing high SNR for TOF sensors and / or camera sensors. For example, target direction 601b can be determined by controller 650 based on data 615 and / or azimuth array 300.

[0089] Figure 6 An incident laser beam 641 and a scattered laser beam 642 are shown. The scattered laser beam 642 can be collected by a lens assembly 620, and the collected light is transmitted to a sensor 610.

[0090] Tracking cameras 100 and 600, including event camera sensors, can provide one or more advantages over conventional / frame cameras, such as reducing the amount of data processed by controllers 150 and 600. The tracked object 190 may only cover a small portion of the image frame from a conventional / frame camera. This may cause the conventional / frame camera to image the tracked object 190 at a reduced resolution, as most of the image frame may be captured by the background. When the object of interest 190 occupies a relatively small portion of the image frame, most of the image frame data from the conventional camera can be filtered / ignored by the tracking algorithm. Alternatively / otherwise, a frame camera, by sending the entire image frame multiple times per second, can provide images that can also be used for analysis. It is conceivable that the tracking algorithm of the tracking cameras 100 and 600 can ignore some data, particularly the background portion of the image from the frame camera.

[0091] The use of the event camera sensor 119 can improve tracking, for example, by providing more relevant data to the controller 150 to determine the position of the tracked object 190. Efficient use of computing power can also help to quickly and accurately determine the optical, mechanical, and / or digital adjustments used to track the tracked object 190, and can reduce power requirements.

[0092] Event camera data may include at least one event data set, which may include pixel x-position, pixel y-position, polarity, and timestamp. Data 115 can be generated asynchronously. Each individual asynchronous event (such as an event from an event camera) may generate a set of data including at least one of the following: pixel x-position, pixel y-position, polarity, timestamp, and any combination thereof. The term "data" may refer to multiple datasets, such as each set of data generated by a corresponding individual (asynchronous) event. Each individual asynchronous event may be timestamped. For example, event camera data corresponding to a single event includes a polarity indicating whether the event is in response to an increase or decrease in intensity; the data also includes the event's pixel x-position, y-position, and timestamp.

[0093] In embodiments such as those described herein, mirror assemblies 130, 630 have at least one actuable mirror. The actuable mirror may be a galvanometer mirror. It is envisioned that the response time of the actuable mirror may be less than 500 μs, 300-mirror, 200-mirror, or 100-mirror, for movements up to 0.5°, 0.2°, or 0.1°. A fast-response mirror can allow for faster tracking. Specifically, a movable mirror for tracking motion can allow for a faster response time compared to tracking the motion of the camera body 111, for example, using a pan / tilt mechanism. Other types of mirrors, such as microelectromechanical (MEMS) mirrors (MEMS mirrors) and piezoelectric reflectors (PZ mirrors), can be considered for actuable mirrors. Furthermore, low-mass mirrors can also improve susceptibility by having lower inertia and therefore a faster response.

[0094] For example, a fast-moving reflector with high bandwidth, short settling time, and / or low latency can work in conjunction with sensors 110, 119, for example, with excellent temporal resolution and / or low latency, to provide rapid tracking. Motion blur can be reduced, for example. Specifically, as described herein, each of the cooperating reflector assembly 130 and sensors 110, 119 can allow accurate / rapid orientation information of the tracked object 190.

[0095] For example, mirror assemblies 130, 630 may include 2D scanning mirrors that rotate about two axes (e.g., a horizontal axis and a vertical axis). Alternatively, mirror assemblies may include two consecutive 1D scanning mirrors pointing on the same optical axis, with the first 1D scanning mirror rotating about a horizontal axis and the second 1D scanning mirror rotating about a vertical axis.

[0096] Example:

[0097] Note that this disclosure can also be configured as follows.

[0098] (1) A tracking camera, comprising:

[0099] Mirror assembly;

[0100] Camera sensor;

[0101] TOF sensor;

[0102] and controller; among which

[0103] The controller is configured as follows:

[0104] Data is received from the camera sensor and the TOF sensor, and

[0105] Adjust the reflector assembly based on data.

[0106] (2) Based on the tracking camera in (1), it also includes:

[0107] Lens assembly used to collect light for TOF sensors and camera sensors.

[0108] (3) Based on the tracking camera in (1) or (2), wherein,

[0109] The camera sensor includes at least one of a frame camera and an event camera, and the camera sensor data includes at least one of image data and event data.

[0110] (4) A tracking camera based on any one of (1) to (3), wherein,

[0111] The mirror assembly includes an actuable mirror configured to be adjusted based on an adjustment signal from a controller.

[0112] (5) A tracking camera based on any one of (4), wherein,

[0113] The actuated mirror of the reflector assembly is a galvanometer mirror, a microelectromechanical microscope (MEMS) or a piezoelectric mirror.

[0114] (6) Based on the tracking cameras in (1) to (5), it also includes:

[0115] The positioning feedback system is communicatively connected to the controller to transmit the status of the reflector assembly.

[0116] (7) A tracking camera based on any one of (4) to (6), wherein,

[0117] The controller is configured as follows:

[0118] The adjustment signal is determined based on the data, and

[0119] An adjustment signal is sent to the mirror assembly for adjustment of the mirror assembly.

[0120] (8) A tracking camera based on any one of (4) to (7), wherein,

[0121] The controller is configured as follows:

[0122] Receive data at time intervals, and

[0123] The azimuth array corresponding to the tracked object is estimated based on the data; whereby...

[0124] The modulation signal is based on the azimuth array.

[0125] (9) A tracking camera based on any one of (4) to (8), wherein,

[0126] The controller is configured as follows:

[0127] An adjustment signal is sent, causing the optical axis of the tracking camera, which extends beyond the tracking camera, to move toward the target direction based on the azimuth array.

[0128] (10) The tracking camera according to any one of (1) to (9) further includes:

[0129] The hybrid sensor includes a camera sensor and a TOF sensor, and the hybrid sensor includes TOF pixels configured to determine the time of flight.

[0130] (11) The tracking camera according to any one of (1) to (10) further includes:

[0131] The beam splitter is configured to separate light from the field of view to each of the camera sensor and the TOF sensor.

[0132] (12) A tracking camera based on any one of (1) to (11), wherein,

[0133] The lens assembly includes a first sub-assembly for collecting light for a camera sensor and a second sub-assembly for collecting light for a TOF sensor.

[0134] (13) The tracking camera according to any one of (1) to (12) further includes:

[0135] A laser is used to illuminate the object of interest and provide a TOF signal to the TOF sensor.

[0136] (14) The tracking camera according to any one of (1) to (13) further includes:

[0137] A beam splitter intersects the optical axis of the tracking camera and is configured to direct light from the laser out of the tracking camera.

[0138] (15) A method of operating a tracking camera, comprising:

[0139] Receive data from the TOF sensor and camera sensor;

[0140] The adjustment signal is determined based on the data;

[0141] Send the adjustment signal to the mirror assembly;

[0142] Adjust the reflector assembly according to the adjustment signal.

[0143] (16) The method of operating the tracking camera according to (15) further includes:

[0144] Light is collected from the pupil position within 2cm of the reflector assembly.

[0145] An image of the tracked object is formed on the camera sensor.

[0146] (17) The method of operating the tracking camera according to (15) or (16) further includes:

[0147] The azimuth array is estimated based on data, and the azimuth array corresponds to the tracked object.

[0148] The adjustment signal is determined based on the azimuth array.

[0149] (18) The method of operating the tracking camera according to any one of (15) to (17) further includes:

[0150] Data on cumulative time intervals;

[0151] Determine the common times within the time interval; where,

[0152] The azimuth array comprises elements based on accumulated data at common times.

[0153] (19) A non-transitory computer-readable medium computer program having program code, which, when executed on a processor, causes to perform the method according to any one of (15) to (18).

[0154] With one or more previously detailed embodiments and appendices Figure 1 The aspects and features mentioned and described may also be combined with one or more other embodiments in order to replace similar features in other embodiments or to additionally introduce the feature into other embodiments.

[0155] In this document, flowcharts, state transition diagrams, pseudocode, etc., may represent various processes, operations, or steps, which may be substantially represented in transient and / or non-transient machine-readable media (e.g., floppy disks, DVDs, Blu-rays, CDs, ROMs, PROMs and EPROMs, EEPROMs, or FLASH memories) and may be executed by a processor or programmable hardware, whether or not such processor or programmable hardware is explicitly shown.

[0156] The methods disclosed in the specification or claims can be implemented by means of a device having components for performing each of the various actions of these methods.

[0157] A non-transitory computer-readable medium computer program may have program code that, when executed on a processor, causes any of the methods described herein to be performed.

[0158] It should be understood that, unless otherwise explicitly or implicitly described, the disclosure of multiple actions, processes, operations, steps, or functions in the specification or claims should not be construed as limiting them in a particular order. In some embodiments, the described actions, functions, processes, operations, or steps may include or be decomposed into multiple dependent actions, functions, processes, operations, and / or steps.

[0159] The figure labels are provided to aid understanding, not to limit it.

[0160] It should be understood that when a feature is referred to as "connected" or "linked" to another element, the feature can be directly connected or linked via one or more intermediate elements.

[0161] In this document, the term "illumination" is used interchangeably with "lighting". In this document, "actuable" and "adjustable" are used interchangeably; for example, an adjustable mirror can be an actuable reflector. In this document, "controller" can include, for example, at least one processor, such as a computer processor, a computing device (such as a programmable electronic device), etc., and may include a network of computing devices. "Controller" and "control unit" are used interchangeably. In this document, "azimuth array" can be a unit element array, a dual element array, or larger; for example, an azimuth array can be the x and y coordinates corresponding to the position of an object of interest in an image plane formed by the lens assembly. In this document, "conventional camera" is used interchangeably with "frame camera". A conventional camera and / or a frame camera can be a high-speed camera. In this document, "object of interest" and "target object" are used interchangeably.

[0162] In this document, the coordinates x, y, z used in azimuth information and / or azimuth arrays can be in the reference frame of the tracking camera, such as in at least one of its sensors.

[0163] The x and y coordinates can be associated with the x and y dimensions of the camera and / or camera sensor (e.g., camera sensor). The z direction can be associated with the distance from the sensor, or with the front of the camera (e.g., a tracking camera), and can be associated with a portion of the optical axis extending from the tracking camera. In another coordinate system, such as an "absolute" coordinate system, such as the user / observer's or a second camera's coordinate system, changes in coordinate direction are possible. For example, the z direction can change in response to a change in the positioning of the tracking camera's mirrors.

[0164] In this context, an aperture and / or pupil can block some light rays while allowing others to pass through. An aperture and / or pupil can restrict light from passing along / near the optical axis. For example, an aperture and / or pupil on the optical axis of a tracking camera, such as an aperture and / or pupil within a lens assembly and / or mirror assembly, can allow light near the optical axis to pass through and block light rays further away from the optical axis (e.g., off-axis). In this context, "pupil position" can refer to the location where an aperture, such as a lens aperture, is placed. Pupil position can refer to the location where light converges along the optical axis. Pupil position can refer to the location of the intermediate image within a tracking camera. In this context, pupil position and pupil point are used interchangeably.

[0165] In this document, the suffix "singular" or "plural" indicates an optional plurality. Thus, for example, "reflector" means "one or more reflectors," "at least one reflector," or "a reflector and an optional plurality of reflectors." In this document, the forward slash " / " indicates "and / or," conveying "and" or "or." Thus, "A / B" means "A and / or B"; equivalently, "A / B" means "at least one of A and B, i.e., possibly only A, possibly only B, or possibly both."

[0166] The specification and accompanying drawings are for illustrative purposes. This description is intended to help the reader understand the subject matter defined in the appended claims. For convenience, and not to be construed as limiting, the reference numerals in the drawings are as follows:

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

Claims

1. A tracking camera, comprising: Mirror assembly; Camera sensor; TOF sensor; and the controller; among which, The controller is configured to: Event data is received from the camera sensor, and time-of-flight data is received from the TOF sensor; Based on the received time-of-flight data, an adjustment signal is asynchronously determined from the received event data to adjust the mirror assembly. Determining the adjustment signal from the received event data includes: asynchronously processing the event data and generating state estimation information of the tracked object using a recursive filter, and... The process of determining the adjustment signal from the received event data further includes: receiving and accumulating event data over time intervals to determine a common time within the time interval, and synthesizing an image frame at the determined common time based on the accumulated event data to perform motion compensation on the accumulated event data.

2. The tracking camera according to claim 1 further includes: A lens assembly for collecting light for the TOF sensor and the camera sensor.

3. The tracking camera according to claim 1, wherein, The mirror assembly includes an actuable mirror configured to be adjusted based on an adjustment signal from the controller.

4. The tracking camera according to claim 3, wherein, The actuable mirror of the mirror assembly is a galvanometer mirror, a microelectromechanical microscope (MEMS) or a piezoelectric mirror.

5. The tracking camera according to claim 1, further comprising: A positioning feedback system is communicatively connected to the controller to transmit the status of the reflector assembly.

6. The tracking camera according to claim 1, wherein, The camera sensor and the TOF sensor are integrated into a hybrid sensor, which includes a TOF pixel configured to determine the time of flight and an event pixel configured to output the event data.

7. The tracking camera according to claim 1, further comprising: A beam splitter is configured to separate light from the field of view to each of the camera sensor and the TOF sensor.

8. The tracking camera according to claim 2, wherein, The lens assembly includes a first sub-assembly for collecting light for the camera sensor and a second sub-assembly for collecting light for the TOF sensor.

9. The tracking camera according to claim 1, further comprising: A laser is used to illuminate the object of interest and provide a TOF signal to the TOF sensor.

10. The tracking camera according to claim 1, further comprising: A beam splitter intersects the optical axis of the tracking camera and is configured to direct light from the laser out of the tracking camera.

11. A method for operating a tracking camera, wherein, The method includes the steps performed by the controller of the tracking camera according to claim 1.

12. The method of operating a tracking camera according to claim 11, further comprising: An image of the tracked object is formed on the camera sensor.

13. The method of operating a tracking camera according to claim 11, further comprising: An azimuth array is estimated based on the event data and the time-of-flight data, the azimuth array corresponding to the tracked object, wherein... The adjustment signal is determined based on the azimuth array.

14. A non-transitory computer-readable medium storing a computer program having program code that, when executed on a processor, causes the method of operating a tracking camera according to claim 11 to be performed.

Images