Characterization of resident space objects using event-based sensors

Abstract

Disclosed herein is a method for remote monitoring, the method including determining, from event signals obtained from an event-based vision sensor, a rate at which those event signals are generated in response to changes associated with a particular object, and optionally determining a brightness of the particular object from exposure measurements associated with the event signals.

Description

[Technical Field]

[0001] Disclosed herein are methods and systems for monitoring remote areas using event-based vision sensors, and products produced using such methods or systems. The methods, systems, and products have particular application in facilitating the characterization of Resident Space Objects (RSOs). [Background technology]

[0002] RSOs are monitored for space domain awareness (SDA), a task traditionally performed using frame-based vision sensors.

[0003] A frame-based sensor is an integral sensor in which the generated photocurrent of each pixel is integrated over the exposure time of a frame. The exposure time is a controllable parameter of the camera and is typically proportional to the light level of the image. A faint object can be made visible by increasing the exposure time as long as the object remains stationary within the field of view. Therefore, to compensate for the Earth's rotation, conventional optical SDA techniques typically use a robotic system to move the telescope mount to keep the star stationary within the field of view in order to precisely adjust the sensor's field of view. Alternatively, to observe a faint object such as a satellite or asteroid, the object can be kept stationary within the sensor's field of view as long as a sufficiently accurate estimate of the object's trajectory is available.

[0004] Frame-based vision sensors are used to characterize RSOs by measuring their brightness, possibly in different color wavelength bands, and monitoring how the brightness changes over various time scales. The characteristics of these measurements, such as brightness magnitude, brightness oscillation frequency, or color ratio, can enable the calculation of RSO rotation rate, help distinguish between different RSOs, or determine whether an RSO is stable or tumbling. Spectroscopic measurements can also indicate the presence of certain material types in some cases. These types of data are important for a variety of reasons, including maintaining track of known RSOs and thereby helping to prevent on-orbit collisions, determining satellite health status, more accurately propagating RSO orbits to assess collision potential, assisting in the proactive removal of non-functioning satellites, and assessing satellite capabilities. However, obtaining such information using frame-based sensors remains a significant challenge due to the limited exposure time required, poor temporal resolution, and pixel saturation in bright reflections.

[0005] Any discussion of documents, acts, materials, devices, articles or the like which has been included in this specification should not be construed as an admission that any or all of such matters form part of the prior art base or were common general knowledge in the art relevant to this disclosure as existing before the priority date of each of the appended claims.

[0006] Throughout this specification, the words "comprise" and "include", and variations such as "comprises", "comprising", "includes" and "including", will be understood to imply the inclusion of a stated element, integer, or step, or group of elements, integers, or steps, but not the exclusion of any other element, integer, or step, or group of elements, integers, or steps. Summary of the Invention

[0007] A first aspect of the disclosure herein relates to a method for remote monitoring, the method comprising: Determining, from event signals obtained from the event-based vision sensor, a rate at which those event signals were generated in response to changes associated with a particular object, and optionally, determining a brightness of the particular object from exposure measurements associated with the event signals.

[0008] The particular object may be located in an area remote from the sensor. The sensor may be configured to generate an event signal, and optionally an associated brightness measurement, in response to a change the sensor detects in the remote area. The sensor may be located on Earth or on the RSO. In some embodiments, the sensor is located on Earth and the remote area is in space. In other embodiments, the sensor is located in space, such as on the RSO, and the remote area is located on Earth. The particular object may be, for example, an RSO, a drone, a missile, an animal, or a structure.

[0009] Variations in rate, and optionally brightness, may be logged over time. Variations in rate over time may be logged relative to GPS time. The method may include inferring or determining information about a particular object based on the variations in rate and / or brightness over time. Inferring or determining information based on the variations in rate and / or brightness over time may include identifying repeating patterns in rate and / or brightness over time and determining a rate of rotation of the particular object about one or more axes based on the frequency at which those patterns repeat. The method may include calculating the duration of spikes in rate and / or brightness to determine the angular width of the specular reflection that caused the spike.

[0010] The method may include determining a rate of event signals generated in response to changes associated with a particular object from event signals acquired contemporaneously from two event-based vision sensors, each having a different color filter, and optionally determining a brightness of the particular object from exposure measurements associated with the event signals. The method may include determining a material property of the particular object based on a difference in the event rate data and / or brightness data from each of the sensors.

[0011] The method includes: (i) an event signal is generated in response to the change occurring within a first predetermined distance from a center of the particular object, the center of the particular object being at a known or predicted location; and / or (ii) the event signal is generated in response to the change occurring (a) along the trajectory of a particular object and / or (b) in close proximity to a previous such event signal associated with the particular object; and / or (iii) Correlating may include correlating based on whether the event signal occurs within a sub-frame of the sensor's pixel array that is visually determined to contain a particular object. The associated change occurring proximate to the most recent previous event signal may include the associated change occurring within a first predetermined time interval and / or the associated change occurring within a second predetermined distance from the center of the specific object, which may be the same as or different from the first predetermined distance. The first predetermined distance may be measured in pixels. The first predetermined distance may be based on the number of pixels of the event-based vision sensor occupied by the specific object. The method may include updating the center and / or velocity of the specific object based on subsequent event signals associated with the specific object. The method may include periodically re-determining the center of the specific object based on subsequent event signals associated with the specific object and calculating the velocity of the specific object across the pixel array from changes in the center's location over time. The center of the specific object may be re-determined approximately every 10 milliseconds.

[0012] The method may include flagging a plurality of event signals generated in response to associated changes occurring (i) along a linear path and (ii) proximate to one another, and associating the plurality of event signals with a particular object. The associated changes occurring proximate to one another may include the associated changes occurring within a first predetermined time interval of one another. The first predetermined time interval may be on the order of one millisecond. The method may further include determining a center and / or velocity of the particular object from one or more of the plurality of event signals. The method may further include associating subsequent event signals with a particular object. (i) the subsequent event signal is generated in response to the change occurring within a predetermined distance from the center of the particular object; and / or (ii) The subsequent event signal may include associating the subsequent event signal with a particular object based on the change being generated in response to the change occurring (a) along a linear path and / or (b) in proximity to an immediately preceding one of the plurality of event signals. The associated change occurring proximate to a nearest one of the plurality of event signals may include the associated change occurring within a second predetermined time interval, which may be the same as or different from the first predetermined time interval, or the associated change occurring within a first predetermined distance from a center of the particular object. The first predetermined distance may be measured in pixels. The first predetermined distance may be based on the number of pixels of the event-based vision sensor occupied by the particular object. The method may include updating the center and / or velocity of the particular object based on subsequent event signals associated with the particular object. The method may include periodically re-determining the center of the particular object based on subsequent event signals associated with the particular object and calculating the velocity of the particular object across the pixel array from changes in the center's location over time. The center of the particular object may be re-determined approximately every 10 milliseconds.

[0013] Determining the rate at which those event signals are generated in response to changes associated with a particular object includes: determining, for each pixel associated with those event signals, a time interval between each of those event signals associated with that pixel; determining a moving average of the time intervals of a predetermined number of event signals; and determining that the rate at each of those event signals is the inverse of the moving average at that time. For example, if a particular object occupies two pixels (A and B), where pixel A generates a first event signal at a first time and generates a second event signal 100 milliseconds later, and pixel B generates a first event signal at a first time and generates a second event signal 100 milliseconds later, and the event signals of pixel A and pixel B are offset in time from each other by 1 millisecond, then two time intervals of 100 milliseconds each are determined (not two time intervals of 1 millisecond, nor two time intervals of 99 milliseconds). The predetermined number of events may be about one-tenth the number of those event signals generated per second.

[0014] Determining the brightness of the particular object from the exposure measurements may include summing all associated exposure measurements within a predetermined distance from the center of the particular object, which may be the same as the first predetermined distance.

[0015] In an alternative embodiment, determining the rate of the event signals generated in response to changes associated with the particular object may include determining the number of the event signals generated within each successive time interval of a predetermined period of time, which may be approximately one-tenth of a second.

[0016] The orientation of the sensor may be adjusted based on the trajectory of the particular object, and the position information associated with the event signal may be adjusted to account for changes in the orientation of the sensor.

[0017] The sensor may be attached to an optical telescope. [Brief explanation of the drawings]

[0018] A method embodying the principles disclosed herein will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0019] [Figure 1] 1 is a plot of event rate data for the low Earth orbit (LEO) satellite Globalstar06, the event rate data being obtained using an embodiment of the method disclosed herein.

[0020] [Figure 2] 1 is a plot of event rate data for an RSO in the form of a rocket body (image inset), depicting event rate data obtained using embodiments of the methods disclosed herein.

[0021] [Figure 3-4]1 is a plot of color event rate data and associated event rate ratios for the LEO satellite Globalstar 22, depicting event rate data obtained using embodiments of the methods disclosed herein.

[0022] [Figure 5] An image of the LEO satellite Globalstar22 is shown. DETAILED DESCRIPTION OF THE INVENTION

[0023] Applicant has developed the use of neuromorphic event-based vision sensors to monitor RSOs for Space Domain Awareness (SDA). Event-based vision sensors have several advantages over frame-based imagers traditionally used for this task, including higher temporal resolution, better imaging while the field of view is moving, significantly lower data rates for sparse scenes, and much higher intra-frame dynamic range.

[0024] Each pixel in an event-based vision sensor is effectively a change detector, signaling a change event if the pixel's generated photocurrent changes by more than a set percentage from the level that last triggered a change event. An on event signals an increase in photocurrent, and an off event signals a decrease in photocurrent. These two types of events have separate parameters that control the percentage change required to trigger an event. These are set by the control software as global parameters for the sensor. In addition, some models of event-based vision sensors perform pixel exposure measurements, each of which is triggered by any change detection event.

[0025] The output of an event-based vision sensor can be represented as a multi-column stream of numbers representing pixel location, event polarity (increase or decrease in brightness), time, and exposure measurement information (for models that perform exposure measurement).

[0026] The systems and methods disclosed herein facilitate the generation and analysis of high-temporal resolution RSO brightness data, known as "light curves," as well as high-temporal resolution event rate plots, both of which leverage the aforementioned advantages of event-based vision sensors to better perform characterization tasks such as those mentioned in paragraph 0004 above.

[0027] The plots shown in Figures 1-4 were obtained from event signals from an event-based vision sensor, specifically a fourth-generation event-based camera manufactured by Prophesee, Inc. In each case, one or more event-based sensors are attached to an optical telescope pointed at a particular object in the form of a respective RSO (i.e., LEO satellite Glablestar06, the rocket body, or LEO satellite Glablestar22), and the telescope, and thus the event-based sensor, is configured to track the RSO.

[0028] The high temporal resolution of event-based sensors results in more information being derived from event rate plots, such as those shown in the accompanying figures, than would be possible from light curves generated using frame-based sensors. However, because every satellite is different, there is no single optimal procedure for analyzing all satellite event rate curves. For example, for geosynchronous or rotating RSOs in LEO, the timing of event rate spikes can be used to calculate the angular width of bright specular reflections. To accomplish this, the satellite rotation period must be calculated by identifying event rate features corresponding to the same satellite surface on successive rotations and measuring the time between them. The duration of the event rate spike can then be measured and divided by the satellite rotation period. Multiplying the result by 360 gives the angular width, in degrees, of the specular reflection that caused the event rate spike. Not every satellite passing overhead will exhibit an event rate spike suitable for this analysis.

[0029] When two cameras are used simultaneously, each fitted with a different color filter, simultaneous multicolor event rate curves can be generated for a given object. By dividing one color event rate by the other, one can see how the relative brightness of one color changes compared to the other. Typically, the apparent color of a material changes during specular reflection compared to when diffuse reflection is observed. The color change varies from material to material, providing a means of distinguishing between some materials.

[0030] Figure 1 shows a 12-second plot of event rate data from the tumbling LEO satellite Globalstar06. There are three event rate spikes due to bright reflections from different surfaces. The central spike is significantly narrower than the other two spikes and is characteristic of a more specular reflective surface. The central red / green (r / v) event rate ratio spike is also narrower and of larger magnitude (more red) than the other two. The characteristics of the central spike are consistent with a smooth, flat aluminum surface or solar panel, while the characteristics of the other two spikes are consistent with an uneven, specular surface such as a multi-layer insulator.

[0031] The aforementioned specular reflection width and color data can be compared to a library of spacecraft materials to classify the type of material responsible for the bright reflection. For unknown objects, including debris, surface material information can aid in understanding the object's origin or its drag coefficient, which can potentially improve orbital propagation accuracy. For tumbling satellites of known configuration, surface material information can aid in determining the satellite's rotation axis by constraining the possible range of orientations during each specular reflection. The high temporal resolution of event rate data also provides much finer detail than is typically available in frame-based sensor light curves. The additional detail reduces ambiguity regarding RSO rotation rates when both sides of the RSO have very similar brightness characteristics.

[0032] In some cases, a satellite's rotation axis can be inferred simply from the presence or absence of an event rate spike. For example, in the event rate plot shown in Figure 2 for a tumbling rocket body, there is a sharp event rate spike every half rotation because the side of the rocket body reflects toward the observer. However, for approximately 25 to 60 seconds after the start of the recording, there is no event rate spike because the rotation axis points toward the sensor, resulting in a relatively unchanged reflection toward the sensor. One example of the importance of understanding satellite rotation state is when a second satellite is connected to a first satellite for maintenance or is used to deorbit the first satellite. Misidentifying the satellite's rotation state could result in the destruction of both satellites, or at least mission failure.

[0033] Figures 3 and 4 depict color event rate plots and event rate ratio plots for the tumbling LEO Globalstar22. Figures 3 and 4 provide further examples of how these data can be used to infer satellite orientation and rotation information. First, the periodic nature of the plots reveals that the satellite's apparent rotation period is 4.97 seconds. Examining the satellite image shown in Figure 5 and analyzing the timing, width, and amplitude of the various event rate and event rate ratio spikes reveals that four prominent spikes at approximately 130 seconds are due to reflections from two solar panels. Furthermore, the temporal separation of the solar panel spikes reveals that the solar panels are oriented with a 48-degree separation angle. The mirror-like solar panels cause very few event rate spikes because they must be very precisely oriented to reflect light toward the sensor. The timing of these spikes, combined with knowledge of the satellite's position relative to the sensor and the location of the sun, therefore indicates the orientation of the solar panels at the time of the event rate spikes. Most of the other spikes are consistent with reflections from the gold-colored multilayer insulator found on the edge of the satellite body. The underside of the satellite body has a shiny, flat metal surface that also requires precise orientation to reflect brightly toward the sensor. The narrow event rate spike at 196 seconds, along with the associated significant event rate ratio spike, is consistent with the satellite surface. While further analysis can yield more information, this example serves to illustrate how plots of event rates and event rate ratios can be used to characterize RSOs.

[0034] Variations in event rate and / or brightness data over time may be logged relative to GPS time to allow this data to be later combined with other data, such as satellite orbit data or data from additional event-based sensors, for further analysis.

[0035] In the example described with reference to FIGS. 1 to 5, the event rate was determined in the following manner. Considering multiple event signals generated (i) along a linear path across the pixel array of the sensor and (ii) in close proximity to one another as being associated with an RSO, where close proximity means that the multiple event signals are generated within a predetermined time interval of one another, e.g., within 1 millisecond or so of one another. b. Determining the center and / or velocity of the RSO from one or more of the plurality of event signals, such as by assigning the center of the RSO to be the location of a pixel of the sensor associated with a last event signal of the plurality of event signals. c. Deem subsequent event signals associated with the RSO using the method described in paragraph 0037 below. d. For each pixel associated with those event signals deemed to be associated with the RSO, determining the time interval between each of those event signals associated with that pixel. e. Determining a moving average of the time intervals for a predetermined number of event signals. Typically, the predetermined number of events is about one-tenth the number of event signals generated per second that are considered to be associated with the RSO. f. Determining the event rate at each of those event signals is the inverse of the moving average at that time.

[0036] With respect to step (d) above, for example, if an RSO occupies two pixels (A and B), where pixel A generates a first event signal at a first time point and generates a second event signal 100 milliseconds later, and pixel B generates a first event signal at a first time point and generates a second event signal 100 milliseconds later, and the event signals of pixel A and pixel B are shifted in time from each other by 1 millisecond, then two time intervals that are 100 milliseconds each are determined (not two time intervals of 1 millisecond, nor two time intervals of 99 milliseconds).

[0037] The event signals output from the sensors were considered to be associated with the RSO based on the following: (i) whether the event signal was generated within a first predetermined distance from the current center of the RSO; and / or (ii) whether the event signal was generated (a) along the trajectory of the RSO across the pixel array of the sensor and / or (b) in close proximity to a previous such event signal associated with the RSO; In this regard, close by means that the event signal is generated within a first predetermined time interval from the most recent previous event signal associated with the RSO, or the event signal is generated within a second predetermined distance from the center of the particular object, which may be the same as or different from the first predetermined distance. The first predetermined distance may be based on the number of pixels of the event-based vision sensor occupied by the RSO, or may be approximately equal to the diameter of a circle of pixels of the sensor occupied by the RSO.

[0038] Regardless of the method used to associate an event signal with each RSO, the center and velocity of the RSO were updated based on the location and timing of subsequent event signals associated with the RSO. The center of the RSO was periodically recalculated based on subsequent event signals associated with the RSO, and the velocity of the RSO across the pixel array was calculated from the change in the center's location over time. The center of a particular object was re-determined approximately every 10 milliseconds.

[0039] In the embodiments described with reference to Figures 1-5, the brightness of the RSO was determined based on exposure measurements corresponding to event signals deemed to be associated with the RSO. Determining the brightness of the RSO from the exposure measurements includes summing all associated exposure measurements within a predetermined distance from the center of the RSO. Again, this predetermined distance may be based on the number of pixels of the event-based vision sensor occupied by the RSO, or may be approximately equal to the diameter of the circle of pixels of the sensor occupied by the RSO.

[0040] As described above, the method facilitates inferring or determining information related to RSO based on variations in event rate and / or brightness over time. Inferring or determining information based on variations in rate and / or brightness over time may include identifying repeating patterns in rate and / or brightness over time and determining a rate of rotation of a particular object about one or more axes based on the frequency with which those patterns repeat. The method may include calculating the duration of spikes in rate and / or brightness to determine the angular width of the specular reflection that caused the spike.

[0041] It will be understood by those skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments without departing from the broad general scope of the present disclosure. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive. Examples of such variations and / or modifications include, but are not limited to, the following: Event signals output from the sensors are associated with respective RSOs based on whether the event signal occurs along the trajectory of the RSO and / or in proximity to the most recent previous event signal associated with the RSO, where proximity may be measured in units of time between event signals or distance between the event signal and the center of the RSO. Determine the event rate by calculating the number of event signals considered to be associated with the RSO generated in each consecutive time interval over a predetermined period of time, which may be approximately one-tenth of a second. The position information associated with the event signal is adjusted to account for changes in sensor orientation, thereby allowing event rate and intensity data to be acquired simultaneously for multiple RSOs. An event signal output from the sensor is associated with each RSO based on whether it occurs within a subframe of a pixel array that is visually determined to contain the RSO.

Claims

1. 1. A method for remote monitoring, comprising:

1. A method comprising: determining, from event signals obtained from an event-based vision sensor, a rate at which those event signals are generated in response to changes associated with a particular object; and optionally determining a brightness of the particular object from exposure measurements associated with the event signals.

2. 2. The method of claim 1, wherein the particular object is located in an area remote from the sensor, the sensor being configured to generate an event signal, and optionally an associated brightness measurement, in response to a change the sensor detects in the remote area.

3. the sensor is located on Earth or on an RSO, and optionally the remote region is in space; or 3. The method of claim 1 or 2, wherein the sensor is located in space, optionally on an RSO, and the remote region is located on Earth.

4. 10. The method of any one of the preceding claims, wherein the particular object is an RSO, a drone, a missile, an animal, or a structure.

5. 10. A method according to any one of the preceding claims, comprising logging the rate and optionally variations in brightness over time, and inferring or determining information about the particular object based on the rate and / or the variations in brightness over time.

6. 6. The method of claim 5, wherein inferring or determining information based on variations in the rate and / or intensity over time comprises identifying repeating patterns in the rate and / or intensity over time and determining a rate of rotation of the particular object about one or more axes based on a frequency with which those patterns repeat.

7. 10. A method according to any one of the preceding claims, comprising calculating the duration of spikes in the rate and / or intensity to determine the angular width of the specular reflection that caused the spike.

8. 10. A method according to any one of the preceding claims, comprising determining a rate at which event signals generated in response to changes associated with a particular object are generated from event signals acquired contemporaneously from two event-based vision sensors, each having a different color filter, and optionally determining a brightness of the particular object from exposure measurements associated with the event signals.

9. The method of claim 8 , comprising determining material properties of the particular object based on differences in event rate data and / or brightness data from each of the sensors.

10. the event signal to a specific object, (i) the event signal is generated in response to the change occurring within a first predetermined distance from a center of the particular object, and optionally, the first predetermined distance is substantially equal to a maximum dimension of the particular object; and / or (ii) the event signal is generated in response to the change occurring (a) along the trajectory of the particular object, and / or (b) proximate to a previous event signal associated with the particular object; and / or 10. The method of claim 9, further comprising: (iii) associating based on whether the event signal occurs within a sub-frame of the pixel array of the sensor that is visually determined to include the particular object.

11. 11. The method of claim 10, wherein the associated change occurring in proximity to the most recent previous event signal comprises the associated change occurring within a first predetermined time interval and / or the associated change occurring within the first predetermined distance from the center of the particular object.

12. 10. A method according to any one of the preceding claims, comprising periodically redetermining the centre of the particular object based on subsequent event signals associated with the particular object, optionally wherein the centre of the particular object is redetermined approximately every 10 milliseconds.

13. determining the rate of those event signals generated in response to changes associated with the particular object; determining, for each pixel associated with those event signals, the time interval between each of those event signals associated with that pixel; determining a moving average of the time intervals of a predetermined number of event signals; and determining that the rate at the time of each of those event signals is the inverse of the moving average at that time.

14. 14. The method of claim 13, wherein the predetermined number of events is approximately one-tenth the number of those event signals generated per second.

15. 10. A method according to any one of the preceding claims, wherein said determining the brightness of said particular object from said exposure measurements comprises summing all associated said exposure measurements within a predetermined distance from the centre of said particular object, optionally said predetermined distance being substantially equal to a maximum dimension of said particular object.

16. 10. A method according to any one of the preceding claims, comprising adjusting the orientation of the sensor based on the trajectory of the particular object.

17. 10. A method according to any one of the preceding claims, comprising adjusting position information associated with the event signal to take into account changes in the orientation of the sensor.

18. 10. A method according to any one of the preceding claims, comprising logging the rate relative to GPS time, and optionally the variation in brightness.

19. 10. A method according to any one of the preceding claims, comprising inferring or determining information about the particular object based on the rate and optionally the variation in brightness over time, together with (i) information about the particular object obtained from sources other than the or each sensor, and / or (ii) information about the particular object obtained from the or each sensor on different days or at different times.

20. The method of claim 19 , wherein the information about the particular object obtained from a source other than the or each sensor includes trajectory data and / or data from other sensors.

Images